Enhancing Patient Safety Event Reporting

Keywords

Incident reporting - clinical information systems - patient safety - user-computer interface - medical informatics applications

Background and Significance

A patient safety event is defined as any process, act of omission, or commission that results in hazardous healthcare conditions and/or unintended harm to the patient [[1]]. Reporting patient safety events is a useful approach for improving patient safety [[2]]. The mechanism of event reporting was first introduced in the high-risk industries such as aviation, nuclear, rail industry, etc. to improve safety and enhance organizational learning from errors. The mechanism was then extended to healthcare systems with additional features such as anonymous reporting, meaningful feedback, ease of reporting, etc [[3], [4]]. Through collecting adverse events and near misses in healthcare, the reporting systems enable safety specialists to analyze events, identify underlying factors, and generate actionable knowledge to mitigate risks [[5]]. Since the emergence of electronic patient safety reporting (e-reporting) systems, the collection and analysis of events tend to become a more efficient manner than traditional paper-based systems [[6], [7]]. However, the promising benefits of such systems in healthcare are yet to be fully seen. Currently, low quality reports have been found as a roadblock hampering the data utility for quality improvement and patient safety research. It is reported that a large amount of inaccurate and incomplete information, such as inconsistent records and mis-classifications, is typically contained in the systems [[8], [9]].

Similar to Electronic Health Record (EHR) systems, data quality in general regarding accuracy, completeness, and timeliness has been a main concern [[10]]. Substantial studies have linked the poor data quality and lack of integration to the design flaws in functionality and usability. A well-designed system tends to generate data of high quality [[11]]. Likewise, a well-designed e-reporting system could serve as a facilitator to enhance data quality for patient safety.

Earlier studies show that quality and rate of event reporting can be greatly affected by user interface associated with human factors [[8], [12]]. It was argued that an effective design of e-reporting system should support social-cognitive process of potential reporters [[13]]. An effective and efficient e-reporting system should guide a reporter to go through the reporting details step-by-step without costing additional time and efforts of the reporter. Unfortunately, by far the design features of e-reporting systems have been addressed in a fragmented way across studies. Although there are reviews and comparative studies regarding extant reporting system design [[13]–[15]], the discussions were limited to organizational culture and never extended to the nuances of interface design. For that reason, the objective of the study was to explore the current status of e-reporting in terms of design features which could improve data quality, to identify the feature hierarchy in terms of the stage of development, and to detect the potential technical gaps and challenges of developing an e-reporting system. To our best knowledge, our study is the first of its kind posing a significant step forward in identifying essential features of e-reporting associated with data quality.

Data quality in an e-reporting system is defined as a multidimensional concept, including accuracy, completeness, and timeliness [[16]]. We define the three data quality dimensions as follows: (1) Accuracy: the degree of proximity of a given patient safety event report to corresponding real world occurrences. The reporting accuracy is subject to user errors and cognitive limitations in memory and reasoning, including but not limited to typographical errors, memory decay, causal attribution and hindsight biases. The accuracy of e-reporting could be improved if these contributing factors are incorporated into design considerations with good usability and functionality. (2) Completeness: the degree to which a given patient safety event report includes necessary information describing the corresponding real world event so as to be sufficiently valid for the purpose of analysis and generation of intervention. The completeness could be enhanced if its criteria are explicitly delineated and properly represented to the reporters with the help of interface features. (3) Timeliness: the degree to which a patient safety event is reported in a timely manner for root cause analysis and the generation of real time intervention. The timeliness could be enhanced by improving the efficiency of the reporting process and offering a smooth review process to generate actionable knowledge.

2. Objectives

The purpose of this study was to reveal the current status of design features of electronic patient safety event reporting (e-reporting) systems, detect potential gaps in system design, and accordingly propose suggestions for future design and implementation.

3. Methods

3.1. Search strategy

Search resources for peer reviewed publications included three databases: Ovid MEDLIINE (1946 to December week 3 2016), PsycINFO (1806 to December week 4 2016), and Health and Psychological Instruments (1985 to December 2016). Search terms including “medical error/incident/event”, “report/reporting system”, “electronic report”, “healthcare”, and “information system” were applied with different combinations to all field search (title, abstract, keywords, etc.). As this strategy may include articles with high sensitivity and low specificity, we set restrictions on the MeSH Subject Heading to match such term clusters as “Risk/Safety Management”/ “Quality of Health Care”/ “Quality Assurance” and “Patients or Medical Records Systems”/ “Computerized or Hospital Information Systems”. The searching process involved multiple steps with step-specific rules (details shown in ►[Table 1]). Final searches were run on Feb 1, 2017.

Table 1

Search Strategy for Peer Reviewed Publications

Steps	Search Type	Searches	Results
1	All field search	(medical error$ or medical incident$ or medical event$ or near miss$ or medication incident$ or medication error$ or patient safety) and report$ system	808
2	All field search	Incident registry	9
3	All field search	(medical error$ or medical incident$ or medical event$ or near miss$ or medication incident$ or medication error$ or patient safety) and electronic report$	41
4	All field search	(electronic incident report$ or electronic error report$ or electronic error report$) and health care	8
5	All field search	patient occurrence and report$	9
6	All field search	health care and information system$ and error report$	53
7	All field search	(incident report$ system or error report$ system or event report$ system or near miss report$ system) and health care	340
8	Combine	1 or 2 or 3 or 4 or 5 or 6 or 7	971
9	Mesh Subject Heading search	Quality of Health Care or Risk Management or Safety Management or Quality Assurance, Health Care or Patient Safety	191351
10	Mesh Subject Heading search	(Academic Medical Center or Nursing Homes or Medical Records Systems, Computerized or Hospital Information Systems or Adverse Drug Reaction Reporting Systems or Medical Errors) and Humans	79179
11	Combine	8 and 9 and 10	287
12	Remove duplicates	Remove duplicate 11	285

3.2. Study selection

The inclusion and exclusion criteria were based upon the review objective, which was to reveal design features of e-reporting systems. Studies that fell into one of the five following categories were included:

1. Evaluation studies of electronic patient safety event reporting systems;

2. Intervention studies to improve medical error reporting rates by introducing a new electronic patient safety event reporting system;

3. Cross-sectional or case-control studies involving reported error rates or error reporting rates;

4. Review studies of electronic patient safety event reporting systems implemented or still in use at healthcare systems or patient safety organizations;

5. Studies introducing the process of developing an electronic patient safety event reporting system.

Studies were excluded if one of the following applied:

1. Studies published in a language other than English;

2. Studies missing descriptions of the system used for collecting error data;

3. Studies focusing on paper-based reporting.

3.3. Data extraction and organization

Following the search strategy described above, we identified 287 articles which were imported into a reference management software Endnote™ X7. After removing two duplicates and reviewing the reference lists of the 287 articles, we identified additional seven articles, defined as grey literature, and appeared in the reference section through manual searches. As a result, 292 articles were included for further screening. Then, one author examined the titles and abstracts against the inclusion and exclusion criteria, 151 articles were excluded, among which 60% were due to lack of system description rather focusing on error data analytics. The rest excluded were due to one of the reasons, i.e. English abstract only, paper-based system or non-major topic in the retrieval articles. The remaining 141 articles with full text were further scrutinized by two researchers to guarantee the consistency in making decisions.

The entire review process is presented as a PRISMA flow diagram [[17]], which is a commonly used technique to ensure the transparency of systematic reviews and meta-analyses (►[Figure 1]). To identify developing levels of e-reporting systems described in the literature, we extracted design attributes and organized them into a hierarchy in an EMR Adoption Model [[18]], which is commonly used to track EMR progress at hospitals and health systems. The model includes eight stages (0–7) for different levels of EMR cumulative capabilities, based on which hospital executives compare their EMR adoption to similar facilities as well as compare to their state’s average level [[19]]. In this study, we matched each attribute with the capability statement of stages and then placed in the hierarchy with the consideration of its prevalence, practical significance, and technical complexity. Specifically, three researchers who are familiar with patient safety event reporting and patient safety data were invited to review the literatures. They also summarized the design features during reviewing. A pre-discussion was conducted to make sure the features in the highest and lowest levels. The rest features were ranked by the experts individually according to the practical significance. A final discussion was conducted to determine the level of each feature. The occurrence of certain feature was counted if a system applies this feature; and the frequencies of identified features were calculated based upon the most recent publication regarding an identical system. Thus, the maximal frequency of certain feature equals the number of identified systems when those systems exhibit such a feature.

A PRISMA flow diagram for the literature review process

3.4. Search for additional e-reporting systems accessible online

In order to go beyond the peer-reviewed literature, we used Google search engine to identify publicly accessible e-reporting systems and other information sources that contain substantial system descriptions (e.g. screen shots or demonstration videos) as supplemental information. Detailed search strategies are shown in Appendix A. The final searches for supplemental materials were run on Feb 1, 2017. This process detected eight web-based reporting systems, four PDF documents containing system screenshots and descriptions, and two videos introducing event-reporting systems. System design features extracted from these additional materials were merged into the results of the literature review.

3.5. Review strategy

A 4-point Likert scale in which 1=irrelevant, 2=somewhat irrelevant, 3=relevant, and 4=highly relevant was adopted to identify literature and system features. Three researchers assigned every literature/feature 1–4 points according to the relational degree. After their individual reviews, the researchers as a panel discussed the results to ensure concordance of the reviews. Three rounds of discussion were performed to reach concordance among the panelists. If the panelists could not reach an agreement about a certain case, the majority would make a decision. Any literature/feature that is labeled with 3 or 4 by majority was regarded as an “agreement” and was included; thus, a literature/feature that is labeled with 1 or 2 by majority was considered an “agreement” but was excluded.

4. Results

After a careful integration by the researchers, 48 unique e-reporting systems were identified (►[Table 2]). Within the 48 systems, we were able to summarize the following features as a trend of the e-reporting systems: widgets, anonymity or confidentiality, hierarchy, validator, review notification, and reference (►[Table 3]).

Table 2

E-reporting Systems of Patient Safety Events; # System information derived from peer reviewed literature, * System information derived from webpages

System	Year of Publication	Country	System feature (see Table III for feature descriptions)
The Quality Assurance (QA) Database Application* [[32]]	2011	United States	Validator; feedback and communication; reference; integrated interface; one pager; widgets; hierarchy; anonymity or confidentiality
Anesthesia Incident Reporting System* [[48]]	2011	United States	Widgets; hierarchy; validator; anonymity or confidentiality
Anesthesia Incident Reporting System (AIRS)# [[25]]	2011	United States	Integrated interface; anonymity or confidentiality
Patient Safety Reporting System (PSRS) Report Form* [[49]]	2010 (updated)	United States	Widgets
Patient Safety Reporting Form* [[50]]	2010	United States	Widgets; validator; anonymity or confidentiality
AHRQ Common Formats Patient Safety Events Reporting System* [[33]]	>2009	United States	Widgets; hierarchy; feedback and communication
Medication Error Quality Initiative– Individual Error (MEQI-IE) Reporting System# [[34]–[36]]	2006 (introduced) 2009 (updated)	United States	Phonetic algorithm; reference; hierarchy; widgets; feedback and communication
Duke University Health System Safety Reporting System (SRS)# [[37], [38]]	2009	United States	Review notification; hierarchy; widgets; one pager; feedback and communication; anonymity or confidentiality
The Family Reporting System (DATIX Software)* [[26]]	2009	Canada	Validator; reference; integrated interface; widgets; hierarchy
A Web-based Medication Report Form* [[51]]	2008	United States	Color coding; widgets; validator; one pager; anonymity or confidentiality
The MEADERS Event Reporting Form* [[52]]	2008	United States	Widgets
Medi-Event System# [[27]]	2006	United Kingdom	Integrated interface; widgets; reference
Incident Registry Form# [[28]]	2006	Netherlands	Widgets; one pager; reference; integrated interface; anonymity or confidentiality
Incident Reporting and Review eForm (version 4)# [[39]]	2006	Canada	Review notification; widgets; feedback and communication
A Microsoft Access Data Collection Tool* [[53]]	2006	United States	Widgets
Non-optimal Care and Safety Event Form* [[54]]	2006	United States	Widgets; anonymity or confidentiality
Critical Incident Reporting System (CIRS)*# [[55], [56]]	2005	Germany	Widgets; one pager; anonymity or confidentiality; validator
DATIX*# [[23], [29], [40], [41]]	2005	United Kingdom	Review notification; hierarchy; widgets; one pager; integrated interface; anonymity or confidentiality; feedback and communication; validator; color coding
Radiation Oncology Incident Reporting System (ROIR)* [[57]]	2005	United States	Widgets; hierarchy; anonymity or confidentiality; color coding; validator; one pager
A CQI Web-page Reporting System* [[58]]	2005	United States	Widgets; one pager; anonymity or confidentiality
Jeder Fehler Zaehlt*# [[59], [60]]	2004	Germany	Hierarchy; widgets; anonymity or confidentiality; reference; validator
Risk Monitor Pro# [[61]]	2004	United States	Review notification; widgets; hierarchy
A Web-based incident reporting system in Taiwan# [[6]]	2004	China	Validator; review notification
Department of Health Patient Safety Reporting System, State of New Jersey* [[62]]	2004	United States	Widgets; hierarchy; validator; anonymity or confidentiality; reference; color coding
UHC Patient Safety Net Event Report* [[63]]	2004	United States	Widgets; validator; color coding; reference
The Medication-Error Reporting System* [[64]]	2004	United States	Validator
National Patient Safety Agency Incident Reporting Form* [[65]]	2003	United Kingdom	Widgets; validator; anonymity or confidentiality; reference; color coding; hierarchy
The University of Texas Close Call Reporting System# [[24]]	2003	United States	Review notification; widgets; feedback and communication
Falls Menu-Driven Incident-Reporting System (MDIRS)# [[66]]	2003	United States	Validator; widgets
The Ohio State University Health System (OSUHS) Event Reporting System# [[30]]	2003	United States	Reference, review notification; hierarchy; widgets; anonymity or confidentiality; feedback and communication; integrated interface
Medication Error Occurrence Report# [[7]]	2003	United States	Validator; review notification; hierarchy; integrated interface; anonymity or confidentiality
An Electronic Medical Error Reporting System* [[67]]	2003	United States	Anonymity or confidentiality
Event Reporting Management System (ERMS)* [[68]]	2003	United States	Review notification; widgets
The Monthly Summary Report (MSR)* [[69]]	2003	United States	Widgets
Pennsylvania Patient Safety Anonymous Report Form* [[70]]	2002	United States	Widgets
Incident Reporting Form# [[31]]	2002	Australia	Widgets; integrated interface
Baylor Health Care Web Forms# [[71], [72]]	2002 (updated)	United States	Reference; review notification; validator; anonymity or confidentiality
Electronic Error Reporting System# [[73], [74]]	2002	United States	Review notification; hierarchy; anonymity or confidentiality
Web-based Intensive Care Unit Safety Reporting System# [[75]]	2002	United States	Reference; hierarchy; widgets; color coding; anonymity or confidentiality
University of Missouri Health Care (MUHC) Patient Safety Network System# [[42]]	2002	United States	Review notification; hierarchy; widgets; feedback and communication
New Anonymous Medical Error Report Form# [[76]]	2002	United States	Widgets; reference
The Occurrence Screen Database* [[77]]	2002	United States	Review notification; widgets; anonymity or confidentiality
Potential Error and Event Reporting System# [[78]]	2001	United States	Review notification; hierarchy; anonymity or confidentiality
Electronic Reporting System (ERS) # [[9], [79]]	2001	United States	Hierarchy; widgets; anonymity or confidentiality
Incident Reporting System* [[80]]	2001	Japan	Widgets; anonymity or confidentiality
The ASIPS Patient Safety Reporting System* [[81]]	2001	United States	Widgets; anonymity or confidentiality
Medical Incident Reporting System (MIRS)* [[82]]	2001	United States	Widgets; review notification
Online Incident Reporting System (OIRS) # [[83], [84]]	2000	Japan	Hierarchy; widgets; anonymity or confidentiality

Table 3

Identified Design Features in the E-reporting Systems

System design feature	Description	Frequency	Percentage
Widgets	Drop-down lists, check boxes, or radio buttons are used to replace plain text input for convenience when appropriate.	41	85.42%
Anonymity or confidentiality	Allow reporters to report anonymously.	29	60.42%
Hierarchy	Relevant information is shown based on previous user choices.	20	41.67%
Validator	A timely validator to prevent leaving mandatory items blank, inconsistent entries, incorrect formats, etc.	17	35.42%
Review notification	Reviewers are notified by the system generated message once the review is completed.	15	31.25%
References	Search box, examples, or instructions are provided.	13	27.08%
Integrated interface	Reporting interface is integrated into other clinical applications, or the data or functionality of other clinical applications can be shared with the reporting system.	9	18.75%
Feedback and communication	Reporters are allowed to track event review or resolution status, see reviewers’ comments on previously reported cases, and get feedback on the aggregate level (e.g. statistics).	9	18.75%
One pager	All items are shown on one page to reduce potential technical delays.	8	16.67%
Color coding	Important numbers or text are highlighted by colors.	6	12.50%
Phonetic algorithm	Words are indexed by pronunciation to reduce spelling errors.	1	2.08%

The systems shown in ►[Table 2] were implemented in healthcare institutions across the world in the United States, Netherlands, Canada, United Kingdom, Germany, Australia, China, and Japan. The use of e-reporting systems was not limited in a particular clinical area. For example, some of them focus on general patient safety events and some others focus on specific areas such as anesthesia events and radiation oncology events. The time spans over ten years with the oldest introduced in 2000 and the latest in 2011.

The design features were derived and summarized in ►[Table 3], which includes detailed description and occurrence frequency of each feature. Among the features, ‘widgets’ (frequency=41) and ‘anonymity or confidentiality’ (frequency=29) are the most popular features. ‘Validator’, ‘Reference’, ‘Review notification’, and ‘Hierarchy’ revealed an intermediate prevalence. The remaining features were considered as non-widely used.

These derived features were then organized into a conceptual hierarchy as shown in ►[Table 4]. Similar to the evolvement of paper chart to EHR, the features show a trend of advancement of user-centered design. The designs at the early stages (stages 0–2) simply transformed paper forms into e-forms where the features ensuring data quality (stages 3–6) were not pervasive. Based upon this model, we found 12 out of all 48 identified e-reporting systems were actually electronic copies of paper-based reporting forms rather than interactive systems.

Table 4

A Feature Hierarchy Identified in the E-reporting Systems

System design feature	Description	Frequency	Percentage
Widgets	Drop-down lists, check boxes, or radio buttons are used to replace plain text input for convenience when appropriate.	41	85.42%
Anonymity or confidentiality	Allow reporters to report anonymously.	29	60.42%
Hierarchy	Relevant information is shown based on previous user choices.	20	41.67%
Validator	A timely validator to prevent leaving mandatory items blank, inconsistent entries, incorrect formats, etc.	17	35.42%
Review notification	Reviewers are notified by the system generated message once the review is completed.	15	31.25%
References	Search box, examples, or instructions are provided.	13	27.08%
Integrated interface	Reporting interface is integrated into other clinical applications, or the data or functionality of other clinical applications can be shared with the reporting system.	9	18.75%
Feedback and communication	Reporters are allowed to track event review or resolution status, see reviewers’ comments on previously reported cases, and get feedback on the aggregate level (e.g. statistics).	9	18.75%
One pager	All items are shown on one page to reduce potential technical delays.	8	16.67%
Color coding	Important numbers or text are highlighted by colors.	6	12.50%
Phonetic algorithm	Words are indexed by pronunciation to reduce spelling errors.	1	2.08%

The development of e-reporting systems over the years in terms of design features that we had identified was summarized in ►[Table 4]. The hierarchy contains three phases and seven stages, which were paper form (phase I, stage 0), e-form (phase II, stages 1 and 2), and e-reporting (Phase III, stages 3–6). The features aiming to improve accuracy, completeness, or timeliness present an increasing trend over time. Accuracy was not adequately considered in the early stage design, i.e. Phase I & II in terms of usability and functionality, which frequently resulted in user errors, and cognitive limitations in memory and reasoning. There has been a clear trend observed in Phase III, where accuracy of e-reporting is greatly facilitated through intelligent features, such as full validator, data entry prediction, etc. Completeness was not guaranteed in Phase I because the paper form-based reporting systems did not have completeness checking features. A systematic consideration of interoperability promotes both completeness and timeliness when a few common fields can be populated by linking patient’s medical record systems. Meanwhile, timeliness is enhanced by instant communication between reporter and reviewer through feedback access or notification.

Similar to the evolution of EHR, e-reporting systems started from an electronic copy of the paper-based reporting form. In this phase, the reporting systems can be viewed as a primitive alteration of paper-based reporting forms. The use of drop-down lists, check boxes, or radio buttons replaces unnecessary free text boxes, which accelerates the electronic entry process and improves data accuracy by reducing data entry errors. To keep abreast of EHR systems, ideal e-reporting systems should be characterized by flexibility, adaptability, reasoning, temporal dynamics and should contain associated functionalities that convert these essentials toward an improved data quality [[20]].

To further enhance e-reporting systems in terms of accuracy, completeness, and timeliness, e-reporting system designers should incorporate the following features we have characterized through this study.

Validator: By setting embedded constrains or alerts in data entry fields, validators prevent the occurrence of future date, leaving mandatory items blank, inconsistent entry, typographical errors, and incorrect data formats, etc. In e-commerce, for example online banking, validators are a widely used technique in electronic form design to enhance accurate and complete data entry [[21]].

Hierarchy: The hierarchical data layout can reduce the appearance of unreasonable answers from the reporters. Such a layout reduces reporters’ memory load for particular task operations and decreases the likelihood of skipping correct answers. Additionally, this format can lead a top-down reporting process starting from the general to the specific in a logical flow, which potentially helps bridge the reporter’s reasoning and memory gap. The efficiency of electronic data entry process, accuracy and completeness of collected data could also be improved through such a hierarchical layout [[22]].

Access to feedback: This design feature enables reporters to detect inaccurate or incomplete content in previous cases based upon reviewers’ feedback, and to improve the quality of their own reporting. This feature can also help detect certain memory-dependent inaccuracy to which a validator is usually not helpful. The comments or feedback from reviewers may also serve as an individualized event reporting instruction and good resource for reporters to learn from. This interactive communication in a long run may form an iterative learning process to facilitate quality reporting in a virtuous circle.

Data sharing: The function of utilizing stored data in other clinical applications (e.g. EHR) may not have an effect on data quality improvement, but may serve as a basis for developing quality-enhanced artifacts. For example, several e-reporting systems can check patient demographic information against pre-stored patient data, or can incorporate search functionality and ID index for auto-populating [[23], [24]]. In addition, sharing event data among e-reporting systems will contribute to a knowledge base which allows implementing technically sophisticated design, such as event prediction or risk analysis based on similar events in the knowledge base.

5. Discussion

5.1. Identified potential technical gaps

Comparing the characteristics discussed above and what we identified through the included patient safety reporting systems, there are technical gaps that can be potentially bridged through system re-engineering without radically changing the systems in use so that accurate and complete patient safety data would be collected in a timely manner.

First of all, the validator function of current reporting systems is still at its infancy. Most of the identified systems only incorporated a few simple edit checks, such as misspelling alerts on the interface. The ratios of the number of edit checks to the number of question items are low. No functionality that targets at internal consistency between questions was identified. Hence, errors of oblivious, mutual contradictory answers are always accepted by the system without any warnings or alerts. Additionally, free text description of events, where major event information is collected, lacks assistant functionality to ensure data entry accuracy and completeness. Of all the identified systems, only one involved an alert of insufficient event description [[23]]. The other systems merely set this section as an optional data entry process.

In addition, current systems exhibit a lack of interoperability and communication. For improving accuracy, completeness and timeliness, it is essential that generic information regarding location and patient should be incorporated with other healthcare information systems, such as EHRs.

Nine identified systems incorporated features that tend to integrate the reporting form into other clinical applications [[7], [25]–[32]]. However, few would allow sharing pre-stored information. For example, in the case when event independent information (e.g. patient demographic data) is required, a reporter needs to manually enter the information that already exists elsewhere in the reporting form, which could cause unnecessary time consumption in a clinical setting. Communication is another feature that is rarely addressed across current systems. Most systems do not contain any description of the mechanism through which an effective communication between reporters and reviewers can be achieved, or merely involved periodical feedback at the aggregate level through meetings, newsletters, or education training programs. Several systems have included the feedback feature in their system design [[23], [24], [29], [30], [32]–[42]]. However, the functionality supporting communication was rarely mentioned.

Overall, current e-reporting systems present a pattern of technical simplicity, which attributes to both technical and non-technical challenges. The adoption of such features should be a technical challenge. Some non-technical reasons may stem, at least partially, from organizational issues [[43]]. For instance, the scarcity of field validators probably indicates a concern about additional security problems came from multiple databases access in a clinical setting. Lacking considerations on system interoperability and communication indicates a poor integration of event reporting procedure into clinicians’ workflow, organizational quality control, and risk management process.

5.2. Important considerations and future direction

On the basis of our review of published and grey literature on 48 e-reporting systems, we proposed several essential considerations for the effective design of future e-reporting systems.

1. Provide embedded validators to assist electronic data entry. Specify pre-determined constrains to reject or warn users when illegal or unlikely values are entered. This feature has been proposed as one of the best practices for electronic form design [[21]]. Along with the advancement of software and users’ experience, some automatic checking on entries are actually essential across all systems [[44]].

2. Structure a hierarchical data entry supporting adaptive interactions. The display of hierarchical information guided by logical rules has already been a pervasive feature in generic user interfaces. More technically sophisticated methods involving adaptive models or Bayesian network are suggested in e-reporting systems in healthcare. These techniques could optimize the sequence of user selection based upon stored answers in knowledge base. Simulation studies regarding these techniques demonstrated an improvement in accuracy, completeness, and efficiency [[21]].

3. Incorporate assistant functionality to facilitate free text entry. We have been developing predictive artifacts built in a prototype e-reporting system. The artifacts of auto-suggestion and solution recommend free text through a case-based reasoning [[45]]. Our evaluation result on the artifacts is promising in improving reporting accuracy, completeness, and efficiency.

4. Provide independent interfaces and recommendation strategies on a role basis in e-reporting, because doctors, nurses, and patients may have different perspectives to provide.

5. Enable user to view reviewer’s feedback and comments on other relevant reports so as to promote learning and sustain quality reporting.

6. Enhance interoperability with other clinical applications and allow data sharing.

Due to the promising value of e-reporting systems in patient safety research and quality improvement, we envision there will be a widespread implementation and application of the systems in healthcare. An increasing amount of data generated from e-reporting systems will be utilized for pattern recognition and root-cause analysis. However, our exploration of current e-reporting systems reveals that the development of such systems is still at the primary stage in terms of system usability and functionality. Large amounts of low quality data generated by poorly designed systems significantly hampered system’s effectiveness [[8]]. Lessons learned from EHR evolution indicate that the development of an effective health information system requires multidisciplinary knowledge, including medicine, human factors, cognitive sciences, usability engineering, and others [[46]]. In our another pioneering study, we developed a novel schema to improve the data quality of patient safety event reporting systems based on the design features proposed in this study [[47]]. We believe these efforts and considerations serve as a basis and reference for the application of domain knowledge to promote a rapid growth of e-reporting systems.

5.3. Limitations

The information about e-reporting systems was obtained from secondary study of published peer-refereed and grey literature instead of empirical research. It is inevitable that certain system features may be overlooked and were not included in the descriptions in the identified articles. This may lead to our underestimation of the prevalence of specific features and overemphasis on their rarity. Additionally, our study reported here merely focuses on system features associated with data quality. Other design elements (e.g., role-based interface design and multiplatform support) may be also necessary for e-reporting systems but not analyzed and discussed in this report. On the other hand, the features included in this study may have limited generalizability due to the reluctance of organizations to discuss these systems publicly owing to the sensitive nature of such systems.

6. Conclusions

In this study, 11 system design features of patient safety reporting systems, i.e., widgets, anonymity or confidentiality, hierarchy, validator, review notification, references, integrated interface, feedback and communication, one pager, color coding, and phonetic algorithm, were identified and discussed. The findings indicate that current e-reporting systems remain at an early stage of development, thus more efforts are needed to address the technical gaps and challenges. Essential features or functionalities that enhance data entry quality were sparsely identified. Accordingly, these features and the proposed considerations serve as a guidance by which an efficient and effective e-reporting system is promising in future development and implementation.

Multiple Choice Questions

Question: As design features of e-reporting systems, which of the following is more advanced than others? Options: A. Feedback and communication, B. Electronic copy of reporting form, C. System interoperability, D. Hierarchy

Answer: C. System interoperability

• References

• 1 International Classification for Patient Safety (ICPS): World Health Organization. Available from: http://www.who.int/patientsafety/implementation/taxonomy/development_site/en
• 2 Pronovost PJ, Morlock LL, Sexton JB, Miller MR, Holzmueller CG, Thompson DA. et al. Improving the Value of Patient Safety Reporting Systems. In: Henriksen K, Battles JB, Keyes MA, Grady ML. editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol 1: Assessment). Advances in Patient Safety.. Rockville (MD): 2008
• 3 Barach P, Small SD. Reporting and preventing medical mishaps: lessons from non-medical near miss reporting systems. BMJ 2000; 320 7237 759-63.
• 4 Mahajan RP. Critical incident reporting and learning. Br J Anaesth 2010; 105 (01) 69-75.
• 5 Shaw R, Drever F, Hughes H, Osborn S, Williams S. Adverse events and near miss reporting in the NHS. Qual Saf Health Care 2005; 14 (04) 279-83.
• 6 Kuo YH, Lee TT, Mills ME, Lin KC. The evaluation of a web-based incident reporting system. Computers, informatics, nursing CIN 2012; 30 (07) 386-94.
• 7 Rudman WJ, Bailey JH, Hope C, Garrett P, Brown CA. The Impact of a Web-based Reporting System on the Collection of Medication Error Occurrence Data. In: Henriksen K, Battles JB, Marks ES, Lewin DI. editors. Advances in Patient Safety: From Research to Implementation (Volume 3: Implementation Issues). Advances in Patient Safety.. Rockville (MD) 2005
• 8 Gong Y. Data consistency in a voluntary medical incident reporting system. Journal of medical systems 2011; 35 (04) 609-15.
• 9 Tuttle D, Holloway R, Baird T, Sheehan B, Skelton WK. Electronic reporting to improve patient safety. Qual Saf Health Care 2004; 13 (04) 281-6.
• 10 Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc 2013; 20 (01) 144-51.
• 11 Wand Y, Wang RY. Anchoring data quality dimensions in ontological foundations. Communications of the ACM 1996; 39 (11) 86-95.
• 12 Johnson C. Failure in safety-critical systems: A handbook of accident and incident reporting. Glasgow: University of Glasgow Press; 2003
• 13 Holden RJ, Karsh BT. A review of medical error reporting system design considerations and a proposed cross-level systems research framework. Hum Factors 2007; 49 (02) 257-76.
• 14 Cheng L, Sun N, Li Y, Zhang Z, Wang L, Zhou J. et al. International comparative analyses of incidents reporting systems for healthcare risk management. Journal of evidence-based medicine 2011; 4 (01) 32-47.
• 15 Benn J, Koutantji M, Wallace L, Spurgeon P, Rejman M, Healey A. et al. Feedback from incident reporting: information and action to improve patient safety. Qual Saf Health Care 2009; 18 (01) 11-21.
• 16 Data quality: Wikipedia. Available from: https://en.wikipedia.org/wiki/Data_quality#Definitions
• 17 PRISMA Transparent Reporting of Systematic Reviews and Meta-Analysis. The PRISMA Flow Diagram. Available from: http://www.prisma-statement.org/statement.htm
• 18 EMR Adoption ModelSM EMR Adoption Model Structure Ensures Objectivity Available from: http://www.himssanalytics.org/hc_providers/emr_adoption.asp
• 19 Hersh W, Wright A. What workforce is needed to implement the health information technology agenda? Analysis from the HIMSS analytics database. AMIA Annu Symp Proc 2008: 303-7.
• 20 Rudas I, Fodor J. Intelligent systems. J of Computers, Communications & Control 2008; 3 (Proceedings of ICCCC 2008): 132-8
• 21 Chen K, Chen H, Conway N, Hellerstein JM, Parikh TS. Usher: Improving data quality with dynamic forms. Knowledge and Data Engineering, IEEE Transactions on 2011; 23 (08) 1138-53.
• 22 Tulving E. Episodic memory: from mind to brain. Annu Rev Psychol 2002; 53: 1-25.
• 23 Handler SM, Perera S, Olshansky EF, Studenski SA, Nace DA, Fridsma DB. et al. Identifying modifiable barriers to medication error reporting in the nursing home setting. J Am Med Dir Assoc 2007; 8 (09) 568-74.
• 24 Martin SK, Etchegaray JM, Simmons D, Belt WT, Clark K. Development and Implementation of The University of Texas Close Call Reporting System. In: Henriksen K, Battles JB, Marks ES, Lewin DI. editors. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Advances in Patient Safety. Rockville (MD) 2005
• 25 Guffey P, Szolnoki J, Caldwell J, Polaner D. Design and implementation of a near-miss reporting system at a large, academic pediatric anesthesia department. Paediatric anaesthesia 2011; 21 (07) 810-4.
• 26 Daniels JP, King AD, Cochrane DD, Carr R, Shaw NT, Lim J. et al. A human factors and survey methodology-based design of a web-based adverse event reporting system for families. International journal of medical informatics 2010; 79 (05) 339-48.
• 27 Haw C, Cahill C. A computerized system for reporting medication events in psychiatry: the first two years of operation. Journal of psychiatric and mental health nursing 2011; 18 (04) 308-15.
• 28 van der Veer S, Cornet R, de Jonge E. Design and implementation of an ICU incident registry. International journal of medical informatics 2007; 76 2–3 103-8.
• 29 Alrwisan A, Ross J, Williams D. Medication incidents reported to an online incident reporting system. European journal of clinical pharmacology 2011; 67 (05) 527-32.
• 30 Mekhjian HS, Bentley TD, Ahmad A, Marsh G. Development of a Web-based event reporting system in an academic environment. J Am Med Inform Assoc 2004; 11 (01) 11-8.
• 31 Haller G, Myles PS, Stoelwinder J, Langley M, Anderson H, McNeil J. Integrating incident reporting into an electronic patient record system. J Am Med Inform Assoc 2007; 14 (02) 175-81.
• 32 Peterfreund RA, Driscoll WD, Walsh JL, Subramanian A, Anupama S, Weaver M. et al. Evaluation of a mandatory quality assurance data capture in anesthesia: a secure electronic system to capture quality assurance information linked to an automated anesthesia record. Anesthesia and analgesia 2011; 112 (05) 1218-25.
• 33 Patient safety zone: Patient safety events reporting system [cited 2017 Feb 1]. Available from: http://www.patientsafetyzone.com/PatientSafetyEventsReportingSystem.html
• 34 Pierson S, Hansen R, Greene S, Williams C, Akers R, Jonsson M. et al. Preventing medication errors in long-term care: results and evaluation of a large scale web-based error reporting system. Qual Saf Health Care 2007; 16 (04) 297-302.
• 35 Crespin DJ, Modi AV, Wei D, Williams CE, Greene SB, Pierson S. et al. Repeat medication errors in nursing homes: Contributing factors and their association with patient harm. The American journal of geriatric pharmacotherapy 2010; 8 (03) 258-70.
• 36 Greene SB, Williams CE, Pierson S, Hansen RA, Carey TS. Online medication error graphic reports: a pilot in North Carolina nursing homes. Journal of patient safety 2011; 7 (02) 92-8.
• 37 Whitehurst JM, Cozart H, Leonard D, Schroder J, Horvath M, Avent S. et al. Tailoring ,,best-of-breed“ safety classification for patient fall voluntary reporting. Journal of patient safety 2010; 6 (03) 192-8.
• 38 Whitehurst JM, Schroder J, Leonard D, Horvath MM, Cozart H, Ferranti J. Towards the creation of a flexible classification scheme for voluntarily reported transfusion and laboratory safety events. Journal of biomedical semantics 2012; 3: 4.
• 39 Tepfers A, Louie H, Drouillard M. Developing an electronic incident report: experiences of a multi-site teaching hospital. Healthcare quarterly (Toronto, Ont) 2007; 10 (02) 117-22.
• 40 Grando VT, Rantz MJ, Maas M. Nursing home staff ‘s views on quality improvement interventions: a follow-up study. J Gerontol Nurs 2007; 33 (01) 40-7.
• 41 Welters ID, Gibson J, Mogk M, Wenstone R. Major sources of critical incidents in intensive care. Critical care (London, England) 2011; 15 (05) R232.
• 42 Kivlahan C, Sangster W, Nelson K, Buddenbaum J, Lobenstein K. Developing a comprehensive electronic adverse event reporting system in an academic health center. Jt Comm J Qual Improv 2002; 28 (11) 583-94.
• 43 Ansar M, Khan TA. editors. Non-technical issues in software designing phase. 2016 Sixth International Conference on Innovative Computing Technology (INTECH); Dublin, Ireland: IEEE.
• 44 Anderson AE, Murphy ED, Nichols EM, Sigman RS, Willimack DK. Designing edits for electronic economic surveys and censuses: Issues and guidelines. Available from: http://www.fcsm.gov/working-papers/Anderson_Murphy_Nichols_AAPOR_1004.pdf
• 45 Gong Y, Hua L, Wang S. Leveraging user‘s performance in reporting patient safety events by utilizing text prediction in narrative data entry. Computer methods and programs in biomedicine 2016; 131: 181-9.
• 46 Stow J. Using medical-error reporting to drive patient safety efforts. AORN journal 2006; 84 (03) 405-20.
• 47 Kang H, Gong Y. A Novel Schema to Enhance Data Quality of Patient Safety Event Reports. AMIA Annu Symp Proc 2016; 2016: 1840-9.
• 48 Anesthesia Qualtiy Institute: Anesthesia incident reporting system [cited 2017 Feb 1]. Available from: https://www.aqihq.org/airs/airsIntro.aspx
• 49 Patient Safety Reporting System (PSRS) Report Form [cited 2017 Feb 1]. Available from: http://www.psrs.arc.nasa.gov/web_docs/PSRS_ExampleForm.pdf
• 50 DANA-FARBER Cancer Institute: Patient Safety Reporting Form [cited 2017 Feb 1]. Available from: https://www.dana-farber.org/apps/patient-safety-form.aspx
• 51 Institute for Safe Medication Practices [cited 2017 Feb 1]. Available from: http://www.consumermed/safety.org/report-a-medication-error
• 52 Hickner J, Zafar A, Kuo GM, Fagnan LJ, Forjuoh SN, Knox LM. et al. Field test results of a new ambulatory care Medication Error and Adverse Drug Event Reporting System –MEADERS. Annals of family medicine 2010; 8 (06) 517-25.
• 53 Good VS, Saldana M, Gilder R, Nicewander D, Kennerly DA. Large-scale deployment of the Global Trigger Tool across a large hospital system: refinements for the characterisation of adverse events to support patient safety learning opportunities. BMJ quality & safety 2011; 20 (01) 25-30.
• 54 Foster PN, Sidhu R, Gadhia DA, DeMusis M. Leveraging computerized sign-out to increase error reporting and addressing patient safety in graduate medical education. Journal of general internal medicine 2008; 23 (04) 481-4.
• 55 Hohenstein C, Hempel D, Schultheis K, Lotter O, Fleischmann T. Critical incident reporting in emergency medicine: results of the prehospital reports. Emergency medicine journal EMJ 2014; 31 (05) 415-8.
• 56 Harris CB, Krauss MJ, Coopersmith CM, Avidan M, Nast PA, Kollef MH. et al. Patient safety event reporting in critical care: a study of three intensive care units. Crit Care Med 2007; 35 (04) 1068-76.
• 57 ROIR Radiation Oncology Incident Reporting System [cited 2014 March 6]. Available from: http://www.cars-pso.org
• 58 Khare RK, Uren B, Wears RL. Capturing more emergency department errors via an anonymous web-based reporting system. Quality management in health care 2005; 14 (02) 91-4.
• 59 Hoffmann B, Beyer M, Rohe J, Gensichen J, Gerlach FM. ,,Every error counts“: a web-based incident reporting and learning system for general practice. Qual Saf Health Care 2008; 17 (04) 307-12.
• 60 Gong Y, Jackson-Thompson J. A Human-centered design of cancer registry system. NAACCR: North American Association of Central Cancer Registries. ; June 8; Detroit, MI2007.
• 61 Levtzion-Korach O, Alcalai H, Orav EJ, Graydon-Baker E, Keohane C, Bates DW. et al. Evaluation of the contributions of an electronic web-based reporting system: enabling action. Journal of patient safety 2009; 5 (01) 9-15.
• 62 State of New Jersey: Department of Health Patient Safety Reporting System [cited 2014 March 6]. Available from: https://www13.state.nj.us/DHSS_PSRS/(S(4zg42dqjape2l3ivzd0wvteo))/welcome/anonymous.aspx
• 63 Reporting of medical errors: Real-time tales [cited 2017 Feb 1]. Available from: http://ocw.jhsph.edu/courses/patientsafety/PDFs/PS_lec10_miller-wu.pdf
• 64 Hansen RA, Greene SB, Williams CE, Blalock SJ, Crook KD, Akers R. et al. Types of medication errors in North Carolina nursing homes: a target for quality improvement. The American journal of geriatric pharmacotherapy 2006; 4 (01) 52-61.
• 65 NHS: Report a patient safety incident here [cited 2017 Feb 1]. Available from: http://www.nrls.npsa.nhs.uk/report-a-patient-safety-incident/healthcare-staff-reporting/
• 66 Wagner LM, Capezuti E, Taylor JA, Sattin RW, Ouslander JG. Impact of a falls menu-driven incident-reporting system on documentation and quality improvement in nursing homes. The Gerontologist. 2005; 45 (06) 835-42.
• 67 Taylor JA, Brownstein D, Klein EJ, Strandjord TP. Evaluation of an anonymous system to report medical errors in pediatric inpatients. Journal of hospital medicine : an official publication of the Society of Hospital Medicine 2007; 2 (04) 226-33.
• 68 Avery J, Beyea SC, Campion P. Active error management: use of a Web-based reporting system to support patient safety initiatives. The Journal of nursing administration 2005; 35 (02) 81-5.
• 69 Davis MA, Rake GW. Implementation of a Data-based Medical Event Reporting System in the U.S. Department of Defense. In: Henriksen K, Battles JB, Marks ES, Lewin DI. editors. Advances in Patient Safety: From Research to Implementation (Volume 3: Implementation Issues). Advances in Patient Safety. Rockville (MD) 2005
• 70 Pennsylvania Patient Safety Reporting System Anonymous Report Form [cited 2017 Feb 1]. Available from: https://www.papsrs.state.pa.us/psa/_pdf/PA-PSRS%20Anonymous%20Report.pdf
• 71 Dixon JF. Going paperless with custom-built Web-based patient occurrence reporting. Jt Comm J Qual Improv 2002; 28 (07) 387-95.
• 72 Joshi MS, Anderson JF, Marwaha S. A systems approach to improving error reporting. Journal of healthcare information management JHIM 2002; 16 (01) 40-5.
• 73 Milch CE, Salem DN, Pauker SG, Lundquist TG, Kumar S, Chen J. Voluntary electronic reporting of medical errors and adverse events. An analysis of 92,547 reports from 26 acute care hospitals. Journal of general internal medicine 2006; 21 (02) 165-70.
• 74 Snydman LK, Harubin B, Kumar S, Chen J, Lopez RE, Salem DN. Voluntary electronic reporting of laboratory errors: an analysis of 37,532 laboratory event reports from 30 health care organizations. American journal of medical quality 2012; 27 (02) 147-53.
• 75 Holzmueller CG, Pronovost PJ, Dickman F, Thompson DA, Wu AW, Lubomski LH. et al. Creating the web-based intensive care unit safety reporting system. J Am Med Inform Assoc 2005; 12 (02) 130-9.
• 76 Suresh G, Horbar JD, Plsek P, Gray J, Edwards WH, Shiono PH. et al. Voluntary anonymous reporting of medical errors for neonatal intensive care. Pediatrics 2004; 113 (06) 1609-18.
• 77 Arroyo DA. A Nonpunitive, Computerized System for Improved Reporting of Medical Occurrences. In: Henriksen K, Battles JB, Marks ES, Lewin DI. editors. Advances in Patient Safety: From Research to Implementation (Volume 4: Programs, Tools, and Products). Advances in Patient Safety. Rockville (MD) 2005
• 78 Conlon P, Havlisch R, Kini N, Porter C. Using an Anonymous Web-Based Incident Reporting Tool to Embed the Principles of a High-Reliability Organization. In: Henriksen K, Battles JB, Keyes MA, Grady ML. editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol 1: Assessment). Advances in Patient Safety. Rockville (MD) 2008
• 79 Holloway RG, Tuttle D, Baird T, Skelton WK. The safety of hospital stroke care. Neurology 2007; 68 (08) 550-5.
• 80 Kanda H. Development of an online incident-reporting system for management of medical risks at hospital. Yakugaku zasshi Journal of the Pharmaceutical Society of Japan 2011; 131 (09) 1353-9.
• 81 Fernald DH, Pace WD, Harris DM, West DR, Main DS, Westfall JM. Event reporting to a primary care patient safety reporting system: a report from the ASIPS collaborative. Annals of family medicine 2004; 2 (04) 327-32.
• 82 Kobus DA, Amundson D, Moses JD, Rascona D, Gubler KD. A computerized medical incident reporting system for errors in the intensive care unit: initial evaluation of interrater agreement. Military medicine 2001; 166 (04) 350-3.
• 83 Takeda H, Matsumura Y, Nakajima K, Kuwata S, Zhenjun Y, Shanmai J. et al. Health care quality management by means of an incident report system and an electronic patient record system. International journal of medical informatics 2003; 69 2–3 285-93.
• 84 Nakajima K, Kurata Y, Takeda H. A web-based incident reporting system and multidisciplinary collaborative projects for patient safety in a Japanese hospital. Qual Saf Health Care 2005; 14 (02) 123-9.