Quality Initiative Using Theory of Change and Visual Analytics to Improve Controlled Substance Documentation Discrepancies in the Operating Room

Abstract

Background Discrepancies in controlled substance documentation are common and can lead to legal and regulatory repercussions. We introduced a visual analytics dashboard to assist in a quality improvement project to reduce the discrepancies in controlled substance documentation in the operating room (OR) of our free-standing pediatric hospital.

Methods Visual analytics were applied to collected documentation discrepancy audit data and were used to track progress of the project, to motivate the OR team, and in analyzing where further improvements could be made. This was part of a seven-step improvement plan based on the Theory of Change with a logic model framework approach.

Results The introduction of the visual analytics dashboard contributed a 24% improvement in controlled substance documentation discrepancy. The project overall reduced documentation errors by 71% over the studied period.

Conclusion We used visual analytics to simultaneously analyze, monitor, and interpret vast amounts of data and present them in an appealing format. In conjunction with quality-improvement principles, this led to a significant improvement in controlled substance documentation discrepancies.

Keywords

quality improvement - visual analytics - clinical informatics - anesthesia - medication documentation discrepancies

Background and Significance

Inconsistencies in prescription information between two or more documentation sources, known as medication documentation discrepancies, are common and problematic on multiple levels.[1] [2] [3] Most medication documentation discrepancies are accidental and benign, but when the medication involved is a controlled substance such as an opioid, diversion for personal use, or illicit commercialization, then it is an important consideration. Controlled substance diversion can cause a risk to patient safety, harm to the diverting health care worker, and legal and regulatory repercussions for the health care organization.[4] [5]

The Controlled Substances Act (CSA)[6] requires accurate accounting of controlled substance transactions by inpatient pharmacies, to assist in preventing and detecting controlled substance diversion. In 2015 Massachusetts General Hospital was required to pay $2.3 million in a settlement to the federal government and to develop a corrective action plan for failing to comply with the Act.[7] In addition to prosecution under the CSA, hospitals may be prosecuted for fraudulent billing by regulatory federal and state entities if they appear to be over-accounting for medications used.

The American Society of Health-System Pharmacists has developed guidelines for preventing diversion of controlled substances.[8] Each health care organization is encouraged to develop a process that complies with state and federal laws and regulations, but despite this, a 15% discrepancy rate between dispensed and administered controlled substances is reported in the literature.[3] [9] Improvement in this rate may best be achieved by leveraging both technology and continued surveillance. Despite having an electronic medical record (EMR) and automated medication dispensing system (AMDS), our institution had a 3% incidence of controlled substance documentation discrepancies in the operating room (OR). This was considered problematic by our institution due to the absolute number of errors—42 per month at the start of the project.

In this article we describe a quality improvement (QI) project by the use of the Theory of Change (ToC) improvement framework and application of the visual analytics approach. Visual analytics is becoming increasingly used in health care[10] [11] and pharmacovigilance[12] to present complex data in a way which reduces cognitive load and augments mental data integration.

Methods

Ethical Considerations

This QI project had no direct contact with patients or families and was considered Nonhuman Subjects Research/Quality Improvement by the institutional review board (IRB, 00209372). Data were obtained from the hospital information technology systems (Cerner EMR and BD Pyxis MedStation AMDS).

Setting

The project was conducted at Johns Hopkins All Children's Hospital (JHACH), a 300 medical and surgical bed, urban pediatric academic and tertiary care facility in western Florida. JHACH surgical services performed 15,225 cases in the calendar year 2016, 14,460 for calendar year 2017, and 14,222 for calendar year 2018.

Improvement Team

As part of a hospital-wide effort to develop a more robust controlled substance diversion prevention program, we formed a core team of individuals that included anesthesiologists, informaticists, and pharmacists. Each of these departments was represented at both staff and director levels. The team's aim was to develop a QI initiative focusing on reducing controlled substance documentation discrepancy in the OR. The initial goal was to reduce the controlled substance documentation discrepancy rate by 20% within 2 years and was inspired by previously published projects at Dupont Nemours Children's Hospital (using a dashboard to track controlled substance documentation discrepancies)[13] and Jefferson University (using a software program to reconcile the AMDS to the anesthesia EMR).[9]

Planning the Interventions

During the course of a wide-reaching project to improve our controlled substance documentation discrepancy rate, the improvement team regularly shared data updates and progress within the anesthesia department using Microsoft Excel. This was found to be a suboptimal process for several reasons—the file was accessible only to one person, making access limited for other individuals; tables of numbers were visually unappealing and difficult for audience members to quickly interpret; data analysis was limited by the ease of use of the software; and the charts that we could produce did not provide a full and clear analysis. We therefore decided to utilize visual analytics concepts to build an interactive dashboard for ongoing internal publication of results and process monitoring.

Beginning in November 2017 the QI project team developed an interactive dashboard using the Tableau business intelligence tool. When designing our dashboard, we discussed general considerations such as:

• What is our main goal in designing the dashboard?

• Who is our primary/target audience?

• How can we design a tool that helps the user to make better decisions about the process in question?

• How can we present the main trends and tell a compelling story about the different numbers of interventions?

• How can we present information from the lowest to the highest granular data?

We presented the prototype dashboard to our monthly departmental meeting and held several informal co-creation discussions in which the target audience of anesthesiologists provided feedback that helped improve our initial design. The dashboard design was finalized using Dowding and Merrill's heuristics for evaluation of dashboard visualization.[14]

Once the dashboard design was finalized, rotating screenshots ([Figs. 1] [2] [3] [4]) were displayed in the anesthesia office immediately adjacent to the OR, making them permanently available to view by all providers in the anesthesia team.

Fig. 1 Overall trend and average of the number of documented errors by month over the time frame of the project (August 2016 through April 2019). Vertical lines represent the initialization of interventions over time.

Fig. 2 Horizontal bar chart of type of error and controlled substance class over time (quarters). User IDs have been removed from view for de-identification purposes.

Fig. 3 Frequency of discrepancies by provider over time, sorted from highest to lowest frequency. User IDs have been masked for de-identification purposes.

Fig. 4 Box plot that presents the total number of discrepancies per month by user.

For the wider QI project, we chose to use the ToC, a methodology for health promotion programs wishing to achieve improvement through individual and social change. ToC provides an intuitive methodology for empowerment of stakeholders to achieve a straightforward goal without rigidly imposing actions or unrealistic budget and time demands when compared with other more complex QI methodologies, such as PDSA (Plan-Do-Study-Act).[15] ToC has been used for a broad range of public health initiatives including integrated mental health care services, community initiatives, substance use interventions, and to generate behavior change in chronic diseases such as stroke and heart disease,[16] so it seemed appropriate to use for a chronic health care system problem, i.e., controlled substance documentation discrepancies.

When using ToC, groups of stakeholders—which may include staff, management, experts, service users, and evaluators—discuss a current challenge and begin by defining their desired long-term goals and outcomes. This may be through workshops, interviews and discussions, working groups, document reviews, surveys, program observation, or literature review, amongst other techniques.[16] They then work toward a detailed explanation of the actions they intend to take (interventions) to change the current processes to reach this goal. This involves mapping backward from the desired goal to envisage precisely at each step of the current system what will need to change, how much, and by whom. Each intervention carries an assumption that when it is implemented it will result in a predictable behavioral change and therefore movement toward the long-term goal. All of the desired outcomes, preconditions, contextual factors, and interventions are mapped out to become the pathway of change, indicating how outcomes are going to be achieved and how they relate to each other.[16] [17]

We used a framework approach to produce a logic model ([Fig. 5]) to help implementers identify these prerequisites, steps, and interventions that promote behavior modification at the individual and group level.[18] The logic model gives a visual map of program implementation—the pathway of change. There are four components to this model: input (physical resources and infrastructure), activity (intervention), output (evidence that will be assessed), and impact or effects (results or outcomes). This then allows for continuous evaluation, implementation, understanding, and tracking of the progress of the QI initiative, and facilitates the development of modifications of the plan based on results.

Fig. 5 Logic model framework used to guide the improvement process. AIMS, anesthesia information management system; AMDS, automated medication dispensing system; OR, operating room.

Quality Improvement Intervention Timeline

We developed a program of interventions based on the logic model in [Fig. 5]. The program is summarized in [Table 1]. The process of dispensing and witnessed wasting of controlled substances using the AMDS is summarized in [Fig. 6]. We defined a controlled substance documentation discrepancy as any event where the medication and dose recorded as dispensed by the AMDS did not match the combined total of medication recorded as administered to the patient in the EMR and that recorded as wasted in the AMDS. The discrepancies were further classified by type as follows:

• Removal without administration—medication dispensed from AMDS but not documented as administered or wasted. Example: fentanyl 100 micrograms dispensed and no further documentation on that patient's record for administration or waste. This often occurs when an incorrect patient is selected on the AMDS touchscreen or when clean unused medication is given to another patient without documented transfer on AMDS. This scenario may suggest a potential controlled substance diversion event.

• Undocumented waste—medication dispensed and partially administered but no waste documented. Example: fentanyl 100 micrograms dispensed, 20 micrograms administered to a patient, and no waste documented. This occurs when the provider forgets to update AMDS with waste. This discrepancy type may suggest a potential controlled substance diversion event.

• Administration without removal—AMDS record of medication dispensed is of a lower dose than the EMR record of medication administered. Example: fentanyl 100 micrograms administered to a patient and documented on the EMR, and no evidence of dispensing against that patient's AMDS record. This suggests the undocumented transfer of clean unused medication dispensed for a previous patient and therefore usually occurs paired with a “removal without administration” error for that prior patient.

• Incorrect waste documentation—documentation of dispensing and administration but the amount documented as waste does not reduce the balance to zero. Example: fentanyl 100 micrograms dispensed, 50 micrograms given, 5 micrograms documented as waste, and 45 micrograms unaccounted. Often occurs when a digit is accidentally omitted or incorrectly selected on the AMDS touchscreen. This scenario may suggest a potential controlled substance diversion event.

• Incorrect drug documentation—name of drug documented does not match the name of drug dispensed. Example: lorazepam 2 mg documented as administered on the EMR and midazolam 2 mg dispensed from AMDS. The “incorrect drug documentation” category represents a documentation error in the EMR and may occur as a result of distraction or memory lapse.

Table 1 Quality improvement interventions with timeline

Step	Activity	Date
1	Monthly meeting to discuss the need for improved discrepancy rates.	July 2016
2	Alert sign added to OR monitors reminding providers to be cautious about documentation.	August 2016
3	Audit of discrepancy data and reporting at monthly department meetings. Private discussions with provider outliers.	September 2016
4	Creation of mandatory annual learning module titled “Controlled Substance Safe Practices.”	September 2016
5	Formal policy change disallowing the transfer of medications from one patient to another.	February 2017
6a	Physicians were notified that narcotic discrepancies would be linked to compensatory plan 80:20.	March 2017
6b	Narcotic discrepancies were formally linked to physician compensation 80:20.	July 2017
7	Use of visual analytics to display results to team.	March 2018

Abbreviation: OR, operating room.

Fig. 6 The process of dispensing and witnessed wasting of controlled substances using the AMDS. AMDS, automated medication dispensing system.

Our first step, starting in July 2016, was the recognition and dissemination of our base problem—controlled substance documentation discrepancies in the OR. The scale and implications of the problem were presented to the anesthesia team during a regular departmental meeting.

In August 2016 we began our second step—the development and placement of printed laminated signs on all the computers in the OR. These signs reminded providers to reconcile their documentation of medications administered with the documentation in the AMDS screen, before closing the EMR.

The third step (September 2016) was to perform a prospective detailed audit of controlled substance documentation discrepancies. The audit included obtaining documentation from the AMDS and reconciling this with the anesthesia EMR using Microsoft Excel. When a discrepancy was identified, the record was sent to three nominated attending anesthesiologists for validation on a monthly basis. They checked for errors, reviewed it with the original care provider if necessary, documented the cause or reason for the discrepancy, and then signed off on the discrepancy. The data after the audit were stored for analysis and were used as variables for the visual analytics tool: provider who removed medication, amount of medication removed from the AMDS, type of medication, date and time of the discrepancy, amount of waste recorded in the AMDS, amount documented as administered in the patient's EMR, and type of error.

The fourth initiative (September 2016) was the development of a training module titled “Controlled Substances Safe Practices” which was made mandatory for every member of the anesthesia team to complete within 1 month of the rollout. The module educated the team about best practices associated with handling narcotics. This module is currently a part of the annual required learning modules for reapplication of hospital privileges.

After the implementation of the educational module, the collected audit data were analyzed using Microsoft Excel. We recognized that a large number of discrepancies were attributed to the transfer of clean and unused controlled substances from the patient for whom they were issued to another patient. Therefore in February 2017 our fifth initiative introduced a formal policy eliminating the transfer of unused medications, including controlled substances, between patients. The policy stipulated that dispensed but unused medicines, even full vials, must be wasted rather than transferred to another patient. This was accomplished by changing the settings of the AMDS to prevent the ability to transfer medications between patients.

The sixth intervention involved linking physician's compensation to QI initiatives. In March of 2017 anesthesiologists were notified that controlled substance documentation discrepancies would be part of a raft of QI and productivity targets tied to their at-risk compensation, with the potential for a 5% reduction of their existing compensation for greater than two controlled substance errors per quarter year. In July 2017 the intervention was implemented and up to the time of writing of this article no anesthesiologist exceeded the two-error threshold.

Results

There was a 24% reduction of controlled substance documentation discrepancies after the implementation of the visual analytics dashboard. The seven QI initiatives of the overall project reduced documentation errors in total by 71% over the studied 34-month period. [Table 2] and [Figs. 1] [2] [3] [4] detail further breakdown of the success of the improvement initiatives over time.

Table 2 Rate of controlled substance documentation discrepancy at various time points during the project

Time point description	Time period	Documentation error	Total number of errors	Total number of OR cases	Improvement
Baseline	August 2016	3.01%	41	1,364	0%
First set of interventions	September 2016 to February 2017	1.79%	22	1,251	41%
Second set of interventions	March 2017 to March 2018	1.14%	14	1,201	36%
Visual analytics intervention	April 2018 to March 2019	0.87%	10	1,162	24%
Overall improvement from baseline					71%

Abbreviation: OR, operating room.

[Fig. 1] exhibits the overall trend and average between the number of documented errors by month over the time frame of the program (August 2016 to April 2019). Vertical lines represent the initialization of interventions over time. The graph is a clear demonstration of the program over time.

[Fig. 2] presents the error type and controlled substance type over time (quarterly). The most common types of errors were removal of medication from the AMDS without documentation in the EMR and lack of documentation of controlled substance waste. Fentanyl is the most commonly used controlled substance in anesthesia practice and was the most common medication involved in the discrepancies, accounting for 66% of errors.

[Fig. 3] presents the frequency of discrepancies by provider over time, sorted from the highest to lowest frequency. Providers are represented along the y-axis by ID numbers and listed in order of decreasing frequency of discrepancies from the top of the graphic to the bottom. A color scale is provided with the graphic to represent frequency count.

[Fig. 4] is a box plot that presents the total number of discrepancies per month by user. The green portion represents the users falling within the interquartile range. The darker green area represents 25 to 50%, and the lighter green area 50 to 75%. The whiskers represent users falling within 1.5× the interquartile range. Each user is represented as a blue dot, making the outliers with most discrepancies easy to identify as blue dots lying above the upper whiskers.

Discussion

The use of visual analytics in conjunction with QI principles led to a 71% improvement in controlled substance documentation discrepancies in the OR at our institution. Interventions that were found to be most helpful in advancing the objective of the wider improvement project were eliminating the ability to transfer medications from one patient to another, tying provider performance to compensation, and the use of visual analytics to provide feedback.

Involving users in a co-creation process seemed to be the most important part of the dashboard design. Motivated individuals were able to articulate their information needs with a clear understanding of the need to reduce the number of controlled substance documentation discrepancy errors in the OR. This dashboard design allowed users to filter or click on one graph/chart and automatically apply the changes to other charts, saving time and creating a fluid interface for data interactivity. These interactive visualizations far outstrip what is possible with conventional spreadsheet software and were critical to understanding the full content of the data. They enable analytic reasoning interactions between our team members, supported by the combination of visually appealing content, automated analysis, and dynamic querying. The dashboard can be updated continuously in near real-time, allowing for the provision of timely visual feedback to users, and can be simultaneously displayed in multiple locations throughout the hospital.

The visual analytics dashboard was introduced at a time during the QI project when all the other initiatives in the ToC logic model had already been introduced. The incidence of error declined further after the introduction of the dashboard. We cannot know if this decline was as a direct result of the dashboard introduction, or indeed of any other intervention. However, behavioral change is difficult to maintain and we feel that the dashboard provided a regular reminder of progress for the QI project team and the wider anesthesia department to continue to motivate efforts. A qualitative survey of feedback from the anesthesia team may have been able to confirm this but was not one of the planned interventions of the project.

The controlled substance documentation discrepancy dashboard named every user in the department with their performances. This allowed members of the team to determine the performance of every individual and then compare their own performance with that of peer benchmark. A threshold for discrepancies per provider was established and communicated by the department leadership. Providers were alerted when their performance was approaching the threshold. The outliers with the highest number of discrepancies each month (1–3 providers/month) were approached with feedback and counseling for improvement. We found that for some providers personal feedback was most effective, but that public display of their individual performance was more effective for others. If, despite intervention, no improvement of their performance was noted, then this was followed by documentation into the provider's personal file and a formal investigation by hospital authorities. No evidence of illicit conduct or diversion of controlled substance was found during the project.

Human error limits our ability to attain a zero controlled substance documentation discrepancy rate and is thought to be the cause of the variability in the number of controlled substance documentation discrepancies in our project. Currently the provider is burdened with accurate data entry into two different systems and accurate documentation of controlled substance waste, while simultaneously caring for a patient. This may lead to inaccurate charting of controlled substance use or, even more critical from the safety perspective, distraction from patient care.[19] Another cause of upward trends in the number of documented errors, such as that seen from November 2018 to March 2019, is thought to be the recruitment of new staff who were not immediately familiar with the system. Although we have leveraged technology, we are unable to automate every step in the process of obtaining controlled substances, drug administration, drug wastage, documentation, and audit, so human error will continue to occur.

While the dashboard was developed for QI, it may also have a small role to play in detecting controlled substance diversion. If a provider was noted to have multiple discrepancies that could not be explained by simple clerical errors, it would prompt a deeper investigation of this individual. However, now that the dashboard is openly displayed and providers are aware that the process is audited, it is possible that a provider engaged in diversion could find ways to deceive our audit system.[13] We would need to introduce further measures if we wish to improve our detection of controlled substance diversion, such as staff urine screening, pain score audit, and returning waste syringes to pharmacy for testing of their contents.

A major barrier of our wider project was the controlled substance documentation discrepancy auditing process. The audit requires tedious manual manipulation of the data as the two software programs from the AMDS and the EMR do not communicate directly with each other. This manual process hinders the current process of reconciling controlled substance documentation discrepancies. Currently there is a 24 to 48-hour lag between the occurrence of the discrepancy and when it is identified through audit. Ideally, identification of a controlled substance documentation discrepancy should be discovered in real time so that an automated message can be delivered to the clinicians to re-examine their documentation and correct the error. The development of a visual analytics dashboard that displays the integrated form of these two systems in real time would be a very helpful next stage of the process.

Conclusion

Visual analytics offers powerful tools to help to reduce controlled substance documentation discrepancies as part of a comprehensive strategy of behavioral and system change. These tools are helpful in the heightened scrutiny of meeting regulatory requirements. Future use of similar visual analytics tools can be applied to other medications that are closely monitored due to high cost or other regulatory requirements.

Multiple Choice Questions

1. The Theory of Change …

   • … uses incremental cycles of improvement.

   • … maps backward from desired goals to identify intermediate steps.

   • … can utilize a logic model framework to summarize the long-term goal.

   • … has been used to generate behavioral change in military, health, and social settings.

   Correct Answer: The correct answer is option b. The Theory of Change is used in health and social care and begins by defining desired long-term outcomes before mapping backward to the current state to identify the conditions and contextual factors necessary to reach intermediate steps of behavioral change necessary to reach the long-term goal. Unlike PDSA it does not work on cycles of incremental change. A logic model framework can be used to help implementers of the theory to identify intermediate steps and interventions.

2. Controlled substance documentation discrepancies …

   • … carry a theoretical risk of prosecution under the Controlled Substances Act.

   • … are rare in the operating room due to the limited use of controlled substances.

   • … cannot be reduced to zero due to human error.

   • … are largely due to personal diversion events.

   Correct Answer: The correct answer is option c. Controlled substance documentation discrepancies are common in the operating room due to the widespread use of controlled substances during anesthesia and for postoperative pain relief. They are largely due to administrative oversights and accidental human error and therefore cannot be reduced to zero. They carry an actual risk of prosecution under the CSA as evidenced by the 2015 Massachusetts General Hospital settlement described further in the introduction to this article.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

The Johns Hopkins Medicine All Children's Hospital Institutional Review Board has determined that this research does not constitute human subjects research under the DHHS or FDA regulations. IRB reference number: 00209372.

• References

• 1 Lesselroth BJ, Adams K, Church VL. , et al. Evaluation of multimedia medication reconciliation software: a randomized controlled, single-blind trial to measure diagnostic accuracy for discrepancy detection. Appl Clin Inform 2018; 9 (02) 285-301
• 2 Anyanwu C, Egwim O. The prevalence and determinants of controlled substance discrepancies in a level I trauma hospital. Am Health Drug Benefits 2016; 9 (03) 128-133
• 3 Vigoda MM, Gencorelli FJ, Lubarsky DA. Discrepancies in medication entries between anesthetic and pharmacy records using electronic databases. Anesth Analg 2007; 105 (04) 1061-1065
• 4 McClure SR, O'Neal BC, Grauer D, Couldry RJ, King AR. Compliance with recommendations for prevention and detection of controlled-substance diversion in hospitals. Am J Health Syst Pharm 2011; 68 (08) 689-694
• 5 Trinkoff AM, Storr CL. Substance use among nurses: differences between specialties. Am J Public Health 1998; 88 (04) 581-585
• 6 Controlled Substances Act, US, 1971. Statutes at Large, 1970–1971, vol. 84, part 1. Government Printing Office, Washington, DC
• 7 Settlement agreement [United States of America and Massachusetts General Hospital]. Available at: https://www.justice.gov/usao-ma/file/778651/download . Accessed July 2, 2019
• 8 Brummond PW, Chen DF, Churchill WW. , et al. ASHP guidelines on preventing diversion of controlled substances. Am J Health Syst Pharm 2017; 74 (05) 325-348
• 9 Epstein RH, Dexter F, Gratch DM, Perino M, Magrann J. Controlled substance reconciliation accuracy improvement using near real-time drug transaction capture from automated dispensing cabinets. Anesth Analg 2016; 122 (06) 1841-1855
• 10 Simpao AF, Ahumada LM, Larru Martinez B. , et al. Design and implementation of a visual analytics electronic antibiogram within an electronic health record system at a tertiary pediatric hospital. Appl Clin Inform 2018; 9 (01) 37-45
• 11 Wanderer JP, Gruss CL, Ehrenfeld JM. Using visual analytics to determine the utilization of preoperative anesthesia assessments. Appl Clin Inform 2015; 6 (04) 629-637
• 12 Sorbello A, Ripple A, Tonning J. , et al. Harnessing scientific literature reports for pharmacovigilance. Prototype software analytical tool development and usability testing. Appl Clin Inform 2017; 8 (01) 291-305
• 13 Brenn BR, Kim MA, Hilmas E. Development of a computerized monitoring program to identify narcotic diversion in a pediatric anesthesia practice. Am J Health Syst Pharm 2015; 72 (16) 1365-1372
• 14 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
• 15 Reed JE, Card AJ. The problem with Plan-Do-Study-Act cycles. BMJ Qual Saf 2016; 25 (03) 147-152
• 16 Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci 2016; 11: 63
• 17 Anderson AA. The Community Builder's Approach to Theory of Change: A Practical Guide to Theory Development. New York, NY: Aspen Institute Roundtable on Community Change; 2006
• 18 McLaughlin JA, Jordan GB. Logic models: a tool for telling your programs performance story. Eval Program Plann 1999; 22 (01) 65-72
• 19 Epstein RH, Dexter F, Gratch DM, Lubarsky DA. Intraoperative handoffs among anesthesia providers increase the incidence of documentation errors for controlled drugs. Jt Comm J Qual Patient Saf 2017; 43 (08) 396-402