Developing and Trialling an Independent, Scalable and Repeatable IT-benchmarking Procedure for Healthcare Organisations^[*]

Summary

Background: Continuous improvements of IT-performance in healthcare organisations require actionable performance indicators, regularly conducted, independent measurements and meaningful and scalable reference groups. Existing IT-benchmarking initiatives have focussed on the development of reliable and valid indicators, but less on the questions about how to implement an environment for conducting easily repeatable and scalable IT-benchmarks.

Objectives: This study aims at developing and trialling a procedure that meets the afore-mentioned requirements.

Methods: We chose a well established, regularly conducted (inter-) national IT-survey of healthcare organisations (IT-Report Health-care) as the environment and offered the participants of the 2011 survey (CIOs of hospitals) to enter a benchmark. The 61 structural and functional performance indicators cov -ered among others the implementation status and integration of IT-systems and functions, global user satisfaction and the resources of the IT-department. Healthcare organisations were grouped by size and ownership. The benchmark results were made available electronically and feedback on the use of these results was requested after several months.

Results: Fifty-nine hospitals participated in the benchmarking. Reference groups consisted of up to 141 members depending on the number of beds (size) and the ownership (public vs. private). A total of 122 charts showing single indicator frequency views were sent to each participant. The evaluation showed that 94.1% of the CIOs who participated in the evaluation considered this benchmarking beneficial and reported that they would enter again. Based on the feedback of the participants we developed two additional views that provide a more consolidated picture.

Conclusion: The results demonstrate that establishing an independent, easily repeatable and scalable IT-benchmarking procedure is possible and was deemed desirable. Based on these encouraging results a new benchmarking round which includes process indicators is currently conducted.

Keywords

Hospital - IT-benchmarking - IT-governance - performance indicator - visualisation - guidance

^* Supplementary online material published on our website www.methods-online.com

• References

• 1 Chan KS, Fowles JB, Weiner JP. Electronic Health Records and the Reliability and Validity of Quality Measures: A Review of the Literature. Med Care Res Rev 2010; 67 (05) 503-527.
• 2 van Rosse F, Maat B, Rademaker CM, van Vught AJ, Egberts AC, Bollen CW. The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics 2009; 123 (04) 1184-1190.
• 3 Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 2008; 15 (05) 585-600.
• 4 Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. J Am Med Inform Assoc 2011; 18 (03) 327-334.
• 5 Häyrinen K, Saranto K, Nykänen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 2008; 77 (05) 291-304.
• 6 Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E. et al Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752.
• 7 Hemens BJ, Holbrook A, Tonkin M, Mackay JA, Weise-Kelly L, Navarro T. et al Computerized clinical decision support systems for drug prescribing and management: a decision-maker-researcher partnership systematic review. Implement Sci 2011; 6: 89
• 8 Augustin S. Information als Wettbewerbsfaktor: Informationslogistik - Herausforderung an das Management. 1st ed. Oberhofer AF. editor Köln: Verlag TÜV Rheinland; 1990.
• 9 Haux R, Winter A, Ammenwerth E, Brigl B. Strategic Information Management in Hospitals - An Introduction to Hospital Information Systems. 1st ed. New York, Berlin, Heidelberg: Springer-Verlag; 2004.
• 10 Bradley RV, Byrd TA, Pridmore JL, Thrasher E, Pratt RM, Mbarika VW. An empirical examination of antecedents and consequences of IT-governance in US hospitals. J Inf Technol 2012; 27 (02) 156-177.
• 11 Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T. et al The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med 2011; 8 (01) e1000387
• 12 Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009; 78 Suppl (Suppl. 01) S69-76.
• 13 Jobst F. IT zur Prozessgestaltung im Kranken-haus - Wie bekommt man die optimale Kombination von IT-Anwendungen. In Schlegel H. editor Steuerung der IT im Klinik-Management: Methoden und Verfahren. Wiesbaden: Vieweg + Teubner; 2010: 29-52.
• 14 Simon A. Die betriebswirtschaftliche Bewertung der IT-Performance im Krankenhaus am Beispiel eines Benchmarking-Projekts. In Schlegel H. editor Steuerung der IT im Klinik-Management: Methoden und Verfahren. Wiesbaden: Vieweg + Teubner; 2010: 73-90.
• 15 Jaana M, Tamim H, Paré G, Teitelbaum M. Key IT management issues in hospitals: Results of a Delphi study in Canada. Int J Med Inform 2011; 80 (12) 828-840.
• 16 Müller U, Winter A. A monitoring infrastructure for supporting strategic information management in hospitals based on key performance indicators. In Hasman A, Haux R, van der Lei J, De Clercq E, France F. editors Ubiquity: Technologies for Better Health in Aging Societies: Proceedings of MIE 2006. Studies in Health Technology and Informatics; 2006 Aug. 27–30; Maastricht, Netherlands. Amsterdam: IOS Press; 2006: 328-322.
• 17 Hübner-Bloder G, Ammenwerth E. Key performance indicators to benchmark hospital information systems - a delphi study. Methods Inf Med 2009; 48 (06) 508-518.
• 18 Otieno GO, Hinako T, Motohiro A, Daisuke K, Keiko N. Measuring effectiveness of electronic medical records systems: towards building a composite index for benchmarking hospitals. Int J Med Inform 2008; 77 (10) 657-669.
• 19 Dugas M, Eckholt M, Bunzemeier H. Benchmarking of hospital information systems: monitoring of discharge letters and scheduling can reveal heterogeneities and time trends. BMC Med Inform Decis Mak 2008; 8: 15
• 20 Parmeter D. Key Performance Indicators (KPI): Developing, Implementing, and Using Winning KPIs. 2nd ed. Hoboken, New Jersey: John Wiley Sons; 2010.
• 21 ISACA. Information Systems Audit and Control Assocation (ISACA): Control Objectives for Information and Related Technology (COBIT) (Internet). 2004 (cited 2013 April 05). Available from http://www.isaca.org/cobit.htm.
• 22 ITIL. Information Technology Infrastructure Library. (Internet). 2009 (cited 2013 April 05). Available from http://www.itil-officialsite.com/home/home.asp.
• 23 Paré G, Sicotte C. Information technology sophistication in health care: an instrument validation study among Canadian hospitals. Int J Med Inform 2001; 63 (03) 205-223.
• 24 Schlegel H. IT-Governance mit COBIT® - Methodenunterstützung für das Management. In Schlegel H. editor Steuerung der IT im Klinik-Management: Methoden und Verfahren. Wiesbaden: Vieweg + Teubner; 2010: 7-27.
• 25 Ammenwerth E, Ehlers F, Hirsch B, Gratl G. HIS-Monitor: an approach to assess the quality of information processing in hospitals. Int J Med Inform 2007; 76 2–3 216-215.
• 26 Amarasingham R, Diener-West M, Plantinga L, Cunningham AC, Gaskin DJ, Powe NR. Hospital characteristics associated with highly automated and usable clinical information systems in Texas, United States. BMC Med Inform Decis Mak 2008; 8: 39
• 27 Jahn F, Winter A. A KPI Framework for Process-based Benchmarking of Hospital Information Systems. Stud Health Technol Inform 2011; 169: 542-546.
• 28 Kütz M. IT-Controlling für die Praxis. 1st ed. Heidelberg: dpunkt. 2005.
• 29 Hübner U, Sellemann B. Current and Future Use of ICT for Patient Care and Management in German Acute Hospitals - a Comparison of the Nursing and the Hospital Managers’ Perspectives. Methods Inf Med 2005; 44 (04) 528-536.
• 30 Hübner U, Ammenwerth E, Flemming D, Schaubmayr C, Sellemann B. IT adoption of clinical information systems in Austrian and German hospitals: results of a comparative survey with a focus on nursing. BMC Med Inform Decis Mak 2010; 10: 8
• 31 Liebe JD, Egbert N, Frey A, Hübner U. Characteristics of German hospitals adopting health IT systems - results from an empirical study. Stud Health Technol Inform 2011; 169: 335-338.
• 32 Liebe JD, Egbert N, Hübner U. Krankenhäuser können Innovationen steuern - Validierte Ergebnisse einer Regressionsanalyse. In Schreier G, Hayn D, Hörbst A, Ammenwerth E. editors Health Informatics meets eHealth - von der Wissenschaft zur Anwendung und zurück - Mobile Health Care - Gesundheitsversorgung immer und überall: Proceedings of eHealth; 2012 May 10–11. Vienna, Austria: [e-mail]. 2012: 69-75.
• 33 Rogers MR. Diffusion of Innovation. 5th ed. New York: Free Press; 2003.
• 34 Egbert N, Hübner U, Ammenwerth E, Schaubmayr C, Sellemann B. Sind die Krankenhäuser in Österreich besser vernetzt als in Deutschland?. In Schreier G, Hayn D, Ammenwerth E. editors Health Informatics meets eHealth - von der Wissenschaft zur Anwendung und zurück - Grenzen überwinden - Continuity of Care: Proceedings of eHealth; 2011 May 26–27. Vienna, Austria: [e-mail]. 235-239.
• 35 Krediet I, Hübner U, Goossen W. Rapport IT-ontwikkelingen in de Nederlandse ziekenhuizen 2011 - Een inventarisatie van IT, EPD en e-Overdracht in de zorg. Windesheim, Netherland: Lectoraat ICT Innovaties in de Zorg; 2012.
• 36 Donabedian A. Evaluating the Quality of Medical Care. Milbank Quarterly 2005; 83 (04) 691-629.