Key Performance Indicators to Benchmark Hospital Information Systems – A Delphi Study

Summary

Objectives: To identify the key performance indicators for hospital information systems (HIS) that can be used for HIS benchmarking.

Methods: A Delphi survey with one qualitative and two quantitative rounds. Forty-four HIS experts from health care IT practice and academia participated in all three rounds.

Results: Seventy-seven performance indicators were identified and organized into eight categories: technical quality, software quality, architecture and interface quality, IT vendor quality, IT support and IT department quality, workflow support quality, IT outcome quality, and IT costs. The highest ranked indicators are related to clinical workflow support and user satisfaction. Isolated technical indicators or cost indicators were not seen as useful. The experts favored an interdisciplinary group of all the stakeholders, led by hospital management, to conduct the HIS benchmarking. They proposed benchmarking activities both in regular (annual) intervals as well as at defined events (for example after IT introduction). Most of the experts stated that in their institutions no HIS benchmarking activities are being performed at the moment.

Conclusion: In the context of IT governance, IT benchmarking is gaining importance in the healthcare area. The found indicators reflect the view of health care IT professionals and researchers. Research is needed to further validate and operationalize key performance indicators, to provide an IT benchmarking framework, and to provide open repositories for a comparison of the HIS benchmarks of different hospitals.

Keywords

Benchmarking - hospital information systems - medical informatics - Delphi technique - quality indicators - key performance indicators - user satisfaction

• References

• 1 Committee on Quality Health Care in America – Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington: National Academy Press; 2001 http://www.nap.edu/books/0309072808/html.
• 2 Haux R, Winter A, Ammenwerth E, Brigl B. Strategic Information Management in Hospitals – An Introduction to Hospital Information Systems. New York, Berlin, Heidelberg: Springer-Verlag; 2004
• 3 Winter A, Ammenwerth E, Bott O, Brigl B, Buchauer A, Gräber S. et al. Strategic Information Management Plans: The Basis for Systematic Information Management in Hospitals. Int J Med Inform 2001; 64 2–3 99-109.
• 4 Kuhn K, Guise D. From Hospital Information Systems to Health Information Systems – Problems, Challenges, Perspectives. Methods Inf Med 2000; 40: 275-286.
• 5 Ward J, Griffiths P. Strategic Planning for Information Systems. 2nd ed. Chichester: John Wiley & Sons; 1996
• 6 ITIL. Information Technology Infrastructure Library. 2009 (cited May 2009). Available from: http://www.itil-officialsite.com/home/home.asp.
• 7 Köhler P. Prince 2. Heidelberg: Spinger; 2006
• 8 Müller U, Winter A. A monitoring infrastructure for supporting strategic information management in hospitals based on key performance indicators. In: Hasman A, Haux R, van der Lei J, De Clercq E, France F. editors. Ubiquity: Technologies for Better Health in Aging Societies. Proceedings of MIE2006. Studies in Health Technology and Informatics vol. 124. Amsterdam: IOS Press; 2006. pp 328-332.
• 9 Friedman C, Wyatt JC. Evaluation Methods in Medical Informatics. 2nd ed. New York: Springer; 2006
• 10 JCAHO. Sentinel Event Glossary of Terms. 2009 (cited May 2009). Available from: http://www.jointcommission.org/SentinelEvents/se_glossary.htm.
• 11 Otieno GO, Hinako T, Motohiro A, Daisuke K, Keiko N. Measuring effectiveness of electronic medical records systems: towards building a composite index for benchmarking hospitals. Int J Med Inform 2008; 77 (10) 657-669. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=18313352.
• 12 JCAHO. Improving America’s Hospitals: The Joint Commission’s Annual Report on Quality and Safety 2008. 2008 (cited May 2009). Available from: http://www.jointcommissionreport.org/introduction/introduction.aspx.
• 13 JCAHO. Joint commission for accreditation for healthcare organizations (JCAHO): Information management standards. 2009 (cited May 2009). Available from: http://www.jcaho.org.
• 14 Ribière V, LaSalle A, Khorramshahgool R, Gousty Y. Hospital Information Systems Quality: A Customer Satisfaction Assessment Tool. In: El-Rewini H. editor. Proc 32nd Hawaii International Conference on System Sciences (HICSS-32), Maui, Hawaii, January 5-8, 1999. IEEE Computer Society Press; 1999. pp 1-9.
• 15 Ammenwerth E, Ehlers F, Hirsch B, Gratl G. HIS-Monitor: an approach to assess the quality of information processing in hospitals. Int J Med Inform 2007; 76 2–3 216-225. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=16777476.
• 16 ISACA. Information Systems Audit and Control Assocation (ISACA): Control Objectives for Information and Related Technology (COBIT). 2004 (cited May 2009). Available from: http://www.isaca.org/cobit.htm.
• 17 Linstone HA, Turoff M. The Delphi Method: Techniques and Applications. 2002 (cited May 2009). Available from: http://is.njit.edu/pubs/delphibook.
• 18 Dalkey NC. The Delphi Method: An Experimental Study of Group Opinion. 1969 (cited May 2009). Available from: http://www.rand.org/pubs/research_memoranda/RM5888/.
• 19 Green KC, Armstrong JS, Graefe A. Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared. Foresight: The International Journal of Applied Forecasting 2007 (issue 8, fall 2007): 17-21.
• 20 Häder M. Delphi Befragungen – Ein Arbeitsbuch (Delphie surveys). Wiesbaden, Germany: West-deutscher Verlag GmbH; 2002
• 21 Bortz J, Döring N. Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler (Research methods and evaluation). 3rd ed. Berlin: Springer; 2002
• 22 Dey I. Qualitative Data Analysis. A User-Friendly Guide for Social Scientists. London, Great Britain: Routledge; 1993
• 23 Gwet K. Handbook of Inter-Rater Reliability. Gaithersburg, MD: STATAXIS Publishing Company; 2001
• 24 Barbour RS. The case for combining qualitative and quantitative approaches in health services research. J Health Serv Res Policy 1999; 4 (01) 39-43.
• 25 Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol 2005; 5: 37. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=16321161.
• 26 Hersh WR. A Stimulus to Define Informatics and Health Information Technology. BMC Med Inform Decis Mak 2009; 9: 24. doi: 10.1186/1472-6947-9-24. http://www.biomedcentral.com/1472-6947/9/24.
• 27 Kütz M. Kennzahlen in der IT – Werkzeuge für Controlling und Management (IT performance indicators). 3 ed. Heidelberg, Germany: dpunkt.verlag GmbH; 2009
• 28 Lutchen M, Collins A. IT Governance in a Health Care Setting: Reinventing the Health Care Industry Preview. Journal of Health Care Compliance 2005; 7 (06) 27-30.
• 29 Dewan N, Lorenzi N. Behavioral Health Information Systems: Evaluating Readiness and User Acceptance. MD Computing 2000; 17 (04) 50-52.
• 30 Brender J, Ammenwerth E, Nykanen P, Talmon J. Factors influencing success and failure of health informatics systems – a pilot Delphi study. Methods Inf Med 2006; 45 (01) 125-136. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=16482383.
• 31 Beynon-Davies P, Lloyd-Williams M. When health information systems fail. Topics Health Inform Manage 1999; 20 (01) 66-79.
• 32 Southon G, Sauer C, Dampney K. Lessons from a failed information systems initiative: issues for complex organisations. Int J Med Inform 1999; 55 (01) 33-46.
• 33 Ribière V, La Salle A, Khorramshahgol R. Measuring Customer Satisfaction: A Case Study in Hospital Information Systems Evaluation. In: The First World Customer Service congress, Oct 29-31, 1997; 1997; Tysons Corner, VA: 1997
• 34 Dugas M, Eckholt M, Bunzemeier H. Benchmarking of hospital information systems: monitoring of discharge letters and scheduling can reveal heterogeneities and time trends. BMC Med Inform Decis Mak 2008; 8: 15. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db= PubMed&dopt=Citation&list_uids=18423046.
• 35 Kaplan R, Norton D. The Balanced Scorecard – Measures that Drive Performance. Harvard Business Review 1992; January-February: 71-79.
• 36 Martin J. Strategic Information Planning Methodologies. 2nd ed. Prentice Hall: Englewood Cliffs; 1989
• 37 Ohmann C, Boy O, Yang Q. A systematic approach to the assessment of user satisfaction with health care systems: constructs, models and instruments. In: Pappas C. editor. Medical Informatics Europe ’97. Conference proceedings. Amsterdam: IOS Press; 1997. pp 781-785.
• 38 Laerum H, Ellingsen G, Faxvaag A. Doctors’ use of electronic medical record systems in hospitals: cross sectional survey. Bmj 2001; 323 7325 1344-1348. http://www.pubmedcentral.nih.gov/tocrender.fcgi?action=cited&artid=61372.
• 39 Nielsen J. Usability Engineering. New York: Academic Press; 1993
• 40 Brigl B, Huebner-Bloder G, Wendt T, Haux R, Winter A. Architectural quality criteria for hospital information systems. AMIA Annu Symp Proc. 2005 pp 81-85. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt= Citation&list_uids=16779006.
• 41 Protti D. A proposal to use a balanced scorecard to evaluate Information for Health: an information strategy for the modern NHS (1998–2005). Comput Biol Med 2002; 32 (03) 221-236.