Evaluation Considerations for Secondary Uses of Clinical Data: Principles for an Evidence-based Approach to Policy and Implementation of Secondary Analysis

Summary

Objectives: To set the scientific context and then suggest principles for an evidence-based approach to secondary uses of clinical data, covering both evaluation of the secondary uses of data and evaluation of health systems and services based upon secondary uses of data.

Method: Working Group review of selected literature and policy approaches.

Results: We present important considerations in the evaluation of secondary uses of clinical data from the angles of governance and trust, theory, semantics, and policy. We make the case for a multi-level and multi-factorial approach to the evaluation of secondary uses of clinical data and describe a methodological framework for best practice. We emphasise the importance of evaluating the governance of secondary uses of health data in maintaining trust, which is essential for such uses. We also offer examples of the re-use of routine health data to demonstrate how it can support evaluation of clinical performance and optimize health IT system design.

Conclusions: Great expectations are resting upon “Big Data” and innovative analytics. However, to build and maintain public trust, improve data reliability, and assure the validity of analytic inferences, there must be independent and transparent evaluation. A mature and evidence-based approach needs not merely data science, but must be guided by the broader concerns of applied health informatics.

Keywords

Data mining - secondary use - informed consent - evidence-based practice - program evaluation

• References

• 1 OECD. Health Data Governance: Privacy, Monitoring and Research. Paris: OECD Publishing; 2015 [cited 2016 21 November]. Available from: http://www.oecd-ilibrary.org/social-issues-migration-health/health-data-governance_9789264244566-en
• 2 Safran C, Bloomrosen M, Hammond WE, Labkoff S, Markel-Fox S, Tang PC. et al. Toward a national framework for the secondary use of health data: an American Medical Informatics Association White Paper. J Am Med Inform Assoc 2007; Jan-Feb; 14 (01) 1-9.
• 3 OECD. ICTs and the Health Sector: Towards Smarter Health and Wellness Models. Paris: OECD Publishing; 2013 [cited 2016 21 November]. Available from: http://www.oecd.org/sti/ieconomy/ ict-and-the-health-sector.htm
• 4 Rigby M, Ronchi E, Graham S. Evidence for building a smarter health and wellness future--key messages and collected visions from a joint OECD and NSF workshop. Int J Med Inform 2013; Apr; 82 (04) 209-19.
• 5 Friedman C, Rigby M. Conceptualising and creating a global learning health system. Int J Med Inform 2013; Apr; 82 (04) e63-71.
• 6 Wyatt JC. Evidence-based Health Informatics and the Scientific Development of the Field. In: Ammenwerth E, Rigby M. editors. Evidence-based Health Informatics: Promoting safety and efficiency through scientific methods and ethical policy. Stud Health Technol Inform Amsterdam. IOS Press; 2016: 14-24.
• 7 Bowling A. Research methods in health: investigating health and health services. 4th edition. Buckingham: McGraw Hill: Open University Press; 2014
• 8 Campos-Castillo C, Anthony DL. The double-edged sword of electronic health records: implications for patient disclosure. J Am Med Inform Assoc 22 (e1) e130-40.
• 9 Open Knowledge Network. The Open Definition [cited 2017 12 April]. Available from: http://open-definition.org/
• 10 Verhulst S, Noveck BS, Caplan R, Brown K, Paz C. The Open Data Era in Health and Social Care. 2014 [cited 2017 12 April]. Available from: http://thegovlab.org/nhs/
• 11 Sanderson SC, Brothers KB, Mercaldo ND, Clayton EW, Antommaria AH, Aufox SA. et al. Public Attitudes toward Consent and Data Sharing in Bio-bank Research: A Large Multi-site Experimental Survey in the US. Am J Hum Genet 2017; Mar 02; 100 (03) 414-27.
• 12 Patil S, Lu H, Saunders CL, Potoglou D, Robinson N. Public preferences for electronic health data storage, access, and sharing — evidence from a pan-European survey. J Am Med Inform Assoc 2016; 23 (06) 1096-106.
• 13 El Emam K, Jonker E, Arbuckle L, Malin B. A systematic review of re-identification attacks on health data. PLoS One 2011; 06 (12) e28071.
• 14 Information Commissioner’s Office. Anonymisation: managing data protection risk code of practice. 2012 [cited 2017 12 April]. Available from https://ico.org.uk/media/1061/anonymisation-code.pdf
• 15 NHS National Services Scotland. About ISD: Confidentiality. 2010 [cited 2016 1 December]. Available from: http://www.isdscotland.org/About-ISD/Confidentiality/#secondaryUse
• 16 WebFinance Inc. Governance. 2016 [cited 2016 21 November]. Available from: http://www.business-dictionary.com/definition/governance.html
• 17 The Data Governance Institute. Defining data governance. 2015 [cited 2016 21 November]. Available from: http://www.datagovernance.com/defining-data-governance/
• 18 Ingenerf J. Telemedicine and terminology: different needs of context information. IEEE Trans Inf Technol Biomed 1999; Jun; 03 (02) 92-100.
• 19 Coiera E. Guide to Health Informatics. 3rd ed. CRC press; 2015
• 20 Smith SW, Koppel R. Healthcare information technology’s relativity problems: a typology of how patients’ physical reality, clinicians’ mental models, and healthcare information technology differ. J Am Med Inform Assoc 2014; 21 (01) 117-31.
• 21 van der Lei J. Use and abuse of computer-stored medical records. Methods Inf Med 1991; Apr; 30 (02) 79-80.
• 22 Nolan J, McNair P, Brender J. Factors influencing transferability of knowledge-based systems. Int J Biomed Comput 1991; 27: 7-26.
• 23 Garza M, Del Fiol G, Tenenbaum J, Walden A, Zozus MN. Evaluating common data models for use with a longitudinal community registry. J Biomed Inform 2016; Dec; 64: 333-41.
• 24 Trifiro G, Fourrier-Reglat A, Sturkenboom MC, Diaz CAcedo, van der Lei J. The EU-ADR project: preliminary results and perspective. Stud Health Technol Inform 2009; 148: 43-9.
• 25 Brender J, McNair P. Meta-level concepts as a basis for structuring data. In: Hejlesen O, Bygholm A, Bertelsen P. editors. 4th Scandinavian Conference on Health Informatics. Aalborg, Denmark: Virtual Centre for Health Informatics, Aalborg University; 2006: 17-20.
• 26 Brender J, McNair P. Enhancing semantic interoperability: a model for making context explicit. In: Harorimana D, Watkins D. editors. 9th European Conference on Knowledge Management. Southampton Solent University; UK: 2008: 83-91.
• 27 Scott PJ, Cornet R, McCowan C, Peek N, Fraccaro P, Geifman N. et al. Informatics for Health 2017: Advancing both science and practice. J Innov Health Inform 2017; 24 (01) 1-2.
• 28 Longhurst CA, Harrington RA, Shah NH. A ‘green button’ for using aggregate patient data at the point of care Health Aff (Millwood). 2014; 33 (07) 1229-35.
• 29 Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med 2011; 365 (19) 1758-9.
• 30 Gabriel SE, Normand SL. Getting the methods right--the foundation of patient-centered outcomes research. N Engl J Med 2012; 367 (09) 787-90.
• 31 Tunis SR, Clarke M, Gorst SL, Gargon E, Blazeby JM, Altman DG. et al. Improving the relevance and consistency of outcomes in comparative effectiveness research. J Comp Eff Res 2016; 05 (02) 193-205.
• 32 Collins FS, Varmus H. A new initiative on precision medicine. N Engl J Med 2015; 372 (09) 793-5.
• 33 Coleman J. Segmenting data privacy. Cross-industry initiative aims to piece out privacy within the health record. J AHIMA 2013; 84 (02) 34-8.
• 34 Clancy CM. Common formats allow uniform collection and reporting of patient safety data by patient safety organizations. Am J Med Qual 2010; 25 (01) 73-5.
• 35 Armstrong S. The computer will assess you now. BMJ 2016; 355: i5680.
• 36 OECD. Improving Health Sector Efficiency: The Role of Information and Communication Technologies Paris: OECD Publishing. 2010 [cited 2016 21 November]. Available from: http://www.oecd-ilibrary.org/social-issues-migration-health/improving-health-sector-efficiency_9789264084612-en
• 37 Bélanger F, Crossler RE. Privacy in the digital age: a review of information privacy research in information systems. MIS Quarterly 2011; 35 (04) 1017-42.
• 38 Pavlou PA. State of the information privacy literature: where are we now and where should we go?. MIS Quarterly 2011; 35 (04) 977-88.
• 39 Ruotsalainen PS, Blobel BG, Seppala AV, Sorvari HO, Nykanen PA. A conceptual framework and principles for trusted pervasive health. J Med Internet Res 2012; 14 (02) e52.
• 40 European Commission. eHealth Action Plan 2012-2020 - Innovative healthcare for the 21st century 2012 [cited 2016 21 November]. Available from: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52012DC0736&from=EN
• 41 OECD. Draft OECD guide to measuring ICTs in the Health Sector 2015 [cited 2016 21 November]. Available from: https://www.oecd.org/health/health-systems/Draft-oecd-guide-to-measuringicts-in-the-health-sector.pdf
• 42 Hyppönen H, Kangas M, Reponen J, Nohr C, Villumsen S, Koch S, et al. Nordic eHealth benchmarking. Status 2014. 2014 [cited 2016 21 November]. Available from: https://norden.diva-portal.org/smash/get/diva2:821230/FULLTEXT01.pdf
• 43 Gude WT, van der Veer SN, de Keizer NF, Coiera E, Peek N. Optimizing Digital Health Informatics Interventions Through Unobtrusive Quantitative Process Evaluations. Stud Health Technol Inform 2016; 228: 594-8.
• 44 Dentler K, Numans ME, ten Teije A, Cornet R, de Keizer NF. Formalization and computation of quality measures based on electronic medical records. J Am Med Inform Assoc 2014; Mar-Apr; 21 (02) 285-91.
• 45 Dentler K, Cornet R, Ten Teije A, Tytgat K, Klinkenbujl J, De Keizer N. The reproducibility of CLIF, a method for clinical quality indicator formalisation. Stud Health Technol Inform 2012; 180: 113-7.
• 46 Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD. et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012 Jun 13(6) CD000259.
• 47 Roshanov PS, Misra S, Gerstein HC, Garg AX, Sebaldt RJ, Mackay JA. et al. Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review. Implement Sci 2011; Aug 03; 06: 92.
• 48 Musen MA, Middleton B, Greenes RA. Clinical Decision-Support Systems. In: Shortliffe EH, Cimino JJ. editors. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. London: Springer; 2014: 643-74.
• 49 Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ. et al. No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci 2014; Jan 17; 09: 14.
• 50 Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD. et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med 2014; Nov; 29 (11) 1534-41.
• 51 Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013; Jan 12; 14: 15.
• 52 Davidson EJ. Evaluation methodology basics: The nuts and bolts of sound evaluation. London: Sage; 2005
• 53 Bureau of Health Information. Data Matters -Linking data to unlock information. The use of linked data in healthcare performance assessment. Sydney, NSW: Bureau of Health Information; 2015 [cited 2016 21 November]. Available from: http://www.bhi.nsw.gov.au/__data/assets/pdf_file/0003/290712/DataMatters.pdf
• 54 Honeyman M, Dunn P, McKenna H. A digital NHS? An introduction to the digital agenda and plans for implementation. London: The King’s Fund; 2016 [cited 2016 21 November]. Available from: https://www.kingsfund.org.uk/publications/digital-nhs
• 55 National Patient Safety Foundation. Free from Harm - Accelerating Patient Safety Improvement Fifteen Years after To Err Is Human. Boston, MA: National Patient Safety Foundation; 2015 [cited 2016 21 November]. Available from: https://psnet.ahrq.gov/resources/resource/29554/free-from-harm-accelerating-patient-safety-improvement-fifteen-years-after-to-err-is-human
• 56 Singh H, Sittig DF. Measuring and improving patient safety through health information technology: The Health IT Safety Framework. BMJ Qual Saf 2016; Apr; 25 (04) 226-32.
• 57 Okun S, McGraw D, Stang PEL, Goldmann D, Kupersmith J. et al. Making the case for continuous learning from routinely collected data. Washington, DC: Institute of Medicine; 2013 [cited 2016 21 November]. Available from: https://nam.edu/wp-content/uploads/2015/06/MakingtheCase.pdf
• 58 Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Health Inf Sci Syst 2014; 02: 3.
• 59 Shaw J, Taylor R, Dix K. Uses and abuses of performance data in healthcare. London: Dr Foster; 2015 [cited 2016 21 November]. Available from: http://www.drfoster.com/updates/recent-publica-tions/uses-and-abuses-of-performance-data-in-healthcare/
• 60 Abdelhak M, Grostick S, Hankin M, Jacobs E. Health Information: Management of a Strategic Resource. Philadelphia: WB Saunders; 1996
• 61 Streiner DL, Norman GR, Cairney J. Health Measurement Scales. A practical guide to their development and use. 5th ed. Oxford: OUP; 2014
• 62 Strome TL. Healthcare analytics for quality and performance improvement. Chichester: John Wiley & Sons; 2013
• 63 Brender J, Talmon J, de Keizer N, Nykanen P, Rigby M, Ammenwerth E. STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration. Appl Clin Inform 2013; 04 (03) 331-58.
• 64 McNair JB. Handbook of evaluation methods for health informatics. London: Elsevier Academic Press; 2006
• 65 Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC. et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform 2011; Dec; 80 (12) 815-27.
• 66 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P, Rigby M. STARE-HI--Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; Jan; 78 (01) 1-9.
• 67 Hackl WO, Ammenwerth E. SPIRIT: Systematic Planning of Intelligent Reuse of Integrated Clinical Routine Data. A Conceptual Best-practice Framework and Procedure Model. Methods Inf Med 2016; 55 (02) 114-24.
• 68 Fayyad U, Piatetsky-Shapiro G, Smyth P. From data mining to knowledge discovery in databases. AI Magazine 1996; 17 (03) 37.
• 69 Codd E, Codd S, Salley C. Providing OLAP (On-line analytical processing) to user-analysts: an IT mandate. Ann Arbor, Michigan: Codd & Associates; 1993
• 70 Ammenwerth E, Rigby M. editors. Evidence-based Health Informatics: Promoting safety and efficiency through scientific methods and ethical policy. Amsterdam: IOS Press; 2016