Numeracy and Understanding of Quantitative Aspects of Predictive Models: A Pilot Study

Gary E. Weissman; Kuldeep N. Yadav; Vanessa Madden; Katherine R. Courtright; Joanna L. Hart; David A. Asch; Marilyn M. Schapira; Scott D. Halpern

doi:10.1055/s-0038-1669457

Applied Clinical Informatics, Inhaltsverzeichnis

Appl Clin Inform 2018; 09(03): 683-692
DOI: 10.1055/s-0038-1669457

Research Article

Georg Thieme Verlag KG Stuttgart · New York

Numeracy and Understanding of Quantitative Aspects of Predictive Models: A Pilot Study

Gary E. Weissman

¹Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States

²Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States

³Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁴Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States

,

Kuldeep N. Yadav

²Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States

³Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁴Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States

,

Vanessa Madden

²Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States

³Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁴Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States

,

Katherine R. Courtright

¹Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States

²Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States

³Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁴Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States

,

Joanna L. Hart

¹Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States

²Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States

³Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁴Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States

,

David A. Asch

¹Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁴Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁵Center for Health Care Innovation, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁶The Center for Health Equity Research and Promotion, Philadelphia VA Medical Center, Philadelphia, Pennsylvania, United States

,

Marilyn M. Schapira

¹Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁶The Center for Health Equity Research and Promotion, Philadelphia VA Medical Center, Philadelphia, Pennsylvania, United States

,

Scott D. Halpern

¹Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States

²Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States

³Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States

⁴Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States

› Institutsangaben

Abstract

Background The assessment of user preferences for performance characteristics of patient-oriented clinical prediction models is lacking. It is unknown if complex statistical aspects of prediction models are readily understandable by a general audience.

Objective A pilot study was conducted among nonclinical audiences to determine the feasibility of interpreting statistical concepts that describe the performance of prediction models.

Methods We conducted a cross-sectional electronic survey using the Amazon Mechanical Turk platform. The survey instrument included educational modules about predictive models, sensitivity, specificity, and confidence intervals (CIs). Follow-up questions tested participants' abilities to interpret these characteristics with both verbatim and gist knowledge. Objective and subjective numeracy were assessed using previously validated instruments. We also tested understanding of these concepts when embedded in a sample discrete choice experiment task to establish feasibility for future elicitation of preferences using a discrete choice experiment design. Multivariable linear regression was used to identify factors associated with correct interpretation of statistical concepts.

Results Among 534 respondents who answered all nine questions, the mean correct responses was 95.9% (95% CI, 93.8–97.4) for sensitivity, 93.1% (95% CI, 90.5–95.0) for specificity, and 86.6% (95% CI, 83.3–89.3) for CIs. Verbatim interpretation was high for all concepts, but significantly higher than gist only for CIs (p < 0.001). Scores on each discrete choice experiment tasks were slightly lower in each category. Both objective and subjective numeracy were positively associated with an increased proportion of correct responses (p < 0.001).

Conclusion These results suggest that a nonclinical audience can interpret quantitative performance measures of predictive models with very high accuracy. Future development of patient-facing clinical prediction models can feasibly incorporate patient preferences for model features into their development.

Keywords

forecasting - health communication - decision support techniques - clinical decision-making - education of patients

Volltext

Referenzen

References
1 Ryan M. Discrete choice experiments in health care. BMJ 2004; 328 (7436): 360-361
2 Bridges JF, Hauber AB, Marshall D. , et al. Conjoint analysis applications in health--a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health 2011; 14 (04) 403-413
3 Ibrahim SA. Decision aids and elective joint replacement — how knowledge affects utilization. N Engl J Med 2017; 376 (26) 2509-2511
4 Elwyn G, Cochran N, Pignone M. Shared decision making—the importance of diagnosing preferences. JAMA Intern Med 2017; 177 (09) 1239-1240
5 Malhotra C, Farooqui MA, Kanesvaran R, Bilger M, Finkelstein E. Comparison of preferences for end-of-life care among patients with advanced cancer and their caregivers: a discrete choice experiment. Palliat Med 2015; 29 (09) 842-850
6 Halpern SD, Berns JS, Israni AK. Willingness of patients to switch from conventional to daily hemodialysis: looking before we leap. Am J Med 2004; 116 (09) 606-612
7 Armstrong K, Putt M, Halbert CH. , et al. The influence of health care policies and health care system distrust on willingness to undergo genetic testing. Med Care 2012; 50 (05) 381-387
8 Wordsworth S, Ryan M, Skåtun D, Waugh N. Women's preferences for cervical cancer screening: a study using a discrete choice experiment. Int J Technol Assess Health Care 2006; 22 (03) 344-350
9 Hol L, de Bekker-Grob EW, van Dam L. , et al. Preferences for colorectal cancer screening strategies: a discrete choice experiment. Br J Cancer 2010; 102 (06) 972-980
10 de Bekker-Grob EW, Rose JM, Donkers B, Essink-Bot ML, Bangma CH, Steyerberg EW. Men's preferences for prostate cancer screening: a discrete choice experiment. Br J Cancer 2013; 108 (03) 533-541
11 Goldstein BA, Navar AM, Pencina MJ, Ioannidis JP. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J Am Med Inform Assoc 2017; 24 (01) 198-208
12 Amarasingham R, Audet A-MJ, Bates DW. , et al. Consensus statement on electronic health predictive analytics: a guiding framework to address challenges. EGEMS (Wash DC) 2016; 4 (01) 1163
13 Krumholz HM. Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system. Health Aff (Millwood) 2014; 33 (07) 1163-1170
14 Charles D, Gabriel M, Searcy T. Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008 - 2014. ONC Data Brief 2015; 23: 1-10
15 Darcy AM, Louie AK, Roberts LW. Machine learning and the profession of medicine. JAMA 2016; 315 (06) 551-552
16 Schapira MM, Imbert D, Oh E, Byhoff E, Shea JA. Public engagement with scientific evidence in health: a qualitative study among primary-care patients in an urban population. Public Underst Sci 2016; 25 (05) 612-626
17 Peters E, Hart PS, Fraenkel L. Informing patients: the influence of numeracy, framing, and format of side effect information on risk perceptions. Med Decis Making 2011; 31 (03) 432-436
18 Bodemer N, Meder B, Gigerenzer G. Communicating relative risk changes with baseline risk: presentation format and numeracy matter. Med Decis Making 2014; 34 (05) 615-626
19 Schapira MM, Aggarwal C, Akers S. , et al. How patients view lung cancer screening. The role of uncertainty in medical decision making. Ann Am Thorac Soc 2016; 13 (11) 1969-1976
20 Hoffrage U, Gigerenzer G. Using natural frequencies to improve diagnostic inferences. Acad Med 1998; 73 (05) 538-540
21 Whiting PF, Davenport C, Jameson C. , et al. How well do health professionals interpret diagnostic information? A systematic review. BMJ Open 2015; 5 (07) e008155
22 Ott M. 2017 . Available at: https://uniqueturker.myleott.com/ . Accessed August 5, 2018
23 Amazon Web Services. Mechanical Turk Concepts; 2017 . Available at: http://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/mechanical-turk-concepts.html . Accessed August 5, 2018
24 Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA 2012; 307 (02) 182-192
25 Cardona-Morrell M, Hillman K. Development of a tool for defining and identifying the dying patient in hospital: Criteria for Screening and Triaging to Appropriate aLternative care (CriSTAL). BMJ Support Palliat Care 2015; 5 (01) 78-90
26 Litman L, Robinson J, Abberbock T. TurkPrime.com: a versatile crowdsourcing data acquisition platform for the behavioral sciences. Behav Res Methods 2017; 49 (02) 433-444
27 Sladakovic J, Jansen J, Hersch J, Turner R, McCaffery K. The differential effects of presenting uncertainty around benefits and harms on treatment decision making. Patient Educ Couns 2016; 99 (06) 974-980
28 Fagerlin A, Zikmund-Fisher BJ, Ubel PA. Helping patients decide: ten steps to better risk communication. J Natl Cancer Inst 2011; 103 (19) 1436-1443
29 Hawley ST, Zikmund-Fisher B, Ubel P, Jancovic A, Lucas T, Fagerlin A. The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns 2008; 73 (03) 448-455
30 Halpern J, Arnold RM. Affective forecasting: an unrecognized challenge in making serious health decisions. J Gen Intern Med 2008; 23 (10) 1708-1712
31 Fagerlin A, Peters E, Schwartz A. , et al. Cognitive and affective influences on health decisions. In: Suls JM, Davidson KW, Kaplan RM, eds. Handbook of Health Psychology and Behavioral Medicine. New York, NY: Guilford Press; 2010: 49-63
32 Reyna VF. A theory of medical decision making and health: fuzzy trace theory. Med Decis Making 2008; 28 (06) 850-865
33 Reyna VF, Nelson WL, Han PK, Dieckmann NF. How numeracy influences risk comprehension and medical decision making. Psychol Bull 2009; 135 (06) 943-973
34 Dolan JG, Cherkasky OA, Li Q, Chin N, Veazie PJ. Should health numeracy be assessed objectively or subjectively?. Med Decis Making 2016; 36 (07) 868-875
35 Schapira MM, Walker CM, Miller T. , et al. Development and validation of the numeracy understanding in Medicine Instrument short form. J Health Commun 2014; 19 (Suppl (Suppl. 02) 240-253
36 Fagerlin A, Zikmund-Fisher BJ, Ubel PA, Jankovic A, Derry HA, Smith DM. Measuring numeracy without a math test: development of the Subjective Numeracy Scale. Med Decis Making 2007; 27 (05) 672-680
37 Wallace E, McDowell R, Bennett K, Fahey T, Smith SM. Comparison of count-based multimorbidity measures in predicting emergency admission and functional decline in older community-dwelling adults: a prospective cohort study. BMJ Open 2016; 6 (09) e013089
38 Brilleman SL, Salisbury C. Comparing measures of multimorbidity to predict outcomes in primary care: a cross sectional study. Fam Pract 2013; 30 (02) 172-178
39 R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2015
40 Brewer NT, Richman AR, DeFrank JT, Reyna VF, Carey LA. Improving communication of breast cancer recurrence risk. Breast Cancer Res Treat 2012; 133 (02) 553-561
41 Windish DM, Huot SJ, Green ML. Medicine residents' understanding of the biostatistics and results in the medical literature. JAMA 2007; 298 (09) 1010-1022
42 Waldrop-Valverde D, Osborn CY, Rodriguez A, Rothman RL, Kumar M, Jones DL. Numeracy skills explain racial differences in HIV medication management. AIDS Behav 2010; 14 (04) 799-806
43 Hall CW, Davis NB, Bolen LM, Chia R. Gender and racial differences in mathematical performance. J Soc Psychol 1999; 139 (06) 677-689
44 Langford AT, Resnicow K, Roberts JS, Zikmund-Fisher BJ. Racial and ethnic differences in direct-to-consumer genetic tests awareness in HINTS 2007: sociodemographic and numeracy correlates. J Genet Couns 2012; 21 (03) 440-447
45 Schapira MM, Fletcher KE, Gilligan MA. , et al. A framework for health numeracy: how patients use quantitative skills in health care. J Health Commun 2008; 13 (05) 501-517
46 Brundage MD, Smith KC, Little EA, Bantug ET, Snyder CF. ; PRO Data Presentation Stakeholder Advisory Board. Communicating patient-reported outcome scores using graphic formats: results from a mixed-methods evaluation. Qual Life Res 2015; 24 (10) 2457-2472
47 Brundage M, Feldman-Stewart D, Leis A. , et al. Communicating quality of life information to cancer patients: a study of six presentation formats. J Clin Oncol 2005; 23 (28) 6949-6956
48 Bromgard GD, Trafimow D, Silvera DH. The influence of presentation format on the “bigger is better” (BIB) effect. Psychol Rep 2013; 112 (02) 458-468
49 NCHS, National Health and Nutrition Examination Survey. See Appendix I, National Health and Nutrition Examination Survey (NHANES). Available at: https://www.cdc.gov/nchs/data/hus/hus16.pdf#079 . Accessed June 15, 2018
50 Paolacci G, Chandler J, Ipeirotis PG. Running experiments on Amazon Mechanical Turk. Judgm Decis Mak 2010; 5: 411-419
51 Korfage IJ, Fuhrel-Forbis A, Ubel PA. , et al. Informed choice about breast cancer prevention: randomized controlled trial of an online decision aid intervention. Breast Cancer Res 2013; 15 (05) R74
52 Daneman M, Hannon B. Using working memory theory to investigate the construct validity of multiple-choice reading comprehension tests such as the SAT. J Exp Psychol Gen 2001; 130 (02) 208-223

Zusatzmaterial

Supplementary Material