Download PDF
Appl Clin Inform 2018; 09(03): 683-692
DOI: 10.1055/s-0038-1669457
Research Article
Georg Thieme Verlag KG Stuttgart · New York

Numeracy and Understanding of Quantitative Aspects of Predictive Models: A Pilot Study

Gary E. Weissman
1   Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
2   Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States
3   Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States
4   Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Kuldeep N. Yadav
2   Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States
3   Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States
4   Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Vanessa Madden
2   Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States
3   Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States
4   Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Katherine R. Courtright
1   Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
2   Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States
3   Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States
4   Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Joanna L. Hart
1   Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
2   Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States
3   Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States
4   Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
David A. Asch
1   Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
4   Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
5   Center for Health Care Innovation, University of Pennsylvania, Philadelphia, Pennsylvania, United States
6   The Center for Health Equity Research and Promotion, Philadelphia VA Medical Center, Philadelphia, Pennsylvania, United States
,
Marilyn M. Schapira
1   Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
6   The Center for Health Equity Research and Promotion, Philadelphia VA Medical Center, Philadelphia, Pennsylvania, United States
,
Scott D. Halpern
1   Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
2   Department of Medicine, Palliative and Advanced Illness Research Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States
3   Fostering Improvement in End-of-Life Decision Science Program, University of Pennsylvania, Philadelphia, Pennsylvania, United States
4   Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
› Author Affiliations Funding This work was supported by the National Institutes of Health/NHLBI (T32-HL098054), the Masters of Science in Health Policy Research Program, and the Center for Health Incentives and Behavioral Economics/Penn Roybal Center at the Leonard Davis Institute through a grant from the National Institutes of Health/NIA (P30-AG034546) at the Perelman School of Medicine, University of Pennsylvania. The views expressed here are those of the authors and not of the funders.
Further Information

Publication History

10 May 2018

17 July 2018

Publication Date:
29 August 2018 (online)

   Permissions and Reprints

Abstract

Background The assessment of user preferences for performance characteristics of patient-oriented clinical prediction models is lacking. It is unknown if complex statistical aspects of prediction models are readily understandable by a general audience.

Objective A pilot study was conducted among nonclinical audiences to determine the feasibility of interpreting statistical concepts that describe the performance of prediction models.

Methods We conducted a cross-sectional electronic survey using the Amazon Mechanical Turk platform. The survey instrument included educational modules about predictive models, sensitivity, specificity, and confidence intervals (CIs). Follow-up questions tested participants' abilities to interpret these characteristics with both verbatim and gist knowledge. Objective and subjective numeracy were assessed using previously validated instruments. We also tested understanding of these concepts when embedded in a sample discrete choice experiment task to establish feasibility for future elicitation of preferences using a discrete choice experiment design. Multivariable linear regression was used to identify factors associated with correct interpretation of statistical concepts.

Results Among 534 respondents who answered all nine questions, the mean correct responses was 95.9% (95% CI, 93.8–97.4) for sensitivity, 93.1% (95% CI, 90.5–95.0) for specificity, and 86.6% (95% CI, 83.3–89.3) for CIs. Verbatim interpretation was high for all concepts, but significantly higher than gist only for CIs (p < 0.001). Scores on each discrete choice experiment tasks were slightly lower in each category. Both objective and subjective numeracy were positively associated with an increased proportion of correct responses (p < 0.001).

Conclusion These results suggest that a nonclinical audience can interpret quantitative performance measures of predictive models with very high accuracy. Future development of patient-facing clinical prediction models can feasibly incorporate patient preferences for model features into their development.

Keywords

forecasting - health communication - decision support techniques - clinical decision-making - education of patients

Protection of Human and Animal Subjects

This study was considered exempt by the Institutional Review Board of the University of Pennsylvania.


Supplementary Material

 

  • References

  • 1 Ryan M. Discrete choice experiments in health care. BMJ 2004; 328 (7436): 360-361
    CrossrefPubMedGoogle Scholar
  • 2 Bridges JF, Hauber AB, Marshall D. , et al. Conjoint analysis applications in health--a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health 2011; 14 (04) 403-413
    CrossrefPubMedGoogle Scholar
  • 3 Ibrahim SA. Decision aids and elective joint replacement — how knowledge affects utilization. N Engl J Med 2017; 376 (26) 2509-2511
    CrossrefPubMedGoogle Scholar
  • 4 Elwyn G, Cochran N, Pignone M. Shared decision making—the importance of diagnosing preferences. JAMA Intern Med 2017; 177 (09) 1239-1240
    CrossrefPubMedGoogle Scholar
  • 5 Malhotra C, Farooqui MA, Kanesvaran R, Bilger M, Finkelstein E. Comparison of preferences for end-of-life care among patients with advanced cancer and their caregivers: a discrete choice experiment. Palliat Med 2015; 29 (09) 842-850
    CrossrefPubMedGoogle Scholar
  • 6 Halpern SD, Berns JS, Israni AK. Willingness of patients to switch from conventional to daily hemodialysis: looking before we leap. Am J Med 2004; 116 (09) 606-612
    CrossrefPubMedGoogle Scholar
  • 7 Armstrong K, Putt M, Halbert CH. , et al. The influence of health care policies and health care system distrust on willingness to undergo genetic testing. Med Care 2012; 50 (05) 381-387
    CrossrefPubMedGoogle Scholar
  • 8 Wordsworth S, Ryan M, Skåtun D, Waugh N. Women's preferences for cervical cancer screening: a study using a discrete choice experiment. Int J Technol Assess Health Care 2006; 22 (03) 344-350
    CrossrefPubMedGoogle Scholar
  • 9 Hol L, de Bekker-Grob EW, van Dam L. , et al. Preferences for colorectal cancer screening strategies: a discrete choice experiment. Br J Cancer 2010; 102 (06) 972-980
    CrossrefPubMedGoogle Scholar
  • 10 de Bekker-Grob EW, Rose JM, Donkers B, Essink-Bot ML, Bangma CH, Steyerberg EW. Men's preferences for prostate cancer screening: a discrete choice experiment. Br J Cancer 2013; 108 (03) 533-541
    CrossrefPubMedGoogle Scholar
  • 11 Goldstein BA, Navar AM, Pencina MJ, Ioannidis JP. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J Am Med Inform Assoc 2017; 24 (01) 198-208
    CrossrefPubMedGoogle Scholar
  • 12 Amarasingham R, Audet A-MJ, Bates DW. , et al. Consensus statement on electronic health predictive analytics: a guiding framework to address challenges. EGEMS (Wash DC) 2016; 4 (01) 1163
    PubMedGoogle Scholar
  • 13 Krumholz HM. Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system. Health Aff (Millwood) 2014; 33 (07) 1163-1170
    CrossrefPubMedGoogle Scholar
  • 14 Charles D, Gabriel M, Searcy T. Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008 - 2014. ONC Data Brief 2015; 23: 1-10
    PubMedGoogle Scholar
  • 15 Darcy AM, Louie AK, Roberts LW. Machine learning and the profession of medicine. JAMA 2016; 315 (06) 551-552
    CrossrefPubMedGoogle Scholar
  • 16 Schapira MM, Imbert D, Oh E, Byhoff E, Shea JA. Public engagement with scientific evidence in health: a qualitative study among primary-care patients in an urban population. Public Underst Sci 2016; 25 (05) 612-626
    CrossrefPubMedGoogle Scholar
  • 17 Peters E, Hart PS, Fraenkel L. Informing patients: the influence of numeracy, framing, and format of side effect information on risk perceptions. Med Decis Making 2011; 31 (03) 432-436
    CrossrefPubMedGoogle Scholar
  • 18 Bodemer N, Meder B, Gigerenzer G. Communicating relative risk changes with baseline risk: presentation format and numeracy matter. Med Decis Making 2014; 34 (05) 615-626
    CrossrefPubMedGoogle Scholar
  • 19 Schapira MM, Aggarwal C, Akers S. , et al. How patients view lung cancer screening. The role of uncertainty in medical decision making. Ann Am Thorac Soc 2016; 13 (11) 1969-1976
    CrossrefPubMedGoogle Scholar
  • 20 Hoffrage U, Gigerenzer G. Using natural frequencies to improve diagnostic inferences. Acad Med 1998; 73 (05) 538-540
    CrossrefPubMedGoogle Scholar
  • 21 Whiting PF, Davenport C, Jameson C. , et al. How well do health professionals interpret diagnostic information? A systematic review. BMJ Open 2015; 5 (07) e008155
    CrossrefPubMedGoogle Scholar
  • 22 Ott M. 2017 . Available at: https://uniqueturker.myleott.com/ . Accessed August 5, 2018
    PubMed
  • 23 Amazon Web Services. Mechanical Turk Concepts; 2017 . Available at: http://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/mechanical-turk-concepts.html . Accessed August 5, 2018
    PubMed
  • 24 Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA 2012; 307 (02) 182-192
    CrossrefPubMedGoogle Scholar
  • 25 Cardona-Morrell M, Hillman K. Development of a tool for defining and identifying the dying patient in hospital: Criteria for Screening and Triaging to Appropriate aLternative care (CriSTAL). BMJ Support Palliat Care 2015; 5 (01) 78-90
    CrossrefPubMedGoogle Scholar
  • 26 Litman L, Robinson J, Abberbock T. TurkPrime.com: a versatile crowdsourcing data acquisition platform for the behavioral sciences. Behav Res Methods 2017; 49 (02) 433-444
    CrossrefPubMedGoogle Scholar
  • 27 Sladakovic J, Jansen J, Hersch J, Turner R, McCaffery K. The differential effects of presenting uncertainty around benefits and harms on treatment decision making. Patient Educ Couns 2016; 99 (06) 974-980
    CrossrefPubMedGoogle Scholar
  • 28 Fagerlin A, Zikmund-Fisher BJ, Ubel PA. Helping patients decide: ten steps to better risk communication. J Natl Cancer Inst 2011; 103 (19) 1436-1443
    CrossrefPubMedGoogle Scholar
  • 29 Hawley ST, Zikmund-Fisher B, Ubel P, Jancovic A, Lucas T, Fagerlin A. The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns 2008; 73 (03) 448-455
    CrossrefPubMedGoogle Scholar
  • 30 Halpern J, Arnold RM. Affective forecasting: an unrecognized challenge in making serious health decisions. J Gen Intern Med 2008; 23 (10) 1708-1712
    CrossrefPubMedGoogle Scholar
  • 31 Fagerlin A, Peters E, Schwartz A. , et al. Cognitive and affective influences on health decisions. In: Suls JM, Davidson KW, Kaplan RM, eds. Handbook of Health Psychology and Behavioral Medicine. New York, NY: Guilford Press; 2010: 49-63
    Google Scholar
  • 32 Reyna VF. A theory of medical decision making and health: fuzzy trace theory. Med Decis Making 2008; 28 (06) 850-865
    CrossrefPubMedGoogle Scholar
  • 33 Reyna VF, Nelson WL, Han PK, Dieckmann NF. How numeracy influences risk comprehension and medical decision making. Psychol Bull 2009; 135 (06) 943-973
    CrossrefPubMedGoogle Scholar
  • 34 Dolan JG, Cherkasky OA, Li Q, Chin N, Veazie PJ. Should health numeracy be assessed objectively or subjectively?. Med Decis Making 2016; 36 (07) 868-875
    CrossrefPubMedGoogle Scholar
  • 35 Schapira MM, Walker CM, Miller T. , et al. Development and validation of the numeracy understanding in Medicine Instrument short form. J Health Commun 2014; 19 (Suppl (Suppl. 02) 240-253
    CrossrefPubMedGoogle Scholar
  • 36 Fagerlin A, Zikmund-Fisher BJ, Ubel PA, Jankovic A, Derry HA, Smith DM. Measuring numeracy without a math test: development of the Subjective Numeracy Scale. Med Decis Making 2007; 27 (05) 672-680
    CrossrefPubMedGoogle Scholar
  • 37 Wallace E, McDowell R, Bennett K, Fahey T, Smith SM. Comparison of count-based multimorbidity measures in predicting emergency admission and functional decline in older community-dwelling adults: a prospective cohort study. BMJ Open 2016; 6 (09) e013089
    CrossrefPubMedGoogle Scholar
  • 38 Brilleman SL, Salisbury C. Comparing measures of multimorbidity to predict outcomes in primary care: a cross sectional study. Fam Pract 2013; 30 (02) 172-178
    CrossrefPubMedGoogle Scholar
  • 39 R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2015
  • 40 Brewer NT, Richman AR, DeFrank JT, Reyna VF, Carey LA. Improving communication of breast cancer recurrence risk. Breast Cancer Res Treat 2012; 133 (02) 553-561
    CrossrefPubMedGoogle Scholar
  • 41 Windish DM, Huot SJ, Green ML. Medicine residents' understanding of the biostatistics and results in the medical literature. JAMA 2007; 298 (09) 1010-1022
    CrossrefPubMedGoogle Scholar
  • 42 Waldrop-Valverde D, Osborn CY, Rodriguez A, Rothman RL, Kumar M, Jones DL. Numeracy skills explain racial differences in HIV medication management. AIDS Behav 2010; 14 (04) 799-806
    CrossrefPubMedGoogle Scholar
  • 43 Hall CW, Davis NB, Bolen LM, Chia R. Gender and racial differences in mathematical performance. J Soc Psychol 1999; 139 (06) 677-689
    CrossrefPubMedGoogle Scholar
  • 44 Langford AT, Resnicow K, Roberts JS, Zikmund-Fisher BJ. Racial and ethnic differences in direct-to-consumer genetic tests awareness in HINTS 2007: sociodemographic and numeracy correlates. J Genet Couns 2012; 21 (03) 440-447
    CrossrefPubMedGoogle Scholar
  • 45 Schapira MM, Fletcher KE, Gilligan MA. , et al. A framework for health numeracy: how patients use quantitative skills in health care. J Health Commun 2008; 13 (05) 501-517
    CrossrefPubMedGoogle Scholar
  • 46 Brundage MD, Smith KC, Little EA, Bantug ET, Snyder CF. ; PRO Data Presentation Stakeholder Advisory Board. Communicating patient-reported outcome scores using graphic formats: results from a mixed-methods evaluation. Qual Life Res 2015; 24 (10) 2457-2472
    CrossrefPubMedGoogle Scholar
  • 47 Brundage M, Feldman-Stewart D, Leis A. , et al. Communicating quality of life information to cancer patients: a study of six presentation formats. J Clin Oncol 2005; 23 (28) 6949-6956
    CrossrefPubMedGoogle Scholar
  • 48 Bromgard GD, Trafimow D, Silvera DH. The influence of presentation format on the “bigger is better” (BIB) effect. Psychol Rep 2013; 112 (02) 458-468
    CrossrefPubMedGoogle Scholar
  • 49 NCHS, National Health and Nutrition Examination Survey. See Appendix I, National Health and Nutrition Examination Survey (NHANES). Available at: https://www.cdc.gov/nchs/data/hus/hus16.pdf#079 . Accessed June 15, 2018
    PubMed
  • 50 Paolacci G, Chandler J, Ipeirotis PG. Running experiments on Amazon Mechanical Turk. Judgm Decis Mak 2010; 5: 411-419
    CrossrefPubMedGoogle Scholar
  • 51 Korfage IJ, Fuhrel-Forbis A, Ubel PA. , et al. Informed choice about breast cancer prevention: randomized controlled trial of an online decision aid intervention. Breast Cancer Res 2013; 15 (05) R74
    CrossrefPubMedGoogle Scholar
  • 52 Daneman M, Hannon B. Using working memory theory to investigate the construct validity of multiple-choice reading comprehension tests such as the SAT. J Exp Psychol Gen 2001; 130 (02) 208-223
    CrossrefPubMedGoogle Scholar