Download PDF
CC BY 4.0 · Methods Inf Med 2023; 62(05/06): 154-164
DOI: 10.1055/a-2155-2021
Original Article for a Focus Theme

A Proposal for a Robust Validated Weighted General Data Protection Regulation-Based Scale to Assess the Quality of Privacy Policies of Mobile Health Applications: An eDelphi Study

Jaime Benjumea
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
,
Jorge Ropero
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
,
Enrique Dorronzoro-Zubiete
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
,
Octavio Rivera-Romero
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
,
Alejandro Carrasco
1   Department of Electronic Technology, Universidad de Sevilla, Sevilla, Spain
› Author Affiliations Funding This work was partially funded by the Cátedra de Telefónica “Inteligencia en la red” of the Universidad de Sevilla and by the Cátedra Indra “Sociedad Digital” of the Universidad de Sevilla. O.R.-R. has received funding from the Universidad de Sevilla and theMinisterio de Universidades of the Spanish Government under the Requalification of Spanish University System Program funded by European Union—NextGenerationEU.
› Further Information
   Permissions and Reprints

Abstract

Background Health care services are undergoing a digital transformation in which the Participatory Health Informatics field has a key role. Within this field, studies aimed to assess the quality of digital tools, including mHealth apps, are conducted. Privacy is one dimension of the quality of an mHealth app. Privacy consists of several components, including organizational, technical, and legal safeguards. Within legal safeguards, giving transparent information to the users on how their data are handled is crucial. This information is usually disclosed to users through the privacy policy document. Assessing the quality of a privacy policy is a complex task and several scales supporting this process have been proposed in the literature. However, these scales are heterogeneous and even not very objective. In our previous study, we proposed a checklist of items guiding the assessment of the quality of an mHealth app privacy policy, based on the General Data Protection Regulation.

Objective To refine the robustness of our General Data Protection Regulation-based privacy scale to assess the quality of an mHealth app privacy policy, to identify new items, and to assign weights for every item in the scale.

Methods A two-round modified eDelphi study was conducted involving a privacy expert panel.

Results After the Delphi process, all the items in the scale were considered “important” or “very important” (4 and 5 in a 5-point Likert scale, respectively) by most of the experts. One of the original items was suggested to be reworded, while eight tentative items were suggested. Only two of them were finally added after Round 2. Eleven of the 16 items in the scale were considered “very important” (weight of 1), while the other 5 were considered “important” (weight of 0.5).

Conclusion The Benjumea privacy scale is a new robust tool to assess the quality of an mHealth app privacy policy, providing a deeper and complementary analysis to other scales. Also, this robust scale provides a guideline for the development of high-quality privacy policies of mHealth apps.

Keywords

participatory health informatics - mHealth apps - privacy policies - assessment scale - GDPR

Note

A first non-peer-reviewed version of this article is available at Research Square.[57] The current version expands and improves the methodology section. It is also more focused on Participatory Health Informatics than the previous version.


Authors' Contribution

J.B. directed the study and took the lead in conducting the Delphi process. J.B. also supported the data analysis, the calculation of statistical values, and the interpretation of the data. J.R. took the lead in drafting the manuscript, supported by J.B., E.D.-Z., O.R.-R., and A.C. J.R. also participated in study direction, contributed to the calculation of statistical values, and supported the data analysis and interpretation of the data. E.D.-Z. participated in study direction, data analysis, and interpretation of the data. O.R.-R. supported the Delphi process, resolved discrepancies, and reviewed the final version of the manuscript. A.C. resolved discrepancies, acquired funding through a research project, participated in the interpretation of data, and reviewed the final version of the manuscript. J.B., J.R., E.D.-Z., O.R.-R., and A.C. selected and contacted experts for the Delphi process.


Supplementary Material



Publication History

Received: 06 July 2022

Accepted: 11 July 2023

Accepted Manuscript online:
17 August 2023

Article published online:
22 December 2023

© 2023. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution License, permitting unrestricted use, distribution, and reproduction so long as the original work is properly cited. (https://creativecommons.org/licenses/by/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Swan M. Health 2050: the realization of personalized medicine through crowdsourcing, the quantified self, and the participatory biocitizen. J Pers Med 2012; 2 (03) 93-118
    CrossrefPubMedSearch in Google Scholar
  • 2 Denecke K, Gabarron E, Petersen C, Merolli M. Defining participatory health informatics - a scoping review. Inform Health Soc Care 2021; 46 (03) 234-243
    CrossrefPubMedSearch in Google Scholar
  • 3 Caulfield BM, Donnelly SC. What is Connected Health and why will it change your practice?. QJM 2013; 106 (08) 703-707
    CrossrefPubMedSearch in Google Scholar
  • 4 Whittaker R. Issues in mHealth: findings from key informant interviews. J Med Internet Res 2012; 14 (05) e129
    CrossrefPubMedSearch in Google Scholar
  • 5 Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task?. Transl Behav Med 2011; 1 (01) 53-71
    CrossrefPubMedSearch in Google Scholar
  • 6 Iribarren SJ, Cato K, Falzon L, Stone PW. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PLoS One 2017; 12 (02) e0170581
    CrossrefPubMedSearch in Google Scholar
  • 7 Ghani Z, Jarl J, Sanmartin Berglund J, Andersson M, Anderberg P. The cost-effectiveness of mobile health (mHealth) interventions for older adults: systematic review. Int J Environ Res Public Health 2020; 17 (15) 5290
    CrossrefPubMedSearch in Google Scholar
  • 8 Rinaldi G, Hijazi A, Haghparast-Bidgoli H. Cost and cost-effectiveness of mHealth interventions for the prevention and control of type 2 diabetes mellitus: a systematic review. Diabetes Res Clin Pract 2020; 162: 108084
    CrossrefPubMedSearch in Google Scholar
  • 9 de Batlle J, Massip M, Vargiu E. et al; CONNECARE-Lleida Group. Implementing mobile health-enabled integrated care for complex chronic patients: intervention effectiveness and cost-effectiveness study. JMIR Mhealth Uhealth 2021; 9 (01) e22135
    CrossrefPubMedSearch in Google Scholar
  • 10 Giunti G, Guisado Fernández E, Dorronzoro Zubiete E, Rivera Romero O. Supply and demand in mhealth apps for persons with multiple sclerosis: systematic search in app stores and scoping literature review. JMIR Mhealth Uhealth 2018; 6 (05) e10512
    CrossrefPubMedSearch in Google Scholar
  • 11 Martín Payo R, Harris J, Armes J. Prescribing fitness apps for people with cancer: a preliminary assessment of content and quality of commercially available apps. J Cancer Surviv 2019; 13 (03) 397-405
    CrossrefPubMedSearch in Google Scholar
  • 12 Byambasuren O, Sanders S, Beller E, Glasziou P. Prescribable mHealth apps identified from an overview of systematic reviews. NPJ Digit Med 2018; 1: 12
    CrossrefPubMedSearch in Google Scholar
  • 13 ISO/TS 82304–2:2021. Health Software. Part 2: Health and Wellness apps – Quality and reliability. Accessed August 22, 2023 at: https://www.iso.org/standard/78182.html
    PubMed
  • 14 Ribeiro N, Moreira L, Barros A, Almeida AM, Santos-Silva F. Guidelines for a cancer prevention smartphone application: a mixed-methods study. Int J Med Inform 2016; 94: 134-142
    CrossrefPubMedSearch in Google Scholar
  • 15 Robertson MC, Tsai E, Lyons EJ. et al. Mobile health physical activity intervention preferences in cancer survivors: a qualitative study. JMIR Mhealth Uhealth 2017; 5 (01) e3
    CrossrefPubMedSearch in Google Scholar
  • 16 Monteiro-Guerra F, Signorelli GR, Rivera-Romero O, Dorronzoro-Zubiete E, Caulfield B. Breast cancer survivors' perspectives on motivational and personalization strategies in mobile app-based physical activity coaching interventions: qualitative study. JMIR Mhealth Uhealth 2020; 8 (09) e18867
    CrossrefPubMedSearch in Google Scholar
  • 17 Giunti G, Kool J, Rivera Romero O, Dorronzoro Zubiete E. Exploring the specific needs of persons with multiple sclerosis for mHealth solutions for physical activity: mixed-methods study. JMIR Mhealth Uhealth 2018; 6 (02) e37
    CrossrefPubMedSearch in Google Scholar
  • 18 Benjumea J, Ropero J, Rivera-Romero O, Dorronzoro-Zubiete E, Carrasco A. Privacy assessment in mobile health apps: scoping review. JMIR Mhealth Uhealth 2020; 8 (07) e18868
    CrossrefPubMedSearch in Google Scholar
  • 19 Minen MT, Stieglitz EJ, Sciortino R, Torous J. Privacy issues in smartphone applications: an analysis of headache/migraine applications. Headache 2018; 58 (07) 1014-1027
    CrossrefPubMedSearch in Google Scholar
  • 20 Leigh S, Ouyang J, Mimnagh C. Effective? Engaging? Secure? Applying the ORCHA-24 framework to evaluate apps for chronic insomnia disorder. Evid Based Ment Health 2017; 20 (04) e20
    CrossrefPubMedSearch in Google Scholar
  • 21 O'Loughlin K, Neary M, Adkins EC, Schueller SM. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv 2018; 15: 110-115
    CrossrefPubMedSearch in Google Scholar
  • 22 Knorr K, Aspinall D, Wolters M. On the privacy, security and safety of blood pressure and diabetes apps. In: Proceedings of the IFIP (International Federation for Information Processing) International Conference on ICT (Information and Communications Technology) Systems Security and Privacy Protection. Cham, Switzerland: Springer; 2015 May 9. Presented at: IFIP (International Federation for Information Processing) International Conference on ICT (Information and Communications Technology) Systems Security and Privacy Protection; May 26–28, 2015; Hamburg, Germany; 571–584
    PubMed
  • 23 Benjumea J, Ropero J, Rivera-Romero O, Dorronzoro-Zubiete E, Carrasco A. Assessment of the fairness of privacy policies of mobile health apps: scale development and evaluation in cancer apps. JMIR Mhealth Uhealth 2020; 8 (07) e17134
    CrossrefPubMedSearch in Google Scholar
  • 24 Papageorgiou A, Strigkos M, Politou E, Alepis E, Solanas A, Patsakis C. Security and privacy analysis of mobile health applications: the alarming state of practice. IEEE Access 2018; 6: 9390-9403
    CrossrefPubMedSearch in Google Scholar
  • 25 Huckvale K, Torous J, Larsen ME. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw Open 2019; 2 (04) e192542
    CrossrefPubMedSearch in Google Scholar
  • 26 Sunyaev A, Dehling T, Taylor PL, Mandl KD. Availability and quality of mobile health app privacy policies. J Am Med Inform Assoc 2015; 22 (e1): e28-e33
    CrossrefPubMedSearch in Google Scholar
  • 27 Scott K, Richards D, Adhikari R. A review and comparative analysis of security risks and safety measures of mobile health apps. AJIS Australas J Inf Syst 2015; 19: 1-18
    PubMedSearch in Google Scholar
  • 28 Zapata B, Hernández Niñirola A, Fernández-Alemán JL, Toval A. Assessing the privacy policies in mobile personal health records. In: Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014). New York, NY: IEEE; 2014; Presented at: 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014); August 26–30, 2014; Chicago, IL; 4956–4959
    PubMed
  • 29 Bachiri M, Idri A, Fernández-Alemán JL, Toval A. Evaluating the privacy policies of mobile personal health records for pregnancy monitoring. J Med Syst 2018; 42 (08) 144
    CrossrefPubMedSearch in Google Scholar
  • 30 Hutton L, Price BA, Kelly R. et al. Assessing the privacy of mHealth apps for self-tracking: heuristic evaluation approach. JMIR Mhealth Uhealth 2018; 6 (10) e185
    CrossrefPubMedSearch in Google Scholar
  • 31 Baumel A, Faber K, Mathur N, Kane JM, Muench F. Enlight: a comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. J Med Internet Res 2017; 19 (03) e82
    CrossrefPubMedSearch in Google Scholar
  • 32 Bondaronek P, Alkhaldi G, Slee A, Hamilton FL, Murray E. Quality of publicly available physical activity apps: review and content analysis. JMIR Mhealth Uhealth 2018; 6 (03) e53
    CrossrefPubMedSearch in Google Scholar
  • 33 Ni Z, Wang Y, Qian Y. Privacy policy compliance of chronic disease management apps in China: scale development and content evaluation. JMIR Mhealth Uhealth 2021; 9 (01) e23409
    CrossrefPubMedSearch in Google Scholar
  • 34 Robillard JM, Feng TL, Sporn AB. et al. Availability, readability, and content of privacy policies and terms of agreements of mental health apps. Internet Interv 2019; 17: 100243
    CrossrefPubMedSearch in Google Scholar
  • 35 Powell AC, Singh P, Torous J. The complexity of mental health app privacy policies: a potential barrier to privacy. JMIR Mhealth Uhealth 2018; 6 (07) e158
    CrossrefPubMedSearch in Google Scholar
  • 36 Adhikari AR, Richards D, Scott K. Security and privacy issues related to the use of mobile health apps. Paper presented at: Proceedings of the 25th Australasian Conference on Information Systems; 2014; Auckland, New Zealand; 8–10
    PubMed
  • 37 Vaughan-Graham J, Cott C. Defining a Bobath clinical framework - a modified e-Delphi study. Physiother Theory Pract 2016; 32 (08) 612-627
    CrossrefPubMedSearch in Google Scholar
  • 38 Robustillo Cortés MdeL, Cantudo Cuenca MR, Morillo Verdugo R, Calvo Cidoncha E. High quantity but limited quality in healthcare applications intended for HIV-infected patients. Telemed J E Health 2014; 20 (08) 729-735
    CrossrefPubMedSearch in Google Scholar
  • 39 Administrative Simplification HIPAA. 45 CFR Parts 160, 162, and 164 (unofficial version, as amended through March 26, 2013). Accessed October 2021 at: http://www.hhs.gov/sites/default/files/ocr/privacy/hipaa/administrative/combined/hipaa-simplification-201303.pdf
    PubMed
  • 40 Savage M, Savage LC. Doctors routinely share health data electronically under HIPAA, and sharing with patients and patients' third-party health apps is consistent: interoperability and privacy analysis. J Med Internet Res 2020; 22 (09) e19818
    CrossrefPubMedSearch in Google Scholar
  • 41 FTC Warns Health Apps and Connected Device Companies to Comply With Health Breach Notification Rule. FTC Press Releases. Accessed October 2021 at: https://www.ftc.gov/news-events/press-releases/2021/09/ftc-warns-health-apps-connected-device-companies-comply-health
    PubMed
  • 42 European Parliament and Council. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation), EUR-Lex. Accessed October 2021 at: https://eur-lex.europa.eu/eli/reg/2016/679/2016-05-04 . Last consulted: October 2021
    PubMed
  • 43 Mense MA, Urbauer P, Sauermann S, Wahl H. Simulation environment for testing security and privacy of mobile health apps. Paper presented at: Modeling and Simulation in Medicine Symposium; 2016; Pasadena, CA; 3–6
    PubMed
  • 44 Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts. Manage Sci 1963; 9 (03) 458-467
    CrossrefPubMedSearch in Google Scholar
  • 45 Linstone HA, Turoff M. The Delphi Method: Techniques and Applications. London: Addison-Wesley; 1975
    Search in Google Scholar
  • 46 Fischer RG. The Delphi method: a description, review and criticism. J Acad Librariansh 1978; 4: 64-70
    PubMedSearch in Google Scholar
  • 47 Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med 2017; 31 (08) 684-706
    CrossrefPubMedSearch in Google Scholar
  • 48 McKenna HP. The Delphi technique: a worthwhile research approach for nursing?. J Adv Nurs 1994; 19 (06) 1221-1225
    CrossrefPubMedSearch in Google Scholar
  • 49 Von der Gracht HA. Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change 2012; 79 (08) 1525-1536
    CrossrefPubMedSearch in Google Scholar
  • 50 Schmidt RC. Managing Delphi surveys using non-parametric statistical techniques. Decis Sci 1997; 28: 763-774
    CrossrefPubMedSearch in Google Scholar
  • 51 Golenko X, Pager S, Holden L. A thematic analysis of the role of the organisation in building allied health research capacity: a senior managers' perspective. BMC Health Serv Res 2012; 12 (276) 276
    CrossrefPubMedSearch in Google Scholar
  • 52 Robles N, Puigdomènech Puig E, Gómez-Calderón C. et al. Evaluation criteria for weight management apps: validation using a modified Delphi process. JMIR Mhealth Uhealth 2020; 8 (07) e16899
    CrossrefPubMedSearch in Google Scholar
  • 53 Studnek JR, Lerner EB, Shah MI. et al. Consensus-based criterion standard for the identification of pediatric patients who need emergency medical services transport to a hospital with higher-level pediatric resources. Acad Emerg Med 2018; 25 (12) 1409-1414
    CrossrefPubMedSearch in Google Scholar
  • 54 Universidad de Sevilla. Record of processing activities for “Evaluación de Políticas de Privacidad: Estudio no experimental con expertos y usuarios” (in Spanish). Accessed August 22, 2023 at: https://sic.us.es/sites/default/files/pd/rateval.pol_.priv_.pdf
    PubMed
  • 55 Birko S, Dove ES, Özdemir V. Evaluation of nine consensus indices in Delphi foresight research and their dependency on Delphi survey characteristics: a simulation study and debate on Delphi design and interpretation. PLoS One 2015; 10 (08) e0135162
    CrossrefPubMedSearch in Google Scholar
  • 56 Brüggemann BT, Hansen J, Dehling T, Sunyaev A. Privacy technologies and policy. In: Schiffner S, Serna J. eds. An Information Privacy Risk Index for mHealth Apps. Cham: Springer; 2016
    Search in Google Scholar
  • 57 Benjumea J, Ropero J, Dorronzoro-Zubiete E, Rivera-Romero O, Carrasco A. A proposal for a robust validated weighted general data protection regulation-based scale to assess the fairness of privacy policies of mobile health applications: a Delphi study. 21 June 2022, PREPRINT (Version 1) available at Research Square
    CrossrefPubMed