PDF herunterladen
CC BY-NC-ND 4.0 · Appl Clin Inform 2022; 13(01): 67-79
DOI: 10.1055/s-0041-1740919
Research Article

Agile, Easily Applicable, and Useful eHealth Usability Evaluations: Systematic Review and Expert-Validation

Irina Sinabell
1   Department of Biomedical Computer Science and Mechatronics, Institute of Medical Informatics, UMIT, Private University of Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria
,
Elske Ammenwerth
1   Department of Biomedical Computer Science and Mechatronics, Institute of Medical Informatics, UMIT, Private University of Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria
› Institutsangaben
› Weitere Informationen
Auch verfügbar auf
   Lizenzen und Reprints

Abstract

Background Electronic health (eHealth) usability evaluations of rapidly developed eHealth systems are difficult to accomplish because traditional usability evaluation methods require substantial time in preparation and implementation. This illustrates the growing need for fast, flexible, and cost-effective methods to evaluate the usability of eHealth systems. To address this demand, the present study systematically identified and expert-validated rapidly deployable eHealth usability evaluation methods.

Objective Identification and prioritization of eHealth usability evaluation methods suitable for agile, easily applicable, and useful eHealth usability evaluations.

Methods The study design comprised a systematic iterative approach in which expert knowledge was contrasted with findings from literature. Forty-three eHealth usability evaluation methods were systematically identified and assessed regarding their ease of applicability and usefulness through semi-structured expert interviews with 10 European usability experts and systematic literature research. The most appropriate eHealth usability evaluation methods were selected stepwise based on the experts' judgements of their ease of applicability and usefulness.

Results Of these 43 eHealth usability evaluation methods identified as suitable for agile, easily applicable, and useful eHealth usability evaluations, 10 were recommended by the experts based on their usefulness for rapid eHealth usability evaluations. The three most frequently recommended eHealth usability evaluation methods were Remote User Testing, Expert Review, and Rapid Iterative Test and Evaluation Method. Eleven usability evaluation methods, such as Retrospective Testing, were not recommended for use in rapid eHealth usability evaluations.

Conclusion We conducted a systematic review and expert-validation to identify rapidly deployable eHealth usability evaluation methods. The comprehensive and evidence-based prioritization of eHealth usability evaluation methods supports faster usability evaluations, and so contributes to the ease-of-use of emerging eHealth systems.

Keywords

telemedicine and telehealth - clinical research informatics - clinical information systems - human–computer interaction - hospital information systems

Protection of Human and Animal Subjects

Ethical approval was not required for this study.


Supplementary Material



Publikationsverlauf

Eingereicht: 24. Mai 2021

Angenommen: 20. Oktober 2021

Artikel online veröffentlicht:
09. März 2022

© 2022. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Lau F, Kuziemsky CE. eds. Handbook of eHealth Evaluation: An Evidence-Based Approach. Victoria, British Columbia Canada: University of Victoria; 2016
    Suche in Google Scholar
  • 2 International Telecommunication Union (ITU). Standardization in e-health. Accessed August 9, 2021 at: https://www.itu.int/itunews/issue/2003/06/standardization.html
  • 3 World Health Organization (WHO). eHealth. Accessed August 9, 2021 at: https://www.euro.who.int__data/assets/pdf_file/0010/261694/6.-eHealth,-Factsheet-for-European-Parliament.pdf
  • 4 Bockhacker M, Syrek H, Elstermann von Elster M, Schmitt S, Roehl H. Evaluating usability of a touchless image viewer in the operating room. Appl Clin Inform 2020; 11 (01) 88-94
    Thieme ConnectPubMedSuche in Google Scholar
  • 5 Richter Lagha R, Burningham Z, Sauer BC. et al. Usability testing a potentially inappropriate medication dashboard: a core component of the dashboard development process. Appl Clin Inform 2020; 11 (04) 528-534
    Thieme ConnectPubMedSuche in Google Scholar
  • 6 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
    Thieme ConnectPubMedSuche in Google Scholar
  • 7 Hron JD, Parsons CR, Williams LA, Harper MB, Bourgeois FC. Rapid implementation of an inpatient telehealth program during the COVID-19 pandemic. Appl Clin Inform 2020; 11 (03) 452-459
    Thieme ConnectPubMedSuche in Google Scholar
  • 8 Rödle W, Wimmer S, Zahn J. et al. User-centered development of an online platform for drug dosing recommendations in pediatrics. Appl Clin Inform 2019; 10 (04) 570-579
    Thieme ConnectPubMedSuche in Google Scholar
  • 9 Farzandipour M, Nabovati E, Heidarzadeh Arani M, Akbari H, Sharif R, Anvari S. Enhancing asthma patients' self-management through smartphone-based application: design, usability evaluation, and educational intervention. Appl Clin Inform 2019; 10 (05) 870-878
    Thieme ConnectPubMedSuche in Google Scholar
  • 10 Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems-a literature survey. Int J Med Inform 2016; 93: 57-69
    CrossrefPubMedSuche in Google Scholar
  • 11 Kushniruk AW, Borycki EM. Low-cost rapid usability engineering: designing and customizing usable healthcare information systems. Healthc Q 2006; 9 (04) 98-102
    PubMedSuche in Google Scholar
  • 12 Ali H, Cole A, Panos G. Transforming patient hospital experience through smart technologies. In: Proceedings of the International Conference on Human-Computer Interaction; HCII '20. Copenhagen, Denmark: Springer International Publishing; 2020: 203-215
    Suche in Google Scholar
  • 13 Coppersmith NA, Sarkar IN, Chen ES. Quality informatics: the convergence of healthcare data, analytics, and clinical excellence. Appl Clin Inform 2019; 10 (02) 272-277
    Thieme ConnectPubMedSuche in Google Scholar
  • 14 Kushniruk A, Borycki E. Low-cost rapid usability testing: its application in both product development and system implementation. Stud Health Technol Inform 2017; 234: 195-200
    PubMedSuche in Google Scholar
  • 15 Price M, Weber J, Bellwood P, Diemert S, Habibi R. Evaluation of eHealth system usability and safety. In: Handbook of EHealth Evaluation: An Evidece-Based Approach. Victoria, British Columbia Canada: University of Victoria; 2016: 337-350
    Suche in Google Scholar
  • 16 Kushniruk AW, Borycki EM. Integrating low-cost rapid usability testing into agile system development of healthcare IT: a methodological perspective. Stud Health Technol Inform 2015; 210: 200-204
    PubMedSuche in Google Scholar
  • 17 Marcilly R, Schiro J, Beuscart-Zéphir MC, Magrabi F. Building usability knowledge for health information technology: a usability-oriented analysis of incident reports. Appl Clin Inform 2019; 10 (03) 395-408
    Thieme ConnectPubMedSuche in Google Scholar
  • 18 Broekhuis M, van Velsen L, Hermens H. Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. Int J Med Inform 2019; 128: 24-31
    CrossrefPubMedSuche in Google Scholar
  • 19 International Organization for Standardization (ISO). Ergonomics of human-system interaction - Part 11: usability: definitions and concepts. Accessed July 26, 2021 at: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
  • 20 Paelke V, Röcker C. User interfaces for cyber-physical systems: challenges and possible approaches. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 75-85
    Suche in Google Scholar
  • 21 Bastien JM. Usability testing: a review of some methodological and technical aspects of the method. Int J Med Inform 2010; 79 (04) e18-e23
    CrossrefPubMedSuche in Google Scholar
  • 22 Speicher M, Both A, Gaedke M. INUIT: the interface usability instrument. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 256-268
    Suche in Google Scholar
  • 23 Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019; 126: 95-104
    CrossrefPubMedSuche in Google Scholar
  • 24 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-353
    CrossrefPubMedSuche in Google Scholar
  • 25 Thompson KE, Rozanski EP, Haake AR. Here, there, anywhere: remote usability testing that works. In: Proceedings of the 5th Conference on Information Technology Education; CITC5 '04. Salt Lake City, UT, USA: ACM; 2004: 132-137
    Suche in Google Scholar
  • 26 Wysocki T, Pierce J, Caldwell C. et al. A web-based coping intervention by and for parents of very young children with type 1 diabetes: user-centered design. JMIR Diabetes 2018; 3 (04) e16
    CrossrefPubMedSuche in Google Scholar
  • 27 González Sánchez JL, Padilla Zea N, Gutiérrez FL. From usability to playability: introduction to player-centred video game development process. In: Proceedings of the International Conference on Human Centered Design; HCD '09. San Diego, CA, USA: Springer; 2009. ;5619: 65-74
    Suche in Google Scholar
  • 28 Dubey A, Singi K, Kaulgud V. Personas and redundancies in crowdsourced testing. In: Proceedings of the 12th International Conference on Global Software Engineering; ICGSE '17. Buenos Aires, Argentinia: IEEE; 2017: 76-80
    Suche in Google Scholar
  • 29 Silva T, Silveira M, Maurer F. Usability evaluation practices within agile development. In: Proceedings of the 48th Hawaii International Conference on System Sciences. HICSS '15. Kauai, HI, USA: IEEE; 2015: 5133-5142
    Suche in Google Scholar
  • 30 Pawson M, Greenberg S. Extremely rapid usability testing. J Usability Stud 2009; 4: 124-135
    PubMedSuche in Google Scholar
  • 31 Kane D. Finding a place for discount usability engineering in agile development: throwing down the gauntlet. In: Proceedings of the Agile Development Conference; ADC '03. Salt Lake City, UT, USA: IEEE; 2003: 40-46
    Suche in Google Scholar
  • 32 Grigoreanu V, Mohanna M. Informal Cognitive Walkthroughs (ICW): paring down and pairing up for an agile world. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '13. Paris, France: ACM; 2013: 3093-3096
    Suche in Google Scholar
  • 33 Marques AB, Figueiredo R, Amorin W, Rabelo J, Barbosa SDJ, Conte T. Do usability and agility combine?: investigating the adoption of usability modeling in an agile software project in the industry. In: Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems; IHC '18. Belém, Brazil: ACM; 2018: 1-11
    Suche in Google Scholar
  • 34 Cavichi de Freitas R, Rodrigues LA, Marques da Cunha A. AGILUS: a method for integrating usability evaluations on agile software development. In: Proceedings of the International Conference on Human-Computer Interaction; HCI '16. Cham, Switzerland: Springer International Publishing; 2016. ;9731: 545-552
    Suche in Google Scholar
  • 35 Magües D, Castro J, Acuna S. HCI usability techniques in agile development. In: Proceedings of the International Conference on Automatica; ICA-ACCA '16. Curico, Chile: IEEE; 2016: 1-7
    Suche in Google Scholar
  • 36 Butt S, Onn A, Butt M, Inam N, Butt S. Incorporation of usability evaluation methods in agile software model. In: Proceedings of the 17th IEEE International Multi Topic Conference; INMIC '14. Karachi, Pakistan: IEEE; 2014: 193-199
    Suche in Google Scholar
  • 37 Federoff M, Villamor C, Miller L. et al. Extreme usability: adapting research approaches for agile development. In: Proceedings of the 26th Annual CHI Conference Extended Abstracts on Human Factors in Computing Systems; CHI EA '08. Florence, Italy: ACM; 2008: 2269-2272
    Suche in Google Scholar
  • 38 Nielsen J. Discount usability: 20 years. Accessed September 13, 2019 at: https://www.nngroup.com/articles/discount-usability-20-years/
  • 39 Schwartz L. Agile-user experience design: an agile and user-centered process?. In: Proceedings of the 8th International Conference on Software Engineering Advances; ICSEA '13. Venice, Italy: IARIA; 2013: 346-351
    Suche in Google Scholar
  • 40 Isa W, Lokman A, Aris S. et al. Engineering rural informatics using agile user centered design. In: Proceedings of the 2nd International Conference on Information and Communication Technology; ICoICT '14. Bandung, Indonesia: IEEE; 2014: 367-372
    Suche in Google Scholar
  • 41 Constantine LL, Lockwood LAD. Usage-centered software engineering: an agile approach to integrating users, user interfaces, and usability into software engineering practice. In: Proceedings of the 25th International Conference on Software Engineering; ICSE '03. Portland, OR, USA: IEEE; 2003: 746-747
    Suche in Google Scholar
  • 42 Verhoeven F, van Gemert-Pijnen J. Discount user-centered e-health design: a quick-but-not-dirty method. In: Proceedings of the Symposium of the Austrian HCI and Usability Engineering Group; USAB '10. Klagenfurt, Austria: Springer; 2010. 638. 101-123
    Suche in Google Scholar
  • 43 Russ AL, Baker DA, Fahner WJ. et al. A rapid usability evaluation (RUE) method for health information technology. AMIA Annu Symp Proc 2010; 2010: 702-706
    PubMedSuche in Google Scholar
  • 44 European Union Agency for Network and Information Security (ENISA). Security and Resilience in eHealth: Security Challenges and Risks. Iraklio, Greece: ENISA; 2015: 1-48
    Suche in Google Scholar
  • 45 Kuziemsky CE, Kushniruk A. Context mediated usability testing. Stud Health Technol Inform 2014; 205: 905-909
    PubMedSuche in Google Scholar
  • 46 Øvad T, Larsen L. The prevalence of UX design in agile development processes in industry. In: Proceedings of the Agile Conference; AGILE '15. Washington, DC, USA: IEEE; 2015: 40-49
    Suche in Google Scholar
  • 47 Duh HB-L, Tan GCB, Chen VH. Usability evaluation for mobile device: a comparison of laboratory and field tests. In: Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services; MobileHCI '06. Helsinki, Finland: ACM; 2006: 181-186
    Suche in Google Scholar
  • 48 Whitenton K. Tools for unmoderated usability testing. Accessed April 28, 2021 at: https://www.nngroup.com/articles/unmoderated-user-testing-tools/
  • 49 Nielsen J. Usability inspection methods. In: Proceedings of the Conference Companion on Human Factors in Computing Systems; CHI '94. Boston, MA, USA: ACM; 1994: 413-414
    Suche in Google Scholar
  • 50 Johnson CM, Johnston D, Crowley PK. et al. EHR Usability Toolkit: A Background Report on Usability and Electronic Health Records. Accessed August 8, 2021 at: https://digital.ahrq.gov/sites/default/files/docs/citation/EHR_Usability_Toolkit_Background_Report.pdf
  • 51 Davis R, Gardner J, Schnall R. A review of usability evaluation methods and their use for testing eHealth HIV interventions. Curr HIV/AIDS Rep 2020; 17 (03) 203-218
    CrossrefPubMedSuche in Google Scholar
  • 52 Wu DTY, Vennemeyer S, Brown K. et al. Usability testing of an interactive dashboard for surgical quality improvement in a large congenital heart center. Appl Clin Inform 2019; 10 (05) 859-869
    Thieme ConnectPubMedSuche in Google Scholar
  • 53 Elbabour F, Alhadreti O, Mayhew P. Eye tracking in retrospective think-aloud usability testing: is there added value?. J Usability Stud 2017; 12 (03) 95-110
    PubMedSuche in Google Scholar
  • 54 Elling S, Lentz L, de Jong M. Retrospective think-aloud method: using eye movements as an extra cue for participants' verbalizations. In: Proceedings of the 29th Annual CHI Conference on Human Factors in Computing Systems; SIGCHI '11. Vancouver, BC, Canada: ACM; 2011: 1161-1170
    Suche in Google Scholar
  • 55 Khajouei R, Zahiri Esfahani M, Jahani Y. Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. J Am Med Inform Assoc 2017; 24 (e1): e55-e60
    CrossrefPubMedSuche in Google Scholar
  • 56 Georgsson M, Staggers N, Årsand E, Kushniruk A. Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: a case study and beginning method validation. J Biomed Inform 2019; 91: 1-15
    CrossrefPubMedSuche in Google Scholar
  • 57 Ghalibaf AK, Jangi M, Habibi MRM, Zangouei S, Khajouei R. Usability evaluation of obstetrics and gynecology information system using cognitive walkthrough method. Electron Physician 2018; 10 (04) 6682-6688
    CrossrefPubMedSuche in Google Scholar
  • 58 Rowley DE, Rhoades DG. The cognitive jogthrough: a fast-paced user interface evaluation procedure. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; CHI '92. Monterey, Calif, USA: ACM; 1992: 389-395
    Suche in Google Scholar
  • 59 Andersen SB, Rasmussen CK, Frøkjær E. Bringing content understanding into usability testing in complex application domains - a case study in eHealth. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 327-341
    Suche in Google Scholar
  • 60 The User Experience Professionals' Association (UXPA). Usability body of knowledge. Accessed August 9, 2021 at: http://www.usabilitybok.org/methods
  • 61 University of Minnesota Duluth (UMD). Usability evaluation toolbox. Accessed August 15, 2021 at: https://www.d.umn.edu/itss/training/online/usability/toolbox.html
  • 62 Maguire M. Methods to support human-centred design. Int J Hum Comput Stud 2001; 55 (04) 587-634
    CrossrefPubMedSuche in Google Scholar
  • 63 Frøkjær E, Hornbæk K. Cooperative usability testing. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '05. Portland, OR, USA: ACM; 2005: 1383-1386
    Suche in Google Scholar
  • 64 Power C, Petrie H, Mitchell R. A framework for remote user evaluation of accessibility and usability of websites. In: Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction; UAHCI '09. San Diego, CA, USA: Springer; 2009. ;5614: 594-601
    Suche in Google Scholar
  • 65 Hammontree M, Weiler P, Nayak N. Remote usability testing. Interact 1994; 1 (03) 21-25
    CrossrefPubMedSuche in Google Scholar
  • 66 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (01) 56-76
    CrossrefPubMedSuche in Google Scholar
  • 67 Medlock MC, Wixon D, McGee M, Welsh D. The rapid iterative test and evaluation method. In: Int Techn, Cost-Justifying Usability. 2nd ed.. Elsevier; 2005: 489-517
    Suche in Google Scholar
  • 68 Wilson C. User Interface Inspection Methods: A User-Centered Design Method. Amsterdam, The Netherlands; Elsevier. 2013
    Suche in Google Scholar
  • 69 Laemller R. Guerrilla Testing: Muting und rasch zu Resultaten. Accessed November 2, 2020 at: https://www.testingtime.com/blog/guerrilla-testing/
  • 70 Firmenich S, Garrido A, Grigera J, Rivero JM, Rossi G. Usability improvement through A/B testing and refactoring. Softw Qual J 2018; 27 (01) 203-240
    CrossrefPubMedSuche in Google Scholar
  • 71 Guaiani F, Muccini H. Crowd and laboratory testing, can they co-exist? An exploratory study. In: Proceedings of the 2nd International Workshop on CrowdSourcing in Software Engineering; CSI-SE '15. Florence, Italy: IEEE; 2015: 32-37
    Suche in Google Scholar
  • 72 Zogaj S, Bretschneider U, Leimeister JM. Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary. J Bus Econ 2014; 84 (03) 375-405
    CrossrefPubMedSuche in Google Scholar
  • 73 Downey LL. Group usability testing: evolution in usability techniques. J Usability Stud 2007; 2 (03) 133-144
    PubMedSuche in Google Scholar
  • 74 Sears A. Heuristic walkthroughs: finding the problems without the noise. Int J Hum Comput Interact 1997; 9 (03) 213-234
    CrossrefPubMedSuche in Google Scholar
  • 75 Nielsen L, Madsen S. The usability expert's fear of agility: an empirical study of global trends and emerging practices. In: Proceedings of the 7th Nordic Conference on Human-Computer Interaction; NordiCHI '12. Copenhagen, Denmark: ACM; 2012: 261-264
    Suche in Google Scholar
  • 76 Höysniemi J, Hämäläinen P, Turkki L. Using peer tutoring in evaluating the usability of a physically interactive computer game with children. Interact Comput 2003; 15 (02) 203-255
    CrossrefPubMedSuche in Google Scholar
  • 77 Khalayli N, Nyhus S, Hamnes K, Terum T. Persona based rapid usability kick-off. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '07. San Jose, CA, USA: ACM; 2007: 1771-1776
    Suche in Google Scholar
  • 78 Usability in Germany eV (UIG). Fokusgruppe. Accessed November 9, 2020 at: https://www.usability-in-germany.de/definition/fokusgruppe
  • 79 Rosenbaum S, Cockton G, Coyne K, Muller M, Rauch T. Focus groups in HCI. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '02. Minneapolis, MN, USA: ACM; 2002: 702-703
    Suche in Google Scholar
  • 80 Usability in Germany eV (UIG). Fragebogen. Accessed November 9, 2020 at: https://www.usability-in-germany.de/definition/fragebogen
  • 81 Krause R. Storyboards help visualize UX ideas. Nielsen Norman Group. Accessed November 9, 2020 at: https://www.nngroup.com/articles/storyboards-visualize-ideas/
  • 82 Piyush J, Sanjay KD, Ajay R. Software usability evaluation method. Int J Adv Res Comput Eng Technol 2012; 1 (02) 28-33
    PubMedSuche in Google Scholar
  • 83 Holzinger A, Slany WXP. + UE → XU Praktische Erfahrungen mit eXtreme Usability. Informatik-Spektrum 2006; 29 (02) 91-97
    CrossrefPubMedSuche in Google Scholar
  • 84 Borycki E, Senathirajah Y, Kushniruk AW. The future of mobile usability, workflow and safety testing. Stud Health Technol Inform 2017; 245: 15-19
    PubMedSuche in Google Scholar
  • 85 Holzinger A. Usability engineering methods for software developers. Commun ACM 2005; 48 (01) 71-74
    CrossrefPubMedSuche in Google Scholar