User Evaluation of an Integrated Medical Workstation for Clinical Data Analysis

Abstract

Results are presented of the user evaluation of an integrated medical workstation for support of clinical research. Twenty-seven users were recruited from medical and scientific staff of the University Hospital Dijkzigt, the Faculty of Medicine of the Erasmus University Rotterdam, and from other Dutch medical institutions; and all were given a written, self-contained tutorial. Subsequently, an experiment was done in which six clinical data analysis problems had to be solved and an evaluation form was filled out. The aim of this user evaluation was to obtain insight in the benefits of integration for support of clinical data analysis for clinicians and biomedical researchers. The problems were divided into two sets, with gradually more complex problems. In the first set users were guided in a stepwise fashion to solve the problems. In the second set each stepwise problem had an open counterpart. During the evaluation, the workstation continuously recorded the user’s actions. From these results significant differences became apparent between clinicians and non-clinicians for the correctness (means 54% and 81%, respectively, p = 0.04), completeness (means 64% and 88%, respectively, p = 0.01), and number of problems solved (means 67% and 90%, respectively, p = 0.02). These differences were absent for the stepwise problems. Physicians tend to skip more problems than biomedical researchers. No statistically significant differences were found between users with and without clinical data analysis experience, for correctness (means 74% and 72%, respectively, p = 0.95), and completeness (means 82% and 79%, respectively, p = 0.40). It appeared that various clinical research problems can be solved easily with support of the workstation; the results of this experiment can be used as guidance for the development of the successor of this prototype workstation and serve as a reference for the assessment of next versions.

Key-Words

Integration - Evaluation - Workstation

^* HERMES is an acronym for HEalth Research MEdical System.

• REFERENCES

• 1 Van Mulligcn EM, Timmers X, De Faria Lcao B. Implementation of a medical workstation for research support in cardiology. In: Miller RA. eds Proceedings 14th Symposium on Computer Applications in Medical Care. New York: IEEE Computer Society Press; 1990: 769-73.
• 2 Van Mulligcn EM, Timrners T, Van Bem-mel JH. A new architecture for integration of heterogeneous software components. Meth Inform Med 1993; 32: 292-301.
• 3 Basic Reference Model of Open Distributed Processing. ISO/IEC JTC1/SC21AVG7 N315. Nederlands Normalisetie-Instituut, December 1990
• 4 SaFran C. Using routinely collected data for clinical research. Stal Med 1991; 10: 559-64.
• 5 Power LR. Post-facto integration technology: new discipline for an old practice. In: Ng PA, Ramamoorthy CV, Seifert LC, Yeh RT. eds. Proceedings First Internaiional Conference on Systems Integration. Los Aiamitos: IEEE Computer Society Press; 1990: 4-13.
• 6 Andrews GR. Paradigm for process interaction in distributed programs. ACM Computing Surveys 1991; 23: 49-90.
• 7 Zimmermann H. OSI reference model — the ISO model of architecture for open systems interconnection. IEEE Trans Commun 1980; 26: 425-32.
• 8 Caschnig J, Klahr Ph, Pople H, Shortliffe EH, Terry A. Evaluation of expert systems: issues and case studies. In: Hayes-Roth F, Waterman DA, Lenat DB. eds. Building expert systems. New York: Addison-Wes-ley; 1983: 241-79.
• 9 Wyatt J, Spiegelhalter D. Evaluating medical decision-a ids: what to test, and how?. In: Talmon j, Fox J. eds. System Engineering in Medicine. Heidelberg: Springer Verlag; 1989: 1-13.
• 10 Bloem RM, Zwavefing A, Stijncn Th. Adenocarcinoma of the colon and rectum: a report on 624 cases. Netherl J Surgery 1988; 40: 121-6.