Download PDF
Methods Inf Med 2005; 44(05): 616-625
DOI: 10.1055/s-0038-1634017
Original Article
Schattauer GmbH

Methods for Evaluation of Medical Terminological Systems

A Literature Review and a Case Study
D. G. T. Arts
1   Academic Medical Center, Department of Medical Informatics, Amsterdam, The Netherlands
,
R. Cornet
1   Academic Medical Center, Department of Medical Informatics, Amsterdam, The Netherlands
,
E. de Jonge
2   Academic Medical Center, Department of Intensive Care, Amsterdam, The Netherlands
,
N. F. de Keizer
1   Academic Medical Center, Department of Medical Informatics, Amsterdam, The Netherlands
› Author Affiliations
Further Information

Publication History

Received: 15 April 2004

accepted: 15 March 2005

Publication Date:
07 February 2018 (online)

  PDF Download  Permissions and Reprints

Summary

Objectives: The usability of terminological systems (TSs) strongly depends on the coverage and correctness of their content. The objective of this study was to create a literature overview of aspects related to the content of TSs and of methods for the evaluation of the content of TSs. The extent to which these methods overlap or complement each other is investigated.

Methods: We reviewed literature and composed definitions for aspects of the evaluation of the content of TSs. Of the methods described in literature three were selected: 1) Concept matching in which two samples of concepts representing a) documentation of reasons for admission in daily care practice and b) aggregation of patient groups for research, are looked up in the TS in order to assess its coverage; 2) Formal algorithmic evaluation in which reasoning on the formally represented content is used to detect inconsistencies; and 3) Expert review in which a random sample of concepts are checked for incorrect and incomplete terms and relations. These evaluation methods were applied in a case study on the locally developed TS DICE (Diagnoses for Intensive Care Evaluation).

Results: None of the applied methods covered all the aspects of the content of a TS. The results of concept matching differed for the two use cases (63% vs. 52% perfect matches). Expert review revealed many more errors and incompleteness than formal algorithmic evaluation.

Conclusions: To evaluate the content of a TS, using a combination of evaluation methods is preferable. Different representative samples, reflecting the uses of TSs, lead to different results for concept matching. Expert review appears to be very valuable, but time consuming. Formal algorithmic evaluation has the potential to decrease the workload of human reviewers but detects only logical inconsistencies. Further research is required to exploit the potentials of formal algorithmic evaluation.

Keywords

Terminological systems - evaluation - methods - definitions
 

  • References

  • 1 De Keizer N, Abu-Hanna A, Zwetsloot-Schonk J. Understanding terminological systems I: terminology and typology. Meth Inform Med 2000; 39 (01) 16-21.
    PubMedGoogle Scholar
  • 2 World Health Organization. International Classification of Diseases, manual of the International Statistical Classification of diseases, injuries and causes of death: 9th revision;. 1977
    Google Scholar
  • 3 Cad R, Robboy S. Progress in medical information management. Systemized nomenclature of medicine (SNOMED). JAMA 1980; 243: 756-62.
    CrossrefPubMedGoogle Scholar
  • 4 Cote R, Rothwell D, Palotay J, Beckett R, Brochu L. The systemized nomenclature of human and veterinary medicine: SNOMED International. College of American Pathologists 1993
    Google Scholar
  • 5 North American Nursing Diagnosis Association. NANDA nursing diagnoses: Definitions and classifications 1999-2000. Philadelphia, PA: NANDA; 1999
    Google Scholar
  • 6 De Keizer N, Abu-Hanna A, Cornet R, Zwetsloot- Schonk J, Stoutenbeek C. Analysis and design of an ontology for intensive care diagnoses. Methods Inf Med 1999; 38 (02) 102-12.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 7 Campbell J, Carpenter P, Sneiderman C, Cohn S, Chute C, Warren J. Phase II evaluation of clinical coding schemes: Completeness, taxonomy, mapping, definitions, and clarity. J Am Med Inform Assoc 1997; 4 (03) 238-51.
    CrossrefPubMedGoogle Scholar
  • 8 Chute C, Cohn S, Campbell J. A framework for comprehensive health terminology systems in the United States. Development guidelines, criteria for selection, and public policy implications. J Am Med Inform Assoc 1998; 5 (06) 503-10.
    CrossrefPubMedGoogle Scholar
  • 9 Cimino J. Desiderata for controlled medical vocabularies in the twenty-first century. Meth Inform Med 2001; 37 4-5 394-403.
    PubMedGoogle Scholar
  • 10 ISO/TC215 WG 3. Standard Specification for Quality Indicators for Controlled Health Vocabularies; 2000 July. Report No.: TS17117
    PubMed
  • 11 Bakken Henry S, Holzemer W, Reilly C, Campbell K. Terms used by nurses to describe patient problems: Can SNOMED III represent nursing concepts in the patient record?. J Am Med Inform Assoc 1994; 1 (01) 61-74.
    CrossrefPubMedGoogle Scholar
  • 12 Chute C, Cohn S, Campbell K, Oliver D, Campbell J. The content coverage of clinical classifications. J Am Med Inform Assoc 1996; 3 (03) 224-33.
    CrossrefPubMedGoogle Scholar
  • 13 Humphreys B, McCray A, Cheh M. Evaluating the coverage of controlled health data terminologies: Report on the results of the NLM/AHCPR large scale vocabulary test. J Am Med Inform Assoc 1997; 4 (06) 484-500.
    CrossrefPubMedGoogle Scholar
  • 14 Bodenreider O, Burgun A, Botti G, Fieschi M, Le Beux P, Kohler F. Evaluation of the Unified Medical Language System as a medical knowledge source. J Am Med Inform Assoc 1998; 5 (01) 76-87.
    CrossrefPubMedGoogle Scholar
  • 15 Cimino J. Auditing the unified medical language system with semantic methods. J Am Med Inform Assoc 1998; 5 (01) 41-51.
    CrossrefPubMedGoogle Scholar
  • 16 Cimino J, McNamara T, Meredith T, Broverman C, Eckert K, Moore M. et al Evaluation of a proposed method for representing drug terminology. In: AMIA 1999 Proceedings of the AMIA Annual Symposium; 1999. Washington, DC. Philadelphia: Hanley & Belfus; 1999: 47-51.
    Google Scholar
  • 17 Bowles K. Application of the Omaha system in acute care. Res Nurs Health 2000; 23: 93-105.
    CrossrefPubMedGoogle Scholar
  • 18 Cimino J, Patel V, Kushniruk A. Studying the human-computer-terminology interface. J Am Med Inform Assoc 2001; 8 (02) 163-73.
    CrossrefPubMedGoogle Scholar
  • 19 Hardiker N, Rector A. Structural validation of nursing terminologies. J Am Med Inform Assoc 2001; 8 (03) 212-21.
    CrossrefPubMedGoogle Scholar
  • 20 Schulz S, Hahn U. Medical knowledge reengineering – converting major portions of the UMLS into a terminological knowledge base. Int J Med Inf 2001; 64: 207-21.
    CrossrefPubMedGoogle Scholar
  • 21 Bodenreider O. Circular hierarchical relationships in the UMLS: etiology, diagnosis, treatment, complications and prevention. In Bakken S. editor AMIA 2001 Proceedings of the AMIA Annual Symposium; 2001. Washington, USA. Philadelphia: Hanley & Belfus; 2001: 57-61.
    Google Scholar
  • 22 Bodenreider O, Mitchell J, McCray A. Evaluation of the UMLS as a terminology and knowledge resource for biomedical informatics. In Kohane I. editor AMIA 2002. Proceedings of the AMIA Annual Symposium; 2002. San Antonio, USA. Philadelphia: Hanley & Belfus; 2002: 61-65.
    Google Scholar
  • 23 Bodenreider O, Burgun A, Rindflesch T. Assessing the consistency of a biomedical terminology through lexical knowledge. Int J Med Inf 2002; 67: 85-95.
    CrossrefPubMedGoogle Scholar
  • 24 Cornet R, Abu-Hanna A. Evaluation of a framebased ontology. A formalization-orriented approach. In Engelbrecht R, Surján G, McNair P. editors. MIE 2002 Proceedings of Medical Infobahn Europe 2002 Budapest.. Amsterdam: IOS Press; 2002: 488-93.
    Google Scholar
  • 25 Moss J, Coenen A, Etta Mills M. Evaluation of the draft international standard for reference terminology model for nursing actions. J Biomed Inform 2003; 36: 271-8.
    CrossrefPubMedGoogle Scholar
  • 26 Hardiker N. Logical ontology for mediating between nursing intervention terminology systems. Methods Inf Med 2003; 42: 265-70.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 27 Hardiker N. Determining sources for formal nursing terminology systems. J Biomed Inform 2003; 36: 279-86.
    CrossrefPubMedGoogle Scholar
  • 28 Bodenreider O. Strength in numbers: Exploring redundancy in hierarchical relations across biomedical terminologies. In Musen M, Friedman C, Teich J. editors. AMIA 2003 Proceedings of the AMIA annual symposium 2003. Washington, USA. Philadelphia: Hanley & Belfus; 2003: 101-5.
    Google Scholar
  • 29 Warnekar P, Carter J. HIV terms coverage by a commercial nomenclature. In Musen M, Friedman C, Teich J. editors. AMIA2003 Proceedings of the AMIA Annual Symposium 2003. Washington, DC. Philadelphia: Hanley & Belfus; 2003: 1046
    Google Scholar
  • 30 Wasserman H, Wang J. An applied evaluation of SNOMED CT as a clinical vocabulary for the computerized diagnosis and problem list. In Musen M, Friedman C, Teich J. editors AMIA2003: Proceedings of the AMIA Annual Symposium 2003. Washington, DC. Philadelphia: Hanley & Belfus; 2003: 699-703.
    Google Scholar
  • 31 Brown S, Elkin P, Rosenbloom S, Husser C, Bauer B, Lincoln M. et al VA national drug reference terminology: A cross-institutional content coverage study. In Fieschi M, Coiera E, Li J. editors Medinfo 2004 Proceedings of the 11th world congress on medical informatics; 2004. San Francisco, USA. Amsterdam: IOS Press; 2004: 477-81.
    Google Scholar
  • 32 Penz J, Brown S, Carter J, Elkin P, Nguyen V, Sims S. Evaluation of SNOMED coverage of Veterans Health Administration terms. In Fieschi M, Coiera E, Li J. editors Medinfo 2004 Proceedings of the 11th world congress on medical informatics; 2004. San Francisco, CA. Amsterdam: IOS Press; 2004: 540-4.
    Google Scholar
  • 33 Ceusters W, Smith B, Kumar A, Dhaen C. Ontology- based error detection in SNOMED-CT. In Fieschi M, Coiera E, Li J. editors Medinfo 2004 Proceedings of the 11th world congress on medical informatics; 2004. San Fransisco, USA. Amsterdam: IOS Press; 2004: 482-6.
    Google Scholar
  • 34 Rothwell D. SNOMED-based knowledge representation. Meth Inform Med 1995; 34: 209-13.
    PubMedGoogle Scholar
  • 35 Rector A, Nowlan W. The GALEN project. Comput Methods Programs Biomed 1994; 45 1-2 75-8.
    CrossrefPubMedGoogle Scholar
  • 36 Young J, Goldfrad C, Rowan K. Development and testing of a hierarchical method to code the reason for admission to intensive care units: the ICNARC coding method. Br J Anaesth 2001; 87 (04) 543-8.
    CrossrefPubMedGoogle Scholar
  • 37 Yeh I, Karp P, Noy N, Altman R. Knowledge acquisition, consistency checking and concurrency control for Gene Ontology (GO). Bioinformatics 2003; 19 (02) 241-8.
    CrossrefPubMedGoogle Scholar
  • 38 Baader F, Calvanese D, McGuinness DL, Nardi D, Patel-Schneider PF. The Description Logic Handbook: Theory, Implementation, and Applications. Cambridge: University Press; 2003
    Google Scholar
  • 39 Haarslev V, Möller R. RACER System Description. In Massacci F. editor IJCAR 2001 Proceedings of the International Joint Conference for Automated Reasoning; 2001. Sienna. Springer- Verlag: 2001: 701-6.
    Google Scholar
  • 40 Cornet R, Abu-Hanna A. Using Description Logics for Managing Medical Terminologies. In Barahona P. editor 9th Conference on Artificial Intelligence in Medicine in Europe, AIME; 2003. Protaras, Cyprus: Springer; 2003: 61-70.
    Google Scholar
  • 41 Hales J, Schoeffler K. Barriers to evaluation of clinical vocabularies. In Cesnik B. editor Medinfo 1998. Proceedings of the 5th world congress on medical informatics; 1998, Seoul. Amsterdam: IOS Press; 1998: 680-4.
    Google Scholar
  • 42 Schlobach S, Cornet R. Logical Support for Terminological Modeling. In Fieschi M, Coiera E, Li J. editors. Medinfo 2004 Proceedings of the 11th world congress on medical informatics; 2004. San Francisco, CA. Amsterdam: IOS Press; 2004: 439-43.
    Google Scholar
  • 43 Schlobach S, Cornet R. Non-Standard Reasoning Services for the Debugging of Description Logic Terminologies. In Cohn A. editor IJCAI 2003. Proceedings of the International Joint Conference on Artificial Intelligence; 2003. Acapulco, Mexico. San Fransisco: Morgan Kaufmann Publishers; 2003
    Google Scholar
  • 44 Prins A, Arts D, Cornet R, de Keizer N. Internetbased terminological knowledge maintenance. In Fieschi M, Coiera E, Li J. editors. Medinfo 2004 Proceedings of the 11th world congress on medical informatics; 2004. San Francisco, CA. Amsterdam: IOS Press; 2004: 1820
    Google Scholar
  • 45 Arts D, Cornet R, De Jonge E, De Keizer N. Comparison of methods for evaluation of medical terminological systems. In Musen M, Friedman C, Teich J. editors. AMIA 2003 Proceedings of the AMIA Annual Symposium 2003. Washington, DC. Philadelphia: Hanley & Belfus; 2003: 779
    Google Scholar
  • 46 Arts D, De Keizer N, De Jonge E, Cornet R. Comparison of methods for evaluation of a medical terminological system. In Fieschi M, Coiera E, Li J. editors. Medinfo 2004 Proceedings of the 11th world congress on medical informatics; 2004. San Francisco, CA. Amsterdam: IOS Press; 2004: 467-71.
    Google Scholar