PDF herunterladen
Methods Inf Med 2014; 53(05): 344-356
DOI: 10.3414/ME13-01-0085
Original Articles
Schattauer GmbH

Learning the Preferences of Physicians for the Organization of Result Lists of Medical Evidence Articles

D. O´Sullivan
1   School of Informatics, City University London, London, United Kingdom
2   Telfer School of Management, University of Ottawa, Ottawa, Canada
,
S. Wilk
2   Telfer School of Management, University of Ottawa, Ottawa, Canada
3   Institute of Computing Science, Poznan University of Technology, Poznan, Poland
,
W. Michalowski
2   Telfer School of Management, University of Ottawa, Ottawa, Canada
,
R. Slowinski
3   Institute of Computing Science, Poznan University of Technology, Poznan, Poland
4   Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland
,
R. Thomas
5   Sprott School of Business, Carleton University, Ottawa, Canada
,
M. Kadzinski
3   Institute of Computing Science, Poznan University of Technology, Poznan, Poland
,
K. Farion
6   Departments of Pediatrics and Emergency Medicine, University of Ottawa, Ottawa, Canada
› Institutsangaben
Weitere Informationen

Publikationsverlauf

received:24. Juli 2013

accepted:24. Februar 2014

Publikationsdatum:
20. Januar 2018 (online)

    Lizenzen und Reprints

Summary

Background: Online medical knowledge repositories such as MEDLINE and The Cochrane Library are increasingly used by physicians to retrieve articles to aid with clinical decision making. The prevailing approach for organizing retrieved articles is in the form of a rank-ordered list, with the assumption that the higher an article is presented on a list, the more relevant it is.

Objectives: Despite this common list-based organization, it is seldom studied how physicians perceive the association between the relevance of articles and the order in which articles are presented. In this paper we describe a case study that captured physician preferences for 3-element lists of medical articles in order to learn how to organize medical knowledge for decision-making.

Methods: Comprehensive relevance evaluations were developed to represent 3-element lists of hypothetical articles that may be retrieved from an online medical knowledge source such as MEDLINE or The Cochrane Library. Comprehensive relevance evalua tions asses not only an article’s relevance for a query, but also whether it has been placed on the correct list position. In other words an article may be relevant and correctly placed on a result list (e.g. the most relevant article appears first in the result list), an article may be relevant for a query but placed on an incorrect list position (e.g. the most relevant article appears second in a result list), or an article may be irrelevant for a query yet still appear in the result list. The relevance evaluations were presented to six senior physi cians who were asked to express their preferences for an article’s relevance and its position on a list by pairwise comparisons representing different combinations of 3-element lists. The elicited preferences were assessed using a novel GRIP (Generalized Regression with Intensities of Preference) method and represented as an additive value function. Value functions were derived for individual physicians as well as the group of physicians.

Results: The results show that physicians assign significant value to the 1st position on a list and they expect that the most relevant article is presented first. Whilst physicians still prefer obtaining a correctly placed article on position 2, they are also quite satisfied with misplaced relevant article. Low consideration of the 3rd position was uniformly confirmed.

Conclusions: Our findings confirm the importance of placing the most relevant article on the 1st position on a list and the importance paid to position on a list significantly diminishes after the 2nd position. The derived value functions may be used by developers of clinical decision support applications to decide how best to organize medical knowledge for decision making and to create personalized evaluation measures that can augment typical measures used to evaluate information retrieval systems.

Keywords

Organization of medical evidence - physician preferences - rank-ordered lists - evidence-based medicine - information retrieval - decision support systems - clinical
 

  • References

  • 1 O’Sullivan D, Wilk S, Michalowski W, Farion K. Automatic indexing and retrieval of encounter-specific evidence for point-of-care support. J Biomed Inform 2010; 43: 623-631.
    CrossrefPubMedGoogle Scholar
  • 2 Manning C, Raghavan P, Schutze H. Introduction to Information Retrieval. New York, NY, USA: Cambridge University Press; 2008
    Google Scholar
  • 3 Dong X, Su L. Search engines on the World Wide Web and information retrieval from the Internet: A review and evaluation. Online Inform Rev 1997; 2: 67-82.
    PubMedGoogle Scholar
  • 4 Harter S, Hert C. Evaluation of information retrieval systems: Approaches, issues, and methods. Online Inform Rev 1997; 32: 3-94.
    PubMedGoogle Scholar
  • 5 Kobayashi M, Takeda K. Information retrieval on the Web. ACM Comput Surv 2000; 32: 144-173.
    CrossrefPubMedGoogle Scholar
  • 6 Denecke K. An architecture for diversity-aware search for medical web content. Methods Inf Med 2012; 51 (06) 549-556.
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 7 Westbrook J, Coiera E, Gosling A. Do online information retrieval systems help experienced clinicians answer clinical questions?. JAMIA 2005; 12: 315-321.
    PubMedGoogle Scholar
  • 8 Moskovitch R, Martins SB, Behiri E, Weiss A, Shahar Y. Application of information technology: A comparative evaluation of full-text, concept-based, and context-sensitive search. JAMIA 2007; 14: 164-174.
    PubMedGoogle Scholar
  • 9 D’Avolio L, Nguyen T, Farwell W, Chen Y, Fitzmeyer F, Harris O, Fiore L. Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC). JAMIA 2010; 17: 375-382.
    PubMedGoogle Scholar
  • 10 García-Remesal M, Maojo V, Billhardt H, Crespo J. Integration of Relational and Textual Biomedical Sources A Pilot Experiment Using a Semi-automated Method for Logical Schema Acquisition. Methods Inf Med 2010; 49 (04) 337-348.
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 11 Büttcher S, Clarke C, Cormack GV. Information Retrieval: Implementing and Evaluating Search Engines. The MIT Press 2010
    Google Scholar
  • 12 Allan J. Hard track overview in TREC 2004 (notebook), high accuracy retrieval from documents. Proceedings of the 13th Thirteenth Text Retrieval Conference (TREC 2004) Notebook). 226-235.
    Google Scholar
  • 13 Buckley C, Voorhees E. Retrieval evaluation with incomplete information. Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2004). 25-32.
    Google Scholar
  • 14 Zhang J. The nature of external representations in problem solving. Cognitive Sci 1997; 21: 179-217.
    PubMedGoogle Scholar
  • 15 Keinan G, Friedland N, Ben-Porath Y. Decision making under stress: Scanning of alternatives under physical threat. Acta Psychol 1987; 64: 219-228.
    CrossrefPubMedGoogle Scholar
  • 16 Levin I, Huneke M, Jasper J. Information processing at successive stages of decision making: Need for cognition and inclusion-exclusion effects. Organ Behav Hum Dec 2000; 82: 171-193.
    CrossrefPubMedGoogle Scholar
  • 17 Starcke K, Brand M. Decision making under stress: A selective review. Neurosci Biobehav R 2009; 9: 1228-1248.
    PubMedGoogle Scholar
  • 18 Spink A, Wolfram D, Jansen B, Saracevic T. Searching the web: The public and their queries. JASIST 2001; 53: 226-234.
    PubMedGoogle Scholar
  • 19 Covell DG, Uman P. Manning, Information needs in office practice. Are they being met? Ann Intern Med 1985; 108: 596-599.
    PubMedGoogle Scholar
  • 20 Sackett D, Strauss S. Finding and applying evidence during clinical rounds: the “evidence cart”. JAMIA 1998; 280: 1336-1338.
    PubMedGoogle Scholar
  • 21 Ely JW, Osheroff J, Chambliss M, Ebell M, Rosenbaum M. Answering physicians’ clinical questions: obstacles and potential solutions. JAMIA 2005; 12: 217-224.
    PubMedGoogle Scholar
  • 22 Cullen R. In search of evidence: Family practitioners’ use of the Internet for clinical information. J Med Libr Assoc 2002; 90: 370-379.
    PubMedGoogle Scholar
  • 23 Schilling L, Steiner J, Lundahl K, Anderson R. Residents’ patient-specific clinical questions: opportunities for evidence-based learning. Acad Med 2005; 80: 51-56.
    CrossrefPubMedGoogle Scholar
  • 24 Muller H, Hanbury A, Shorbaji NA. Health information search to deal with the exploding amount of health information produced. Methods Inf Med 2012; 51: 516-518.
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 25 Berner ES, McGowan J. Use of diagnostic decision support systems in medical education. Methods Inf Med 2010; 49: 412-417.
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 26 Davies K. The information-seeking behaviour of doctors: a review of the evidence. Health Info Libr J 2007; 24: 78-94.
    CrossrefPubMedGoogle Scholar
  • 27 Murdock B. The serial position effect of free recall. J Exp Psychol 1962; 64: 482-488.
    CrossrefPubMedGoogle Scholar
  • 28 Hunt R. The subtlety of distinctiveness. Psychonom Bull Rev 1995; 2: 105-112.
    CrossrefPubMedGoogle Scholar
  • 29 Henson R. Short-term memory for serial order: the start-end model. Cognitive Psychol 1998; 36: 73-137.
    CrossrefPubMedGoogle Scholar
  • 30 Terry W. Serial position effects in recall of television commercials. J Gen Psychol 2005; 132: 151-163.
    CrossrefPubMedGoogle Scholar
  • 31 Zamir O, Etzioni O. Grouper: a dynamic clustering interface to web search results. Proceedings of the 8th International Conference on World Wide Web (WWW’09). 1361-1374.
    Google Scholar
  • 32 Senathirajah Y, Bakken S. Visual clustering analysis of CIS logs to inform creation of a user-configurable web CIS interface. Methods Inf Med 2011; 50: 337-348.
    Artikel in Thieme ConnectPubMedGoogle Scholar
  • 33 Klockner K, Wirschum N, Jameson A. Depth- and breadth-first processing of search result lists. Proceedings of the 22nd SIGCHI Conference on Human Factors in Computing Systems (CHI ‘04). 1539
    Google Scholar
  • 34 Payne JW, Bettman JR, Johnson E. The Adaptive Decision Maker. New York: Cambridge University Press; 1993
    Google Scholar
  • 35 Joachims T, Granka L, Pang B, Hembrooke H, Gay G. Accurately interpreting click-through data as implicit feedback. Proceedings of the 28th ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’05). 154-161.
    Google Scholar
  • 36 Cutrell E, Guan Z. What are you looking for? An eye-tracking study of information usage in web search. Proceedings of Human Factors in Computing Systems (CHI’07). 407-416.
    Google Scholar
  • 37 Keane M, O’Brien M, Smyth B. Are people biased in their use of search engines?. Commun ACM 2008; 51: 49-52.
    CrossrefPubMedGoogle Scholar
  • 38 Jansen BJ, Spink A, Bateman J, Saracevic T. Real life information retrieval: A study of user queries on the web. SIGIR Forum 1998; 33: 5-17.
    PubMedGoogle Scholar
  • 39 Jansen BJ, Spink A, Saracevic T. Real life, real users, and real needs: a study and analysis of user queries on the web. Inform Process Manag 2000; 36: 207-227.
    CrossrefPubMedGoogle Scholar
  • 40 Silverstein C, Henzinger M, Marais H, Moricz M. Analysis of a very large Web search engine query log. ACM SIGIR Forum 1999; 33: 6-12.
    CrossrefPubMedGoogle Scholar
  • 41 Cho J, Roy S. Impact of search engines on page popularity. Proceedings of the 13th International World Wide Web Conference (WWW‘04). 20-29.
    Google Scholar
  • 42 Pandey S, Roy S, Olston C, Cho J, Chakrabarti S. Shuffling a stacked deck: the case for partially randomized ranking of search engine results. Proceedings of the 31st international Conference on Very Large Data Bases (VLDB ‘05). 781-792.
    Google Scholar
  • 43 Teevan J. How people recall search result lists. Proceedings of the 24th SIGCHI Conference on Human Factors in Computing Systems (CHI ‘06). 1415-1420.
    Google Scholar
  • 44 Moskovitch R, Shahar Y. Vaidurya: A multiple-ontology, concept-based, context-sensitive clinical-guideline search engine. J Biomed Inform 2009; 42: 11-21.
    CrossrefPubMedGoogle Scholar
  • 45 Lin Y, Li W, Chen K, Liu Y. A Document Clustering and Ranking System for Exploring MEDLINE. J Am Med Inform Assoc 2007; 14: 651-661.
    CrossrefPubMedGoogle Scholar
  • 46 Malo P, Sinha A, Wallenius J, Korhonen P. Concept-based document classification using Wikipedia and value function. JASIST 2011; 62: 2496-2511.
    CrossrefPubMedGoogle Scholar
  • 47 Roy A, Mackin P, Wallenius J, Corner J, Keith M, Schmick G, Arora H. An interactive search method based on user preferences. Dec Anal 2009; 5: 203-229.
    PubMedGoogle Scholar
  • 48 Siskos Y, Grigoroudis V, Matsatsinis N. UTA methods. In: Figueira SJ, Greco M. Ehrgott (eds.) Multiple criteria decision analysis: state of the art surveys New York: Springer Science + Business Media Inc.; 2005: 297-343.
    Google Scholar
  • 49 Greco S, Słowiñski R, Figueira J, Mousseau VM. Robust ordinal regression. In: Ehrgott J, Figueira S. Greco (eds.) Trends in multiple criteria decision analysis New York: Springer Science + Business Media Inc; 2010: 241-283.
    Google Scholar
  • 50 Figueira J, Greco S, Słowiñski R. Building a set of additive value functions representing a reference preorder and intensities of preference: GRIP method. Eur J Oper Res 2009; 195: 460-486.
    CrossrefPubMedGoogle Scholar
  • 51 Greco S, Mousseau V, Słowiñski R. Ordinal regression revisited: multiple criteria ranking with a set of additive value functions. Eur J Oper Res 2008; 191: 415-435.
    PubMedGoogle Scholar
  • 52 Kadziñski M, Greco S, Słowiñski R. Selection of a representative value function in robust multiple criteria ranking and choice. Eur J Oper Res 2012; 217: 541-553.
    CrossrefPubMedGoogle Scholar
  • 53 Kadziñski M, Greco S, Słowiñski R. Selection of a representative value function for robust ordinal regression in group decision making. Group Decis Negot 2013; 22 (03) 429-462.
    CrossrefPubMedGoogle Scholar