Download PDF
Methods Inf Med 2014; 53(05): 389-405
DOI: 10.3414/ME13-02-0033
Focus Theme – Original Articles
Schattauer GmbH

Real-time Feedback on Nonverbal Clinical Communication

Theoretical Framework and Clinician Acceptance of Ambient Visual Design
A. L. Hartzler
1   The Information School, University of Washington, Seattle, Washington, USA
,
R. A. Patel
2   Department of Biomedical Informatics and Medical Education, Division of Biomedical and Health Informatics, University of Washington, Seattle, Washington, USA
,
M. Czerwinski
3   Microsoft Research, Redmond, Washington, USA
,
W. Pratt
1   The Information School, University of Washington, Seattle, Washington, USA
2   Department of Biomedical Informatics and Medical Education, Division of Biomedical and Health Informatics, University of Washington, Seattle, Washington, USA
,
A. Roseway
3   Microsoft Research, Redmond, Washington, USA
,
N. Chandrasekaran
3   Microsoft Research, Redmond, Washington, USA
,
A. Back
4   Department of Medicine, Division of Oncology, University of Washington, Seattle, Washington, USA
› Author Affiliations
Further Information

Publication History

received:21 September 2013

accepted:25 April 2014

Publication Date:
20 January 2018 (online)

    Permissions and Reprints

Summary

Introduction: This article is part of the Focus Theme of Methods of Information in Medicine on “Pervasive Intelligent Technologies for Health”.

Background: Effective nonverbal communication between patients and clinicians fosters both the delivery of empathic patient-centered care and positive patient outcomes. Although nonverbal skill training is a recognized need, few efforts to enhance patient-clinician communication provide visual feedback on nonverbal aspects of the clinical encounter.

Objectives: We describe a novel approach that uses social signal processing technology (SSP) to capture nonverbal cues in real time and to display ambient visual feedback on control and affiliation – two primary, yet distinct dimensions of interpersonal nonverbal communication. To examine the design and clinician acceptance of ambient visual feedback on nonverbal communication, we 1) formulated a model of relational communication to ground SSP and 2) conducted a formative user study using mixed methods to explore the design of visual feedback.

Methods: Based on a model of relational communication, we reviewed interpersonal communication research to map nonverbal cues to signals of affiliation and control evidenced in patient-clinician interaction. Corresponding with our formulation of this theoretical framework, we designed ambient real-time visualizations that reflect variations of affiliation and control. To explore clinicians’ acceptance of this visual feedback, we conducted a lab study using the Wizard-of-Oz technique to simulate system use with 16 healthcare professionals. We followed up with seven of those participants through interviews to iterate on the design with a revised visualization that addressed emergent design considerations.

Results: Ambient visual feedback on non-verbal communication provides a theoretically grounded and acceptable way to provide clinicians with awareness of their nonverbal communication style. We provide implications for the design of such visual feedback that encourages empathic patient-centered communication and include considerations of metaphor, color, size, position, and timing of feedback.

Conclusions: Ambient visual feedback from SSP holds promise as an acceptable means for facilitating empathic patient-centered nonverbal communication.

Keywords

Nonverbal communication - patient-provider communication - information display - social signal processing - formative evaluation - feedback - user-computer interface
 

  • References

  • 1 Lelorain S, Brédart A, Dolbeault S, Sultan S. A systematic review of the associations between empathy measures and patient outcomes in cancer care. Psychooncology 2012; 21 (12) 1255-1264.
    CrossrefPubMedGoogle Scholar
  • 2 Neumann M, Bensing J, Mercer S, Ernstmann N, Ommen O, Pfaff H. Analyzing the “nature” and “specific effectiveness” of clinical empathy: a theoretical overview and contribution towards a theory-based research agenda. Patient Education & Counseling 2009; 74 (03) 339-346.
    CrossrefPubMedGoogle Scholar
  • 3 Epstein RM, Street Jr. RL. Patient-Centered Communication in Cancer Care: Promoting healing and reducing suffering. National Cancer Institute, NIH Publication No. 2007. Bethesda: MD; 07-6225.
    Google Scholar
  • 4 Henry SG, Fuhrel-Forbis A, Rogers MA, Eggly S. Association between nonverbal communication during clinical interactions and outcomes: a systematic review and meta-analysis. Patient Educucation & Counseling 2012; 86 (03) 297-315.
    PubMedGoogle Scholar
  • 5 Ong LM, de Haes JC, Hoos AM, Lammes FB. Doctor-patient communication: a review of the literature. Social Science & Medicine 1995; 40 (07) 903-918.
    CrossrefPubMedGoogle Scholar
  • 6 Street RL, Makoul G, Arora NK, Epstein RM. How does communication heal? Pathways linking clinician-patient communication to health outcomes. Patient Education & Counseling 2009; 74: 295-301.
    CrossrefPubMedGoogle Scholar
  • 7 DiMatteo MR, Taranta T, Friedman HS, Prince LM. Predicting patient satisfaction from physicians’ nonverbal communication skills. Medical Care 1980; 18 (04) 376-387.
    CrossrefPubMedGoogle Scholar
  • 8 Zolnierek KB, DiMatteo MR. Physician communication and patient adherence to treatment: a meta-analysis. Medical Care 2009; 47 (08) 826-834.
    CrossrefPubMedGoogle Scholar
  • 9 Roter DL, Frankel RM, Hall JA, Sluyter D. The expression of emotion through nonverbal behavior in medical visits. Mechanisms and outcomes. Journal of General Internal Medicine 2006; 21 S1 S28-34.
    PubMedGoogle Scholar
  • 10 Buller DB, Street RL. Physician-patient relationships. In: Feldman RS. editor Applications of nonverbal theories and research. London: Psychology Press; 1992: 119-141.
    Google Scholar
  • 11 Hall JA, Harrigan JA, Rosenthal R. Nonverbal behavior in clinician-patient interaction. Applied & Preventive Psychology 1995; 4: 21-37.
    CrossrefPubMedGoogle Scholar
  • 12 Hojat M. Empathy in clinical care: antecedents, development, measurement, and outcomes. New York: Springer; 2007
    Google Scholar
  • 13 Martin L, Friedman HS. Nonverbal communication in health care. In: Riggio RE, Feldman RS. (eds) Applications of nonverbal communication. London: Lawrence Erlbaum; 2005: 3-16.
    Google Scholar
  • 14 Burgoon JK, Guerrero LK, Floyd K. Nonverbal communication. Boston: Pearson; 2010
    Google Scholar
  • 15 Pentland A. Honest signals: How they shape our world. Cambridge, MA: MIT Press; 2010
    Google Scholar
  • 16 Cegala J, Lenzmeier Broz S. Physician communication skills training: a review of theoretical backgrounds, objectives and skills. Medical Education 2002; 36 (11) 1004-1016.
    CrossrefPubMedGoogle Scholar
  • 17 Dickson DA, Hargie O, Morrow NC. Communication skills training for health professionals. London: Nelson Thornes; 2003
    Google Scholar
  • 18 Silverman J. Doctors’ non-verbal behaviour in consultations: look at the patient before you look at the computer. British Journal of General Practice 2010; 60 (571) 76-78.
    CrossrefPubMedGoogle Scholar
  • 19 Patel RA, Hartzler AL, Czerwinski M, Pratt W, Back AL, Roseway A. Leveraging visual feedback from social signal processing to enhance clinicians’ nonverbal skills. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’13); 2013 Apr 27-May 2. Paris, France: ACM; 2013: 421-426.
    Google Scholar
  • 20 Haddawy P. Suebnukarn S. Intelligent Clinical Training Systems. Methods Inf Med 2010; 49: 388-389.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 21 Patel RA, Hartzler AL, Czerwinski M, Pratt W, Back AL, Roseway A. Visual Feedback on Nonverbal Communication: A Design Exploration with Healthcare Professionals. Proceedings of Pervasive health 2013; 7th International Conference on Pervasive Computing Technologies for Healthcare; 2013 May 5-8. Venice, Italy: IEEE; 2013: 105-112.
    Google Scholar
  • 22 Byun B, Awasthi A, Chou PA, Kapoor A, Lee B, Czerwinski M. Honest signals in video conferencing. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’11); 2011 July 11-15. Barcelona, Spain: IEEE; 2011: 1-6.
    Google Scholar
  • 23 Buller MK, Buller DB. Physicians’ communication style and patient satisfaction. Journal of Health & Social Behavior 1987; 28: 375-388.
    CrossrefPubMedGoogle Scholar
  • 24 Gould JD, Conti J, Hovanyecz T. Composing letters with a simulated listening typewriter. Communications of the ACM 1983; 26 (04) 295-308.
    CrossrefPubMedGoogle Scholar
  • 25 Kelley JF. An empirical methodology for writing user-friendly natural language computer applications. In: Janda A. (ed) Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’83). New York, NY, USA: ACM; 193-196.
    Google Scholar
  • 26 Kelley JF. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Office Information Systems 1984; 2 (01) 26-41.
    CrossrefPubMedGoogle Scholar
  • 27 Wilson J, Rosenberg D. Rapid prototyping for user interface design. In: Helander M. (ed). Handbook of Human-Computer Interaction. New York: North-Holland; 1988: 859-875.
    Google Scholar
  • 28 Masthoff J. The user as wizard: A method for early involvement in the design and evaluation of adaptive systems. Fifth Workshop on User-Centred Design and Evaluation of Adaptive Systems. Vol. 1 2006
    Google Scholar
  • 29 Vinciarelli A, Pantic M, Bourlard H, Pentland A. Social signals, their function, and automatic analysis: a survey. In: Proceedings of the 10th international conference on Multimodal interfaces (ICMI’08); 2008 Oct 20-22. Crete, Greece: ACM; 2008: 61-68.
    Google Scholar
  • 30 Pantic M, Cowie R, D’Errico F, Heylen D, Mehu M, Pelachaud C, Poggi I, Schroder M, Vinciarelli A. Social signal processing: The research agenda. In: Moeslund TB, Hilton A, Kruger V, Sigal L. (eds) Visual Analysis of Humans London: Springer; 2011: 511-538.
    Google Scholar
  • 31 Burgoon JK, Hale JL. The fundamental topoi of relational communication. Communication Monographs 1984; 51 (03) 193-214.
    CrossrefPubMedGoogle Scholar
  • 32 Burgoon JK, Hale JL. Validation and measurement of the fundamental themes of relational communication. Communication Monographs 1987; 54 (01) 19-41.
    CrossrefPubMedGoogle Scholar
  • 33 Burgoon JK, Le Poire BA. Nonverbal cues and interpersonal judgements: Participant observer perceptions of intimacy, dominance, composure, and formalitiy. Communication Monographs 1999; 66: 105-124.
    CrossrefPubMedGoogle Scholar
  • 34 Burgoon JK, Pfau M, Parrott R, Birk T, Coker R, Burgoon M. Relational communication, satisfaction, compliance-gaining strategies, and compliance in communication between physicians and patients. Communication Monographs 1987; 54: 307-324.
    CrossrefPubMedGoogle Scholar
  • 35 Gallagher TJ, Hartung PJ, Gregory SW. Assessment of a measure of relational communication for doctor-patient interactions. Patient Education & Counseling 2001; 45 (03) 211-218.
    CrossrefPubMedGoogle Scholar
  • 36 Kiesler DJ, Auerbach SM. Integrating measurement of control and affiliation in studies of physician-patient interaction: the interpersonal circumplex. Social Science & Medicine 2003; 57 (09) 1707-1722.
    CrossrefPubMedGoogle Scholar
  • 37 Street R, Buller D. Nonverbal response patterns in physician-patient interactions: a functional analysis. Journal of Nonverbal Behavior 1987; 11: 234-253.
    CrossrefPubMedGoogle Scholar
  • 38 Gatica-Perez D. Automatic nonverbal analysis of social interaction in small groups: A review. Image and Vision Computing 2009; 27 (12) 1775-1787.
    CrossrefPubMedGoogle Scholar
  • 39 Kim T, Chang A, Holland L, Pentland AS. Meeting mediator: enhancing group collaboration using sociometric feedback. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW’08); 2008 Nov 8-12. San Diego, CA: ACM; 2008: 457-466.
    Google Scholar
  • 40 Sturm J, Herwijnen OH, Eyck A, Terken J. Influencing social dynamics in meetings through a peripheral display. In: Proceedings of the 9th international conference on Multimodal interfaces (ICMI ’07) 2007 Nov 12-15. Nagoya, Japan: ACM; 2007: 263-270.
    Google Scholar
  • 41 DiMicco JM, Pandolfo A, Bender W. Influencing group participation with a shared display. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW’04) 2004 Nov 6-10. Chicago: IL; 614-623.
    Google Scholar
  • 42 Bachour K, Kaplan F, Dillenbourg P. Reflect: An interactive table for regulating face-to-face collaborative learning. In: Dillenbourg P, Specht M. (eds) Lecture Notes in Computer Science 2008; 5192: 39-48.
    PubMedGoogle Scholar
  • 43 Mercer SW, Maxwell M, Heaney D, Watt GC. The consultation and relational empathy (CARE) measure: Development and preliminary validation and reliability of an empathy-based consultation process measure. Family Practice 2004; 21 (06) 699-705.
    CrossrefPubMedGoogle Scholar
  • 44 D’Agostino TA, Bylund CL. The Nonverbal Accommodation Analysis System (NAAS): Initial application and evaluation. Patient Education & Counseling 2011; 85 (01) 33-39.
    CrossrefPubMedGoogle Scholar
  • 45 Roter DL, Larson S. The Roter interaction analysis system (RIAS): Utility and flexibility for analysis of medical interactions. Patient Education & Counseling 2002; 46 (04) 243-251.
    CrossrefPubMedGoogle Scholar
  • 46 Ambady N, Rosenthal R. Thin slices of expressive behavior as predictors of interpersonal consequences—a meta analysis. Psychological Bulletin 1992; 111 (02) 256-274.
    CrossrefPubMedGoogle Scholar
  • 47 Angus D, Watson B, Smith A, Gallois C, Wiles J. Visualising Conversation Structure across Time: Insights into Effective Doctor-Patient Consultations. PLoS ONE 2012; 7 (06) e38014
    CrossrefPubMedGoogle Scholar
  • 48 Hayes-Roth B, Saker R, Amano K. Automating Individualized coaching and authentic role-play practice for brief intervention training. Methods Inf Med 2010; 49: 406-411.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 49 Griol D, Molina JM, Callejas Z. Modeling the user state for context-aware spoken interaction in ambient assisted living. Applied Intelligence 2014 (in press)
    Google Scholar
  • 50 Tentori M, Hayes GR, Reddy M. Pervasive Computing for Hospital, Chronic, and Preventive Care. Foundations and Trends in Human Computer Interaction 2012; 5 (01) 1-95.
    PubMedGoogle Scholar
  • 51 Favela J, Tentori M, Segura D, Berzunza G. Adaptive awareness of hospital patient information through multiple sentient displays. International Journal of Ambient Computing and Intelligence 2009; 1 (01) 27-38.
    CrossrefPubMedGoogle Scholar
  • 52 Montague E, Xu J, Chen PY, Asan O, Barrett BP, Chewning B. Modeling eye gaze patterns in clinician-patient interaction with lag sequential analysis. Human Factors 2011; 53 (05) 502-516.
    CrossrefPubMedGoogle Scholar
  • 53 Stepanikova I, Zhang Q, Wieland D, Eleazer P. Stewart P. Non-verbal communication between primary care physicians and older patients: How does race matter?. Journal of General Internal Medicine 2012; 27 (05) 576-581.
    CrossrefPubMedGoogle Scholar
  • 54 Weibel N, Emmenegger C, Lyons J, Dixit R, Hill LL, Hollan JD. Interpreter-Mediated Physician-Patient Communication: Opportunities for Multimodal Healthcare Interfaces. In: Proceedings of Pervasive health 2013 7th International Conference on Pervasive Computing Technologies for Healthcare; 2013 May 5-8. Venice, Italy: IEEE; 2013: 113-120.
    Google Scholar
  • 55 White A, Danis M. Enhancing patient-centered communication and collaboration by using the electronic health record in the examination room. JAMA 2013; 309 (22) 2327-2328.
    CrossrefPubMedGoogle Scholar
  • 56 Knapp ML, Hall JA. Nonverbal Communication in Human Interaction (7th ed). New York: Harcourt Brace College Publishers; 2010
    Google Scholar
  • 57 Burgoon JK, Buller DB, Woodall WG. Nonverbal communication: The unspoken dialogue. New York: McGraw; 1996
    Google Scholar
  • 58 Hall JA, Roter DL, CS Rand. Communication of Affect between Patient and Physician. Journal of Health and Social Behavior 1981; 22 (01) 18-30.
    CrossrefPubMedGoogle Scholar
  • 59 Robinson JD. Nonverbal communication in doctor-patient relationships. In: Guerrero LK, Hecht M. editors The nonverbal communication reader: Classic and contemporary Readings (3rd edition). Long Grove, IL: Waveland Press; 2007: 384-394.
    Google Scholar
  • 60 Mast MS, Hall JA, Roter DL. Caring and dominance affect participants’ perceptions and behaviors during a virtual medical visit. Journal of General Internal Medicine 2008; 23 (05) 523-527.
    CrossrefPubMedGoogle Scholar
  • 61 Aghabati N, Mohammadi E, Esmaiel ZP. The effect of therapeutic touch on pain and fatigue of cancer patients undergoing chemotherapy. Evidence-based Complementary & Alternative Medicine 2010; 7 (03) 375-381.
    PubMedGoogle Scholar
  • 62 Wong J, Ghiasuddin A, Kimata C, Patelesio B, Siu A. The impact of healing touch on pediatric oncology patients. Integrative Cancer Therapies 2013; 12 (01) 25-30.
    CrossrefPubMedGoogle Scholar
  • 63 Manusov VL. The sourcebook of nonverbal measures. Going beyond words. Mahwah, NJ: Taylor & Francis; 2004
    Google Scholar
  • 64 Larsen KM, Smith CK. Assessment on nonverbal communication in the patient-physician interview. Journal of Family Practice 1981; 12: 481-488.
    PubMedGoogle Scholar
  • 65 Harrigan JA, Kues JR, Weber JG. Impressions of hand movements: Self-touching and gestures. Perceptual and Motor Skills 1986; 63: 503-516.
    CrossrefPubMedGoogle Scholar
  • 66 Harrigan JA, Oxman TE, Rosenthal RR. Rapport expressed through nonverbal behavior. Journal of Nonverbal Behavior 1985; 9 (02) 95-110.
    CrossrefPubMedGoogle Scholar
  • 67 Duggan AP, Parrott RL. Physicians’ nonverbal rapport building and patients’ talk about the subjective component of illness. Human Communication Research 2001; 27 (02) 299-311.
    PubMedGoogle Scholar
  • 68 Tickle-Degnen L, Rosenthal R. Nonverbal aspects of therapeutic rapport. Feldman RS. editor In: Applications of nonverbal theories and research London: Psychology Press; 1992: 142-164.
    Google Scholar
  • 69 Aguinis H, Simonsen MM, Pierce CA. Effects of nonverbal behavior on perceptions of power. Journal of Social Psychology 1998; 138: 455-469.
    CrossrefPubMedGoogle Scholar
  • 70 Hall JA, Roter DL, Katz NR. Meta-analysis of correlates of provider behavior in medical encounters. Medical Care 1988; 26 (07) 657-675.
    CrossrefPubMedGoogle Scholar
  • 71 Ambady N, Koo J, Rosenthal R, Winograd CH. Physical therapists’ nonverbal communication predicts geriatric patients’ health outcomes. Psychology and Aging 2002; 17 (03) 443-452.
    CrossrefPubMedGoogle Scholar
  • 72 Harrigan JA, Rosenthal RR. Physicians’ head and body positions as determinants of perceived rapport. Journal of Applied Social Psychology 1983; 13 (06) 496-509.
    CrossrefPubMedGoogle Scholar
  • 73 Weinberger M, Greene JY, Mamlin JH. The impact of clinicial encounter events on patient and physician satisfaction. Social Science and Medicine (Part E) 1981; 15 (03) 239-244.
    PubMedGoogle Scholar
  • 74 Beck RS, Daughtridge R, Sloane PD. Physician-patient communication in the primary care office: a systematic review. Journal of the American Board Family Practice 2002; 15: 25-38.
    PubMedGoogle Scholar
  • 75 Roter DL, Hall JA, Katz NR. Relations between physicians’ behaviors and analogue patients’ satisfaction, recall, and impressions. Medical Care 1987; 25 (05) 437-451.
    CrossrefPubMedGoogle Scholar
  • 76 Slovák P, Janssen J, Fitzpatrick G. Understanding heart rate sharing: towards unpacking physiosocial space. In: Proceedings of the SIGCHI Confer- ence on Human Factors in Computing Systems (CHI ’12); 2012 May 5-10. Austin, TX: ACM; 2012: 859-868.
    Google Scholar
  • 77 Chung H, Lee H, Chang D, Kim H, Lee H, Park H, Chae Y. Doctor’s attire influences perceived empathy in the patient-doctor relationship. Patient Education & Counseling 2012; 89: 387-391.
    CrossrefPubMedGoogle Scholar