Real-time Feedback on Nonverbal Clinical Communication

A. L. Hartzler; R. A. Patel; M. Czerwinski; W. Pratt; A. Roseway; N. Chandrasekaran; A. Back

doi:10.3414/ME13-02-0033

Methods of Information in Medicine, Table of Contents

Methods Inf Med 2014; 53(05): 389-405
DOI: 10.3414/ME13-02-0033

Focus Theme – Original Articles

Schattauer GmbH

Real-time Feedback on Nonverbal Clinical Communication

Theoretical Framework and Clinician Acceptance of Ambient Visual Design

A. L. Hartzler

¹The Information School, University of Washington, Seattle, Washington, USA

,

R. A. Patel

²Department of Biomedical Informatics and Medical Education, Division of Biomedical and Health Informatics, University of Washington, Seattle, Washington, USA

,

M. Czerwinski

³Microsoft Research, Redmond, Washington, USA

,

W. Pratt

¹The Information School, University of Washington, Seattle, Washington, USA

²Department of Biomedical Informatics and Medical Education, Division of Biomedical and Health Informatics, University of Washington, Seattle, Washington, USA

,

A. Roseway

³Microsoft Research, Redmond, Washington, USA

,

N. Chandrasekaran

³Microsoft Research, Redmond, Washington, USA

,

A. Back

⁴Department of Medicine, Division of Oncology, University of Washington, Seattle, Washington, USA

› Author Affiliations

Abstract

Summary

Introduction: This article is part of the Focus Theme of Methods of Information in Medicine on “Pervasive Intelligent Technologies for Health”.

Background: Effective nonverbal communication between patients and clinicians fosters both the delivery of empathic patient-centered care and positive patient outcomes. Although nonverbal skill training is a recognized need, few efforts to enhance patient-clinician communication provide visual feedback on nonverbal aspects of the clinical encounter.

Objectives: We describe a novel approach that uses social signal processing technology (SSP) to capture nonverbal cues in real time and to display ambient visual feedback on control and affiliation – two primary, yet distinct dimensions of interpersonal nonverbal communication. To examine the design and clinician acceptance of ambient visual feedback on nonverbal communication, we 1) formulated a model of relational communication to ground SSP and 2) conducted a formative user study using mixed methods to explore the design of visual feedback.

Methods: Based on a model of relational communication, we reviewed interpersonal communication research to map nonverbal cues to signals of affiliation and control evidenced in patient-clinician interaction. Corresponding with our formulation of this theoretical framework, we designed ambient real-time visualizations that reflect variations of affiliation and control. To explore clinicians’ acceptance of this visual feedback, we conducted a lab study using the Wizard-of-Oz technique to simulate system use with 16 healthcare professionals. We followed up with seven of those participants through interviews to iterate on the design with a revised visualization that addressed emergent design considerations.

Results: Ambient visual feedback on non-verbal communication provides a theoretically grounded and acceptable way to provide clinicians with awareness of their nonverbal communication style. We provide implications for the design of such visual feedback that encourages empathic patient-centered communication and include considerations of metaphor, color, size, position, and timing of feedback.

Conclusions: Ambient visual feedback from SSP holds promise as an acceptable means for facilitating empathic patient-centered nonverbal communication.

Keywords

Nonverbal communication - patient-provider communication - information display - social signal processing - formative evaluation - feedback - user-computer interface

Full Text

References

References
1 Lelorain S, Brédart A, Dolbeault S, Sultan S. A systematic review of the associations between empathy measures and patient outcomes in cancer care. Psychooncology 2012; 21 (12) 1255-1264.
2 Neumann M, Bensing J, Mercer S, Ernstmann N, Ommen O, Pfaff H. Analyzing the “nature” and “specific effectiveness” of clinical empathy: a theoretical overview and contribution towards a theory-based research agenda. Patient Education & Counseling 2009; 74 (03) 339-346.
3 Epstein RM, Street Jr. RL. Patient-Centered Communication in Cancer Care: Promoting healing and reducing suffering. National Cancer Institute, NIH Publication No. 2007. Bethesda: MD; 07-6225.
4 Henry SG, Fuhrel-Forbis A, Rogers MA, Eggly S. Association between nonverbal communication during clinical interactions and outcomes: a systematic review and meta-analysis. Patient Educucation & Counseling 2012; 86 (03) 297-315.
5 Ong LM, de Haes JC, Hoos AM, Lammes FB. Doctor-patient communication: a review of the literature. Social Science & Medicine 1995; 40 (07) 903-918.
6 Street RL, Makoul G, Arora NK, Epstein RM. How does communication heal? Pathways linking clinician-patient communication to health outcomes. Patient Education & Counseling 2009; 74: 295-301.
7 DiMatteo MR, Taranta T, Friedman HS, Prince LM. Predicting patient satisfaction from physicians’ nonverbal communication skills. Medical Care 1980; 18 (04) 376-387.
8 Zolnierek KB, DiMatteo MR. Physician communication and patient adherence to treatment: a meta-analysis. Medical Care 2009; 47 (08) 826-834.
9 Roter DL, Frankel RM, Hall JA, Sluyter D. The expression of emotion through nonverbal behavior in medical visits. Mechanisms and outcomes. Journal of General Internal Medicine 2006; 21 S1 S28-34.
10 Buller DB, Street RL. Physician-patient relationships. In: Feldman RS. editor Applications of nonverbal theories and research. London: Psychology Press; 1992: 119-141.
11 Hall JA, Harrigan JA, Rosenthal R. Nonverbal behavior in clinician-patient interaction. Applied & Preventive Psychology 1995; 4: 21-37.
12 Hojat M. Empathy in clinical care: antecedents, development, measurement, and outcomes. New York: Springer; 2007
13 Martin L, Friedman HS. Nonverbal communication in health care. In: Riggio RE, Feldman RS. (eds) Applications of nonverbal communication. London: Lawrence Erlbaum; 2005: 3-16.
14 Burgoon JK, Guerrero LK, Floyd K. Nonverbal communication. Boston: Pearson; 2010
15 Pentland A. Honest signals: How they shape our world. Cambridge, MA: MIT Press; 2010
16 Cegala J, Lenzmeier Broz S. Physician communication skills training: a review of theoretical backgrounds, objectives and skills. Medical Education 2002; 36 (11) 1004-1016.
17 Dickson DA, Hargie O, Morrow NC. Communication skills training for health professionals. London: Nelson Thornes; 2003
18 Silverman J. Doctors’ non-verbal behaviour in consultations: look at the patient before you look at the computer. British Journal of General Practice 2010; 60 (571) 76-78.
19 Patel RA, Hartzler AL, Czerwinski M, Pratt W, Back AL, Roseway A. Leveraging visual feedback from social signal processing to enhance clinicians’ nonverbal skills. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’13); 2013 Apr 27-May 2. Paris, France: ACM; 2013: 421-426.
20 Haddawy P. Suebnukarn S. Intelligent Clinical Training Systems. Methods Inf Med 2010; 49: 388-389.
21 Patel RA, Hartzler AL, Czerwinski M, Pratt W, Back AL, Roseway A. Visual Feedback on Nonverbal Communication: A Design Exploration with Healthcare Professionals. Proceedings of Pervasive health 2013; 7th International Conference on Pervasive Computing Technologies for Healthcare; 2013 May 5-8. Venice, Italy: IEEE; 2013: 105-112.
22 Byun B, Awasthi A, Chou PA, Kapoor A, Lee B, Czerwinski M. Honest signals in video conferencing. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’11); 2011 July 11-15. Barcelona, Spain: IEEE; 2011: 1-6.
23 Buller MK, Buller DB. Physicians’ communication style and patient satisfaction. Journal of Health & Social Behavior 1987; 28: 375-388.
24 Gould JD, Conti J, Hovanyecz T. Composing letters with a simulated listening typewriter. Communications of the ACM 1983; 26 (04) 295-308.
25 Kelley JF. An empirical methodology for writing user-friendly natural language computer applications. In: Janda A. (ed) Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’83). New York, NY, USA: ACM; 193-196.
26 Kelley JF. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Office Information Systems 1984; 2 (01) 26-41.
27 Wilson J, Rosenberg D. Rapid prototyping for user interface design. In: Helander M. (ed). Handbook of Human-Computer Interaction. New York: North-Holland; 1988: 859-875.
28 Masthoff J. The user as wizard: A method for early involvement in the design and evaluation of adaptive systems. Fifth Workshop on User-Centred Design and Evaluation of Adaptive Systems. Vol. 1 2006
29 Vinciarelli A, Pantic M, Bourlard H, Pentland A. Social signals, their function, and automatic analysis: a survey. In: Proceedings of the 10th international conference on Multimodal interfaces (ICMI’08); 2008 Oct 20-22. Crete, Greece: ACM; 2008: 61-68.
30 Pantic M, Cowie R, D’Errico F, Heylen D, Mehu M, Pelachaud C, Poggi I, Schroder M, Vinciarelli A. Social signal processing: The research agenda. In: Moeslund TB, Hilton A, Kruger V, Sigal L. (eds) Visual Analysis of Humans London: Springer; 2011: 511-538.
31 Burgoon JK, Hale JL. The fundamental topoi of relational communication. Communication Monographs 1984; 51 (03) 193-214.
32 Burgoon JK, Hale JL. Validation and measurement of the fundamental themes of relational communication. Communication Monographs 1987; 54 (01) 19-41.
33 Burgoon JK, Le Poire BA. Nonverbal cues and interpersonal judgements: Participant observer perceptions of intimacy, dominance, composure, and formalitiy. Communication Monographs 1999; 66: 105-124.
34 Burgoon JK, Pfau M, Parrott R, Birk T, Coker R, Burgoon M. Relational communication, satisfaction, compliance-gaining strategies, and compliance in communication between physicians and patients. Communication Monographs 1987; 54: 307-324.
35 Gallagher TJ, Hartung PJ, Gregory SW. Assessment of a measure of relational communication for doctor-patient interactions. Patient Education & Counseling 2001; 45 (03) 211-218.
36 Kiesler DJ, Auerbach SM. Integrating measurement of control and affiliation in studies of physician-patient interaction: the interpersonal circumplex. Social Science & Medicine 2003; 57 (09) 1707-1722.
37 Street R, Buller D. Nonverbal response patterns in physician-patient interactions: a functional analysis. Journal of Nonverbal Behavior 1987; 11: 234-253.
38 Gatica-Perez D. Automatic nonverbal analysis of social interaction in small groups: A review. Image and Vision Computing 2009; 27 (12) 1775-1787.
39 Kim T, Chang A, Holland L, Pentland AS. Meeting mediator: enhancing group collaboration using sociometric feedback. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW’08); 2008 Nov 8-12. San Diego, CA: ACM; 2008: 457-466.
40 Sturm J, Herwijnen OH, Eyck A, Terken J. Influencing social dynamics in meetings through a peripheral display. In: Proceedings of the 9th international conference on Multimodal interfaces (ICMI ’07) 2007 Nov 12-15. Nagoya, Japan: ACM; 2007: 263-270.
41 DiMicco JM, Pandolfo A, Bender W. Influencing group participation with a shared display. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW’04) 2004 Nov 6-10. Chicago: IL; 614-623.
42 Bachour K, Kaplan F, Dillenbourg P. Reflect: An interactive table for regulating face-to-face collaborative learning. In: Dillenbourg P, Specht M. (eds) Lecture Notes in Computer Science 2008; 5192: 39-48.
43 Mercer SW, Maxwell M, Heaney D, Watt GC. The consultation and relational empathy (CARE) measure: Development and preliminary validation and reliability of an empathy-based consultation process measure. Family Practice 2004; 21 (06) 699-705.
44 D’Agostino TA, Bylund CL. The Nonverbal Accommodation Analysis System (NAAS): Initial application and evaluation. Patient Education & Counseling 2011; 85 (01) 33-39.
45 Roter DL, Larson S. The Roter interaction analysis system (RIAS): Utility and flexibility for analysis of medical interactions. Patient Education & Counseling 2002; 46 (04) 243-251.
46 Ambady N, Rosenthal R. Thin slices of expressive behavior as predictors of interpersonal consequences—a meta analysis. Psychological Bulletin 1992; 111 (02) 256-274.
47 Angus D, Watson B, Smith A, Gallois C, Wiles J. Visualising Conversation Structure across Time: Insights into Effective Doctor-Patient Consultations. PLoS ONE 2012; 7 (06) e38014
48 Hayes-Roth B, Saker R, Amano K. Automating Individualized coaching and authentic role-play practice for brief intervention training. Methods Inf Med 2010; 49: 406-411.
49 Griol D, Molina JM, Callejas Z. Modeling the user state for context-aware spoken interaction in ambient assisted living. Applied Intelligence 2014 (in press)
50 Tentori M, Hayes GR, Reddy M. Pervasive Computing for Hospital, Chronic, and Preventive Care. Foundations and Trends in Human Computer Interaction 2012; 5 (01) 1-95.
51 Favela J, Tentori M, Segura D, Berzunza G. Adaptive awareness of hospital patient information through multiple sentient displays. International Journal of Ambient Computing and Intelligence 2009; 1 (01) 27-38.
52 Montague E, Xu J, Chen PY, Asan O, Barrett BP, Chewning B. Modeling eye gaze patterns in clinician-patient interaction with lag sequential analysis. Human Factors 2011; 53 (05) 502-516.
53 Stepanikova I, Zhang Q, Wieland D, Eleazer P. Stewart P. Non-verbal communication between primary care physicians and older patients: How does race matter?. Journal of General Internal Medicine 2012; 27 (05) 576-581.
54 Weibel N, Emmenegger C, Lyons J, Dixit R, Hill LL, Hollan JD. Interpreter-Mediated Physician-Patient Communication: Opportunities for Multimodal Healthcare Interfaces. In: Proceedings of Pervasive health 2013 7th International Conference on Pervasive Computing Technologies for Healthcare; 2013 May 5-8. Venice, Italy: IEEE; 2013: 113-120.
55 White A, Danis M. Enhancing patient-centered communication and collaboration by using the electronic health record in the examination room. JAMA 2013; 309 (22) 2327-2328.
56 Knapp ML, Hall JA. Nonverbal Communication in Human Interaction (7th ed). New York: Harcourt Brace College Publishers; 2010
57 Burgoon JK, Buller DB, Woodall WG. Nonverbal communication: The unspoken dialogue. New York: McGraw; 1996
58 Hall JA, Roter DL, CS Rand. Communication of Affect between Patient and Physician. Journal of Health and Social Behavior 1981; 22 (01) 18-30.
59 Robinson JD. Nonverbal communication in doctor-patient relationships. In: Guerrero LK, Hecht M. editors The nonverbal communication reader: Classic and contemporary Readings (3rd edition). Long Grove, IL: Waveland Press; 2007: 384-394.
60 Mast MS, Hall JA, Roter DL. Caring and dominance affect participants’ perceptions and behaviors during a virtual medical visit. Journal of General Internal Medicine 2008; 23 (05) 523-527.
61 Aghabati N, Mohammadi E, Esmaiel ZP. The effect of therapeutic touch on pain and fatigue of cancer patients undergoing chemotherapy. Evidence-based Complementary & Alternative Medicine 2010; 7 (03) 375-381.
62 Wong J, Ghiasuddin A, Kimata C, Patelesio B, Siu A. The impact of healing touch on pediatric oncology patients. Integrative Cancer Therapies 2013; 12 (01) 25-30.
63 Manusov VL. The sourcebook of nonverbal measures. Going beyond words. Mahwah, NJ: Taylor & Francis; 2004
64 Larsen KM, Smith CK. Assessment on nonverbal communication in the patient-physician interview. Journal of Family Practice 1981; 12: 481-488.
65 Harrigan JA, Kues JR, Weber JG. Impressions of hand movements: Self-touching and gestures. Perceptual and Motor Skills 1986; 63: 503-516.
66 Harrigan JA, Oxman TE, Rosenthal RR. Rapport expressed through nonverbal behavior. Journal of Nonverbal Behavior 1985; 9 (02) 95-110.
67 Duggan AP, Parrott RL. Physicians’ nonverbal rapport building and patients’ talk about the subjective component of illness. Human Communication Research 2001; 27 (02) 299-311.
68 Tickle-Degnen L, Rosenthal R. Nonverbal aspects of therapeutic rapport. Feldman RS. editor In: Applications of nonverbal theories and research London: Psychology Press; 1992: 142-164.
69 Aguinis H, Simonsen MM, Pierce CA. Effects of nonverbal behavior on perceptions of power. Journal of Social Psychology 1998; 138: 455-469.
70 Hall JA, Roter DL, Katz NR. Meta-analysis of correlates of provider behavior in medical encounters. Medical Care 1988; 26 (07) 657-675.
71 Ambady N, Koo J, Rosenthal R, Winograd CH. Physical therapists’ nonverbal communication predicts geriatric patients’ health outcomes. Psychology and Aging 2002; 17 (03) 443-452.
72 Harrigan JA, Rosenthal RR. Physicians’ head and body positions as determinants of perceived rapport. Journal of Applied Social Psychology 1983; 13 (06) 496-509.
73 Weinberger M, Greene JY, Mamlin JH. The impact of clinicial encounter events on patient and physician satisfaction. Social Science and Medicine (Part E) 1981; 15 (03) 239-244.
74 Beck RS, Daughtridge R, Sloane PD. Physician-patient communication in the primary care office: a systematic review. Journal of the American Board Family Practice 2002; 15: 25-38.
75 Roter DL, Hall JA, Katz NR. Relations between physicians’ behaviors and analogue patients’ satisfaction, recall, and impressions. Medical Care 1987; 25 (05) 437-451.
76 Slovák P, Janssen J, Fitzpatrick G. Understanding heart rate sharing: towards unpacking physiosocial space. In: Proceedings of the SIGCHI Confer- ence on Human Factors in Computing Systems (CHI ’12); 2012 May 5-10. Austin, TX: ACM; 2012: 859-868.
77 Chung H, Lee H, Chang D, Kim H, Lee H, Park H, Chae Y. Doctor’s attire influences perceived empathy in the patient-doctor relationship. Patient Education & Counseling 2012; 89: 387-391.