Facial Expressions as an Index of Listening Difficulty and Emotional Response

Abstract

Knowledge about listening difficulty experienced during a task can be used to better understand speech perception processes, to guide amplification outcomes, and can be used by individuals to decide whether to participate in communication. Another factor affecting these decisions is individuals' emotional response which has not been measured objectively previously. In this study, we describe a novel method of measuring listening difficulty and affect of individuals in adverse listening situations using automatic facial expression algorithm. The purpose of our study was to determine if facial expressions of confusion and frustration are sensitive to changes in listening difficulty. We recorded speech recognition scores, facial expressions, subjective listening effort scores, and subjective emotional responses in 33 young participants with normal hearing. We used the signal-to-noise ratios of −1, +2, and +5 dB SNR and quiet conditions to vary the difficulty level. We found that facial expression of confusion and frustration increased with increase in difficulty level, but not with change in each level. We also found a relationship between facial expressions and both subjective emotion ratings and subjective listening effort. Emotional responses in the form of facial expressions show promise as a measure of affect and listening difficulty. Further research is needed to determine the specific contribution of affect to communication in challenging listening environments.

Keywords

emotional responses - facial expressions - listening effort - confusion

Publication History

Article published online:
04 April 2023

• References

• 1 Pichora-Fuller MK, Kramer SE, Eckert MA. et al. Hearing impairment and cognitive energy: the framework for understanding effortful listening (FUEL). Ear Hear 2016; 37 (Suppl. 01) 5S-27S
• 2 Francis AL, Love J. Listening effort: Are we measuring cognition or affect, or both?. Wiley Interdiscip Rev Cogn Sci 2020; 11 (01) e1514
• 3 Winn MB, Teece KH. Listening effort is not the same as speech intelligibility score. Trends Hear 2021;25:23312165211027688
• 4 Wendt D, Hietkamp RK, Lunner T. Impact of noise and noise reduction on processing effort: a pupillometry study. Ear Hear 2017; 38 (06) 690-700
• 5 Desjardins JL, Doherty KA. The effect of hearing aid noise reduction on listening effort in hearing-impaired adults. Ear Hear 2014; 35 (06) 600-610
• 6 Hughes SE, Hutchings HA, Rapport FL, McMahon CM, Boisvert I. Social connectedness and perceived listening effort in adult cochlear implant users: a grounded theory to establish content validity for a new patient-reported outcome measure. Ear Hear 2018; 39 (05) 922-934
• 7 Francis AL, Oliver J. Psychophysiological measurement of affective responses during speech perception. Hear Res 2018; 369: 103-119
• 8 Wu Y-H, Stangl E, Zhang X, Perkins J, Eilers E. Psychometric functions of dual-task paradigms for measuring listening effort. Ear Hear 2016; 37 (06) 660-670
• 9 Colby S, McMurray B. Cognitive and physiological measures of listening effort during degraded speech perception: relating dual-task and pupillometry paradigms. J Speech Lang Hear Res 2021; 64 (09) 3627-3652
• 10 Sarampalis A, Kalluri S, Edwards B, Hafter E. Objective measures of listening effort: effects of background noise and noise reduction. J Speech Lang Hear Res 2009; 52 (05) 1230-1240
• 11 McGarrigle R, Munro KJ, Dawes P. et al. Listening effort and fatigue: what exactly are we measuring? A British Society of Audiology Cognition in Hearing Special Interest Group ‘white paper’. Int J Audiol 2014; 53 (07) 433-440
• 12 Giuliani NP, Brown CJ, Wu Y-H. Comparisons of the sensitivity and reliability of multiple measures of listening effort. Ear Hear 2021; 42 (02) 465-474
• 13 Zekveld AA, Kramer SE, Festen JM. Cognitive load during speech perception in noise: the influence of age, hearing loss, and cognition on the pupil response. Ear Hear 2011; 32 (04) 498-510
• 14 Mackersie CL, MacPhee IX, Heldt EW. Effects of hearing loss on heart rate variability and skin conductance measured during sentence recognition in noise. Ear Hear 2015; 36 (01) 145-154
• 15 Seeman S, Sims R. Comparison of psychophysiological and dual-task measures of listening effort. J Speech Lang Hear Res 2015; 58 (06) 1781-1792
• 16 Mackersie CL, Cones H. Subjective and psychophysiological indexes of listening effort in a competing-talker task. J Am Acad Audiol 2011; 22 (02) 113-122
• 17 Ohlenforst B, Wendt D, Kramer SE, Naylor G, Zekveld AA, Lunner T. Impact of SNR, masker type and noise reduction processing on sentence recognition performance and listening effort as indicated by the pupil dilation response. Hear Res 2018; 365: 90-99
• 18 Moore TM, Picou EM. A potential bias in subjective ratings of mental effort. J Speech Lang Hear Res 2018; 61 (09) 2405-2421
• 19 Alhanbali S, Dawes P, Millman RE, Munro KJ. Measures of listening effort are multidimensional. Ear Hear 2019; 40 (05) 1084-1097
• 20 Strand JF, Brown VA, Merchant MB, Brown HE, Smith J. Measuring listening effort: convergent validity, sensitivity, and links with cognitive and personality measures. J Speech Lang Hear Res 2018; 61 (06) 1463-1486
• 21 Zekveld AA, Kramer SE, Festen JM. Pupil response as an indication of effortful listening: the influence of sentence intelligibility. Ear Hear 2010; 31 (04) 480-490
• 22 Zekveld AA, Kramer SE. Cognitive processing load across a wide range of listening conditions: insights from pupillometry. Psychophysiology 2014; 51 (03) 277-284
• 23 Gosselin PA, Gagné JP. Older adults expend more listening effort than young adults recognizing audiovisual speech in noise. Int J Audiol 2011; 50 (11) 786-792
• 24 Lench HC, Carpenter ZK. What do emotions do for us? In: Gonzales M, ed. The Function of Emotions. Springer; 2018: 1-7
• 25 Chirico A, Yaden DB. Awe: a self-transcendent and sometimes transformative emotion. In: Gonzales M, ed. The Function of Emotions. Springer; 2018: 221-233
• 26 Storbeck J, Wylie J. The functional and dysfunctional aspects of happiness: cognitive, physiological, behavioral, and health considerations. In: Gonzales M, ed. The Function of Emotions. Springer; 2018: 195-220
• 27 Picou EM, Buono GH. Emotional responses to pleasant sounds are related to social disconnectedness and loneliness independent of hearing loss. Trends Hear 2018; 22: 2331216518813243
• 28 Picou EM. How hearing loss and age affect emotional responses to nonspeech sounds. J Speech Lang Hear Res 2016; 59 (05) 1233-1246
• 29 Ciuk D, Troy A, Jones M. Measuring emotion: Self-reports vs. physiological indicators. Human Cognition in Evolution & Development 2015;
• 30 Krosnick JA. The role of attitude importance in social evaluation: a study of policy preferences, presidential candidate evaluations, and voting behavior. J Pers Soc Psychol 1988; 55 (02) 196-210
• 31 Berinsky AJ. Silent Voices: Public Opinion and Political Participation in America. Princeton University Press; 2004
• 32 McDermott R. Cognitive neuroscience and politics: next steps. In: Neuman WR, Marcus GE, MacKuen M, Crigler AN, eds. The Affect Effect Dynamics of Emotion in Political Thinking and Behavior. 2007: 375-397
• 33 Mauss IB, Levenson RW, McCarter L, Wilhelm FH, Gross JJ. The tie that binds? Coherence among emotion experience, behavior, and physiology. Emotion 2005; 5 (02) 175-190
• 34 Rosenberg EL, Ekman P. Coherence between expressive and experiential systems in emotion. Cogn Emotion 1994; 8 (03) 201-229
• 35 Ruch W. Will the real relationship between facial expression and affective experience please stand up: the case of exhilaration. Cogn Emotion 1995; 9 (01) 33-58
• 36 Bradley MM, Lang PJ. Measuring emotion: behavior, feeling, and physiology. Cognitive Neuroscience of Emotion. 2000; 25: 49-59
• 37 Li S, Walters G, Packer J, Scott N. A comparative analysis of self-report and psychophysiological measures of emotion in the context of tourism advertising. J Travel Res 2018; 57 (08) 1078-1092
• 38 Matsumoto D, Keltner D, Shiota MN, O'Sullivan M, Frank M. Facial expressions of emotion. In: Lewis M, Haviland-Jones JM, Barrett LF, eds. Handbook of Emotions. The Guilford Press; 2008: 211-234
• 39 Di Leo I, Muis KR, Singh CA, Psaradellis C. Curiosity… Confusion? Frustration! The role and sequencing of emotions during mathematics problem solving. Contemp Educ Psychol 2019; 58: 121-137
• 40 Dolgunsöz E. The effect of taboo content on incidental vocabulary acquisition in a foreign language: a facial expression analysis study. Stud Psychol (Bratisl) 2019; 61 (01) 3-16
• 41 Bartlett MS, Littlewort G, Fasel I, Movellan JR. Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction. 2003 Conference on Computer Vision and Pattern Recognition Workshop; 2003
• 42 Craig SD, D'Mello S, Witherspoon A, Graesser A. Emote aloud during learning with AutoTutor: applying the Facial Action Coding System to cognitive–affective states during learning. Cogn Emotion 2008; 22 (05) 777-788
• 43 Gutierrez-Garcia A, Beltran D, Calvo MG. Facial attractiveness impressions precede trustworthiness inferences: lower detection thresholds and faster decision latencies. Cogn Emot 2019; 33 (02) 378-385
• 44 Rozin P, Cohen AB. High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. Emotion 2003; 3 (01) 68-75
• 45 Darwin C, Prodger P. The Expression of the Emotions in Man and Animals. USA: Oxford University Press; 1998
• 46 Ekman P. Darwin and Facial Expression: A Century of Research in Review. ISHK; 2006
• 47 Ekman P, Davidson RJ, Friesen WV. The Duchenne smile: emotional expression and brain physiology. II. J Pers Soc Psychol 1990; 58 (02) 342-353
• 48 Bonanno GA, Keltner D, Noll JG. et al. When the face reveals what words do not: facial expressions of emotion, smiling, and the willingness to disclose childhood sexual abuse. J Pers Soc Psychol 2002; 83 (01) 94-110
• 49 Chesney MA, Ekman P, Friesen WV, Black GW, Hecker MH. Type A behavior pattern: facial behavior and speech components. Psychosom Med 1990; 52 (03) 307-319
• 50 Jack RE, Garrod OG, Yu H, Caldara R, Schyns PG. Facial expressions of emotion are not culturally universal. Proc Natl Acad Sci U S A 2012; 109 (19) 7241-7244
• 51 Matsumoto D. American-Japanese cultural differences in the recognition of universal facial expressions. J Cross Cult Psychol 1992; 23 (01) 72-84
• 52 Gendron M, Crivelli C, Barrett LF. Universality reconsidered: diversity in making meaning of facial expressions. Curr Dir Psychol Sci 2018; 27 (04) 211-219
• 53 Nelson NL, Russell JA. Universality revisited. Emot Rev 2013; 5 (01) 8-15
• 54 Baron RA. Facial expressions of emotion while watching televised violence as predictors of subsequent aggression. Television and Social Behavior: Reports and Papers; a Technical Report to the Surgeon General's Scientific Advisory Committee on Television and Social Behavior. 1972;5:22
• 55 Marsh AA, Ambady N, Kleck RE. The effects of fear and anger facial expressions on approach- and avoidance-related behaviors. Emotion 2005; 5 (01) 119-124
• 56 Winkielman P, Berridge KC, Wilbarger JL. Unconscious affective reactions to masked happy versus angry faces influence consumption behavior and judgments of value. Pers Soc Psychol Bull 2005; 31 (01) 121-135
• 57 Helfer KS, Merchant GR, Wasiuk PA. Age-related changes in objective and subjective speech perception in complex listening environments. J Speech Lang Hear Res 2017; 60 (10) 3009-3018
• 58 Graesser AC, Lu S, Olde BA, Cooper-Pye E, Whitten S. Question asking and eye tracking during cognitive disequilibrium: comprehending illustrated texts on devices when the devices break down. Mem Cognit 2005; 33 (07) 1235-1247
• 59 Marinelli A. A Qualitative Examination of the Listening Effort Experience of Adults with Hearing Loss; 2017
• 60 Picou EM, Ricketts TA. Increasing motivation changes subjective reports of listening effort and choice of coping strategy. Int J Audiol 2014; 53 (06) 418-426
• 61 Ekman P, Friesen WV. Measuring facial movement. Environ Psychol Nonverbal Behav 1976; 1 (01) 56-75
• 62 Ortony A, Turner TJ. What's basic about basic emotions?. Psychol Rev 1990; 97 (03) 315-331
• 63 Scherer KR, Schorr A, Johnstone T. Appraisal Processes in Emotion: Theory, Methods, Research. Oxford University Press; 2001
• 64 Stöckli S, Schulte-Mecklenbeck M, Borer S, Samson AC. Facial expression analysis with AFFDEX and FACET: a validation study. Behav Res Methods 2018; 50 (04) 1446-1460
• 65 Ellsworth PC, Scherer KR. Appraisal Processes in Emotion. Oxford University Press; 2003
• 66 Scherer KR, Ellgring H. Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal?. Emotion 2007; 7 (01) 113-130
• 67 D'Mello S, Picard RW, Graesser A. Toward an affect-sensitive AutoTutor. IEEE Intell Syst 2007; 22 (04) 53-61
• 68 Lehman B, D'Mello S, Person N. All alone with your emotions: an analysis of student emotions during effortful problem solving activities. Paper presented at: Workshop on Emotional and Cognitive issues in ITS at the Ninth International Conference on Intelligent Tutoring Systems; 2008
• 69 Giuliani NP, Venkitakrishnan S, Wu YH. Input-related demands: vocoded sentences evoke different pupillometrics and subjective listening effort than sentences in speech-shaped noise. International Journal of Audiology 2022; 1-8
• 70 Snaith RP. The hospital anxiety and depression scale. Health Qual Life Outcomes 2003; 1 (01) 29
• 71 Stern AF. The hospital anxiety and depression scale. Occup Med (Lond) 2014; 64 (05) 393-394
• 72 Rothauser E. IEEE recommended practice for speech quality measurements. IEEE Trans Audio Electroacoust 1969; 17: 225-246
• 73 Littlewort G, Whitehill J, Wu T. et al. The computer expression recognition toolbox (CERT). Paper presented at: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG); 2011
• 74 Bartlett MS, Littlewort GC, Frank MG, Lee K. Automatic decoding of facial movements reveals deceptive pain expressions. Curr Biol 2014; 24 (07) 738-743
• 75 Ekman P, Oster H. Facial expressions of emotion. Annu Rev Psychol 1979; 30 (01) 527-554
• 76 Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I. The extended Cohn-Kanade dataset (CK + ): a complete dataset for action unit and emotion-specified expression. Paper presented at: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops; 2010
• 77 Gross R, Matthews I, Cohn J, Kanade T, Baker S. Multi-PIE. Proc Int Conf Autom Face Gesture Recognit 2010; 28 (05) 807-813
• 78 Haines N, Southward MW, Cheavens JS, Beauchaine T, Ahn WY. Using computer-vision and machine learning to automate facial coding of positive and negative affect intensity. PLoS One 2019; 14 (02) e0211735
• 79 Krumhuber EG, Küster D, Namba S, Shah D, Calvo MG. Emotion recognition from posed and spontaneous dynamic expressions: human observers versus machine analysis. Emotion 2021; 21 (02) 447-451
• 80 Calvo MG, Fernández-Martín A, Recio G, Lundqvist D. Human observers and automated assessment of dynamic emotional facial expressions: KDEF-DYN database validation. Front Psychol 2018; 9: 2052
• 81 Pekrun R, Vogl E, Muis KR, Sinatra GM. Measuring emotions during epistemic activities: the Epistemically-Related Emotion Scales. Cogn Emotion 2017; 31 (06) 1268-1276
• 82 Nilsson M, Soli SD, Sullivan JA. Development of the Hearing in Noise Test for the measurement of speech reception thresholds in quiet and in noise. J Acoust Soc Am 1994; 95 (02) 1085-1099
• 83 Winn MB, Wendt D, Koelewijn T, Kuchinsky SE. Best practices and advice for using pupillometry to measure listening effort: an introduction for those who want to get started. Trends Hear 2018; 22: 2331216518800869
• 84 Piquado T, Isaacowitz D, Wingfield A. Pupillometry as a measure of cognitive effort in younger and older adults. Psychophysiology 2010; 47 (03) 560-569
• 85 Makowski D, Ben-Shachar MS, Patil I, Lüdecke D. Methods and algorithms for correlation analysis in R. J Open Source Softw 2020; 5 (51) 2306
• 86 Studebaker GAA. A “rationalized” arcsine transform. J Speech Hear Res 1985; 28 (03) 455-462
• 87 Koelewijn T, Shinn-Cunningham BG, Zekveld AA, Kramer SE. The pupil response is sensitive to divided attention during speech processing. Hear Res 2014; 312: 114-120
• 88 Mackersie CL, Calderon-Moultrie N. Autonomic nervous system reactivity during speech repetition tasks: heart rate variability and skin conductance. Ear Hear 2016; 37 (Suppl. 01) 118S-125S
• 89 Baumeister RF, Vohs KD, DeWall CN, Zhang L. How emotion shapes behavior: feedback, anticipation, and reflection, rather than direct causation. Pers Soc Psychol Rev 2007; 11 (02) 167-203
• 90 Bandyopadhyay D, Pammi VC, Srinivasan N. Role of affect in decision making. In: Pammi VSC, Srinivasan N. eds. Progress in Brain Research. Vol 202. Elsevier; 2013: 37-53
• 91 Koelewijn T, Zekveld AA, Lunner T, Kramer SE. The effect of reward on listening effort as reflected by the pupil dilation response. Hear Res 2018; 367: 106-112
• 92 Carolan PJ, Heinrich A, Munro KJ, Millman RE. Financial reward has differential effects on behavioural and self-report measures of listening effort. Int J Audiol 2021; 60 (11) 900-910
• 93 Pielage H, Zekveld AA, Saunders GH, Versfeld NJ, Lunner T, Kramer SE. The presence of another individual influences listening effort, but not performance. Ear Hear 2021; 42 (06) 1577-1589
• 94 Plain B, Pielage H, Richter M. et al. Social observation increases the cardiovascular response of hearing-impaired listeners during a speech reception task. Hear Res 2021; 410: 108334
• 95 Kensinger EA. Remembering emotional experiences: the contribution of valence and arousal. Rev Neurosci 2004; 15 (04) 241-251
• 96 Kensinger EA. Remembering the details: effects of emotion. Emot Rev 2009; 1 (02) 99-113
• 97 Hamann S. Cognitive and neural mechanisms of emotional memory. Trends Cogn Sci 2001; 5 (09) 394-400
• 98 Nachtegaal J, Smit JH, Smits C. et al. The association between hearing status and psychosocial health before the age of 70 years: results from an internet-based national survey on hearing. Ear Hear 2009; 30 (03) 302-312
• 99 McDaniel B, D'Mello S, King B, Chipman P, Tapp K, Graesser A. Facial features for affective state detection in learning environments. Paper presented at: Proceedings of the Annual Meeting of the Cognitive Science Society; 2007
• 100 Ekman P. Universal facial expressions in emotion. Studia Psychologica 1973; 15 (02) 140-147
• 101 Friesen WV. Cultural Differences in Facial Expressions in a Social Situation: An Experimental Test of the Concept of Display Rules. San Francisco: University of California; 1972
• 102 Ekman P, Friesen WV. Constants across cultures in the face and emotion. J Pers Soc Psychol 1971; 17 (02) 124-129