Download PDF
Semin Hear 2023; 44(02): 095-105
DOI: 10.1055/s-0043-1767669
Review Article

Combining Multiple Psychophysiological Measures of Listening Effort: Challenges and Recommendations

Michael Richter
1   School of Psychology, Liverpool John Moores University, Liverpool, United Kingdom
,
Tanveer Buhiyan
2   R&D, Demant A/S, Kongebakken, Denmark
,
Lars Bramsløw
3   Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
,
Hamish Innes-Brown
3   Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
8   Hearing Systems Section, Department of Health Technology, Technical University of Denmark, Kongens Lyngby, Denmark
,
Lorenz Fiedler
3   Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
,
Lauren V. Hadley
4   Hearing Sciences, School of Medicine, University of Nottingham, Nottingham, United Kingdom
,
Graham Naylor
4   Hearing Sciences, School of Medicine, University of Nottingham, Nottingham, United Kingdom
,
Gabrielle H. Saunders
5   Manchester Centre for Audiology and Deafness, University of Manchester, Manchester, United Kingdom
,
Dorothea Wendt
3   Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
6   Department of Health Technology, Technical University of Denmark, Lyngby, Denmark
,
William M. Whitmer
4   Hearing Sciences, School of Medicine, University of Nottingham, Nottingham, United Kingdom
,
Adriana A. Zekveld
7   Section of Ear and Hearing, Department of Otolaryngology – Head and Neck Surgery, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
,
Sophia E. Kramer
7   Section of Ear and Hearing, Department of Otolaryngology – Head and Neck Surgery, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
› Author Affiliations Funding This project has received funding from the European Union's Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement no. 765329; Medical Research Council (grant number MR/S003576/1). Author GS was supported by the NIHR Manchester Biomedical Research Centre.
› Further Information
Also available at
   Permissions and Reprints

Abstract

About one-third of all recently published studies on listening effort have used at least one physiological measure, providing evidence of the popularity of such measures in listening effort research. However, the specific measures employed, as well as the rationales used to justify their inclusion, vary greatly between studies, leading to a literature that is fragmented and difficult to integrate. A unified approach that assesses multiple psychophysiological measures justified by a single rationale would be preferable because it would advance our understanding of listening effort. However, such an approach comes with a number of challenges, including the need to develop a clear definition of listening effort that links to specific physiological measures, customized equipment that enables the simultaneous assessment of multiple measures, awareness of problems caused by the different timescales on which the measures operate, and statistical approaches that minimize the risk of type-I error inflation. This article discusses in detail the various obstacles for combining multiple physiological measures in listening effort research and provides recommendations on how to overcome them.

Keywords

listening effort - psychophysiological measure - listening demand


Publication History

Article published online:
28 March 2023

© 2023. Thieme. All rights reserved.

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

 

  • References

  • 1 Paul BT, Chen J, Le T, Lin V, Dimitrijevic A. Cortical alpha oscillations in cochlear implant users reflect subjective listening effort during speech-in-noise perception. PLoS One 2021; 16 (07) e0254162
    CrossrefPubMedGoogle Scholar
  • 2 Wisniewski MG, Zakrzewski AC, Bell DR, Wheeler M. EEG power spectral dynamics associated with listening in adverse conditions. Psychophysiology 2021; 58 (09) e13877
    CrossrefPubMedGoogle Scholar
  • 3 Hunter CR. Tracking cognitive spare capacity during speech perception with EEG/ERP: effects of cognitive load and sentence predictability. Ear Hear 2020; 41 (05) 1144-1157
    CrossrefPubMedGoogle Scholar
  • 4 Silcox JW, Payne BR. The costs (and benefits) of effortful listening on context processing: a simultaneous electrophysiology, pupillometry, and behavioral study. Cortex 2021; 142: 296-316
    CrossrefPubMedGoogle Scholar
  • 5 Rovetti J, Goy H, Pichora-Fuller MK, Russo FA. Functional near-infrared spectroscopy as a measure of listening effort in older adults who use hearing aids. Trends Hear 2019; 23: 23 31216519886722
    PubMedGoogle Scholar
  • 6 White BE, Langdon C. The cortical organization of listening effort: new insight from functional near-infrared spectroscopy. Neuroimage 2021; 240: 118324
    CrossrefPubMedGoogle Scholar
  • 7 Alhanbali S, Dawes P, Millman RE, Munro KJ. Measures of listening effort are multidimensional. Ear Hear 2019; 40 (05) 1084-1097
    CrossrefPubMedGoogle Scholar
  • 8 Giuliani NP, Brown CJ, Wu YH. Comparisons of the sensitivity and reliability of multiple measures of listening effort. Ear Hear 2021; 42 (02) 465-474
    CrossrefPubMedGoogle Scholar
  • 9 Koelewijn T, Zekveld AA, Lunner T, Kramer SE. The effect of monetary reward on listening effort and sentence recognition. Hear Res 2021; 406: 108255
    CrossrefPubMedGoogle Scholar
  • 10 Zekveld AA, van Scheepen JAM, Versfeld NJ, Veerman ECI, Kramer SE. Please try harder! The influence of hearing status and evaluative feedback during listening on the pupil dilation response, saliva-cortisol and saliva alpha-amylase levels. Hear Res 2019; 381: 107768
    CrossrefPubMedGoogle Scholar
  • 11 Plain B, Pielage H, Richter M. et al. Social observation increases the cardiovascular response of hearing-impaired listeners during a speech reception task. Hear Res 2021; 410: 108334
    CrossrefPubMedGoogle Scholar
  • 12 Slade K, Kramer SE, Fairclough S, Richter M. Effortful listening: sympathetic activity varies as a function of listening demand but parasympathetic activity does not. Hear Res 2021; 410: 108348
    CrossrefPubMedGoogle Scholar
  • 13 Plain B, Richter M, Zekveld AA, Lunner T, Bhuiyan T, Kramer SE. Investigating the influences of task demand and reward on cardiac pre-ejection period reactivity during a speech-in-noise task. Ear Hear 2021; 42 (03) 718-731
    CrossrefPubMedGoogle Scholar
  • 14 Strand JF, Ray L, Dillman-Hasso NH, Villanueva J, Brown VA. Understanding speech amid the jingle and jangle: recommendations for improving measurement practices in listening effort research. Audit Percept Cogn 2020; 3 (04) 169-188
    CrossrefPubMedGoogle Scholar
  • 15 Książek P, Zekveld AA, Wendt D, Fiedler L, Lunner T, Kramer SE. Effect of speech-to-noise ratio and luminance on a range of current and potential pupil response measures to assess listening effort. Trends Hear 2021; 25: 23 312165211009351
    PubMedGoogle Scholar
  • 16 Pielage H, Zekveld AA, Saunders GH, Versfeld NJ, Lunner T, Kramer SE. The presence of another individual influences listening effort, but not performance. Ear Hear 2021; 42 (06) 1577-1589
    CrossrefPubMedGoogle Scholar
  • 17 Seifi Ala T, Graversen C, Wendt D, Alickovic E, Whitmer WM, Lunner T. An exploratory study of EEG alpha oscillation and pupil dilation in hearing-aid users during effortful listening to continuous speech. PLoS One 2020; 15 (07) e0235782
    CrossrefPubMedGoogle Scholar
  • 18 Fiedler L, Seifi Ala T, Graversen C, Alickovic E, Lunner T, Wendt D. Hearing aid noise reduction lowers the sustained listening effort during continuous speech in noise - a combined pupillometry and EEG study. Ear Hear 2021; 42 (06) 1590-1601
    CrossrefPubMedGoogle Scholar
  • 19 Shields C, Willis H, Nichani J, Sladen M, Kluk-de Kort K. Listening effort: WHAT is it, HOW is it measured and WHY is it important?. Cochlear Implants Int 2022; 23 (02) 114-117
    CrossrefPubMedGoogle Scholar
  • 20 McGarrigle R, Munro KJ, Dawes P. et al. Listening effort and fatigue: what exactly are we measuring? A British Society of Audiology Cognition in Hearing Special Interest Group ‘white paper’. Int J Audiol 2014; 53 (07) 433-440
    CrossrefPubMedGoogle Scholar
  • 21 Francis AL, Love J. Listening effort: Are we measuring cognition or affect, or both?. Wiley Interdiscip Rev Cogn Sci 2020; 11 (01) e1514
    CrossrefPubMedGoogle Scholar
  • 22 Podsakoff PM, MacKenzie SB, Podsakoff NP. Recommendations for creating better concept definitions in the organizational, behavioral, and social sciences. Organ Res Methods 2016; 19 (02) 159-203
    CrossrefPubMedGoogle Scholar
  • 23 Goertz G. Social Science Concepts: A User's Guide. Princeton University Press; 2006
    CrossrefGoogle Scholar
  • 24 DiRenzo GJ. Concepts, Theory, and Explanation in the Behavioral Sciences. Random House; 1966
    Google Scholar
  • 25 Krueger M, Schulte M, Zokoll MA. et al. Relation between listening effort and speech intelligibility in noise. Am J Audiol 2017; 26 (3S): 378-392
    CrossrefPubMedGoogle Scholar
  • 26 Wild CJ, Yusuf A, Wilson DE, Peelle JE, Davis MH, Johnsrude IS. Effortful listening: the processing of degraded speech depends critically on attention. J Neurosci 2012; 32 (40) 14010-14021
    CrossrefPubMedGoogle Scholar
  • 27 Piquado T, Isaacowitz D, Wingfield A. Pupillometry as a measure of cognitive effort in younger and older adults. Psychophysiology 2010; 47 (03) 560-569
    Google Scholar
  • 28 Obleser J, Wöstmann M, Hellbernd N, Wilsch A, Maess B. Adverse listening conditions and memory load drive a common α oscillatory network. J Neurosci 2012; 32 (36) 12376-12383
    CrossrefPubMedGoogle Scholar
  • 29 Mackersie CL, Calderon-Moultrie N. Autonomic nervous system reactivity during speech repetition tasks: heart rate variability and skin conductance. Ear Hear 2016; 37 (Suppl. 01) 118S-125S
    CrossrefPubMedGoogle Scholar
  • 30 Ohlenforst B, Zekveld AA, Lunner T. et al. Impact of stimulus-related factors and hearing impairment on listening effort as indicated by pupil dilation. Hear Res 2017; 351: 68-79
    CrossrefPubMedGoogle Scholar
  • 31 Wendt D, Koelewijn T, Książek P, Kramer SE, Lunner T. Toward a more comprehensive understanding of the impact of masker type and signal-to-noise ratio on the pupillary response while performing a speech-in-noise test. Hear Res 2018; 369: 67-78
    CrossrefPubMedGoogle Scholar
  • 32 Winn MB, Teece KH. Listening effort is not the same as speech intelligibility score. Trends Hear 2021; 25: 23 312165211027688
    PubMedGoogle Scholar
  • 33 Richter M, Slade K. Interpretation of physiological indicators of motivation: caveats and recommendations. Int J Psychophysiol 2017; 119: 4-10
    CrossrefPubMedGoogle Scholar
  • 34 Picou EM, Ricketts TA, Hornsby BWY. Visual cues and listening effort: individual variability. J Speech Lang Hear Res 2011; 54 (05) 1416-1430
    CrossrefPubMedGoogle Scholar
  • 35 Pichora-Fuller MK, Kramer SE, Eckert MA. et al. Hearing impairment and cognitive energy: the framework for understanding effortful listening (FUEL). Ear Hear 2016; 37 (Suppl. 01) 5S-27S
    Google Scholar
  • 36 Richter M. The moderating effect of success importance on the relationship between listening demand and listening effort. Ear Hear 2016; 37 (Suppl. 01) 111S-117S
    CrossrefPubMedGoogle Scholar
  • 37 Meijer JH, Elbertse E, Boesveldt S, Berendse HW, Verdaasdonk RM. Using the Initial Systolic Time Interval to assess cardiac autonomic nervous function in Parkinson's disease. J Electr Bioimpedance 2011; 2 (01) 98-101
    CrossrefPubMedGoogle Scholar
  • 38 Sherwood A, McFetridge J, Hutcheson JS. Ambulatory impedance cardiography: a feasibility study. J Appl Physiol (1985) 1998; 85 (06) 2365-2369
    CrossrefPubMedGoogle Scholar
  • 39 Uchitel J, Vidal-Rosas EE, Cooper RJ, Zhao H. Wearable, integrated EEG-fNIRS technologies: a review. Sensors (Basel) 2021; 21 (18) 6106
    CrossrefPubMedGoogle Scholar
  • 40 von Luhmann A, Muller KR. Why build an integrated EEG-NIRS? About the advantages of hybrid bio-acquisition hardware. Annu Int Conf IEEE Eng Med Biol Soc 2017; 2017: 4475-4478
    PubMedGoogle Scholar
  • 41 Cieslak M, Ryan WS, Macy A. et al. Simultaneous acquisition of functional magnetic resonance images and impedance cardiography. Psychophysiology 2015; 52 (04) 481-488
    CrossrefPubMedGoogle Scholar
  • 42 Oster J, Clifford GD. Acquisition of electrocardiogram signals during magnetic resonance imaging. Physiol Meas 2017; 38 (07) R119-R142
    CrossrefPubMedGoogle Scholar
  • 43 Khan MJ, Ghafoor U, Hong KS. Early detection of hemodynamic responses using EEG: a hybrid EEG-fNIRS study. Front Hum Neurosci 2018; 12: 479
    CrossrefPubMedGoogle Scholar
  • 44 Al-Shargie F, Kiguchi M, Badruddin N, Dass SC, Hani AF, Tang TB. Mental stress assessment using simultaneous measurement of EEG and fNIRS. Biomed Opt Express 2016; 7 (10) 3882-3898
    CrossrefPubMedGoogle Scholar
  • 45 Dalton KM, Davidson RJ. The concurrent recording of electroencephalography and impedance cardiography: effects on EEG. Psychophysiology 1997; 34 (04) 488-493
    CrossrefPubMedGoogle Scholar
  • 46 Keidser G, Naylor G, Brungart DS. et al. The quest for ecological validity in hearing science: what it is, why it matters, and how to advance it. Ear Hear 2020; 41 Suppl 1 (Suppl. 01) 5S-19S
    CrossrefPubMedGoogle Scholar
  • 47 van der Mee DJ, Gevonden MJ, Westerink JHDM, de Geus EJC. Validity of electrodermal activity-based measures of sympathetic nervous system activity from a wrist-worn device. Int J Psychophysiol 2021; 168: 52-64
    CrossrefPubMedGoogle Scholar
  • 48 Calamia M. Practical considerations for evaluating reliability in ambulatory assessment studies. Psychol Assess 2019; 31 (03) 285-291
    CrossrefPubMedGoogle Scholar
  • 49 Winn MB, Wendt D, Koelewijn T, Kuchinsky SE. Best practices and advice for using pupillometry to measure listening effort: an introduction for those who want to get started. Trends Hear 2018; 22: 23 31216518800869
    PubMedGoogle Scholar
  • 50 Kyong JS, Kwak C, Han W, Suh MW, Kim J. Effect of speech degradation and listening effort in reverberating and noisy environments given N400 responses. J Audiol Otol 2020; 24 (03) 119-126
    CrossrefPubMedGoogle Scholar
  • 51 Boudoulas H. Systolic time intervals. Eur Heart J 1990; 11 (Suppl I): 93-104
    CrossrefPubMedGoogle Scholar
  • 52 Francis AL, Bent T, Schumaker J, Love J, Silbert N. Listener characteristics differentially affect self-reported and physiological measures of effort associated with two challenging listening conditions. Atten Percept Psychophys 2021; 83 (04) 1818-1841
    CrossrefPubMedGoogle Scholar
  • 53 Miles K, McMahon C, Boisvert I. et al. Objective assessment of listening effort: coregistration of pupillometry and EEG. Trends Hear 2017; 21: 23 31216517706396
    PubMedGoogle Scholar
  • 54 Wisniewski MG, Thompson ER, Iyer N, Estepp JR, Goder-Reiser MN, Sullivan SC. Frontal midline θ power as an index of listening effort. Neuroreport 2015; 26 (02) 94-99
    CrossrefPubMedGoogle Scholar
  • 55 Moyer JT, Gnatkovsky V, Ono T. et al. Standards for data acquisition and software-based analysis of in vivo electroencephalography recordings from animals. A TASK1-WG5 report of the AES/ILAE Translational Task Force of the ILAE. Epilepsia 2017; 58, Suppl 4 (Suppl. 04) 53-67
    CrossrefPubMedGoogle Scholar
  • 56 Berntson GG, Bigger Jr JT, Eckberg DL. et al. Heart rate variability: origins, methods, and interpretive caveats. Psychophysiology 1997; 34 (06) 623-648
    CrossrefPubMedGoogle Scholar
  • 57 Mathôt S, Fabius J, Van Heusden E, Van der Stigchel S. Safe and sensible preprocessing and baseline correction of pupil-size data. Behav Res Methods 2018; 50 (01) 94-106
    CrossrefPubMedGoogle Scholar
  • 58 Ayasse ND, Wingfield A. Anticipatory baseline pupil diameter is sensitive to differences in hearing thresholds. Front Psychol 2020; 10: 2947
    CrossrefPubMedGoogle Scholar
  • 59 Kuipers M, Richter M, Scheepers D, Immink MA, Sjak-Shie E, van Steenbergen H. How effortful is cognitive control? Insights from a novel method measuring single-trial evoked beta-adrenergic cardiac reactivity. Int J Psychophysiol 2017; 119: 87-92
    CrossrefPubMedGoogle Scholar
  • 60 Koelewijn T, de Kluiver H, Shinn-Cunningham BG, Zekveld AA, Kramer SE. The pupil response reveals increased listening effort when it is difficult to focus attention. Hear Res 2015; 323: 81-90
    CrossrefPubMedGoogle Scholar
  • 61 Mokrane A, Nadeau R. Dynamics of heart rate response to sympathetic nerve stimulation. Am J Physiol 1998; 275 (03) H995-H1001
    PubMedGoogle Scholar
  • 62 Warner HR, Cox A. A mathematical model of heart rate control by sympathetic and vagus efferent information. J Appl Physiol 1962; 17: 349-355
    CrossrefPubMedGoogle Scholar
  • 63 Mathôt S. Pupillometry: psychology, physiology, and function. J Cogn 2018; 1 (01) 16
    CrossrefPubMedGoogle Scholar
  • 64 Picton TW, Hillyard SA, Krausz HI, Galambos R. Human auditory evoked potentials. I. Evaluation of components. Electroencephalogr Clin Neurophysiol 1974; 36 (02) 179-190
    CrossrefPubMedGoogle Scholar
  • 65 Ohlenforst B, Wendt D, Kramer SE, Naylor G, Zekveld AA, Lunner T. Impact of SNR, masker type and noise reduction processing on sentence recognition performance and listening effort as indicated by the pupil dilation response. Hear Res 2018; 365: 90-99
    CrossrefPubMedGoogle Scholar
  • 66 Kuchinsky SE, Ahlstrom JB, Vaden Jr KI. et al. Pupil size varies with word listening and response selection difficulty in older adults with hearing loss. Psychophysiology 2013; 50 (01) 23-34
    CrossrefPubMedGoogle Scholar
  • 67 Bird KD, Hadzi-Pavlovic D. Controlling the maximum familywise Type I error rate in analyses of multivariate experiments. Psychol Methods 2014; 19 (02) 265-280
    CrossrefPubMedGoogle Scholar
  • 68 Schroeder MA. Diagnosing and dealing with multicollinearity. West J Nurs Res 1990; 12 (02) 175-184 , discussion 184–187
    CrossrefPubMedGoogle Scholar
  • 69 Slinker BK, Glantz SA. Multiple regression for physiological data analysis: the problem of multicollinearity. Am J Physiol 1985; 249 (1, Pt 2): R1-R12
    PubMedGoogle Scholar
  • 70 Behnke M, Kaczmarek LD. Successful performance and cardiovascular markers of challenge and threat: a meta-analysis. Int J Psychophysiol 2018; 130: 73-79
    CrossrefPubMedGoogle Scholar
  • 71 Miyake S. Multivariate workload evaluation combining physiological and subjective measures. Int J Psychophysiol 2001; 40 (03) 233-238
    CrossrefPubMedGoogle Scholar
  • 72 Lowenstein O, Loewenfeld IE. Role of sympathetic and parasympathetic systems in reflex dilation of the pupil; pupillographic studies. Arch Neurol Psychiatry 1950; 64 (03) 313-340
    CrossrefPubMedGoogle Scholar
  • 73 Winer BJ, Brown DR, Michels KM. Statistical Principles in Experimental Design. 3rd ed. McGraw-Hill; 1991
    Google Scholar
  • 74 Rosenthal R, Rosnow RL. Contrast Analysis: Focused Comparisons in the Analysis of Variance. Cambridge University Press; 1985
    Google Scholar
  • 75 Richter M. Residual tests in the analysis of planned contrasts: problems and solutions. Psychol Methods 2016; 21 (01) 112-120
    CrossrefPubMedGoogle Scholar
  • 76 Moisl H. Variable scaling in cluster analysis of linguistic data. Corpus Linguist Linguist Theor 2010; 6 (01) DOI: 10.1515/cllt.2010.004.
    CrossrefPubMedGoogle Scholar
  • 77 Zhang M, Alamatsaz N, Ihlefeld A. Hemodynamic responses link individual differences in informational masking to the vicinity of superior temporal gyrus. Front Neurosci 2021; 15: 675326
    CrossrefPubMedGoogle Scholar