Combining Multiple Psychophysiological Measures of Listening Effort: Challenges and Recommendations

Abstract

About one-third of all recently published studies on listening effort have used at least one physiological measure, providing evidence of the popularity of such measures in listening effort research. However, the specific measures employed, as well as the rationales used to justify their inclusion, vary greatly between studies, leading to a literature that is fragmented and difficult to integrate. A unified approach that assesses multiple psychophysiological measures justified by a single rationale would be preferable because it would advance our understanding of listening effort. However, such an approach comes with a number of challenges, including the need to develop a clear definition of listening effort that links to specific physiological measures, customized equipment that enables the simultaneous assessment of multiple measures, awareness of problems caused by the different timescales on which the measures operate, and statistical approaches that minimize the risk of type-I error inflation. This article discusses in detail the various obstacles for combining multiple physiological measures in listening effort research and provides recommendations on how to overcome them.

Keywords

listening effort - psychophysiological measure - listening demand

Publikationsverlauf

Artikel online veröffentlicht:
28. März 2023

• References

• 1 Paul BT, Chen J, Le T, Lin V, Dimitrijevic A. Cortical alpha oscillations in cochlear implant users reflect subjective listening effort during speech-in-noise perception. PLoS One 2021; 16 (07) e0254162
• 2 Wisniewski MG, Zakrzewski AC, Bell DR, Wheeler M. EEG power spectral dynamics associated with listening in adverse conditions. Psychophysiology 2021; 58 (09) e13877
• 3 Hunter CR. Tracking cognitive spare capacity during speech perception with EEG/ERP: effects of cognitive load and sentence predictability. Ear Hear 2020; 41 (05) 1144-1157
• 4 Silcox JW, Payne BR. The costs (and benefits) of effortful listening on context processing: a simultaneous electrophysiology, pupillometry, and behavioral study. Cortex 2021; 142: 296-316
• 5 Rovetti J, Goy H, Pichora-Fuller MK, Russo FA. Functional near-infrared spectroscopy as a measure of listening effort in older adults who use hearing aids. Trends Hear 2019; 23: 23 31216519886722
• 6 White BE, Langdon C. The cortical organization of listening effort: new insight from functional near-infrared spectroscopy. Neuroimage 2021; 240: 118324
• 7 Alhanbali S, Dawes P, Millman RE, Munro KJ. Measures of listening effort are multidimensional. Ear Hear 2019; 40 (05) 1084-1097
• 8 Giuliani NP, Brown CJ, Wu YH. Comparisons of the sensitivity and reliability of multiple measures of listening effort. Ear Hear 2021; 42 (02) 465-474
• 9 Koelewijn T, Zekveld AA, Lunner T, Kramer SE. The effect of monetary reward on listening effort and sentence recognition. Hear Res 2021; 406: 108255
• 10 Zekveld AA, van Scheepen JAM, Versfeld NJ, Veerman ECI, Kramer SE. Please try harder! The influence of hearing status and evaluative feedback during listening on the pupil dilation response, saliva-cortisol and saliva alpha-amylase levels. Hear Res 2019; 381: 107768
• 11 Plain B, Pielage H, Richter M. et al. Social observation increases the cardiovascular response of hearing-impaired listeners during a speech reception task. Hear Res 2021; 410: 108334
• 12 Slade K, Kramer SE, Fairclough S, Richter M. Effortful listening: sympathetic activity varies as a function of listening demand but parasympathetic activity does not. Hear Res 2021; 410: 108348
• 13 Plain B, Richter M, Zekveld AA, Lunner T, Bhuiyan T, Kramer SE. Investigating the influences of task demand and reward on cardiac pre-ejection period reactivity during a speech-in-noise task. Ear Hear 2021; 42 (03) 718-731
• 14 Strand JF, Ray L, Dillman-Hasso NH, Villanueva J, Brown VA. Understanding speech amid the jingle and jangle: recommendations for improving measurement practices in listening effort research. Audit Percept Cogn 2020; 3 (04) 169-188
• 15 Książek P, Zekveld AA, Wendt D, Fiedler L, Lunner T, Kramer SE. Effect of speech-to-noise ratio and luminance on a range of current and potential pupil response measures to assess listening effort. Trends Hear 2021; 25: 23 312165211009351
• 16 Pielage H, Zekveld AA, Saunders GH, Versfeld NJ, Lunner T, Kramer SE. The presence of another individual influences listening effort, but not performance. Ear Hear 2021; 42 (06) 1577-1589
• 17 Seifi Ala T, Graversen C, Wendt D, Alickovic E, Whitmer WM, Lunner T. An exploratory study of EEG alpha oscillation and pupil dilation in hearing-aid users during effortful listening to continuous speech. PLoS One 2020; 15 (07) e0235782
• 18 Fiedler L, Seifi Ala T, Graversen C, Alickovic E, Lunner T, Wendt D. Hearing aid noise reduction lowers the sustained listening effort during continuous speech in noise - a combined pupillometry and EEG study. Ear Hear 2021; 42 (06) 1590-1601
• 19 Shields C, Willis H, Nichani J, Sladen M, Kluk-de Kort K. Listening effort: WHAT is it, HOW is it measured and WHY is it important?. Cochlear Implants Int 2022; 23 (02) 114-117
• 20 McGarrigle R, Munro KJ, Dawes P. et al. Listening effort and fatigue: what exactly are we measuring? A British Society of Audiology Cognition in Hearing Special Interest Group ‘white paper’. Int J Audiol 2014; 53 (07) 433-440
• 21 Francis AL, Love J. Listening effort: Are we measuring cognition or affect, or both?. Wiley Interdiscip Rev Cogn Sci 2020; 11 (01) e1514
• 22 Podsakoff PM, MacKenzie SB, Podsakoff NP. Recommendations for creating better concept definitions in the organizational, behavioral, and social sciences. Organ Res Methods 2016; 19 (02) 159-203
• 23 Goertz G. Social Science Concepts: A User's Guide. Princeton University Press; 2006
• 24 DiRenzo GJ. Concepts, Theory, and Explanation in the Behavioral Sciences. Random House; 1966
• 25 Krueger M, Schulte M, Zokoll MA. et al. Relation between listening effort and speech intelligibility in noise. Am J Audiol 2017; 26 (3S): 378-392
• 26 Wild CJ, Yusuf A, Wilson DE, Peelle JE, Davis MH, Johnsrude IS. Effortful listening: the processing of degraded speech depends critically on attention. J Neurosci 2012; 32 (40) 14010-14021
• 27 Piquado T, Isaacowitz D, Wingfield A. Pupillometry as a measure of cognitive effort in younger and older adults. Psychophysiology 2010; 47 (03) 560-569
• 28 Obleser J, Wöstmann M, Hellbernd N, Wilsch A, Maess B. Adverse listening conditions and memory load drive a common α oscillatory network. J Neurosci 2012; 32 (36) 12376-12383
• 29 Mackersie CL, Calderon-Moultrie N. Autonomic nervous system reactivity during speech repetition tasks: heart rate variability and skin conductance. Ear Hear 2016; 37 (Suppl. 01) 118S-125S
• 30 Ohlenforst B, Zekveld AA, Lunner T. et al. Impact of stimulus-related factors and hearing impairment on listening effort as indicated by pupil dilation. Hear Res 2017; 351: 68-79
• 31 Wendt D, Koelewijn T, Książek P, Kramer SE, Lunner T. Toward a more comprehensive understanding of the impact of masker type and signal-to-noise ratio on the pupillary response while performing a speech-in-noise test. Hear Res 2018; 369: 67-78
• 32 Winn MB, Teece KH. Listening effort is not the same as speech intelligibility score. Trends Hear 2021; 25: 23 312165211027688
• 33 Richter M, Slade K. Interpretation of physiological indicators of motivation: caveats and recommendations. Int J Psychophysiol 2017; 119: 4-10
• 34 Picou EM, Ricketts TA, Hornsby BWY. Visual cues and listening effort: individual variability. J Speech Lang Hear Res 2011; 54 (05) 1416-1430
• 35 Pichora-Fuller MK, Kramer SE, Eckert MA. et al. Hearing impairment and cognitive energy: the framework for understanding effortful listening (FUEL). Ear Hear 2016; 37 (Suppl. 01) 5S-27S
• 36 Richter M. The moderating effect of success importance on the relationship between listening demand and listening effort. Ear Hear 2016; 37 (Suppl. 01) 111S-117S
• 37 Meijer JH, Elbertse E, Boesveldt S, Berendse HW, Verdaasdonk RM. Using the Initial Systolic Time Interval to assess cardiac autonomic nervous function in Parkinson's disease. J Electr Bioimpedance 2011; 2 (01) 98-101
• 38 Sherwood A, McFetridge J, Hutcheson JS. Ambulatory impedance cardiography: a feasibility study. J Appl Physiol (1985) 1998; 85 (06) 2365-2369
• 39 Uchitel J, Vidal-Rosas EE, Cooper RJ, Zhao H. Wearable, integrated EEG-fNIRS technologies: a review. Sensors (Basel) 2021; 21 (18) 6106
• 40 von Luhmann A, Muller KR. Why build an integrated EEG-NIRS? About the advantages of hybrid bio-acquisition hardware. Annu Int Conf IEEE Eng Med Biol Soc 2017; 2017: 4475-4478
• 41 Cieslak M, Ryan WS, Macy A. et al. Simultaneous acquisition of functional magnetic resonance images and impedance cardiography. Psychophysiology 2015; 52 (04) 481-488
• 42 Oster J, Clifford GD. Acquisition of electrocardiogram signals during magnetic resonance imaging. Physiol Meas 2017; 38 (07) R119-R142
• 43 Khan MJ, Ghafoor U, Hong KS. Early detection of hemodynamic responses using EEG: a hybrid EEG-fNIRS study. Front Hum Neurosci 2018; 12: 479
• 44 Al-Shargie F, Kiguchi M, Badruddin N, Dass SC, Hani AF, Tang TB. Mental stress assessment using simultaneous measurement of EEG and fNIRS. Biomed Opt Express 2016; 7 (10) 3882-3898
• 45 Dalton KM, Davidson RJ. The concurrent recording of electroencephalography and impedance cardiography: effects on EEG. Psychophysiology 1997; 34 (04) 488-493
• 46 Keidser G, Naylor G, Brungart DS. et al. The quest for ecological validity in hearing science: what it is, why it matters, and how to advance it. Ear Hear 2020; 41 Suppl 1 (Suppl. 01) 5S-19S
• 47 van der Mee DJ, Gevonden MJ, Westerink JHDM, de Geus EJC. Validity of electrodermal activity-based measures of sympathetic nervous system activity from a wrist-worn device. Int J Psychophysiol 2021; 168: 52-64
• 48 Calamia M. Practical considerations for evaluating reliability in ambulatory assessment studies. Psychol Assess 2019; 31 (03) 285-291
• 49 Winn MB, Wendt D, Koelewijn T, Kuchinsky SE. Best practices and advice for using pupillometry to measure listening effort: an introduction for those who want to get started. Trends Hear 2018; 22: 23 31216518800869
• 50 Kyong JS, Kwak C, Han W, Suh MW, Kim J. Effect of speech degradation and listening effort in reverberating and noisy environments given N400 responses. J Audiol Otol 2020; 24 (03) 119-126
• 51 Boudoulas H. Systolic time intervals. Eur Heart J 1990; 11 (Suppl I): 93-104
• 52 Francis AL, Bent T, Schumaker J, Love J, Silbert N. Listener characteristics differentially affect self-reported and physiological measures of effort associated with two challenging listening conditions. Atten Percept Psychophys 2021; 83 (04) 1818-1841
• 53 Miles K, McMahon C, Boisvert I. et al. Objective assessment of listening effort: coregistration of pupillometry and EEG. Trends Hear 2017; 21: 23 31216517706396
• 54 Wisniewski MG, Thompson ER, Iyer N, Estepp JR, Goder-Reiser MN, Sullivan SC. Frontal midline θ power as an index of listening effort. Neuroreport 2015; 26 (02) 94-99
• 55 Moyer JT, Gnatkovsky V, Ono T. et al. Standards for data acquisition and software-based analysis of in vivo electroencephalography recordings from animals. A TASK1-WG5 report of the AES/ILAE Translational Task Force of the ILAE. Epilepsia 2017; 58, Suppl 4 (Suppl. 04) 53-67
• 56 Berntson GG, Bigger Jr JT, Eckberg DL. et al. Heart rate variability: origins, methods, and interpretive caveats. Psychophysiology 1997; 34 (06) 623-648
• 57 Mathôt S, Fabius J, Van Heusden E, Van der Stigchel S. Safe and sensible preprocessing and baseline correction of pupil-size data. Behav Res Methods 2018; 50 (01) 94-106
• 58 Ayasse ND, Wingfield A. Anticipatory baseline pupil diameter is sensitive to differences in hearing thresholds. Front Psychol 2020; 10: 2947
• 59 Kuipers M, Richter M, Scheepers D, Immink MA, Sjak-Shie E, van Steenbergen H. How effortful is cognitive control? Insights from a novel method measuring single-trial evoked beta-adrenergic cardiac reactivity. Int J Psychophysiol 2017; 119: 87-92
• 60 Koelewijn T, de Kluiver H, Shinn-Cunningham BG, Zekveld AA, Kramer SE. The pupil response reveals increased listening effort when it is difficult to focus attention. Hear Res 2015; 323: 81-90
• 61 Mokrane A, Nadeau R. Dynamics of heart rate response to sympathetic nerve stimulation. Am J Physiol 1998; 275 (03) H995-H1001
• 62 Warner HR, Cox A. A mathematical model of heart rate control by sympathetic and vagus efferent information. J Appl Physiol 1962; 17: 349-355
• 63 Mathôt S. Pupillometry: psychology, physiology, and function. J Cogn 2018; 1 (01) 16
• 64 Picton TW, Hillyard SA, Krausz HI, Galambos R. Human auditory evoked potentials. I. Evaluation of components. Electroencephalogr Clin Neurophysiol 1974; 36 (02) 179-190
• 65 Ohlenforst B, Wendt D, Kramer SE, Naylor G, Zekveld AA, Lunner T. Impact of SNR, masker type and noise reduction processing on sentence recognition performance and listening effort as indicated by the pupil dilation response. Hear Res 2018; 365: 90-99
• 66 Kuchinsky SE, Ahlstrom JB, Vaden Jr KI. et al. Pupil size varies with word listening and response selection difficulty in older adults with hearing loss. Psychophysiology 2013; 50 (01) 23-34
• 67 Bird KD, Hadzi-Pavlovic D. Controlling the maximum familywise Type I error rate in analyses of multivariate experiments. Psychol Methods 2014; 19 (02) 265-280
• 68 Schroeder MA. Diagnosing and dealing with multicollinearity. West J Nurs Res 1990; 12 (02) 175-184 , discussion 184–187
• 69 Slinker BK, Glantz SA. Multiple regression for physiological data analysis: the problem of multicollinearity. Am J Physiol 1985; 249 (1, Pt 2): R1-R12
• 70 Behnke M, Kaczmarek LD. Successful performance and cardiovascular markers of challenge and threat: a meta-analysis. Int J Psychophysiol 2018; 130: 73-79
• 71 Miyake S. Multivariate workload evaluation combining physiological and subjective measures. Int J Psychophysiol 2001; 40 (03) 233-238
• 72 Lowenstein O, Loewenfeld IE. Role of sympathetic and parasympathetic systems in reflex dilation of the pupil; pupillographic studies. Arch Neurol Psychiatry 1950; 64 (03) 313-340
• 73 Winer BJ, Brown DR, Michels KM. Statistical Principles in Experimental Design. 3rd ed. McGraw-Hill; 1991
• 74 Rosenthal R, Rosnow RL. Contrast Analysis: Focused Comparisons in the Analysis of Variance. Cambridge University Press; 1985
• 75 Richter M. Residual tests in the analysis of planned contrasts: problems and solutions. Psychol Methods 2016; 21 (01) 112-120
• 76 Moisl H. Variable scaling in cluster analysis of linguistic data. Corpus Linguist Linguist Theor 2010; 6 (01)
• 77 Zhang M, Alamatsaz N, Ihlefeld A. Hemodynamic responses link individual differences in informational masking to the vicinity of superior temporal gyrus. Front Neurosci 2021; 15: 675326