The Temporal Response Function — a New Method for Investigating Neurophysiological Mechanisms of Speech Perception under Ecologically Valid Conditions

The temporal response function is a new method that allows to investigate the brain mechanisms of perception of natural, naturalistic speech stimuli. In contrast to other methods for studying brain activity (e.g., evoked potentials), the temporal response function does not require the presentation of a large number of uniform stimuli to produce a robust brain response - recordings of narrative speech lasting 10 minutes or more can be used in experimental paradigms, increasing their ecological validity. The temporal response function can be used to study brain mechanisms of online processing of different components of natural speech: acoustic (physical properties of the audio signal such as envelope and spectrogram), phonological (individual phonemes and their combinations), lexical (contextual characteristics of individual words) and semantic (semantic meaning of words), as well as the interaction between these components processing mechanisms. The article presents the history of the method, its advantages in comparison with other methods and limitations, mathematical basis, features of natural speech components extraction, and a brief review of the main studies using this method.

1. Alday P.M. M/EEG analysis of naturalistic stories: a review from speech to language processing. Language, cognition and neuroscience, 2019. Vol. 34, no. 4, pp. 457—473. DOI:10.1080/23273798.2018.1546882
2. Di Liberto G.M., Peter V., Kalashnikova M., Goswami U., Burnham D., Lalor E.C. Atypical cortical entrainment to speech in the right hemisphere underpins phonemic deficits in dyslexia. NeuroImage, 2018. Vol. 175, pp. 70—79. DOI:10.1016/j.neuroimage.2018.03.072
3. Broderick M.P., Anderson A.J., Lalor E.C. Semantic Context Enhances the Early Auditory Encoding of Natural Speech. Journal of Neuroscience, 2019. Vol. 39, no. 38, pp. 7564—7575. DOI:10.1523/JNEUROSCI.0584-19.2019
4. Castles A., Rastle K., Nation K. Ending the Reading Wars: Reading Acquisition From Novice to Expert. Psychological Science in the Public Interest, 2018. Vol. 19, no. 1, pp. 5—51. DOI:10.1177/1529100618772271
5. Crosse M.J., Liberto G.M.D., Lalor E.C. Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term Crossmodal Temporal Integration. Journal of Neuroscience, 2016. Vol. 36, no. 38, pp. 9888—9895. DOI:10.1523/JNEUROSCI.1396-16.2016
6. Mirkovic B., Debener S., Jaeger M., De Vos M. Decoding the attended speech stream with multi-channel EEG: implications for online, daily-life applications. Journal of Neural Engineering, 2015. Vol. 12, no. 4, article ID 046007. 9 p. DOI:10.1088/1741-2560/12/4/046007
7. Di Liberto G.M., Hjortkjær J., Mesgarani N. Editorial: Neural Tracking: Closing the Gap Between Neurophysiology and Translational Medicine. Frontiers in Neuroscience, 2022. Vol. 16, article ID 872600. 4 p. DOI:10.3389/fnins.2022.872600
8. Ding N., Simon J. Cortical entrainment to continuous speech: functional roles and interpretations. Frontiers in Human Neuroscience, 2014. Vol. 8, article ID 311. 7 p. DOI:10.3389/fnhum.2014.00311
9. Ding N., Simon J.Z. Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech. Journal of Neuroscience, 2013. Vol. 33, no. 13, pp. 5728—5735. DOI:10.1523/JNEUROSCI.5297-12.2013
10. Broderick M.P., Di Liberto G.M., Anderson A.J., Rofes A., Lalor E.C. Dissociable electrophysiological measures of natural language processing reveal differences in speech comprehension strategy in healthy ageing. Scientific Reports, 2021. Vol. 11, no. 1, article ID 4963. 12 p. DOI:10.1038/s41598-021-84597-9
11. Mikolov T., Sutskever I., Chen K., Corrado G.S., Dean J. Distributed Representations of Words and Phrases and their Compositionality. In Burges C.J., Bottou L., Welling M., Ghahramani Z., Weinberger K.Q. (eds.), Advances in Neural Information Processing Systems: 27th Annual Conference on Neural Information Processing Systems 2013: Held 5-10 December 2013, Lake Tahoe, Nevada, USA. New York: Curran Associates Inc. Proceedings.com, 2013. Vol. 26. 9 p. DOI:10.48550/arXiv.1310.4546
12. Broderick M.P., Anderson A.J., Di Liberto G.M., Crosse M.J., Lalor E.C. Electrophysiological Correlates of Semantic Dissimilarity Reflect the Comprehension of Natural, Narrative Speech. Current Biology, 2018. Vol. 28, no. 5, pp. 803—809. DOI:10.1016/j.cub.2018.01.080
13. Hamilton L.S., Huth A.G. The revolution will not be controlled: natural stimuli in speech neuroscience. Language, Cognition and Neuroscience, 2020. Vol. 35, no. 5, pp. 573—582. DOI:10.1080/23273798.2018.1499946
14. Klimovich-Gray A., Di Liberto G., Amoruso L., Barrena A., Agirre E., Molinaro N. Increased top-down semantic processing in natural speech linked to better reading in dyslexia. NeuroImage, 2023. Vol. 273, article ID 120072. 11 p. DOI:10.1016/j.neuroimage.2023.120072
15. Kalashnikova M., Peter V., Di Liberto G.M., Lalor E.C., Burnham D. Infant-directed speech facilitates seven-month-old infants’ cortical tracking of speech. Scientific Reports, 2018. Vol. 8, article ID 13745. 8 p. DOI:10.1038/s41598-018-32150-6
16. Khalighinejad B., da Silva G.C., Mesgarani N. Dynamic Encoding of Acoustic Features in Neural Responses to Continuous Speech. Journal of Neuroscience, 2017. Vol. 37, no. 8, pp. 2176—2185. DOI:10.1523/JNEUROSCI.2383-16.2017
17. Crosse M.J., Zuk N.J., Di Liberto G.M., Nidiffer A.R., Molholm S., Lalor E.C. Linear Modeling of Neurophysiological Responses to Speech and Other Continuous Stimuli: Methodological Considerations for Applied Research. Frontiers in Neuroscience, 2021. Vol. 15, article ID 705621. 25 p. DOI:10.3389/fnins.2021.705621
18. Maddox R.K., Lee A.K.C. Auditory Brainstem Responses to Continuous Natural Speech in Human Listeners. eNeuro, 2018. Vol. 5, no. 1, article ID e0441-17.2018, 13 p. DOI:10.1523/ENEURO.0441-17.2018
19. Broderick M.P., Zuk N.J., Anderson A.J., Lalor E.C. More than words: Neurophysiological correlates of semantic dissimilarity depend on comprehension of the speech narrative. European Journal of Neuroscience, 2022. Vol. 56, no. 8, pp. 5201—5214. DOI:10.1111/ejn.15805
20. Gillis M., Vanthornhout J., Simon J.Z., Francart T., Brodbeck C. Neural Markers of Speech Comprehension: Measuring EEG Tracking of Linguistic Speech Representations, Controlling the Speech Acoustics. Journal of Neuroscience, 2021. Vol. 41, no. 50, pp. 10316—10329. DOI:10.1523/JNEUROSCI.0812-21.2021
21. Di Liberto G.M., Nie J., Yeaton J., Khalighinejad B., Shamma S.A., Mesgarani N. Neural representation of linguistic feature hierarchy reflects second-language proficiency. NeuroImage, 2021. Vol. 227, article ID 117586. 13 p. DOI:10.1016/j.neuroimage.2020.117586
22. Brodbeck C., Bhattasali S., Heredia A.A.C., Resnik P., Simon J.Z., Lau E. Parallel processing in speech perception with local and global representations of linguistic context. eLife, 2022. Vol. 11, article ID e72056. 28 p. DOI:10.7554/eLife.72056
23. Pasley B.N., David S.V., Mesgarani N., Flinker A., Shamma S.A., Crone N.E., Knight R.T., Chang E.F. Reconstructing Speech from Human Auditory Cortex. PLOS Biology, 2012. Vol. 10, no. 1, article ID e1001251. 13 p. DOI:10.1371/journal.pbio.1001251
24. Lalor E.C., Power A.J., Reilly R.B., Foxe J.J. Resolving Precise Temporal Processing Properties of the Auditory System Using Continuous Stimuli. Journal of Neurophysiology, 2009. Vol. 102, no. 1, pp. 349—359. DOI:10.1152/jn.90896.2008
25. Sassenhagen J. How to analyse electrophysiological responses to naturalistic language with time-resolved multiple regression. Language, Cognition and Neuroscience, 2019. Vol. 34, no. 4, pp. 474—490. DOI:10.1080/23273798.2018.1502458
26. Seyednozadi Z., Pishghadam R., Pishghadam M. Functional Role of the N400 and P600 in Language-Related ERP Studies with Respect to Semantic Anomalies: An Overview. Archives of Neuropsychiatry, 2021. Vol. 58, no. 3, pp. 249—252. DOI:10.29399/npa.27422
27. Crosse M.J., Di Liberto G.M., Bednar A., Lalor E.C. The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for Relating Neural Signals to Continuous Stimuli. Frontiers in Human Neuroscience, 2016. Vol. 10, article ID 604. 14 p. DOI:10.3389/fnhum.2016.00604
28. Luck S.J., Kappenman E.S. (eds.), The Oxford Handbook of Event-Related Potential Components. Oxford: Oxford University Press, 2011. 664 p. DOI:10.1093/oxfordhb/9780195374148.001.0001
29. Gwilliams L., Marantz A., Poeppel D., King J.R. Top-down information shapes lexical processing when listening to continuous speech. Language, Cognition and Neuroscience, 2023, pp. 1—14. DOI:10.1080/23273798.2023.2171072
30. Fahmie T.A., Rodriguez N.M., Luczynski K.C., Rahaman J.A., Charles B.M., Zangrillo A.N. Toward an explicit technology of ecological validity. Journal of Applied Behavior Analysis, 2023. Vol. 56, no. 2, pp. 302—322. DOI:10.1002/jaba.972
31. Van Petten C., Luka B.J. Prediction during language comprehension: Benefits, costs, and ERP components: Predictive information processing in the brain: Principles, neural mechanisms and models. International Journal of Psychophysiology, 2012. Vol. 83, no. 2, pp. 176—190. DOI:10.1016/j.ijpsycho.2011.09.015
32. Verschueren E., Vanthornhout J., Francart T. The Effect of Stimulus Choice on an EEG-Based Objective Measure of Speech Intelligibility. Ear and Hearing, 2020. Vol. 41, no. 6, pp. 1586—1597. DOI:10.1097/AUD.0000000000000875
33. Weissbart H., Reichenbach J., Kandylaki K. Cortical tracking of surprisal during continuous speech comprehension. Journal of Cognitive Neuroscience, 2020. Vol. 32, no. 1, pp. 155—166. DOI:10.1162/jocn_a_01467