The Temporal Response Function — a New Method for Investigating Neurophysiological Mechanisms of Speech Perception under Ecologically Valid Conditions

73

Abstract

The temporal response function is a new method that allows to investigate the brain mechanisms of perception of natural, naturalistic speech stimuli. In contrast to other methods for studying brain activity (e.g., evoked potentials), the temporal response function does not require the presentation of a large number of uniform stimuli to produce a robust brain response - recordings of narrative speech lasting 10 minutes or more can be used in experimental paradigms, increasing their ecological validity. The temporal response function can be used to study brain mechanisms of online processing of different components of natural speech: acoustic (physical properties of the audio signal such as envelope and spectrogram), phonological (individual phonemes and their combinations), lexical (contextual characteristics of individual words) and semantic (semantic meaning of words), as well as the interaction between these components processing mechanisms. The article presents the history of the method, its advantages in comparison with other methods and limitations, mathematical basis, features of natural speech components extraction, and a brief review of the main studies using this method.

General Information

Keywords: temporal response function (TRF), EEG, speech, brain mechanisms, naturalistic stimuli, ecological validity

Journal rubric: Neurosciences and Cognitive Studies

Article type: scientific article

DOI: https://doi.org/10.17759/jmfp.2024130108

Funding. This work is supported by the Ministry of Science and Higher Education of the Russian Federation (Agreement No. 075-10-2021-093; Project COG-RND-2262).

Received: 31.01.2024

Accepted:

For citation: Rogachev A.O., Sysoeva O.V. The Temporal Response Function — a New Method for Investigating Neurophysiological Mechanisms of Speech Perception under Ecologically Valid Conditions [Elektronnyi resurs]. Sovremennaia zarubezhnaia psikhologiia = Journal of Modern Foreign Psychology, 2024. Vol. 13, no. 1, pp. 92–100. DOI: 10.17759/jmfp.2024130108. (In Russ., аbstr. in Engl.)

References

  1. Alday P.M. M/EEG analysis of naturalistic stories: a review from speech to language processing. Language, cognition and neuroscience, 2019. Vol. 34, no. 4, pp. 457—473. DOI:10.1080/23273798.2018.1546882
  2. Di Liberto G.M., Peter V., Kalashnikova M., Goswami U., Burnham D., Lalor E.C. Atypical cortical entrainment to speech in the right hemisphere underpins phonemic deficits in dyslexia. NeuroImage, 2018. Vol. 175, pp. 70—79. DOI:10.1016/j.neuroimage.2018.03.072
  3. Broderick M.P., Anderson A.J., Lalor E.C. Semantic Context Enhances the Early Auditory Encoding of Natural Speech. Journal of Neuroscience, 2019. Vol. 39, no. 38, pp. 7564—7575. DOI:10.1523/JNEUROSCI.0584-19.2019
  4. Castles A., Rastle K., Nation K. Ending the Reading Wars: Reading Acquisition From Novice to Expert. Psychological Science in the Public Interest, 2018. Vol. 19, no. 1, pp. 5—51. DOI:10.1177/1529100618772271
  5. Crosse M.J., Liberto G.M.D., Lalor E.C. Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term Crossmodal Temporal Integration. Journal of Neuroscience, 2016. Vol. 36, no. 38, pp. 9888—9895. DOI:10.1523/JNEUROSCI.1396-16.2016
  6. Mirkovic B., Debener S., Jaeger M., De Vos M. Decoding the attended speech stream with multi-channel EEG: implications for online, daily-life applications. Journal of Neural Engineering, 2015. Vol. 12, no. 4, article ID 046007. 9 p. DOI:10.1088/1741-2560/12/4/046007
  7. Di Liberto G.M., Hjortkjær J., Mesgarani N. Editorial: Neural Tracking: Closing the Gap Between Neurophysiology and Translational Medicine. Frontiers in Neuroscience, 2022. Vol. 16, article ID 872600. 4 p. DOI:10.3389/fnins.2022.872600
  8. Ding N., Simon J. Cortical entrainment to continuous speech: functional roles and interpretations. Frontiers in Human Neuroscience, 2014. Vol. 8, article ID 311. 7 p. DOI:10.3389/fnhum.2014.00311
  9. Ding N., Simon J.Z. Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech. Journal of Neuroscience, 2013. Vol. 33, no. 13, pp. 5728—5735. DOI:10.1523/JNEUROSCI.5297-12.2013
  10. Broderick M.P., Di Liberto G.M., Anderson A.J., Rofes A., Lalor E.C. Dissociable electrophysiological measures of natural language processing reveal differences in speech comprehension strategy in healthy ageing. Scientific Reports, 2021. Vol. 11, no. 1, article ID 4963. 12 p. DOI:10.1038/s41598-021-84597-9
  11. Mikolov T., Sutskever I., Chen K., Corrado G.S., Dean J. Distributed Representations of Words and Phrases and their Compositionality. In Burges C.J., Bottou L., Welling M., Ghahramani Z., Weinberger K.Q. (eds.), Advances in Neural Information Processing Systems: 27th Annual Conference on Neural Information Processing Systems 2013: Held 5-10 December 2013, Lake Tahoe, Nevada, USA. New York: Curran Associates Inc. Proceedings.com, 2013. Vol. 26. 9 p. DOI:10.48550/arXiv.1310.4546
  12. Broderick M.P., Anderson A.J., Di Liberto G.M., Crosse M.J., Lalor E.C. Electrophysiological Correlates of Semantic Dissimilarity Reflect the Comprehension of Natural, Narrative Speech. Current Biology, 2018. Vol. 28, no. 5, pp. 803—809. DOI:10.1016/j.cub.2018.01.080
  13. Hamilton L.S., Huth A.G. The revolution will not be controlled: natural stimuli in speech neuroscience. Language, Cognition and Neuroscience, 2020. Vol. 35, no. 5, pp. 573—582. DOI:10.1080/23273798.2018.1499946
  14. Klimovich-Gray A., Di Liberto G., Amoruso L., Barrena A., Agirre E., Molinaro N. Increased top-down semantic processing in natural speech linked to better reading in dyslexia. NeuroImage, 2023. Vol. 273, article ID 120072. 11 p. DOI:10.1016/j.neuroimage.2023.120072
  15. Kalashnikova M., Peter V., Di Liberto G.M., Lalor E.C., Burnham D. Infant-directed speech facilitates seven-month-old infants’ cortical tracking of speech. Scientific Reports, 2018. Vol. 8, article ID 13745. 8 p. DOI:10.1038/s41598-018-32150-6
  16. Khalighinejad B., da Silva G.C., Mesgarani N. Dynamic Encoding of Acoustic Features in Neural Responses to Continuous Speech. Journal of Neuroscience, 2017. Vol. 37, no. 8, pp. 2176—2185. DOI:10.1523/JNEUROSCI.2383-16.2017
  17. Crosse M.J., Zuk N.J., Di Liberto G.M., Nidiffer A.R., Molholm S., Lalor E.C. Linear Modeling of Neurophysiological Responses to Speech and Other Continuous Stimuli: Methodological Considerations for Applied Research. Frontiers in Neuroscience, 2021. Vol. 15, article ID 705621. 25 p. DOI:10.3389/fnins.2021.705621
  18. Maddox R.K., Lee A.K.C. Auditory Brainstem Responses to Continuous Natural Speech in Human Listeners. eNeuro, 2018. Vol. 5, no. 1, article ID e0441-17.2018, 13 p. DOI:10.1523/ENEURO.0441-17.2018
  19. Broderick M.P., Zuk N.J., Anderson A.J., Lalor E.C. More than words: Neurophysiological correlates of semantic dissimilarity depend on comprehension of the speech narrative. European Journal of Neuroscience, 2022. Vol. 56, no. 8, pp. 5201—5214. DOI:10.1111/ejn.15805
  20. Gillis M., Vanthornhout J., Simon J.Z., Francart T., Brodbeck C. Neural Markers of Speech Comprehension: Measuring EEG Tracking of Linguistic Speech Representations, Controlling the Speech Acoustics. Journal of Neuroscience, 2021. Vol. 41, no. 50, pp. 10316—10329. DOI:10.1523/JNEUROSCI.0812-21.2021
  21. Di Liberto G.M., Nie J., Yeaton J., Khalighinejad B., Shamma S.A., Mesgarani N. Neural representation of linguistic feature hierarchy reflects second-language proficiency. NeuroImage, 2021. Vol. 227, article ID 117586. 13 p. DOI:10.1016/j.neuroimage.2020.117586
  22. Brodbeck C., Bhattasali S., Heredia A.A.C., Resnik P., Simon J.Z., Lau E. Parallel processing in speech perception with local and global representations of linguistic context. eLife, 2022. Vol. 11, article ID e72056. 28 p. DOI:10.7554/eLife.72056
  23. Pasley B.N., David S.V., Mesgarani N., Flinker A., Shamma S.A., Crone N.E., Knight R.T., Chang E.F. Reconstructing Speech from Human Auditory Cortex. PLOS Biology, 2012. Vol. 10, no. 1, article ID e1001251. 13 p. DOI:10.1371/journal.pbio.1001251
  24. Lalor E.C., Power A.J., Reilly R.B., Foxe J.J. Resolving Precise Temporal Processing Properties of the Auditory System Using Continuous Stimuli. Journal of Neurophysiology, 2009. Vol. 102, no. 1, pp. 349—359. DOI:10.1152/jn.90896.2008
  25. Sassenhagen J. How to analyse electrophysiological responses to naturalistic language with time-resolved multiple regression. Language, Cognition and Neuroscience, 2019. Vol. 34, no. 4, pp. 474—490. DOI:10.1080/23273798.2018.1502458
  26. Seyednozadi Z., Pishghadam R., Pishghadam M. Functional Role of the N400 and P600 in Language-Related ERP Studies with Respect to Semantic Anomalies: An Overview. Archives of Neuropsychiatry, 2021. Vol. 58, no. 3, pp. 249—252. DOI:10.29399/npa.27422
  27. Crosse M.J., Di Liberto G.M., Bednar A., Lalor E.C. The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for Relating Neural Signals to Continuous Stimuli. Frontiers in Human Neuroscience, 2016. Vol. 10, article ID 604. 14 p. DOI:10.3389/fnhum.2016.00604
  28. Luck S.J., Kappenman E.S. (eds.), The Oxford Handbook of Event-Related Potential Components. Oxford: Oxford University Press, 2011. 664 p. DOI:10.1093/oxfordhb/9780195374148.001.0001
  29. Gwilliams L., Marantz A., Poeppel D., King J.R. Top-down information shapes lexical processing when listening to continuous speech. Language, Cognition and Neuroscience, 2023, pp. 1—14. DOI:10.1080/23273798.2023.2171072
  30. Fahmie T.A., Rodriguez N.M., Luczynski K.C., Rahaman J.A., Charles B.M., Zangrillo A.N. Toward an explicit technology of ecological validity. Journal of Applied Behavior Analysis, 2023. Vol. 56, no. 2, pp. 302—322. DOI:10.1002/jaba.972
  31. Van Petten C., Luka B.J. Prediction during language comprehension: Benefits, costs, and ERP components: Predictive information processing in the brain: Principles, neural mechanisms and models. International Journal of Psychophysiology, 2012. Vol. 83, no. 2, pp. 176—190. DOI:10.1016/j.ijpsycho.2011.09.015
  32. Verschueren E., Vanthornhout J., Francart T. The Effect of Stimulus Choice on an EEG-Based Objective Measure of Speech Intelligibility. Ear and Hearing, 2020. Vol. 41, no. 6, pp. 1586—1597. DOI:10.1097/AUD.0000000000000875
  33. Weissbart H., Reichenbach J., Kandylaki K. Cortical tracking of surprisal during continuous speech comprehension. Journal of Cognitive Neuroscience, 2020. Vol. 32, no. 1, pp. 155—166. DOI:10.1162/jocn_a_01467

Information About the Authors

Anton O. Rogachev, PhD Student, junior researcher, Scientific Center for Cognitive Research, Sirius University of Science and Technology, Federal territory "Sirius", Russia, ORCID: https://orcid.org/0000-0002-7645-4354, e-mail: aorogachev@gmail.com

Olga V. Sysoeva, PhD in Psychology, Leading Researcher, Sirius University of Science and Technology, Head of the Laboratory, Scientific Center for Cognitive Research, Federal territory "Sirius", Russia, ORCID: https://orcid.org/0000-0002-4005-9512, e-mail: olga.v.sysoeva@gmail.com

Metrics

Views

Total: 275
Previous month: 30
Current month: 18

Downloads

Total: 73
Previous month: 9
Current month: 7