Perception of the Prosodic Formative of Multimodal Affective States

The features of the expression and perception of vocal expressions as one of the forming multimodal affective states of a person are studied. The experiment, designed on the basis of the Russian-language version of the Geneva Emotion Recognition Test (GERT), involved two groups of women aged 18-45, 72 women each. One group was shown audio-video clips of 14 affective states, lasting 3-5 seconds, played by 10 professional actors according to the Stanislavsky system. The other group was presented with audio clips extracted from the same clips through headphones. It was demanded to recognize the state of the actors using an interactive image of the Geneva Emotion Wheel. It is shown that vocal expressions differ from the corresponding multimodal states by a low average accuracy of expression and identification, a different configuration and a high spread of estimates, a closer dependence on the degree of arousal, individual ways of expression and a fuzzy extended structure of categorical fields. The range of possibilities that vocal expressions have, ready to be included in cross-modal integrations, is indicated.

1. Andreeva E.V. Fenomen krossmodal'nykh vzaimodeistvii: sovremennoe sostoyanie problemy v zarubezhnoi psikhologii. Rossiiskii nauchnyi zhurnal = Russian Scientific Journal, 2013. Vol. 1, no. 32, pp. 219—225. DOI:10.17759/rsj.20130132219 (In Russ.).
2. Barabanschikov V.A., Zhegallo A.V., Korolkova O.A. Pertseptivnaya kategorizatsiya vyrazhenii litsa [Perceptual categorization of facial expressions]. Moscow: Publ. Kogito-Tsentr, 2016. 376 p. DOI:10.17759/9785990501513 (In Russ.).
3. Barabanshchikov V.A., Korolkova O.A. Vospriyatie ekspressii “zhivogo” litsa. Eksperimental'naya psikhologiya = Experimental Psychology, 2020. Vol. 13, no. 3, pp. 55—73. DOI:10.17759/exppsy.2020130305 (In Russ.).
4. Barabanshchikov V.A., Suvorova E.V. Ocenka emocional'nogo sostoyaniya cheloveka po ego videoizobrazheniyam. Eksperimental'naya psikhologiya = Experimental Psychology, 2020. Vol. 13, no. 4, pp. 4—24. DOI:10.17759/exppsy.2020130401 (In Russ.).
5. Barabanshchikov V.A., Suvorova E.V. Ocenka multimodal'nykh ekspressii litsa v laboratorii I onlain. Lico cheloveka v kontekstakh prirody, tekhnologii i kul'tury = The Human Face in Contexts of Nature, Technology, and Culture / Eds. K.I. Ananyeva, V.A. Barabanshchikov, A.A. Demidov. Moscow: Kogito-Tsentr, 2021. Pp. 310—322. DOI:10.17759/kogito.202103220 (In Russ.).
6. Barabanshchikov V.A., Suvorova E.V. Gendernyi faktor v raspoznavanii emocional'nogo sostoyaniya cheloveka po ego audiovideoizobrazheniyam. Rossiiskii psikhologicheskii zhurnal = Russian Psychological Journal, 2022. Vol. 19, no. 2, pp. 6—20. DOI:10.17759/rpj.2022190201 (In Russ.).
7. Barabanshchikov V.A., Suvorova E.V. Individual'nye formy vyrazheniya I identifikaciya multimodal'nykh dinamicheskikh sostoyanii cheloveka. Poznanie i perezhivanie = Cognition and Experience, 2022. Vol. 3, no. 2, pp. 6—35. DOI:10.17759/kogito.202203220 (In Russ.).
8. Barabanshchikov V.A., Suvorova E.V. Vyrazhenie I vospriyatie multimodal'nykh emocional'nykh sostoyanii. Natsional'nyi psikhologicheskii zhurnal = National Psychological Journal, 2023. Vol. 18, no. 3(51), pp. 106—127. DOI:10.11621/npj.2023.0311 (In Russ.).
9. Borisova A.A. Vospriyatie emocional'nogo sostoyaniya cheloveka po intonacionnomu risunku rechi. Voprosy psikhologii = Questions of Psychology, 1989. No. 1, pp. 117—124. DOI:10.17759/voppsy.1989010117 (In Russ.).
10. Vitt N. Lichnostno-situacionnaya oposredovannost' vyrazheniya I raspoznavaniya emocii v rechi. Voprosy psikhologii = Questions of Psychology, 1991. No. 2, pp. 95—107. DOI:10.17759/vp.1991020095 (In Russ.).
11. Lupenko E.A., Korolkova O.A. Semanticheskaya èkvivalentnost' kak osnova intermodal'noi integracii. Eksperimental'naya psikhologiya = Experimental Psychology, 2022. Vol. 15, no. 3, pp. 159—177. DOI:10.17759/exppsy.20221503159 (In Russ.).
12. Morozov V.P. Neverbal'naya kommunikaciya: ėksperimental'no-psikhologicheskie issledovaniya: monografiya. Moscow: Institut psikhologii RAN, 2011. 528 p. DOI:10.17759/rusbook.20110528 (In Russ.).
13. Potapova R.K., Potapov V.V., Komalova L.R. Vospriyatie multimodal'noi mono- I polietnicheskoi kommunikacii: monografiya / Ed. L.R. Komalova. Moscow: INION RAN, 2020. 210 p. DOI:10.17759/iniom.2020210 (In Russ.).
14. Yureva M.V., Khoze E.G. Sozdanie klassifikatora vokal'nykh emocional'nykh ekspressii: personalizirovannye bazy dannykh. Lico cheloveka v kontekstakh prirody, tekhnologii i kul'tury = The Human Face in Contexts of Nature, Technology, and Culture/ Eds. K.I. Ananyeva, V.A. Barabanshchikov. Moscow: Kogito-Tsentr; Moskovskii institute psikhoanaliza, 2020. Pp. 364—372. DOI:10.17759/kogito.2020364372 (In Russ.).
15. Baart M., Vroomen J. Recalibration of Vocal Affect by a Dynamic Face. Experimental Brain Research, Vol. 236, no. 1, pp. 1911—1918. DOI:10.1007/s00221-018-5270-5
16. Barkhuysen P., Krahmer E., Swerts M. Crossmodal and Incremental Perception of Audiovisual Cues to Emotional Speech. Language and Speech, Vol. 53, no. 1, pp. 3—30. DOI:10.1177/0023830910362212
17. Bänziger T., Mortillaro M., Scherer K.R. Introducing the Geneva Multimodal Expression Corpus for Experimental Research on Emotion Perception. Emotion, Vol. 12, no. 5, pp. 1161—1179. DOI:10.1037/a0025827
18. Bänziger T., Patel S., Scherer K.R. The Role of Perceived Voice and Speech Characteristics in Vocal Emotion Communication. Journal of Nonverbal Behavior, 2014. Vol. 38, no. 1, pp. 31—52. DOI:10.1007/s10919-013-0165-x
19. Bryant G.A. The Evolution of Human Vocal Emotion. Emotion Review, 2021. Vol. 13, no. 1, pp. 25—33. DOI:10.1177/1754073919870985
20. Chronaki G., Hadwin J.A., Garner M., Maurage P., Sonuga-Barke E.J.S. The Development of Emotion Recognition from Facial Expressions and Non-Linguistic Vocalizations During Childhood. The British Journal of Developmental Psychology, 2015. Vol. 33, no. 1, pp. 218—236. DOI:10.1111/bjdp.12075
21. Cortes D.S., Tornberg C., Bänziger T., Elfenbein H.A., Fischer H., Laukka P. Effects of Aging on Emotion Recognition from Dynamic Multimodal Expressions and Vocalizations. Scientific Reports, Vol. 11, no. 1, pp. 1—12. DOI:10.1038/s41598-021-94993-4
22. Cowen A.S., Elfenbein H.A., Laukka P., Keltner D. Mapping 24 Emotions Conveyed by Brief Human Vocalization. The American Psychologist, Vol. 74, no. 1, pp. 698—712. DOI:10.1037/amp0000399
23. Dolan R.J., Morris J.S., de Gelder B. Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences of the United States of America, 2001. Vol. 98, no. 17, pp. 9465—9470. DOI:10.1073/pnas.171288598
24. El Ayadi M., Kamel M.S., Karray F. Survey on speech emotion recognition: Features, classification schemes and databases. Pattern Recognition, Vol. 44, No. 3, pp. 572—587. DOI:10.1016/j.patcog.2010.09.014
25. Gelder B. De, Vroomen J. The Perception of Emotions by Ear and by Eye. Cognition &Emotion, Vol. 14, no. 3, pp. 289—311. DOI:10.1080/026999300378824
26. Jürgens R., Grass A., Drolet M., Fischer J. Effect of Acting Experience on Emotion Expression and Recognition in Voice: Non-Actors Provide Better Stimuli Than Expected. Journal of Nonverbal Behavior, Vol. 39, no. 1, pp. 195—214. DOI:10.1007/s10919-014-0201-6
27. Lambrecht L., Kreifelts B., Wildgruber D. Age-Related Decrease in Recognition of Emotional Facial and Prosodic Expressions. Emotion, Vol. 12, no. 1, pp. 529—539. DOI:10.1037/a0026827
28. Lausen A., Schacht A. Gender Differences in the Recognition of Vocal Emotions. Frontiers in Psychology, Vol. 9, no. 1, Article 882. DOI:10.3389/fpsyg.2018.00882
29. Patel S., Scherer K.R., Björkner E., Sundberg J. Mapping Emotions into Acoustic Space: The Role of Voice Production. Biological Psychology, Vol. 87, no. 1, pp. 93—98. DOI:10.1016/j.biopsycho.2011.02.010
30. Paulmann S., Pell M.D. Is There an Advantage for Recognizing Multi-Modal Emotional Stimuli? Motivation and Emotion, Vol. 35, no. 1, pp. 192—201. DOI:10.1007/s11031-011-9211-y
31. Pittam J., Scherer K.R. Vocal Expression and Communication of Emotion/ In Lewis M., Haviland J.M. (Eds.). Handbook of Emotions. The Guilford Press. 1993. Pp. 185—197.
32. Sauter D.A., Eisner F., Calder A., Scott S. Perceptual Cues in Nonverbal Vocal Expressions of Emotion. Quarterly Journal of Experimental Psychology, Vol. 63, no. 1, pp. 2251—2272. DOI:10.1080/17470211003721642
33. Scherer K.R. Vocal Affect Expression: A Review and a Model for Future Research. Psychological Bulletin, Vol. 99, no. 2, pp. 143—165. DOI:10.1037/0033-2909.99.2.143
34. Scherer K.R. What Are Emotions? And How Can They Be Measured. Social Science Information, 2005. Vol. 44, no. 1, pp. 695—729. DOI:10.1177/0539018405058216
35. Scherer K., Banse R. Acoustic Profiles in Vocal Emotion Expression. Journal of Personality and Social Psychology, 1996. Vol. 70, no. 1, pp. 614—636. DOI:10.1037/0022-3514.70.3.614
36. Scherer K.R., Banse R., Wallbott H.G., Goldbeck T. Vocal Cues in Emotion Encoding and Decoding. Motivation and Emotion, Vol. 15, no. 2, pp. 123—148. DOI:10.1007/BF00995674
37. Schirmer A., Adolphs R. Emotion Perception from Face, Voice, and Touch: Comparisons and Convergence. Trends in Cognitive Sciences, 2017. Vol. 21, no. 1, pp. 216—228.DOI:10.1016/j.tics.2017.01.001
38. Schlegel K., SchererK.R.The nomological network of emotion knowledge and emotion understanding in adults: evidence from two new performance-based tests. Cognition and Emotion, 2017. 32. pp. 1514—1530. DOI:10.1080/02699931.2017.1414687
39. Schlegel K., Grandjean D., Scherer K.R. Emotion Recognition: Unidimensional Ability or a Set of Modality- and Emotion-Specific Skills? Personality and Individual Differences, 2012. Vol. 53, no. 1, pp. 16—21.DOI:10.1016/j.paid.2012.01.026
40. Schlegel K., Grandjean D., Scherer K. R. Introducing the Geneva Emotion Recognition Test: An Example of Rasch-Based Test Development. Psychological Assessment, Vol. 26, no. 2, pp. 666—672. DOI:10.1037/a0025827
41. Sen A., Isaacowitz D., Schirmer A. Age Differences in Vocal Emotion Perception: On the Role of Speaker Age and Listener Sex. Cognition & Emotion, Vol. 32, no. 1, pp. 1189—1204. DOI:10.1080/02699931.2017.1390725
42. Williams C., Stevens K. Emotions and speech: some acoustical correlates.The Journal of the Acoustical Society of America, 1972. Vol. 52, no. 4(2), pp. 1238—1250. DOI:10.1121/1.1913238
43. Wilting J., Krahmer E., Swerts M. Real vs acted emotional speech. Proceedings of Interspeech 2006. Pittsburgh, 2006. DOI:10.21437/Interspeech.2006-250