Vivid Face Perception as a Constructive Component of Multimodal Affective States

The features of expression and perception of vivid facial expressions in the system of multimodal affective states of a person are studied. The study is based on the Russian-language version of the Geneva Emotion Recognition Test (GERT) and consists of two series. The first serie of experiment was devoted to demonstration of short audio—video clips of 14 affective states expressed by specially trained actors, the second —to the same videos without sound accompaniment (intonation of voice in pseudolinguistic utterances). The subjects — 72 women in each series of the experiment, age: 18-45 years (M1=22.4 g, SD1=5.4; M2=27.47 l, SD2=5.7) - identified the states of the actors—sitters (five women and five men), using an interactive image of the Geneva Wheel of Emotions on the display screen for evaluation. The analysis of the characteristics of the accuracy and semantic structure of the expression and identification of emotional expressions, depending on the type of modality and exposure conditions, is carried out. The main expression and identification metrics of facial, prosodic and multimodal states are correlated. According to experimental data, within the framework of bimodal (face+voice) expressions of emotions, a mobile face has a higher expressive potential and is the basis of cross-modal integration.

1. Barabanschikov V.A., Zhegallo A.V., Korolkova O.A. Pertseptivnaya kategorizatsiya vyrazhenii litsa [Perceptual categorization of facial expressions]. Moscow: Publ. Kogito-Tsentr, 2016. 376 p. DOI:10.17759/9785990501513 (In Russ.).
2. Barabanshchikov V.A., Korolkova O.A. Vospriyatie ekspressii “zhivogo” litsa. Eksperimental'naya psikhologiya = Experimental Psychology, 2020. Vol. 13, no. 3, pp. 55—73. DOI:10.17759/exppsy.2020130305 (In Russ.).
3. Barabanshchikov V.A., Suvorova E.V. Ocenka emocional'nogo sostoyaniya cheloveka po ego videoizobrazheniyam. Eksperimental'naya psikhologiya = Experimental Psychology, 2020. Vol. 13, no. 4, pp. 4—24. DOI:10.17759/exppsy.2020130401 (In Russ.).
4. Barabanshchikov V.A., Suvorova E.V. Ocenka multimodal'nykh ekspressii litsa v laboratorii i onlain. Lico cheloveka v kontekstakh prirody, tekhnologii i kul'tury = The Human Face in Contexts of Nature, Technology, and Culture / Eds. K.I. Ananyeva, V.A. Barabanshchikov, A.A. Demidov. Moscow: Kogito-Tsentr, 2021. Pp. 310—322. DOI:10.17759/kogito.202103220 (In Russ.).
5. Barabanshchikov V.A., Suvorova E.V. Gendernyi faktor v raspoznavanii emocional'nogo sostoyaniya cheloveka po ego audiovideoizobrazheniyam. Rossiiskii psikhologicheskii zhurnal = Russian Psychological Journal, 2022. Vol. 19, no. 2, pp. 6—20. DOI:10.17759/rpj.2022190201 (In Russ.).
6. Barabanshchikov V.A., Suvorova E.V. Individual'nye formy vyrazheniya i identifikaciya multimodal'nykh dinamicheskikh sostoyanii cheloveka. Poznanie i perezhivanie = Cognition and Experience, 2022. Vol. 3, no. 2, pp. 6—35. DOI:10.17759/kogito.202203220 (In Russ.).
7. Barabanshchikov V.A., Suvorova E.V. Vyrazhenie i vospriyatie multimodal'nykh emocional'nykh sostoyanii. Natsional'nyi psikhologicheskii zhurnal = National Psychological Journal, 2023. Vol. 18, no. 3(51), pp. 106—127. DOI:10.11621/npj.2023.0311 (In Russ.).
8. Barabanschikov V.A., Suvorova E.V., Malionok A.V. Perception of the Prosodic Formative of Multimodal Affective States. Eksperimental’naya psikhologiya = Experimental Psychology, 2024. Vol. 17, no. 3, pp. 30—51. DOI:10.17759/exppsy.2024170303 (In Russ.).
9. Lupenko E.A., Korolkova O.A. Semanticheskaya èkvivalentnost' kak osnova intermodal'noi integracii. Eksperimental'naya psikhologiya = Experimental Psychology, 2022. Vol. 15, no. 3, pp. 159—177. DOI:10.17759/exppsy.20221503159 (In Russ.).
10. Adams R.B., Kleck R.E. Perceived Gaze Direction and the Processing of Facial Displays of Emotion. Psychological Science, 2003. Vol. 14, no. 6, pp. 644—647. DOI:10.1046/j.0956-7976.2003.psci_1479.x
11. Ambadar Z., Cohn J.F., Reed L.I. All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous. Journal of Nonverbal Behavior, 2009. Vol. 33, no. 1, pp. 17—34. DOI:10.1007/s10919-008-0059-5
12. Aviezer H., Hassin R.R., Ryan J., Grady C., Susskind J., Anderson A., Moscovitch M., Bentin S. Angry, disgusted, or afraid? Studies on the malleability of emotion perception. Psychological Science, 2008. Vol. 19, no. 7, pp. 724—732. DOI:10.1111/j.1467-9280.2008.02148.x
13. Baart M., Vroomen J. Recalibration of Vocal Affect by a Dynamic Face. Experimental Brain Research, 2018. Vol. 236, no. 1, pp. 1911—1918. DOI:10.1007/s00221-018-5270-5
14. Bänziger T., Mortillaro M., Scherer K.R. Introducing the Geneva Multimodal Expression Corpus for Experimental Research on Emotion Perception. Emotion, 2012. Vol. 12, no. 5, pp. 1161—1179. DOI:10.1037/a0025827
15. Barrett L.F., Russell J.A. The structure of current affect: Controversies and emerging consensus. Current Directions in Psychological Science, 1999. Vol. 8(1), pp. 10—14. DOI:10.1111/1467-8721.00003
16. Bindemann M., Burton A.M., Langton S.R.H. How do eye gaze and facial expression interact? Visual Cognition, 2008. Vol. 16, no. 6, pp. 708—733. DOI:10.1080/13506280701269318
17. Birmingham E., Bischof W.F., Kingstone A. Why do we look at people’s eyes? Journal of Eye Movement Research, 2007. Vol. 1, no. 1 (SE-Articles). DOI:10.16910/jemr.1.1.1
18. Bould E., Morris N., Wink B. Recognising subtle emotional expressions: The role of facial movements. Cognition & Emotion, 2008. Vol. 22, no. 8, pp. 1569—1587.DOI:10.1080/02699930801921156
19. Bruce V., Young A. In the eye of the beholder: the science of face perception. Oxford: Oxford University Press, 1998.
20. Brody L.R., Hall J.A. Gender and emotion in context. / M. Lewis, J.M. Haviland-Jones, L. Feldman Barrett (Eds.). Handbook of emotions (3rd ed). NY: The Guilford Press, 2010. Pp. 395—408.
21. Campanella S., Belin P. Integrating face and voice in person perception. Trends in Cognitive Sciences, 2007. Vol. 11, no. 12, pp. 535—543. DOI:10.1016/j.tics.2007.10.001
22. Cosker D., Krumhuber E.G., Hilton A. Perception of linear and nonlinear motion properties using a FACS validated 3D facial model. Proceedings of the 7th Symposium on Applied Perception in Graphics and Visualization (APGV 2010). New York, New York, USA: ACM Press, 2010. P. 101. DOI:10.1145/1836248.1836268
23. Cunningham D.W., Wallraven C. Dynamic information for the recognition of conversational expressions. Journal of Vision, 2009. Vol. 9, no. 13, pp. 1—17. DOI:10.1167/9.13.7
24. Cunningham D.W., Wallraven C. The interaction between motion and form in expression recognition. Proceedings of the 6th Symposium on Applied Perception in Graphics and Visualization (APGV 2009). New York, New York, USA: ACM Press, 2009. P. 41. DOI:10.1145/1620993.1621002
25. Cutting J.E., Kozlowski L.T. Recognizing Friends by Their Walk: Gait Perception Without Familiarity Cues. Bulletin of the Psychonomic Society, 1977. Vol. 9, pp. 353—356. DOI:10.3758/BF03337021
26. Ekman P. Basic emotions. In Dalgleish T., Power M. (Eds.). Handbook of Cognition and Emotion. : Wiley, 1999. Pp. 301—320.
27. Gelder B. De, Vroomen J. The Perception of Emotions by Ear and by Eye. Cognition & Emotion, 2000. Vol. 14, no. 3, pp. 289—311. DOI:10.1080/026999300378824
28. Hess U., Kleck R.E. The cues decoders use in attempting to differentiate emotion - elicited and posed facial expressions. European Journal of Social Psychology, 1994. Vol. 24, no. 3, pp. 367—381. DOI:10.1002/ ejsp.2420240306
29. Hill H.C.H., Troje N.F., Johnston A. Range- and domain-specific exaggeration of facial speech. Journal of vision, 2005. Vol. 5, no. 10, pp. 793—807. DOI:10.1167/5.10.4
30. Hyde J.S. The gender similarities hypothesis. American Psychologist, 2005. Vol. 60(6), pp. 581—592. DOI:10.1037/0003-066X.60.6.581
31. Izard C.E. Emotions, personality, and psychotherapy. The psychology of emotions. NY.: Plenum Press,1991. 451 p.
32. Korolkova O.A. The role of temporal inversion in the perception of realistic and morphed dynamic transitions between facial expressions. Vision Research, 2018. Vol. 143, pp. 42—51. DOI:10.1016/j.visres.2017.10.007
33. Kamachi M., Bruce V., Mukaida S., Gyoba J., Yoshikawa S., Akamatsu S. Dynamic properties influence the perception of facial expressions. Perception, 2001. Vol. 30, no. 7, pp. 875—887. DOI:10.1068/p3131
34. Krumhuber E.G., Kappas A. Moving Smiles: The Role of Dynamic Components for the Perception of the Genuineness of Smiles. Journal of Nonverbal Behavior, 2005. Vol. 29, no. 1, pp. 3—24. DOI:10.1007/s10919-004-0887-x
35. Krumhuber E.G., Scherer K.R. The Look of Fear from the Eyes Varies with the Dynamic Sequence of Facial Actions. Swiss Journal of Psychology, 2016. Vol. 75, no. 1, pp. 5—14. DOI:10.1024/1421-0185/a000166
36. Langton S.R. Gaze perception and visually mediated attention. In J. Adams, R. B., N. Ambady, K. Nakayama, S. Shimojo (ed.). The Science of Social Vision. N.Y.: Oxford University Press, 2011. Pp. 108—132.
37. Maringer M., Krumhuber E.G., Fischer A.H., Niedenthal P.M. Beyond smile dynamics: Mimicry and beliefs in judgments of smiles. Emotion, 2011. Vol. 11, no. 1, pp. 181—187. DOI:10.1037/a0022596
38. McGurk H., McDonald J. Hearing lips and seeing voices. Nature, 1976. Vol. 264, no. 5588, pp. 746—748. DOI:10.1038/264746a0
39. Massaro D.W., Egan P.B. Perceiving affect from the voice and the face. Psychonomic Bulletin & Review, 1996. Vol. 3, no. 2, pp. 215—221. DOI:10.3758/BF03212421
40. Niedenthal P.M., Brauer M., Halberstadt J.B., Innes-Ker Å.H. When did her smile drop? Facial mimicry and the influences of emotional state on the detection of change in emotional expression. Cognition and Emotion, 2001. Vol. 15, no. 6, pp. 853—864. DOI:10.1080/02699930143000194
41. Recio G., Schacht A., Sommer W. Classification of dynamic facial expressions of emotion presented briefly. Cognition & Emotion, 2013. Vol. 27, no. 8, pp. 1486—1494. DOI:10.1080/02699931.2013.794128
42. Reinl M., Bartels A. Perception of temporal asymmetries in dynamic facial expressions. Frontiers in Psychology, 2015. Vol. 6, pp. 1—8. DOI:10.3389/fpsyg.2015.01107
43. Russell J.A. Pancultural aspects of the human conceptual organization of emotions. Journal of Personality and Social Psychology, 1983. Vol. 45(6), pp. 1281—1288. DOI:10.1037/0022-3514.45.6.1281
44. Sato W., Yoshikawa S. The dynamic aspects of emotional facial expressions. Cognition & Emotion, 2004. Vol. 18, no. 5, pp. 701—710. DOI:10.1080/02699930341000176
45. Scherer K.R. What Are Emotions? And How Can They Be Measured. Social Science Information, 2005. Vol. 44, no. 1, pp. 695—729. DOI:10.1177/0539018405058216
46. Schlegel K., Scherer K.R. The nomological network of emotion knowledge and emotion understanding in adults: evidence from two new performance-based tests. Cognition and Emotion, 2017. Vol. 32, pp. 1514— DOI:10.1080/02699931.2017.1414687
47. Schlegel K., Grandjean D., Scherer K.R. Introducing the Geneva Emotion Recognition Test: An Example of Rasch-Based Test Development. Psychological Assessment, 2014. Vol. 26, no. 2, pp. 666—672. DOI:10.1037/a0025827
48. Wallraven C., Breidt M., Cunningham D.W., Bülthoff H.H. Evaluating the perceptual realism of animated facial expressions. ACM Transactions on Applied Perception, 2008. Vol. 4, no. 4, pp. 1—20. DOI:10.1145/1278760.1278764