Perception of dynamic natural facial expressions of participants in a structured interview

Context and relevance. Studies of emotion perception based on facial expressions are primarily conducted using photo/video materials obtained from trained models or induced emotions under specially controlled conditions. It is clear that in everyday life situations, spontaneous emotions, expressed on the face through natural expressions, appear less expressive than staged prototypes. Consequently, in real-life communicative situations, emotion recognition is much more difficult, which can easily lead to erroneous conclusions by communication partners. The aim of this study was to investigate the nature of the perception of dynamic natural expressions demonstrated by participants in dyadic interactions in structured interview situations. Hypotheses were tested regarding the presence of specific assessment profiles: for each type of situation; for each model; for each video recording; for each observer. The main independent variables were: interview conditions and the natural expressions of the models. Dependent variables were the structure of observers' assessments. The stimulus material consisted of 35 videos of 7 sitters, one for each of the 5 interview conditions, each 5 seconds long. The primary assessment instrument was the Izard Differential Emotion Scale (DES — 33 Likert scales from 1 to 5). Results. Significant differences between conditions across all videos were obtained for 15 primary DES scales. The scales of interest, surprise, grief/sadness, disgust, contempt, shame, fear, and guilt were insignificant. Based on the obtained results, it can be assumed that the interview participants focused on the topics of the discussed issues that were unexpected for them, did not evoke approval, and put them in an awkward position. Cluster analysis distributed the entire set of assessments into 4 groups (k-means method). The variability between groups was at the level of 11% (SSB / SST = 0.111). Significant differences were found for virtually all DES scales, with the exception of individual scales related to the emotions of interest and contempt. Conclusions. It has been shown that natural expressions displayed by people in interviews are ambiguous and are perceived by observers inconsistently. However, despite individual variability in assessments, patterns are evident in the assessments made by naive observers, likely related to the context of the social situation, which sets the emotional tone of the communication partners.

1. Барабанщиков, В.А., Носуленко, В.Н., Самойленко, Е.С., Хозе, Е.Г. (2015). Полипозиционное наблюдение за поведением людей в процессе целенаправленной структурированной беседы. В: Н.Б. Карабущенко, Н.П. Сунгурова (Ред.), Психология и педагогика XXI века: теория, практика и перспективы (с. 109—131). М.: РУДН.
   Barabanshchikov, V.A., Nosulenko, V.N., Samoylenko, E.S., Khoze, E.G. (2015). Polypositional observation of people's behavior during a purposeful structured conversation. In: N.B. Karabushchenko, N.P. Sungurova (Eds.), Psychology and pedagogy of the 21st century: theory, practice and prospects (pp. 109—131). Moscow: RUDN. (In Russ.)
2. Леонова, А.Б., Капица, М.С. (2004). Методы субъективной оценки функциональных состояний человека. В: Ю.К. Стрелкова (Ред.), Практикум по инженерной психологии и эргономике (с. 134—166). М.: Академия.
   Leonova, A.B., Kapitsa, M.S. (2004). Methods of subjective assessment of human functional states. In: Yu.K. Strelkova (Ed.), Workshop on engineering psychology and ergonomics (pp. 134—166). Moscow: Academy. (In Russ.)
3. Куракова, О.А. (2012). Создание новой базы фотоизображений естественных переходов между базовыми эмоциональными экспрессиями лица. В: В.А. Барабанщиков, А.А. Демидов, Д.А. Дивеев (Ред.), Лицо человека как средство общения: Междисциплинарный подход (с. 287—309). Москва: Когито-Центр; Изд-во «Институт психологии РАН».
   Kurakova, O.A. (2012). Creation of a new database of photo images of natural transitions between basic emotional facial expressions. In: V.A. Barabanshchikov, A.A. Demidov, D.A. Diveev (Eds.), The human face as a means of communication: An interdisciplinary approach (pp. 287—309). Moscow: Cogito-Center; Publishing house “Institute of Psychology of the Russian Academy of Sciences”. (In Russ.)
4. Buck, R., VanLear, C.A. (2002). Verbal and nonverbal communication: distinguishing symbolic, spontaneous, and pseudo-spontaneous nonverbal behavior. Journal of Communication, 52(3), 522— https://doi.org/10.1111/j.1460-2466.2002.tb02560.x
5. Carroll, J.M., Russell, J.A. (1997). Facial expressions in Hollywood’s portrayal of emotion. Journal of Personality and Social Psychology, 72(1), 164—176.
6. Costa-Abreu, M.Da, Bezerra, G.S. (2019). FAMOS: a framework for investigating the use of face features to identify spontaneous emotions. Pattern Analysis and Applications, 22, 683—701. https://doi.org/10.1007/s10044-017-0675-y
7. Ekman P., Friesen W.V. (1976). Pictures of Facial Affect. Palo Alto, CA: Consulting Psychologists Press.
8. Ekman, P., Friesen, W.V. (1978). Facial action coding system. Palo Alto: Consulting Psychologists Press.
9. Ekman, P., Friesen, W. (1976). Measuring facial movement. Environmental psychology and nonverbal behavior, 1(1), 56—75. https://doi.org/10.1007/BF01115465
10. Golan, O., Baron-Cohen, S., Hill, J. (2006). The Cambridge Mindreading (CAM) Face-Voice Battery: Testing Complex Emotion Recognition in Adults with and without Asperger Syndrome. Journal of Autism and Developmental Disorders, 36(2)? 169—183. https://doi.org/10.1007/s10803-005-0057-y
11. Gosselin, P., Kirouac, G., Doré, F.Y. (1995). Components and recognition of facial expression in the communication of emotion by actors. Journal of Personality and Social Psychology, 68(1), 83—96. 
12. Hess, U., Hareli, S. (2019). The Social Nature of Emotion Expression. What emotions can tell us about the world? Cham: Springer. https://doi.org/10.1007/978-3-030-32968-6
13. Jack, R.E., Garrod, O.G., Schyns, P.G. (2014). Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time. Current Biology, 24(2), 187—192.
14. Johnston, L., Miles, L., Macrae, C.N. (2010). Why are you smiling at me? Social functions of enjoyment and non-enjoyment smiles. British Journal of Social Psychology, 49(Pt 1), 107—127. https://doi.org/10.1348/014466609X412476
15. Korolkova, O.A., Lobodinskaya, E.A. (2022). Database of video images of natural emotional facial expressions: perception of emotions and automated analysis of facial structure. Journal of Optical Technology, 89(8), 498—501. https://doi.org/10.1364/JOT.89.000498
16. Krumhuber, E.G., Scherer, K.R. (2011). Affect bursts: dynamic patterns of facial expression. Emotion, 11(4), 825—841. https://doi.org/10.1037/a0023856
17. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H.J., Hawk, S.T., van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition & Emotion, 24(8), 1377—1388.
18. LoBue, V., Thrasher, C. (2015). The Child Affective Facial Expression (CAFE) Set: Validity and Reliability from Untrained Adults. Frontiers in Emotion Science, 5.
19. Matsumoto, D., Ekman, P. (1988). Japanese and Caucasian facial expressions of emotion (IACFEE). San Francisco, CA: Intercultural and Emotion Research Laboratory, Department of Psychology, San Francisco State University.
20. Motley, M.T., Camden, C.T. (1988). Facial expression of emotion: A comparison of posed expressions versus spontaneous expressions in an interpersonal communications setting. Western Journal of Speech Communication, 52, 1—220.
21. Mühlberger, A., Wieser, M.J., Gerdes, A.B., Frey, M.C., Weyers, P., Pauli, P. (2011). Stop looking angry and smile, please: start and stop of the very same facial expression differentially activate threat- and reward-related brain networks. Social Cognitive and Affective Neuroscience, 6(3), 321—329. https://doi.org/1093/scan/nsq039
22. Namba, S., Makihara, S., Kabir, R.S., Miyatani, M., Nakao, T. (2017). Spontaneous Facial Expressions Are Different from Posed Facial Expressions: Morphological Properties and Dynamic Sequences. Current Psychology, 36, 593—605. https://doi.org/10.1007/s12144-016-9448-9
23. Negrão, J.G., Osorio, A.A.C., Siciliano, R.F., Lederman, V.R.G., Kozasa, E.H., D’Antino, M.E.F., Tamborim, A., Santos, V., de Leucas, D.L.B., Camargo, P.S., Mograbi, D.C., Mecca, T.P., Schwartzman, J.S. (2021). The Child Emotion Facial Expression Set: A Database for Emotion Recognition in Children. Frontiers in psychology, 12, 666245. https://doi.org/3389/fpsyg.2021.666245
24. Romani-Sponchiado, A., Sanvicente-Vieira, B., Mottin, C., Hertzog-Fonini, D., Arteche, A. (2015). Child emotions picture set (CEPS): development of a database of children's emotional expressions. Psychology & Neuroscience, 8(4), 467—478. https://doi.org/10.1037/h0101430
25. Ross, E.D., Pulusu, V.K. (2013). Posed versus spontaneous facial expressions are modulated by opposite cerebral hemispheres. Cortex, 49(5), 1280—1291. https://doi.org/10.1016/j.cortex.2012.05.002
26. Scherer, K.R., Ellgring, H. (2007). Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? Emotion, 7(1), 113—130. https://doi.org/10.1037/1528-3542.7.1.113
27. Wehrle, T., Kaiser, S., Schmidt, S., Scherer, K.R. (2000). Studying the dynamics of emotional expression using synthesized facial muscle movements. Journal of Personality and Social Psychology, 78(1), 105—119.