Classification of facial expressions of emotional faces by people and a neural network based on an ecologically valid image database

This paper investigates the problem of universal facial expression patterns for emotion valences, emotion families, and culturally specific emotions. Many studies have been conducted to address this problem, but they sometimes infer the universality of emotion expression from respondents’ recognition of emotions and use traditional image databases that do not meet the conditions of ecological validity. Four groups of theories about the origin of emotions are compared: the theory of universality of basic emotions, the theory of cultural specificity of basic emotions, the theory of social learning of emotions, and the theory of social construction of emotions. Purpose of the study: to determine whether people’s facial expressions have specific patterns for positive and negative valences of emotion (degree of positive or negative affective response), emotion families (includes basic emotion and variations of basic emotion), and specific emotions using an image database whose authors strived for ecological validity: subjects were not given a standard of facial expression patterns to demonstrate emotions. A teachable machine neural network was used to examine the expression process in a respondent-independent manner. Heterogeneous transfer learning was used. In our work, respondents and the neural network categorized images of emotional faces into 14 emotions. Respondents were more likely than the neural network to correctly classify specific emotions. Both the neural network and humans were more efficient at recognizing valence and emotion families than recognizing specific emotions. This may suggest specific patterns of emotion families and emotion valence. Weak specificity at the level of specific emotions is possible. The results are consistent with Scarantino’s “New BET” theory of basic emotions. An interpretation in line with the social learning theory of emotion is possible. The limitations of the neural network in recognizing the full range of variations in prototypical emotion expressions are discussed. Further possible cross-cultural studies to refine the results are described.

Akers, R. L., Jennings, W. G. (2015). Social learning theory. The handbook of criminological theory, 230–240.

Badrulhisham, N. A. S., Mangshor, N. N. A. (2021). Emotion recognition using convolutional neural network (CNN). Journal of Physics: Conference Series, IOP Publishing, 1962, 1, 012040.

Barabanshhikov, V. A., Korol'kova, O. A., Lobodinskaja, E. A. (2018). Emotion recognition under staggered stroboscopic exposure of facial expressions. Eksperimental'naia psikhologiia, 11 (4), 50–69. (In Russian)

Barabanshhikov, V. A., Suvorova, E. V. (2020). Estimation of a person's emotional state from his video image. Eksperimental'naia psikhologiia, 13 (4), 4–24. (In Russian)

Barrett, L. F. (2011). Constructing emotion. Psihologijske teme, 20 (3), 359–380.

Barrett, L. F. (1998). Discrete emotions or dimensions? The role of valence focus and arousal focus. Cognition & Emotion, 12 (4), 579–599.

Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological science in the public interest, 20 (1), 1–68.

Barrett, L. F., Simmons, W. K. (2015). Interoceptive predictions in the brain. Nature Reviews Neuroscience, 16 (7), 419–429.

Carney, M., Webster, B., Alvarado, I., Phillips, K., Howell, N., Griffith, J., Jongejan, J., Pitaru, A., Chen, A. (2020). Teachable machine: Approachable Web-based tool for exploring machine learning classification. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems (pp. 1–8).

Dailey, M. N., Cottrell, G. W., Padgett, C., Adolphs, R. (2002). EMPATH: A neural network that categorizes facial expressions. Journal of cognitive neuroscience, 14 (8), 1158–1173.

Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition (pp. 248–255).

Ekman, P. (1971). Universals and cultural differences in facial expressions of emotion. In: Nebraska symposium on motivation (pp. 207–283). University of Nebraska Press.

Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6 (3–4), 169–200.

Ekman, P. (1999a). Basic emotions. In: Handbook of cognition and emotion (p. 16). Chichester: Wiley

Ekman, P. (1999b). Facial expressions. In: Handbook of cognition and emotion (p. 320). Chichester: Wiley

Ekman, P., Keltner, D. (1970). Universal facial expressions of emotion. California mental health research digest, 8 (4), 151–158.

Ekman, P., Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of personality and social psychology, 17 (2), 124.

Fedoseeva, E. V., Terekhova, V. A., Tsesarenko, O. V., Gladkova, M. M. (2015). Processing the results of toxicological studies in a statistical program R. Printsipy ekologii, 3 (15), 12–26. (In Russian)

Gendron, M., Hoemann, K., Crittenden, A. N., Mangola, S. M., Ruark, G. A., Barrett, L. F. (2020). Emotion perception in Hadza hunter-gatherers. Scientific reports, 10 (1), 3867.

Gendron, M., Roberson, D., van der Vyver, J. M., Barrett, L. F. (2014). Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. Emotion, 14 (2), 251.

Gruhn, D., Sharifian, N. (2016). Lists of emotional stimuli. In: Emotion measurement (pp. 145–164). Sawston, Woodhead Publishing

Korol'kova, O. A., Lobodinskaia, E. A. (2022). A video database of natural emotional expressions: emotion perception and automated analysis of facial expressions. Opticheskii zhurnal, 89 (8), 97–103. (In Russian)

Kretzschmar, R., Karayiannis, N. B., Eggimann, F. (2003). Handling class overlap with variance-controlled neural networks. In: Proceedings of the International Joint Conference on Neural Networks (vol. 1, pp. 517–522). New York: IEEE

Leon, E., Clarke, G., Callaghan, V., Sepulveda, F. (2004). Real-time detection of emotional changes for inhabited environments. Computers & Graphics, 28 (5), 635–642.

Oatley, K., Johnson-Laird, P. N. (2014). The communicative theory of emotions: Empirical tests, mental models, and implications for social interaction. In: Striving and feeling (pp. 363–393). Hove, Psychology Press

Petrakova, A. V., Lebedeva, E. I., Kuz'mina, Iu. V., Iurchik, E. N. (2024). Experience of creating a Russian database of faces depicting different emotions: The first stage. Psikhologiia. Zhurnal Vysshei shkoly ekonomiki, 21 (2), 423–431. (In Russian)

Rjumina, E. V., Karpov, A. A. (2020). Analytical review of methods for recognizing emotions from human facial expressions. Nauchno-tehnicheskii vestnik informatsionnykh tehnologii, mekhaniki i optiki, 20 (2), 163–176. (In Russian)

Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115 (1), 102.

Sato, W., Hyniewska, S., Minemoto, K., Yoshikawa, S. (2019). Facial expressions of basic emotions in Japanese laypeople. Frontiers in Psychology, 10, 259.

Scarantino, A. (2015). Basic emotions, psychological construction, and the problem of variability. In: The psychological construction of emotion (pp. 334–376). New York: The Guilford Press

Scherer, K. R., Wallbott, H. G. (1994). Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology, 66 (2), 310.

Trommsdorff, G. (2006). Development of emotions as organized by culture. ISSBD Newsletter, 49 (1), 1–4.