Recognition of Emotional States of Children with down Syndrome by Facial Expression: Perceptual and Automatic Analysis of Dynamic Images

The study is devoted to the investigation of the recognition of the emotional state of children with Down syndrome (DS) by their facial expression. For this purpose, a series of perceptual experiments involving adults (n=75) and automatic analysis of the facial expressions of children (n=35, aged 5—16 years) were carried out using the FaceReader program. The ability of adults to recognize the emotional states of children: joy — neutral (calm state) — sadness — anger, by open faces and faces with masks over the eyes and mouth is shown. Better recognition of the state of joy and neutral state under the condition of an open face and a decrease in recognition accuracy in a mask in the eye area compared to the absence of a mask and a mask in the mouth area were found. Automatic recognition of the states of joy and neutral states is better than the states of sadness and anger, if the face is open and the mask in the mouth area of the child. The conditions for use the automatic recognition of facial expression in children with DS and for applying the method of perceptual analysis for identifying the specificity of the child emotional sphere development are discussed.

1. Barabanshchikov V.A., Korol’kova O.A., Lobodinskaya E.A. Vospriyatie emotsional’nykh ekspressii litsa pri ego maskirovke i kazhushchemsya dvizhenii [Perception of facial expressions during masking and apparent motion] // Eksperimental’naya psikhologiya [Experimental Psychology], 2015. Vol. 8, no. 1. pp. 7—27. (In Russ.).
2. Golosovoi portret rebenka s tipichnym i atipichnym razvitiem [Voice portrait of a child with typical and atypical development] / Lyakso E.E., Frolova O.V. (eds.) StPetersburg.: Izdatel’sko-poligraficheskaya assotsiatsiya vysshikh uchebnykh zavedenii, 2020. 204 p. (In Russ.).
3. Sinel’nikov R.D. Atlas anatomii cheloveka: v 3 t. T.1. Uchenie o kostyakh, soedinenii kostei i myshtsakh [Atlas of human anatomy: 3 vol. Vol.1.The doctrine of bones, the connection of bones, and muscles]. 7thed. Moscow.: Novaya volna, 2009. 344 p. (In Russ.).
4. Ambadar Z., Schooler J.W., Cohn J.F. Deciphering the enigmatic face: the importance of facial dynamics in interpreting subtle facial expressions. Psychological Science, 2005. Vol. 16, no. 5, pp. 403—410. DOI:10.1111/j.0956-7976.2005.01548.x
5. Fridenson-Hayo S., Berggren S., Lassalle A., Tal S., Pigat D., Bölte S., Baron-Cohen S., Golan O. Basic and complex emotion recognition in children with autism: cross-cultural findings. Molecular Autism, 2016. Vol. 7:52. DOI 10.1186/s13229-016-0113-9
6. Bobicev V., Sokolova M. Inter-annotator agreement in sentiment analysis: Machine learning perspective. Proceedings of Recent Advances in Natural Language Processing (Varna, Bulgaria, September 4-6, 2017), 2017, pp. 97—102. DOI:10.26615/978-954-452-049-6_015
7. Bojanić M., Delić V., Karpov A. Call Redistribution for a Call Center Based on Speech Emotion Recognition. Applied Sciences, 2020. Vol. 10, no. 13:4653. DOI:10.3390/app10134653
8. Carvajal F., Iglesias J. Judgements of facial and vocal signs of emotion in infants with Down syndrome. Developmental Psychobiology, 2006. Vol. 48, no. 8, pp. 644—652. DOI:10.1002/dev.20173
9. Chaplin T.M., Aldao A. Gender differences in emotion expression in children: A meta-analytic review. Psychological Bulletin, 2013. Vol. 139, no. 4, pp. 735—765. DOI:10.1037/a0030737
10. Kilts C., Egan G., Gideon D., Ely T., Hoffman J. Dissociable neural pathways areinvolved in the recognition of emotion in static and dynamic facial expressions. Neuroimage, 2003. Vol. 18, no. 1, pp. 156—168. DOI:10.1006/nimg.2002.1323
11. Ekman P. Basic emotions. / In DalgleishT., PowerM.J. (eds.) Handbook of cognition and emotion. New Jersey, John Wiley & Sons Ltd, Hoboken, 1999, pp. 45—60.
12. Kuusikko-Gauffin S., Elsheikh S., Bölte S., Omar M., RiadG., Ebeling H., Rautio A., Moilanen I. Emotion recognition from the eye region in children with and without Autism Spectrum Disorder in Arab and Scandinavian countries. Scandinavian Journal of Child and Adolescent Psychiatry and Psychology, 2018. Vol. 6, no. 4, pp. 159—169. DOI:10.21307/sjcapp-2018-015
13. Izard C.E., Youngstrom E.A., Fine S.E., Mostow A.J., Trentacosta C.J. Emotions and developmental psychopathology. In Cicchetti D., Cohen D.J. (Eds.). Developmental psychopathology. New York: John Wiley & Sons, Inc, 2006, pp. 244—292. DOI:10.1002/9780470939383.ch8
14. Hart S., Jacobsen S.L. The Emotional Development Scale: Assessing the emotional capacity of 4—12 year olds. Journal of Infant, Child, and Adolescent Psychotherapy, 2019. Vol. 18, no. 2, pp. 185—195. DOI:10.1080/15289168.2019.1583056
15. Hippolyte L., Barisnikov K., Van der Linden M. Face processing and facial emotion recognition in adults with Down syndrome. American Journal of Mental Retardation, 2008. Vol. 113, no. 4, pp. 292—306. DOI:10.1352/0895-8017(2008)113[292:FPAFER]2.0.CO;2
16. Md Juremi N.R., Zulkifley M.A., Hussain A., Zaki W. Inter-rater reliability of actual tagged emotion categories validation using Cohen’s Kappa coefficient. Journal of Theoretical and Applied Information Technology, 2017. Vol. 95, no. 2, pp. 259—264.
17. Kumin L. Early communication skills for children with Down syndrome: A guide for parents and professionals. Bethesda, MD: Woodbine House, 2003, 368 p.
18. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics, 1977. Vol. 33, no. 1, pp. 159—174.
19. LoBue V., Thrasher, C. The Child Affective Facial Expression (CAFE) set: validity and reliability from untrained adults. Frontiers in psychology, 2015. Vol. 5:1532. DOI:10.3389/fpsyg.2014.01532
20. Luneski A., Konstantinidis E., Bamidis P. Affective medicine: a review of affective computing efforts in medical informatics. Methods of information in medicine, 2010. Vol. 49, no. 3, pp. 207—218. DOI: 10.3414/ME0617
21. Lyakso E.E., Frolova O.V. Early development indicators predict speech features of autistic children. In Companion Publication of the 2020 International Conference on Multimodal Interaction (ICMI’20 Companion), 2020, pp. 514—521. DOI:10.1145/3395035.3425183
22. Lyakso E.E., Frolova O.V., Matveev Yu.N. Facial Expression: Psychophysiological Study. In Raj Alex Noel Joseph, Vijayalakshmi G. V. Mahesh, Ruban Nersisson (eds.). Handbook of Research on Deep Learning-Based Image Analysis under Constrained and Unconstrained Environments. Hershey, PA: IGI Global, 2021, Сhapter 14, pp. 266—289. DOI:10.4018/978-1-7998-6690-9
23. Markaki M., Stylianou Y. Voice pathology detection and discrimination based on modulation spectral features. IEEE Transactions on Audio, Speech, and Language Processing, 2011. Vol. 19, no.7, pp. 1938—1948. DOI:10.1109/TASL.2010.2104141
24. Verkhodanova V., Trckova D., Coler M., Lowie W. More than words: Cross-linguistic exploration of Parkinson’s disease identification from speech. Lecture Notes in Computer Science, 2020.Vol. 12335, pp. 613—623. DOI:10.1007/978-3-030-60276-5_59
25. Pochon R., Declercq Ch. Emotion recognition by children with Down syndrome: A longitudinal study. Journal of Intellectual & Developmental Disability, 2013. Vol. 38, no. 4, pp. 332—343. DOI:10.3109/13668250.2013.826346
26. Giuliani N.R., Flournoy J.C., Ivie E.J., Von Hippel A., Pfeifer J. H.Presentation and validation of the Duck EES child and adolescent dynamic facial expressions stimulus set. International Journal of Methods in Psychiatric Research, 2017. Vol. 26, no. 1:e1553. DOI:10.1002/mpr.1553
27. Carvajal F., Fernández-Alcaraz C., Rueda M., Sarrión L.Processing of facial expressions of emotions by adults with Down syndrome and moderate intellectual disability. Research in developmental disabilities, 2012. Vol. 33, no. 3, pp. 783—790. DOI:10.1016/j.ridd.2011.12.004
28. Virji-Babul N., Watt K., Nathoo F., Johnson P. Recognition of facial expressions of emotion in adults with Down syndrome. Physical and Occupational Therapy in Pediatrics, 2012. Vol. 32, no. 3, pp. 333—343. DOI:10.3109/01942638.2011.653626
29. Lyakso E., Frolova O. Gorodnyi V., Grigorev A., Nikolaev A., Matveev Y. Reflection of the emotional state in the characteristics of voice and speech of children with Down syndrome. Proceedings of 10th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2019 (Timisoara, Romania, 10-12 October 2019). 2019, pp. 1—6. DOI:10.1109/SPED.2019.8906579
30. Sorce J.F., Emde R.N. The meaning of infant emotional expressions: regularities in caregiving responses in normal and Down’s syndrome infants.Journal of Child Psychology and Psychiatry, and allied disciplines, 1982. Vol. 23, no. 2, pp. 145—158. DOI:10.1111/j.1469-7610.1982.tb00059.x
31. Wehrle T., Kaiser S., Schmidt S., Scherer, K.R. Studying the dynamics of emotional expression using synthesized facial muscle movements. Journal of Personality and Social Psychology, 2000. Vol. 78, no. 1, pp. 105—119.
32. Terzis V., Moridis Chr. N., Economides A. A. Measuring instant emotions during a self-assessment test: The use of FaceReader. Proceedings of the 7th International Conference on Methods and Techniques in Behavioral Research (Eindhoven, The Netherlands, August, 2010), 2010, pp. 1—4. DOI:10.1145/1931344.1931362
33. Baron-Cohen S., Wheelwright S., Hill J., Raste Y., Plumb I. The “Reading the Mind in the Eyes” Test Revised Version: A Study with Normal Adults, and Adults with Asperger Syndrome or High-functioning Autism. Journal of Child Psychology and Psychiatry, 2001. Vol. 42, no. 2, pp. 241—251.
34. Negrão J.G., Osorio A., Siciliano R.F., Lederman V., Kozasa E.H., D’Antino M., Tamborim A., Santos V., de Leucas D., Camargo P.S., Mograbi D.C., Mecca T.P., Schwartzman, J.S. The Child Emotion Facial Expression Set: A Database for emotion recognition in children. Frontiers in Psychology, 2021. Vol. 12:666245. DOI:10.3389/fpsyg.2021.666245
35. Kalantarian H., Jedoui K., Dunlap K., Schwartz J., Washington P., Husic A., Tariq Q., Ning M., Kline A., Wall P.D.The Performance of Emotion Classifiers for Children with Parent-Reported Autism: Quantitative Feasibility Study. JMIR Mental Health, 2020. Vol. 7, no. 4, e13174. DOI:10.2196/13174
36. Vandevelde S., Morisse F., Dosen A., Poppe L., Jonckheere B., van Hove G., Maes B., van Loon J., Claes C. The scale for emotional development-revised (SED-R) for persons with intellectual disabilities and mental health problems: development, description, and reliability. International Journal of Developmental Disabilities, 2016. Vol. 62, no. 1, pp. 11—23. DOI:10.1179/2047387714Y.0000000062