Skip to Main Content

Experimental Psychology (Russia)
2019. Vol. 12, no. 1, 98–118
https://doi.org/10.17759/exppsy.2019120108
ISSN: 2072-7593 / 2311-7036 (online)

Follow the hands of the interlocutor! (on strategies for the distribution of visual attention)

 
Audio is AI-generated
1196

,

PDF (RUS)
Online translation in english
GS
Information
in Google Scholar
Metrics

Abstract

This article contributes to the development of multichannel linguistics — a new approach that explores the interaction of multiple informational channels between interlocutors, including verbal units, prosody, gestures, facial expressions, direction of gaze. The study was carried out on the material of the Russian multichannel resource “Russian pear chats and stories”, collected with the help of modern equipment, including industrial video cameras and eyetrackers for registering eye movements. Based on the quantitative and qualitative results of the study, the authors describe the distribution of visual attention of the Narrator in the situation of perception of information from two gesticulating interlocutors, as well as various styles of oculomotor activity of the Narrator.

General Information

Keywords: multichannel linguistics, natural interaction, visual attention, gesticulation

Journal rubric: Psycholinguistics

OpenAlex citations: 3

OpenAlex trends: Hearing Impairment and Communication, Language, Metaphor, and Cognition, Hand Gesture Recognition Systems

Information about the work in OpenAlex

Number of citations: 3

Topics

Hearing Impairment and Communication

This cluster of papers focuses on the acquisition and development of sign language, including topics such as the role of gesture in language acquisition, cognitive development in deaf children, bilingualism, neural correlates of sign language processing, and strategies for early intervention and literacy development. The research encompasses various aspects of sign language acquisition and its impact on cognitive and linguistic development in deaf individuals.

Number of works: 99483  |  Total number of citations: 719605

Topic detailsв OpenAlex

Language, Metaphor, and Cognition

This cluster of papers explores the metaphorical representation of time in language and cognition, investigating the influence of language on conceptualizations of time, the role of embodiment and spatial metaphors in understanding time, and the neural basis of understanding metaphoric expressions related to time. It also delves into cross-cultural differences in mental representations of time and the impact of metaphor on reasoning and communication.

Number of works: 142455  |  Total number of citations: 1609852

Topic detailsв OpenAlex

Hand Gesture Recognition Systems

This cluster of papers focuses on the research and development of gesture recognition systems, particularly in the context of human-computer interaction. The topics covered include hand gesture recognition, sign language recognition, depth sensor technology (e.g., Kinect), neural networks, real-time tracking, deep learning, and continuous recognition.

Number of works: 51835  |  Total number of citations: 448345

Topic detailsв OpenAlex

Work details in OpenAlex

Article type: scientific article

DOI: https://doi.org/10.17759/exppsy.2019120108

Funding. This work was supported by the Russian Sience Foundation (project № 14-18-03819 “Language as is: Russian multimodal discourse”).

Published

For citation: Fedorova, O.V., Zherdev, I.Y. (2019). Follow the hands of the interlocutor! (on strategies for the distribution of visual attention). Experimental Psychology (Russia), 12(1), 98–118. (In Russ.). https://doi.org/10.17759/exppsy.2019120108

© Fedorova O.V., Zherdev I.Y., 2019

License: CC BY-NC 4.0

References

  1. Barabanshchikov V.A. Vospriyatie vyrazhenii litsa. M.: Izd-vo IP RAN, 2009.
  2. Barabanshchikov V.A., Zhegallo A.V. Raspoznavanie ekspressii litsa v blizhnei periferii zritel’nogo polya. Eksperimental’naya psikhologiya, 2013. T. 6. № 2. P. 58—83.
  3. Kibrik A.A. Mul’timodal’naya lingvistika. Kognitivnye issledovaniya, 2010. vyp. IV. Moscow. P. 134—152.
  4. Kibrik A.A. Prosodicheskie portrety govoryashchikh. Kibrik A. A., Podlesskaya V. I. (red.). Rasskazy o snovideniyakh: korpusnoe issledovanie ustnogo russkogo diskursa. Moscow: YaSK, 2009. P. 478—487.
  5. Kibrik A.A. Russkiy mul’tikanal’nyy diskurs. Chast’ I. Postanovka problemy // Psikhologicheskiy zhurnal, 2018a. Vol. 39 (1), P. 70—80.
  6. Kibrik A.A. Russkiy mul’tikanal’nyy diskurs. Chast’ II. Razrabotka korpusa i napravleniya issledovaniy // Psikhologicheskiy zhurnal, 2018b. Vol. 39 (2), P. 78—89.
  7. Kibrik A.A., Podlesskaya V. I. (Eds.). Rasskazy o snovideniyakh: korpusnoe issledovanie ustnogo russkogo diskursa. Moscow: YaSK, 2009.
  8. Kodzasov S.V. Issledovaniya v oblasti russkoi prosodii. Moscow: YaSK. 2009.
  9. Litvinenko A.O., Nikolaeva Yu.V., Kibrik A.A. Annotirovanie russkikh manual’nykh zhestov: teoreticheskie i prakticheskie voprosy. Komp’yuternaya lingvistika i intellektual’nye tekhnologii: Po materialam ezhegodnoi Mezhdunarodnoi konferentsii «Dialog 2017», Moscow, 2017. P. 271—286.
  10. Fedorova O.V., Kibrik A.A., Korotaev N.A., Litvinenko A.O., Nikolaeva Yu.V. Vremennaya koordinatsiya mezhdu zhestovymi i rechevymi edinitsami v mul’timodal’noi kommunikatsii. Komp’yuternaya lingvistika i intellektual’nye tekhnologii: Po materialam ezhegodnoi Mezhdunarodnoi konferentsii Dialog (Moskva, 1—4 iyunya 2016). Vyp. 15 (22). Moscow: Izd-vo RGGU, 2016. P. 159—170.
  11. Fedorova O.V. Raspredelenie zritel’nogo vnimaniya sobesednikov v estestvennoi kommunikatsii: 50 let spustya. E.V. Pechenkova, M.V. Falikman (red.) Kognitivnaya nauka v Moskve: novye issledovaniya. Materialy konferentsii 15 iyunya 2017. Moscow: BukiVedi, IPPiP, 2017. P. 370—375.
  12. Yazykov S.A., Fedorova O.V., Filimonova I.V., Potanina Yu.D., Pogodaev I.S. Issledovanie napravleniya vzora v dialoge na materiale mul’timodal’nogo korpusa «Rasskazy i razgovory o grushakh». Sed’maya mezhdunarodnaya konferentsiya po kognitivnoi nauke: Tezisy dokladov. Svetlogorsk, 20—24 iyunya 2016. Moscow: Institut psikhologii RAN, 2016. P. 682—683.
  13. Abuczki A., Esfandiari B.G. An overview of multimodal corpora, annotation tools and schemes // Argumentum, 2013. Vol. 9. P. 86—98.
  14. Beattie G., Webster K., Ross J. The fixation and processing of the iconic gestures that accompany talk // Journal of Language and Social Psychology. 2010. Vol. 29. P. 194—213.
  15. Brône G., Oben B. InSight Interaction. A multimodal and multifocal dialogue corpus // Language Resources and Evaluation. 2015. Vol. 49(1). P. 195—214.
  16. Carletta J. Announcing the AMI Meeting Corpus // The ELRA Newsletter. 2003. Vol. 11 (1), January- March. P. 3—5.
  17. Chafe W. (Ed.). The pear stories: Cognitive, cultural, and linguistic aspects of narrative production. Norwood, 1980.
  18. Engbert R. Microsaccades: A microcosm for research on oculomotor control, attention, and visual perception // Progress in Brain Research, 2006. Vol. 154. P. 177—192.
  19. Foster M.E., Oberlander J.Corpus-based generation of head and eyebrow motion for an embodied conversational agent // Language Resources and Evaluation, 2007. Vol. 41 (3/4). P. 305—323.
  20. Gullberg M. Gestures, referents, and anaphoric linkage in learner varieties // C. Dimroth, M. Starren (Eds.) Information Structure, Linguistic Structure and the Dynamics of Language Acquisition. Amsterdam: John Benjamins, 2003. P. 311—328.
  21. Gullberg M., Holmqvist K. Keeping an eye on gestures: Visual perception of gestures in face-to-face communication // Pragmatics and Cognition, 1999. Vol. 7. P. 35—63.
  22. Gullberg M., Holmqvist K. Visual attention towards gestures in face-to-face interaction vs. on screen // I. Wachsmuth and T. Sowa (Eds.) Gesture and Sign Language in Human—Computer Interaction. Berlin Heidelberg: Springer, 2002. P. 206—214.
  23. Gullberg M., Holmqvist K. What speakers do and what addressees look at: Visual attention to gestures in human interaction live and on video // Pragmatics & Cognition, 2006. Vol. 14 (1), P. 53—82.
  24. Gullberg M., Kita S. Attention to speech-accompanying gestures: Eye movements and information uptake // Journal of Nonverbal Behavior, 2009. Vol. 33. P. 251—277.
  25. Holler J., Kendrick K.H.Unaddressed participants’ gaze in multi-person interaction: Optimizing recipiency // Frontiers in Psychology, 2015. Vol. 6 (98).
  26. Hostetter A.B. When do gestures communicate? A meta-analysis, Psychological Bulletin, 2001. Vol. 137 (2). P. 297—315.
  27. Jacob R., Karn K. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises // R. Radach, J. Hyona, and H. Deubel (Eds.). The mind’seye: Cognitive and applied aspects of eye movement research. Oxford: Elsevier Science, 2003.
  28. Jokinen K. Turn taking, Utterance Density, and Gaze Patterns as Cues to Conversational Activity // Proceedings of The International Conference on Multimodal Interaction, Alicante, Spain, 2011.
  29. Kendon A. Some functions of gaze direction in social interaction // Acta Psychologica, 1967. Vol. 26. P. 22—63.
  30. Kendon A. Gesture. Visible action as utterance. Cambridge, 2004.
  31. Kita S. How representational gestures help speaking // D. McNeill (ed.) Language and gesture, Cambridge University Press; Cambridge, UK, 2000. P. 162—185.
  32. Kita S., Ozyurek A. What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: evidence for an interface representation of spatial thinking and speaking // Journal of Memory and Language, 2003. Vol. 48 (1). P. 16—32.
  33. Knight D. Multimodality and active listenership: A corpus approach, Bloomsbury, London, 2011.
  34. Krauss R.M., Chen Y., Gottesman R. Lexical gestures and lexical access: A process model // D. McNeill D (ed.) Language and gesture, Cambridge University Press, Cambridge, UK, 2000. P. 261—283.
  35. McNeill D. Hand and Mind: What Gestures Reveal about Thought. The University of Chicago Press, Chicago, 1992.
  36. McNeill D. Gesture and thought, Chicago, 2005.
  37. Mol L., Krahmer E., Maes A., Swerts M. Seeing and being seen: The effects on gesture production // Journal of Computer-Mediated Communication, 2011. Vol. 17. P. 77—100.
  38. Mondada L.Challenges of multimodality: Language and the body in social interaction // Journal of sociolinguistics, 2016. Vol. 20& P. 336—366.
  39. Müller C., Fricke E., Cienki A., McNeill D. (Eds.). Body — Language — Communication, Mouton de Gruyter, Berlin, 2014.
  40. Scollon R. Multimodality and the language of politics // K. Brown (ed.) Encyclopedia of language and linguistics, Elsevier, 2006. Vol. 9. P. 386—387.
  41. Smith D.T., Schenk T.The premotor theory of attention: time to move on? // Neuropsychologia, 2012. Vol. 50, P. 1104—1114.
  42. Taylor M. The Structure of Multimodal Dialogue. Amsterdam: Elsevier, 1989.
  43. Vertegaal R., Slagter R., Van der Veer G., Nijholt A. Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes // Proceedings of the Conference on Human Factors in Computing Systems, 2001.

Information About the Authors

Olga V. Fedorova, Doctor of Philology, Professor, Department of Theoretical and Applied Linguistics, Faculty of Philology, Lomonosov Moscow State University, Leading Researcher of the Institute of Linguistics of RAS, Moscow, Russian Federation, e-mail: olga.fedorova@msu.ru

Ivan Y. Zherdev, associated researcher, software developer, Moscow State University of psychology and education, Moscow, Russian Federation, ORCID: https://orcid.org/0000-0001-6810-9297, e-mail: ivan866@mail.ru

Metrics

 Web Views

Whole time: 2790
Previous month: 10
Current month: 14

 PDF Downloads

Whole time: 1196
Previous month: 10
Current month: 7

 Total

Whole time: 3986
Previous month: 20
Current month: 21