Skip to Main Content

Experimental Psychology (Russia)
2019. Vol. 12, no. 1, 98–118
doi:10.17759/exppsy.2019120108
ISSN: 2072-7593 / 2311-7036 (online)

Follow the hands of the interlocutor! (on strategies for the distribution of visual attention)

1106

O.V. Fedorova, I.Y. Zherdev

PDF (RUS)
Online translation in english
GS
Information
in Google Scholar

Abstract

This article contributes to the development of multichannel linguistics — a new approach that explores the interaction of multiple informational channels between interlocutors, including verbal units, prosody, gestures, facial expressions, direction of gaze. The study was carried out on the material of the Russian multichannel resource “Russian pear chats and stories”, collected with the help of modern equipment, including industrial video cameras and eyetrackers for registering eye movements. Based on the quantitative and qualitative results of the study, the authors describe the distribution of visual attention of the Narrator in the situation of perception of information from two gesticulating interlocutors, as well as various styles of oculomotor activity of the Narrator.

General Information

Keywords: multichannel linguistics, natural interaction, visual attention, gesticulation

Journal rubric: Psycholinguistics

Article type: scientific article

DOI: https://doi.org/10.17759/exppsy.2019120108

Funding. This work was supported by the Russian Sience Foundation (project № 14-18-03819 “Language as is: Russian multimodal discourse”).

For citation: Fedorova O.V., Zherdev I.Y. Follow the hands of the interlocutor! (on strategies for the distribution of visual attention). Eksperimental'naâ psihologiâ = Experimental Psychology (Russia), 2019. Vol. 12, no. 1, pp. 98–118. DOI: 10.17759/exppsy.2019120108. (In Russ., аbstr. in Engl.)

References

  1. Barabanshchikov V.A. Vospriyatie vyrazhenii litsa. M.: Izd-vo IP RAN, 2009.
  2. Barabanshchikov V.A., Zhegallo A.V. Raspoznavanie ekspressii litsa v blizhnei periferii zritel’nogo polya. Eksperimental’naya psikhologiya, 2013. T. 6. № 2. P. 58—83.
  3. Kibrik A.A. Mul’timodal’naya lingvistika. Kognitivnye issledovaniya, 2010. vyp. IV. Moscow. P. 134—152.
  4. Kibrik A.A. Prosodicheskie portrety govoryashchikh. Kibrik A. A., Podlesskaya V. I. (red.). Rasskazy o snovideniyakh: korpusnoe issledovanie ustnogo russkogo diskursa. Moscow: YaSK, 2009. P. 478—487.
  5. Kibrik A.A. Russkiy mul’tikanal’nyy diskurs. Chast’ I. Postanovka problemy // Psikhologicheskiy zhurnal, 2018a. Vol. 39 (1), P. 70—80.
  6. Kibrik A.A. Russkiy mul’tikanal’nyy diskurs. Chast’ II. Razrabotka korpusa i napravleniya issledovaniy // Psikhologicheskiy zhurnal, 2018b. Vol. 39 (2), P. 78—89.
  7. Kibrik A.A., Podlesskaya V. I. (Eds.). Rasskazy o snovideniyakh: korpusnoe issledovanie ustnogo russkogo diskursa. Moscow: YaSK, 2009.
  8. Kodzasov S.V. Issledovaniya v oblasti russkoi prosodii. Moscow: YaSK. 2009.
  9. Litvinenko A.O., Nikolaeva Yu.V., Kibrik A.A. Annotirovanie russkikh manual’nykh zhestov: teoreticheskie i prakticheskie voprosy. Komp’yuternaya lingvistika i intellektual’nye tekhnologii: Po materialam ezhegodnoi Mezhdunarodnoi konferentsii «Dialog 2017», Moscow, 2017. P. 271—286.
  10. Fedorova O.V., Kibrik A.A., Korotaev N.A., Litvinenko A.O., Nikolaeva Yu.V. Vremennaya koordinatsiya mezhdu zhestovymi i rechevymi edinitsami v mul’timodal’noi kommunikatsii. Komp’yuternaya lingvistika i intellektual’nye tekhnologii: Po materialam ezhegodnoi Mezhdunarodnoi konferentsii Dialog (Moskva, 1—4 iyunya 2016). Vyp. 15 (22). Moscow: Izd-vo RGGU, 2016. P. 159—170.
  11. Fedorova O.V. Raspredelenie zritel’nogo vnimaniya sobesednikov v estestvennoi kommunikatsii: 50 let spustya. E.V. Pechenkova, M.V. Falikman (red.) Kognitivnaya nauka v Moskve: novye issledovaniya. Materialy konferentsii 15 iyunya 2017. Moscow: BukiVedi, IPPiP, 2017. P. 370—375.
  12. Yazykov S.A., Fedorova O.V., Filimonova I.V., Potanina Yu.D., Pogodaev I.S. Issledovanie napravleniya vzora v dialoge na materiale mul’timodal’nogo korpusa «Rasskazy i razgovory o grushakh». Sed’maya mezhdunarodnaya konferentsiya po kognitivnoi nauke: Tezisy dokladov. Svetlogorsk, 20—24 iyunya 2016. Moscow: Institut psikhologii RAN, 2016. P. 682—683.
  13. Abuczki A., Esfandiari B.G. An overview of multimodal corpora, annotation tools and schemes // Argumentum, 2013. Vol. 9. P. 86—98.
  14. Beattie G., Webster K., Ross J. The fixation and processing of the iconic gestures that accompany talk // Journal of Language and Social Psychology. 2010. Vol. 29. P. 194—213.
  15. Brône G., Oben B. InSight Interaction. A multimodal and multifocal dialogue corpus // Language Resources and Evaluation. 2015. Vol. 49(1). P. 195—214.
  16. Carletta J. Announcing the AMI Meeting Corpus // The ELRA Newsletter. 2003. Vol. 11 (1), January- March. P. 3—5.
  17. Chafe W. (Ed.). The pear stories: Cognitive, cultural, and linguistic aspects of narrative production. Norwood, 1980.
  18. Engbert R. Microsaccades: A microcosm for research on oculomotor control, attention, and visual perception // Progress in Brain Research, 2006. Vol. 154. P. 177—192.
  19. Foster M.E., Oberlander J.Corpus-based generation of head and eyebrow motion for an embodied conversational agent // Language Resources and Evaluation, 2007. Vol. 41 (3/4). P. 305—323.
  20. Gullberg M. Gestures, referents, and anaphoric linkage in learner varieties // C. Dimroth, M. Starren (Eds.) Information Structure, Linguistic Structure and the Dynamics of Language Acquisition. Amsterdam: John Benjamins, 2003. P. 311—328.
  21. Gullberg M., Holmqvist K. Keeping an eye on gestures: Visual perception of gestures in face-to-face communication // Pragmatics and Cognition, 1999. Vol. 7. P. 35—63.
  22. Gullberg M., Holmqvist K. Visual attention towards gestures in face-to-face interaction vs. on screen // I. Wachsmuth and T. Sowa (Eds.) Gesture and Sign Language in Human—Computer Interaction. Berlin Heidelberg: Springer, 2002. P. 206—214.
  23. Gullberg M., Holmqvist K. What speakers do and what addressees look at: Visual attention to gestures in human interaction live and on video // Pragmatics & Cognition, 2006. Vol. 14 (1), P. 53—82.
  24. Gullberg M., Kita S. Attention to speech-accompanying gestures: Eye movements and information uptake // Journal of Nonverbal Behavior, 2009. Vol. 33. P. 251—277.
  25. Holler J., Kendrick K.H.Unaddressed participants’ gaze in multi-person interaction: Optimizing recipiency // Frontiers in Psychology, 2015. Vol. 6 (98).
  26. Hostetter A.B. When do gestures communicate? A meta-analysis, Psychological Bulletin, 2001. Vol. 137 (2). P. 297—315.
  27. Jacob R., Karn K. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises // R. Radach, J. Hyona, and H. Deubel (Eds.). The mind’seye: Cognitive and applied aspects of eye movement research. Oxford: Elsevier Science, 2003.
  28. Jokinen K. Turn taking, Utterance Density, and Gaze Patterns as Cues to Conversational Activity // Proceedings of The International Conference on Multimodal Interaction, Alicante, Spain, 2011.
  29. Kendon A. Some functions of gaze direction in social interaction // Acta Psychologica, 1967. Vol. 26. P. 22—63.
  30. Kendon A. Gesture. Visible action as utterance. Cambridge, 2004.
  31. Kita S. How representational gestures help speaking // D. McNeill (ed.) Language and gesture, Cambridge University Press; Cambridge, UK, 2000. P. 162—185.
  32. Kita S., Ozyurek A. What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: evidence for an interface representation of spatial thinking and speaking // Journal of Memory and Language, 2003. Vol. 48 (1). P. 16—32.
  33. Knight D. Multimodality and active listenership: A corpus approach, Bloomsbury, London, 2011.
  34. Krauss R.M., Chen Y., Gottesman R. Lexical gestures and lexical access: A process model // D. McNeill D (ed.) Language and gesture, Cambridge University Press, Cambridge, UK, 2000. P. 261—283.
  35. McNeill D. Hand and Mind: What Gestures Reveal about Thought. The University of Chicago Press, Chicago, 1992.
  36. McNeill D. Gesture and thought, Chicago, 2005.
  37. Mol L., Krahmer E., Maes A., Swerts M. Seeing and being seen: The effects on gesture production // Journal of Computer-Mediated Communication, 2011. Vol. 17. P. 77—100.
  38. Mondada L.Challenges of multimodality: Language and the body in social interaction // Journal of sociolinguistics, 2016. Vol. 20& P. 336—366.
  39. Müller C., Fricke E., Cienki A., McNeill D. (Eds.). Body — Language — Communication, Mouton de Gruyter, Berlin, 2014.
  40. Scollon R. Multimodality and the language of politics // K. Brown (ed.) Encyclopedia of language and linguistics, Elsevier, 2006. Vol. 9. P. 386—387.
  41. Smith D.T., Schenk T.The premotor theory of attention: time to move on? // Neuropsychologia, 2012. Vol. 50, P. 1104—1114.
  42. Taylor M. The Structure of Multimodal Dialogue. Amsterdam: Elsevier, 1989.
  43. Vertegaal R., Slagter R., Van der Veer G., Nijholt A. Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes // Proceedings of the Conference on Human Factors in Computing Systems, 2001.

Information About the Authors

Olga V. Fedorova, Doctor of Philology, Professor, Department of Theoretical and Applied Linguistics, Faculty of Philology, Lomonosov Moscow State University, Leading Researcher of the Institute of Linguistics of RAS, Moscow, Russia, e-mail: olga.fedorova@msu.ru

Ivan Y. Zherdev, associated researcher, software developer, Moscow State University of Psychology & Education, Moscow, Russia, ORCID: https://orcid.org/0000-0001-6810-9297, e-mail: ivan866@mail.ru

Metrics

Views

Total: 2462
Previous month: 23
Current month: 7

Downloads

Total: 1106
Previous month: 5
Current month: 2