Cover not available
Part of
Eye-tracking in Interaction: Studies on the role of eye gaze in dialogue
Edited by Geert Brône and Bert Oben
[Advances in Interaction Studies 10] 2018
► pp. 47–66

Chapter 3
Effects of a speaker’s gaze on language comprehension and acquisition

This chapter discusses the role of a speaker’s gaze on a listener’s language processing, recall of information, and on child language learning. Speaker gaze facilitates performance in all of these domains, which suggests that it plays an important role in communication. Indeed, the findings indicate that speaker gaze can facilitate not just referential but also compositional processes such as syntactic structuring and thematic role assignment in listeners. Speaker gaze even guided comprehenders’ attention to a target object more rapidly than other aspects of the visual context (action event depictions); but unlike the action depictions, it had no beneficial effects on comprehenders’ post-experiment memory in socially non-interactive settings (when comprehenders inspect the speaker on a computer display). In socially interactive settings, however, speaker gaze seemed to affect memory in that joint object-based attention between a caregiver and child was beneficial for language learning. We conclude that speaker gaze effects are important for both language processing and learning, and are potentially boosted in a socially interactive context.

Keywords: language processing, language learning, recall and memory, communicative function of gaze
Article outline
  • 1.Introduction
  • 2.The role of speaker eye gaze: Syntactic structuring and thematic role assignment
  • 3.Immediate attention and later recall: Speaker gaze vs. action depictions
  • 4.Speaker gaze and joint attention in child language processing and learning
  • 5.The role of speaker gaze: Social and pragmatic considerations
  • Acknowledgements
  • References
Published online: 13 November 2018
DOI logo
https://doi.org/10.1075/ais.10.03kno
References
Baldwin, D.
(1993a) Infants’ ability to consult the speaker for clues to word reference. Journal of Child Language, 2, 395–418.DOI logoGoogle Scholar
(1993b) Early referential understanding: Young children’s ability to recognize referential acts for what they are. Developmental Psychology, 29, 1–12.DOI logoGoogle Scholar
Baldwin, D. A.
(2000) Interpersonal understanding fuels knowledge acquisition. Current Directions in Psychological Science, 9, 40–45.DOI logoGoogle Scholar
Brooks, R., & Meltzoff, A. N.
(2005) The development of gaze following and its relation to language. Developmental Science, 8, 535–543.DOI logoGoogle Scholar
Carpenter, M., Nagell, K., & Tomasello, M.
(1998) Social cognition, joint attention, and communicative competence from 9 to 15 months of age. Monographs of the Society for Research in Child Development, 63, 1–174. DOI logoGoogle Scholar
Chambers, C. G., Tanenhaus, M. K., & Magnuson, J. S.
(2004) Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 687–696.DOI logoGoogle Scholar
Conboy, B. T., Brooks, R., Meltzoff, A. N., & Kuhl, P. K.
(2015) Social Interaction in Infants’ Learning of Second-Language Phonetics: An Exploration of Brain-Behavior Relations, Developmental Neuropsychology, 40, 216–229.DOI logoGoogle Scholar
Cooper, R. M.
(1974) The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84–107.DOI logoGoogle Scholar
Dahan, D., & Tanenhaus, M.
(2005) Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychonomic Bulletin & Review, 12, 453–459.DOI logoGoogle Scholar
Doherty-Sneddon, G., & Phelps, F. G.
(2005) Gaze aversion: A response to cognitive or social difficulty? Memory & Cognition, 33, 727–733.DOI logoGoogle Scholar
(2007) Teachers’ responses to children’s eye gaze. Educational Psychology, 27, 93–109.DOI logoGoogle Scholar
Driver, J., Davis, G., Ricciardelli, P., Kidd, P., Maxwell, E., & Baron-Cohen, S.
(1999) Gaze perception triggers reflexive visuospatial orienting. Visual Cognition, 6, 509–540.DOI logoGoogle Scholar
Farroni, T., Csibra, G., Simion, F., & Johnson, M. H.
(2002) Eye contact detection in humans from birth. Proceedings of the National Academy of Sciences of the United States of America, 99, 9602–9605. DOI logoGoogle Scholar
Griffin, Z. M., & Bock, K.
(2000) What the eyes say about speaking. Psychological Science, 11, 274–279.DOI logoGoogle Scholar
Garrod, S., & Pickering, M. J.
(2009) Joint action, interactive alignment, and dialog. Topics in Cognitive Science, 1, 292–304.DOI logoGoogle Scholar
Glenberg, A. M., Shroeder, J. L., & Robertson, D. A.
(1998) Averting the gaze disengages the environment and facilitates remembering. Memory & Cognition, 26, 651–658.DOI logoGoogle Scholar
Gleitman. L. R.
(1990) The structural sources of verb meaning. Language Acquisition 1, 3–55.DOI logoGoogle Scholar
Hanna, J., and Brennan, S.
(2007) Speaker’s eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language, 57, 596–615.DOI logoGoogle Scholar
Huettig, F., Rommers, J., & Meyer, A. S.
(2011) Using the visual world paradigm to study language processing: A review and critical evaluation. Acta psychologica, 137, 151–171.DOI logoGoogle Scholar
Knoeferle, P., Crocker, M. W., Scheepers, C., & Pickering, M. J.
(2005) The influence of the immediate visual context on incremental thematic role-assignment: evidence from eye- movements in depicted events. Cognition, 95, 95–127.DOI logoGoogle Scholar
Knoeferle, P., & Crocker, M. W.
(2007) The influence of recent scene events on spoken comprehension: evidence from eye movements. Journal of Memory and Language, 57, 519–543.DOI logoGoogle Scholar
(2006) The coordinated interplay of scene, utterance, and world knowledge: evidence from eye-tracking. Cognitive Science, 30, 481–529.DOI logoGoogle Scholar
Knoeferle, P. & Guerra, E.
(2012) What’s non-linguistic visual context? – A view from language comprehension. In: Rita Finkbeiner, Jörg Meibauer, & Petra Schuhmacher (eds) What is a context? Linguistic approaches and challenges (pp.129–149). Amsterdam: John Benjamins.DOI logoGoogle Scholar
(2016) Visually situated language comprehension. Language and Linguistics Compass, 10, 66–82.DOI logoGoogle Scholar
Knoeferle, P., Habets, B., Crocker, M. W., Muente, T. F.
(2008) Visual scenes trigger immediate syntactic reanalysis: evidence from ERPs during situated spoken comprehension. Cerebral Cortex, 18, 789–795.DOI logoGoogle Scholar
Knoeferle, P. & Kreysa, H.
(2012) Effects of speaker gaze on syntactic structuring. Frontiers in Psychology, 2:376. DOI logo.Google Scholar
Knoeferle, P., Urbach, T, & Kutas, M
(2011) Comprehending visual context influences on incremental sentence comprehension: insights from ERPs and picture-sentence verification. Psychophysiology, 48, 495–506.DOI logoGoogle Scholar
Knoeferle, P., Urbach, T., & Kutas, M.
(2014) Different mechanisms for role relations versus verb-action congruence effects: Evidence from ERPs in picture-sentence verification. Acta Psychologica, 152, 133–148.DOI logoGoogle Scholar
Kreysa, H., & Knoeferle, P.
(2011) Effects of speaker gaze on spoken language comprehension: Task matters. In: L. Carlson, C. Hölscher & T.F. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp.1557–1562), Austin, TX: The Cognitive Science Society.Google Scholar
Kreysa, H., Knoeferle, P., & Nunnemann, E.
(2014) Effects of speaker gaze versus depicted actions on visual attention during sentence comprehension. In: Paul Bello, Marcello Guarini, Marjorie McShane & Brian Scassellati (Eds). Proceedings of the 36th Annual Meeting of the Cognitive Science Society, (pp.2513–2518). Austin, TX: Cognitive Science Society.Google Scholar
Kreysa, H., Nunnemann, E., & Knoeferle, P.
(2018) Distinct effects of different visual cues on sentence comprehension and later recall: The case of speaker gaze versus depicted actions. Acta Psychologica. DOI logoGoogle Scholar
Kreysa, H., & Pickering M. J.
(2011) Eye movements in dialogue. In S. P. Liversedge, I. D. Gilchrist, & S. Everling (Eds.), The Oxford Handbook of Eye Movements (p.943–959). Oxford: OUP.Google Scholar
Kuhl, P. K., Tsao, F. -M., & Liu, H. -M.
(2003) Foreign-language experience in infancy: Effects of short-term exposure and social interaction on phonetic learning. Proceedings of the National Academy of Sciences, 100, 9096–9101.DOI logoGoogle Scholar
Kuhl, P. K.
(2007) Is speech learning “gated” by the social brain? Developmental Science, 10, 110–120.DOI logoGoogle Scholar
(2011) Social mechanisms in early language acquisition: Understanding integrated brain systems supporting language. In J. Decety & J. Cacioppo (Eds.), The Oxford handbook of social neuroscience (pp.649–667). Oxford, UK: Oxford University Press.Google Scholar
Langton, S. R. H. and Bruce, V.
(2000) You must see the point: automatic processing of cues to the direction of social attention. Journal of Experimental Psychology: HPP, 26, 747–757.Google Scholar
Langton, S., Watt, R. J., & Bruce, V.
(2000) Do the eyes have it? Cues to the direction of social attention. Trends in Cognitive Sciences, 4, 50–59.DOI logoGoogle Scholar
Macdonald, R. G., & Tatler, B. W.
(2013) Do as eye say: Gaze cueing and language in a real-world social interaction. Journal of Vision, 13, 1–12.DOI logoGoogle Scholar
Meyer, A. S., Sleiderink, A. M., & Levelt, W. J. M.
(1998) Viewing and naming objects: Eye movements during noun phrase production. Cognition, 66, B25–B33.DOI logoGoogle Scholar
Münster K. and Knoeferle P.
(2018) Extending Situated Language Comprehension (Accounts) with Speaker and Comprehender Characteristics: Toward Socially Situated Interpretation. Frontiers in Psychology. 8: 2267. DOI logo.Google Scholar
Nappa, R., Wessel, A., McEldoon, K. L., Gleitman, L., & Trueswell, J. C.
(2009) Use of Speaker’s Gaze and Syntax in Verb Learning. Language Learning and Development, 5, 203–234.DOI logoGoogle Scholar
Neider, M. B., Chen, X., Dickinson, C. A., Brennan, S., & Zelinsky, G.
(2010) Coordinating spatial referencing using shared gaze. Psychological Bulletin & Review, 17, 718–724.DOI logoGoogle Scholar
Phillips A. T., Wellman H. M., & Spelke E. S.
(2002) Infants’ ability to connect gaze and emotional expression to intentional action. Cognition, 85, 53–78.DOI logoGoogle Scholar
Pickering, M. J., & Garrod, S.
(2004) Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27, 169–225.DOI logoGoogle Scholar
Pickering, M., & Garrod, S.
(2007) Do people use language production to make predictions during comprehension? Trends in Cognitive Sciences, 11, 105–110.DOI logoGoogle Scholar
Pickering, M. J., & Garrod, S.
(2009) Language, interaction and embodiment. European Journal of Social Psychology, 39, 1178–1179.DOI logoGoogle Scholar
Ricciardelli, P., Bricolo, E., Aglioti, S., & Chelazzi, L.
(2002) My eyes want to look where your eyes are looking: Exploring the tendency to imitate another individual’s gaze. NeuroReport, 13, 2259–2264.DOI logoGoogle Scholar
Rosenthal Rollins, P., & Snow, C.
(1998) Shared attention and grammatical development in typical children and children with autism. Journal of Child Language, 25, 653–673.DOI logoGoogle Scholar
Samson, D., Apperly, I. A., Braithwaite, J. J., Andrews, B. J., & Bodley Scott, S. E.
(2010) Seeing it their way: evidence for rapid and involuntary computation of what other people see. Journal of Experimental Psychology: Human Perception and Performance, 36, 1255–1266.Google Scholar
Santiesteban, I., Catmur, C., Hopkins, S. C., Bird, G., & Heyes, C.
(2014) Avatars and arrows: Implicit mentalizing or domain-general processing? Journal of Experimental Psychology: Human Perception and Performance, 40, 929–937.DOI logoGoogle Scholar
Saxton, M.
(2010) Child Language: Acquisition and Development. London: Sage Publications.Google Scholar
Scaife, M., & Bruner, J. S.
(1975) The capacity for joint visual attention in the infant. Nature, 253, 265–266.DOI logoGoogle Scholar
Spivey, M., Tanenhaus, M., Eberhard, K., & Seidvy, J.
(2002) Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology, 45, 447–481.DOI logoGoogle Scholar
Staudte, M., and Crocker, M. W.
(2011) Investigating joint attention mechanisms through spoken human–robot interaction. Cognition 120, 268–291.DOI logoGoogle Scholar
Staudte, M., Crocker, M. W., Heloir, A., & Kipp, M.
(2014) The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts, Cognition, 133, 317–328.DOI logoGoogle Scholar
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K., & Sedivy, J. C.
(1995) Integration of visual and linguistic information in spoken language comprehension. Science, 268, 632–634.DOI logoGoogle Scholar
Tomasello, M.
(2003) Constructing a language: A usage-based theory of language acquisition. Cambridge, MA: Harvard University Press.Google Scholar
Tomasello, M., Carpenter, M., Call, J., Behne, T., & Moll, H.
(2005) Understanding and sharing intentions: The origins of cultural cognition. Behavioral and Brain Sciences, 28, 675–691.DOI logoGoogle Scholar
Tomasello, M., & Farrar, J.
(1986) Joint attention and early language. Child Development, 57, 1454–1463.DOI logoGoogle Scholar
Tomasello, M. & Todd, J.
(1983) Joint attention and lexical acquisition style. First Language, 4, 197–211. DOI logoGoogle Scholar
Trueswell, J. C., Sekerina, I., Hill, N. M., & Logrip, M. L.
(1999) The kindergarten-path effect: Studying on-line sentence processing in young children. Cognition, 73, 89–13.DOI logoGoogle Scholar
Woodward, A., & Sommerville, J.
(2000) Twelve-month-old infants interpret actions in context. Psychological Science, 11, 73–77.DOI logoGoogle Scholar
Woodward, A. L.
(2003) Infants’ developing understanding of the link between looker and object. Developmental Science, 6, 297–311.DOI logoGoogle Scholar
Woodward, A.
(1998) Infants selectively encode the goal object of an actor’s reach. Cognition, 69, 1–34.DOI logoGoogle Scholar
Wu, R., Gopnik, A., Richardson, D. C., & Kirkham, N. Z.
(2011) Infants learn about objects from statistics and people. Developmental Psychology, 47, 1220–1229.DOI logoGoogle Scholar
Yu, C., Ballard, D. H., & Aslin, R. N.
(2005) The role of embodied intention in early lexical acquisition. Cognitive Science, 29, 961–1005.DOI logoGoogle Scholar
Zhang, Lu & Knoeferle, P.
(2012) Visual Context Effects on Thematic Role Assignment in Children versus Adults: Evidence from Eye Tracking in German. In: Naomi Miyake, David Peebles, & Richard P. Cooper, Proceedings of the Annual Meeting of the Cognitive Science Society (pp.2593–2598), Boston, USA: The Cognitive Science Society.Google Scholar