Chapter 3
Effects of a speaker’s gaze on language comprehension and acquisition
This chapter discusses the role of a speaker’s gaze on a listener’s language processing, recall of information, and on child language learning. Speaker gaze facilitates performance in all of these domains, which suggests that it plays an important role in communication. Indeed, the findings indicate that speaker gaze can facilitate not just referential but also compositional processes such as syntactic structuring and thematic role assignment in listeners. Speaker gaze even guided comprehenders’ attention to a target object more rapidly than other aspects of the visual context (action event depictions); but unlike the action depictions, it had no beneficial effects on comprehenders’ post-experiment memory in socially non-interactive settings (when comprehenders inspect the speaker on a computer display). In socially interactive settings, however, speaker gaze seemed to affect memory in that joint object-based attention between a caregiver and child was beneficial for language learning. We conclude that speaker gaze effects are important for both language processing and learning, and are potentially boosted in a socially interactive context.
Article outline
- 1.Introduction
- 2.The role of speaker eye gaze: Syntactic structuring and thematic role assignment
- 3.Immediate attention and later recall: Speaker gaze vs. action depictions
- 4.Speaker gaze and joint attention in child language processing and learning
- 5.The role of speaker gaze: Social and pragmatic considerations
-
Acknowledgements
-
References
References
Baldwin, D.
(
1993a)
Infants’ ability to consult the speaker for clues to word reference.
Journal of Child Language, 2, 395–418.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Baldwin, D.
(
1993b)
Early referential understanding: Young children’s ability to recognize referential acts for what they are.
Developmental Psychology, 29, 1–12.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Baldwin, D. A.
(
2000)
Interpersonal understanding fuels knowledge acquisition.
Current Directions in Psychological Science, 9, 40–45.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brooks, R., & Meltzoff, A. N.
(
2005)
The development of gaze following and its relation to language.
Developmental Science, 8, 535–543.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Carpenter, M., Nagell, K., & Tomasello, M.
(
1998)
Social cognition, joint attention, and communicative competence from 9 to 15 months of age.
Monographs of the Society for Research in Child Development, 63, 1–174.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Chambers, C. G., Tanenhaus, M. K., & Magnuson, J. S.
(
2004)
Actions and affordances in syntactic ambiguity resolution.
Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 687–696.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Conboy, B. T., Brooks, R., Meltzoff, A. N., & Kuhl, P. K.
(
2015)
Social Interaction in Infants’ Learning of Second-Language Phonetics: An Exploration of Brain-Behavior Relations,
Developmental Neuropsychology, 40, 216–229.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cooper, R. M.
(
1974)
The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing.
Cognitive Psychology, 6, 84–107.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Dahan, D., & Tanenhaus, M.
(
2005)
Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition.
Psychonomic Bulletin & Review, 12, 453–459.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Doherty-Sneddon, G., & Phelps, F. G.
(
2005)
Gaze aversion: A response to cognitive or social difficulty? Memory & Cognition, 33, 727–733.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Doherty-Sneddon, G., & Phelps, F. G.
(
2007)
Teachers’ responses to children’s eye gaze.
Educational Psychology, 27, 93–109.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Driver, J., Davis, G., Ricciardelli, P., Kidd, P., Maxwell, E., & Baron-Cohen, S.
(
1999)
Gaze perception triggers reflexive visuospatial orienting.
Visual Cognition, 6, 509–540.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Farroni, T., Csibra, G., Simion, F., & Johnson, M. H.
(
2002)
Eye contact detection in humans from birth.
Proceedings of the National Academy of Sciences of the United States of America, 99, 9602–9605.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Griffin, Z. M., & Bock, K.
(
2000)
What the eyes say about speaking.
Psychological Science, 11, 274–279.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Garrod, S., & Pickering, M. J.
(
2009)
Joint action, interactive alignment, and dialog.
Topics in Cognitive Science, 1, 292–304.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Glenberg, A. M., Shroeder, J. L., & Robertson, D. A.
(
1998)
Averting the gaze disengages the environment and facilitates remembering.
Memory & Cognition, 26, 651–658.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Gleitman. L. R.
(
1990)
The structural sources of verb meaning.
Language Acquisition 1, 3–55.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hanna, J., and Brennan, S.
(
2007)
Speaker’s eye gaze disambiguates referring expressions early during face-to-face conversation.
Journal of Memory and Language, 57, 596–615.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Huettig, F., Rommers, J., & Meyer, A. S.
(
2011)
Using the visual world paradigm to study language processing: A review and critical evaluation.
Acta psychologica, 137, 151–171.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Knoeferle, P., Crocker, M. W., Scheepers, C., & Pickering, M. J.
(
2005)
The influence of the immediate visual context on incremental thematic role-assignment: evidence from eye- movements in depicted events.
Cognition, 95, 95–127.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Knoeferle, P., & Crocker, M. W.
(
2007)
The influence of recent scene events on spoken comprehension: evidence from eye movements.
Journal of Memory and Language, 57, 519–543.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Knoeferle, P., & Crocker, M. W.
(
2006)
The coordinated interplay of scene, utterance, and world knowledge: evidence from eye-tracking.
Cognitive Science, 30, 481–529.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Knoeferle, P. & Guerra, E.
Knoeferle, P. & Guerra, E.
(
2016)
Visually situated language comprehension.
Language and Linguistics Compass, 10, 66–82.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Knoeferle, P., Habets, B., Crocker, M. W., Muente, T. F.
(
2008)
Visual scenes trigger immediate syntactic reanalysis: evidence from ERPs during situated spoken comprehension.
Cerebral Cortex, 18, 789–795.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Knoeferle, P. & Kreysa, H.
(
2012)
Effects of speaker gaze on syntactic structuring.
Frontiers in Psychology, 2:376.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Knoeferle, P., Urbach, T, & Kutas, M
(
2011)
Comprehending visual context influences on incremental sentence comprehension: insights from ERPs and picture-sentence verification.
Psychophysiology, 48, 495–506.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Knoeferle, P., Urbach, T., & Kutas, M.
(
2014)
Different mechanisms for role relations versus verb-action congruence effects: Evidence from ERPs in picture-sentence verification.
Acta Psychologica, 152, 133–148.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kreysa, H., & Knoeferle, P.
(
2011)
Effects of speaker gaze on spoken language comprehension: Task matters. In:
L. Carlson,
C. Hölscher &
T.F. Shipley (Eds.),
Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp.1557–1562), Austin, TX: The Cognitive Science Society.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kreysa, H., Knoeferle, P., & Nunnemann, E.
(
2014)
Effects of speaker gaze versus depicted actions on visual attention during sentence comprehension. In:
Paul Bello,
Marcello Guarini,
Marjorie McShane &
Brian Scassellati (Eds).
Proceedings of the 36th Annual Meeting of the Cognitive Science Society, (pp.2513–2518). Austin, TX: Cognitive Science Society.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kreysa, H., Nunnemann, E., & Knoeferle, P.
(
2018)
Distinct effects of different visual cues on sentence comprehension and later recall: The case of speaker gaze versus depicted actions.
Acta Psychologica.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kreysa, H., & Pickering M. J.
(
2011)
Eye movements in dialogue. In
S. P. Liversedge,
I. D. Gilchrist, &
S. Everling (Eds.),
The Oxford Handbook of Eye Movements (p.943–959). Oxford: OUP.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kuhl, P. K., Tsao, F. -M., & Liu, H. -M.
(
2003)
Foreign-language experience in infancy: Effects of short-term exposure and social interaction on phonetic learning.
Proceedings of the National Academy of Sciences, 100, 9096–9101.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kuhl, P. K.
(
2007)
Is speech learning “gated” by the social brain? Developmental Science, 10, 110–120.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kuhl, P. K.
(
2011)
Social mechanisms in early language acquisition: Understanding integrated brain systems supporting language. In
J. Decety &
J. Cacioppo (Eds.),
The Oxford handbook of social neuroscience (pp.649–667). Oxford, UK: Oxford University Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Langton, S. R. H. and Bruce, V.
(
2000)
You must see the point: automatic processing of cues to the direction of social attention.
Journal of Experimental Psychology: HPP, 26, 747–757.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Langton, S., Watt, R. J., & Bruce, V.
(
2000)
Do the eyes have it? Cues to the direction of social attention.
Trends in Cognitive Sciences, 4, 50–59.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Macdonald, R. G., & Tatler, B. W.
(
2013)
Do as eye say: Gaze cueing and language in a real-world social interaction.
Journal of Vision, 13, 1–12.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Meyer, A. S., Sleiderink, A. M., & Levelt, W. J. M.
(
1998)
Viewing and naming objects: Eye movements during noun phrase production.
Cognition, 66, B25–B33.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Münster K. and Knoeferle P.
(
2018)
Extending Situated Language Comprehension (Accounts) with Speaker and Comprehender Characteristics: Toward Socially Situated Interpretation.
Frontiers in Psychology. 8: 2267.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Nappa, R., Wessel, A., McEldoon, K. L., Gleitman, L., & Trueswell, J. C.
(
2009)
Use of Speaker’s Gaze and Syntax in Verb Learning.
Language Learning and Development, 5, 203–234.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Neider, M. B., Chen, X., Dickinson, C. A., Brennan, S., & Zelinsky, G.
(
2010)
Coordinating spatial referencing using shared gaze.
Psychological Bulletin & Review, 17, 718–724.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Phillips A. T., Wellman H. M., & Spelke E. S.
(
2002)
Infants’ ability to connect gaze and emotional expression to intentional action.
Cognition, 85, 53–78.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pickering, M. J., & Garrod, S.
(
2004)
Toward a mechanistic psychology of dialogue.
Behavioral and Brain Sciences, 27, 169–225.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pickering, M., & Garrod, S.
(
2007)
Do people use language production to make predictions during comprehension? Trends in Cognitive Sciences, 11, 105–110.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pickering, M. J., & Garrod, S.
(
2009)
Language, interaction and embodiment.
European Journal of Social Psychology, 39, 1178–1179.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Ricciardelli, P., Bricolo, E., Aglioti, S., & Chelazzi, L.
(
2002)
My eyes want to look where your eyes are looking: Exploring the tendency to imitate another individual’s gaze.
NeuroReport, 13, 2259–2264.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Rosenthal Rollins, P., & Snow, C.
(
1998)
Shared attention and grammatical development in typical children and children with autism.
Journal of Child Language, 25, 653–673.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Samson, D., Apperly, I. A., Braithwaite, J. J., Andrews, B. J., & Bodley Scott, S. E.
(
2010)
Seeing it their way: evidence for rapid and involuntary computation of what other people see.
Journal of Experimental Psychology: Human Perception and Performance, 36, 1255–1266.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Santiesteban, I., Catmur, C., Hopkins, S. C., Bird, G., & Heyes, C.
(
2014)
Avatars and arrows: Implicit mentalizing or domain-general processing? Journal of Experimental Psychology: Human Perception and Performance, 40, 929–937.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Saxton, M.
(
2010)
Child Language: Acquisition and Development. London: Sage Publications.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Scaife, M., & Bruner, J. S.
(
1975)
The capacity for joint visual attention in the infant.
Nature, 253, 265–266.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Spivey, M., Tanenhaus, M., Eberhard, K., & Seidvy, J.
(
2002)
Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution.
Cognitive Psychology, 45, 447–481.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Staudte, M., and Crocker, M. W.
(
2011)
Investigating joint attention mechanisms through spoken human–robot interaction.
Cognition 120, 268–291.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Staudte, M., Crocker, M. W., Heloir, A., & Kipp, M.
(
2014)
The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts,
Cognition, 133, 317–328.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K., & Sedivy, J. C.
(
1995)
Integration of visual and linguistic information in spoken language comprehension.
Science, 268, 632–634.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tomasello, M.
(
2003)
Constructing a language: A usage-based theory of language acquisition. Cambridge, MA: Harvard University Press.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tomasello, M., Carpenter, M., Call, J., Behne, T., & Moll, H.
(
2005)
Understanding and sharing intentions: The origins of cultural cognition.
Behavioral and Brain Sciences, 28, 675–691.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tomasello, M., & Farrar, J.
(
1986)
Joint attention and early language.
Child Development, 57, 1454–1463.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Tomasello, M. & Todd, J.
(
1983)
Joint attention and lexical acquisition style.
First Language, 4, 197–211.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Trueswell, J. C., Sekerina, I., Hill, N. M., & Logrip, M. L.
(
1999)
The kindergarten-path effect: Studying on-line sentence processing in young children.
Cognition, 73, 89–13.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Woodward, A., & Sommerville, J.
(
2000)
Twelve-month-old infants interpret actions in context.
Psychological Science, 11, 73–77.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Woodward, A. L.
(
2003)
Infants’ developing understanding of the link between looker and object.
Developmental Science, 6, 297–311.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Woodward, A.
(
1998)
Infants selectively encode the goal object of an actor’s reach.
Cognition, 69, 1–34.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Wu, R., Gopnik, A., Richardson, D. C., & Kirkham, N. Z.
(
2011)
Infants learn about objects from statistics and people.
Developmental Psychology, 47, 1220–1229.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Yu, C., Ballard, D. H., & Aslin, R. N.
(
2005)
The role of embodied intention in early lexical acquisition.
Cognitive Science, 29, 961–1005.
![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Zhang, Lu & Knoeferle, P.
(
2012)
Visual Context Effects on Thematic Role Assignment in Children versus Adults: Evidence from Eye Tracking in German. In:
Naomi Miyake,
David Peebles, &
Richard P. Cooper,
Proceedings of the Annual Meeting of the Cognitive Science Society (pp.2593–2598), Boston, USA: The Cognitive Science Society.
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cited by
Cited by 1 other publications
Janjić, Paula, Gozdem Arikan, Harmen B. Gudde, Joseph J.C. Murphy, Laksha Sivaram & Kenny R. Coventry
2024.
The role of pointing gestures and eye gaze in second language vocabulary learning.
Discourse Processes ► pp. 1 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
This list is based on CrossRef data as of 26 june 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.