Characterising visual context effects
Active, pervasive, but resource-limited
In this chapter, I will review recent research on visually situated language comprehension, and in doing so identify key characteristics of situated language comprehension. More specifically I will argue that both active visual context effects and the temporally coordinated interplay between visual attention and language comprehension are characteristic of situated comprehension, and are robust across a broad range of comprehension situations, spanning (a) different comprehension modalities (reading and spoken comprehension) and situations in which language is (versus isn’t) in accord with visual context; (b) different kinds of visual contexts (clipart depictions, photographs, and real-world objects and events); (c) speaker-based information such as eye-gaze and gestures; and (d) both concrete and abstract language. Because of their broad coverage ((a)–(d)), situated language comprehension paradigms are, in principle, well suited for developing a relatively comprehensive theory of situated language comprehension. One challenge in further specifying model predictions is the development of more detailed linking hypotheses between comprehension processes and one of the key measures used to examine situated comprehension (visual attention to objects across time).
References
Abashidze, D., Carminati, M.N., & Knoeferle, P
(
2014)
How robust is the recent-event preference? In
P. Bello,
M. Guarini,
M. McShane, &
B. Scassellati (Eds.),
Proceedings of the 36th Annual Meeting of the Cognitive Science Society (pp. 92-97). Cognitive Science Society.

Abashidze, D., Knoeferle, P., Carminati, M.N., & Essig, K
(
2011)
The role of recent real-world versus future events in the comprehension of referentially ambiguous sentences: Evidence from eye tracking. In
B. Kokinov,
A. Karmiloff-Smith, &
N.J. Nersessian (Eds.),
Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press.

Allopenna, P., Magnuson, J., & Tanenhaus, M
(
1998)
Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models.
Journal of Memory & Language, 38, 419-439.


Altmann, G.T.M
(
2004)
Language-mediated eye-movements in the absence of a visual world: The ‘blank screen paradigm’.
Cognition, 93, B79-B87.


Altmann, G.T.M., & Kamide, Y
(
1999)
Incremental interpretation at verbs: Restricting the domain of subsequent reference.
Cognition, 73, 247-264.


Altmann, G.T.M., & Kamide, Y
(
2004)
Now you see it, now you don’t: Mediating the mapping between language and the visual world. In
J. Henderson &
F. Ferreira (Eds.),
The integration of language, vision and action (pp. 347-386). New York: Psychology Press.

Altmann, G.T.M., & Kamide, Y
(
2007)
The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing.
Journal of Memory and Language, 57, 502-518.


Altmann, G.T.M., & Kamide, Y
(
2009)
Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation.
Cognition, 111, 55-71.


Andersson, R., Ferreira, F., & Henderson, J.M
(
2011)
I see what you’re saying: The integration of complex speech and scenes during language comprehension.
Acta Psychologica, 137, 208-216.


Arai, M., van Gompel, R., & Scheepers, C
(
2007)
Priming ditransitive structures in comprehension.
Cognitive Psychology, 54, 218-250.


Arbib, M
(
2005)
From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics.
Behavioral and Brain Sciences, 28, 105-167.


Barsalou, L.W
(
1999)
Perceptual and symbol systems.
Behavioral and Brain Sciences, 22, 577-609.

Brennan, S., Chen, X., Dickinson, C., Neider, M., & Zelinsky, G
(
2007)
Coordinating cognition: The costs and benefits of shared gaze during collaborative search.
Cognition, 106, 1465-1477.


Burigo, M., & Knoeferle, P
(
2011)
Visual attention during spatial language comprehension: Reference alone isn’t enough. In
L. Carlson,
C. Holscher, &
T. Shipley (Eds.),
Proceedings of the 33rd Annual Conference of the Cognitive Science Society. Cognitive Science Society.

Carminati, M.N., & Knoeferle, P
(
2013)
Effects of speaker emotional facial expression and listener age on incremental sentence processing.
PLoS ONE, 8, e72559.


Carstensen, L.L., Fung, H.H., & Charles, S.T
(
2003)
Socioemotional selectivity theory and the regulation of emotion in the second half of life.
Motivation and Emotion, 27, 103-123.


Casasanto, C
(
2008)
Similarity and proximity: When does close in space mean close in mind? Memory & Cognition, 36, 1047-1056.


Castelhano, M., Mack, M., & Henderson, J
(
2009)
Viewing task influences eye movement control during active scene perception.
Journal of Vision, 9(3), 1-15.


Chambers, C.G., & San Juan, V
(
2008)
Perception and presupposition in real-time language comprehension: Insights from anticipatory processing.
Cognition, 108, 26-50.


Chambers, C.G., Tanenhaus, M.K., Filip, H., & Carlson, G.N
(
2002)
Circumscribing referential domains during real time language comprehension.
Journal of Memory and Language, 47, 30-49.


Chambers, C.G., Tanenhaus, M.K., & Magnuson, J.S
(
2004)
Actions and affordances in syntactic ambiguity resolution.
Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 687-696.


Cook, S.W., & Tanenhaus, M.K
(
2009)
Embodied communication: Speakers’ gestures affect listeners’ actions.
Cognition, 113, 98-104.


Cooper, R
(
1974)
The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing.
Cognitive Psychology, 6, 84-107.


Crocker, M.W., Knoeferle, P., & Mayberry, M
(
2010)
Situated sentence comprehension: The coordinated interplay account and a neurobehavioral model.
Brain and Language, 112, 189-201.


Dahan, D., & Tanenhaus, M
(
2005)
Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition.
Psychonomic Bulletin & Review, 12, 453-459.


Duñabeitia, J.A., Aviles, A., Afonso, O., Scheepers, C., & Carreiras, M
(
2008)
Qualitative differences in the representation of abstract versus concrete words: Evidence from the visual-world paradigm.
Cognition, 110, 284-292.


Engelhardt, P.E., Ferreira, F., & Patsenko, E.G
(
2010)
Pupillometry reveals processing load during spoken language comprehension.
Quarterly Journal of Experimental Psychology, 63, 639-664.


Ferreira, F., Foucart, A., & Engelhardt, P.E
(
2013)
Language processing in the visual world: Effects of preview, visual complexity, and prediction.
Journal of Memory and Language, 69, 165-182.


Fodor, J
(
1983)
Modularity of mind. Cambridge, MA: MIT Press.

Frazier, L., & Fodor, J.D
(
1978)
The sausage machine: A new two-stage parsing model.
Cognition, 6, 291-325.


Friederici, A.D
(
2002)
Towards a neural basis of auditory sentence processing.
Trends in Cognitive Sciences, 6, 78-84.


Ganis, G., Kutas, M., & Sereno, M.I
(
1996)
The search for ”common sense”: An electrophysiological study of the comprehension of words and pictures in reading.
Journal of Cognitive Neuroscience, 8, 89-106.


Goolkasian, P
(
1996)
Picture-word differences in a sentence verification task.
Memory & Cognition, 24, 584-594.


Griffin, Z., & Bock, K
(
2000)
What the eyes say about speaking.
Psychological Science, 11, 274-279.


Guerra, E., & Knoeferle, P
(
2014)
Effects of object distance on incremental semantic interpretation: Similarity is closeness.
Cognition, 133, 535-552.


Habets, B., Kita, S., Shao, Z., Ozyurek, A., & Hagoort, P
(
2011)
The role of synchrony and ambiguity in speech-gesture integration during comprehension.
Journal of Cognitive Neuroscience, 23, 1845-1854.


Hagoort, P., Brown, C., & Groothusen, J
(
1993)
The syntactic positive shift (sps) as an ERP measure of syntactic processing.
Language and Cognitive Processes, 8, 439-483.


Hanna, J., & Brennan, S
(
2007)
Speakers’ eye gaze disambiguates referring expressions early during face-to-face conversation.
Journal of Memory and Language, 57, 596-615.


Henderson, J.M., & Ferreira, F
(
2004)
Scene perception for psycholinguists. In
J.M. Henderson &
F. Ferreira (Eds.),
The interface of language, vision, and action: Eye movements and the visual world (pp. 1-58). New York: Psychology Press.

Huang, Y.T., & Snedeker, J
(
2009)
Online interpretation of scalar quantifiers: Insight into the semantics –pragmatics interface.
Cognitive Psychology, 58, 376-415.


Huettig, F., & Altmann, G.T.M
(
2005)
Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm.
Cognition, 96, 23-32.


Huettig, F., Rommers, J., & Meyer, A.S
(
2011)
Using the visual world paradigm to study language processing: A review and critical evaluation.
Acta Psychologica, 137, 151-171.


Isaacowitz, D.M., Allard, E.S., Murphy, N.A., & Schlangel, M
(
2009)
The time course of age-related preferences toward positive and negative stimuli.
The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 64B, 188-192.


Kelly, S.D., Barr, D.J., Church, R.B., & Lynch, K
(
1999)
Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory.
Journal of Memory and Language, 40, 577-592.


Kelly, S.D., Creigh, P., & Bartolotti, J
(
2010)
Integrating speech and iconic gestures in a stroop-like task: Evidence for automatic processing.
Journal of Cognitive Neuroscience, 22, 683-694.


Kelly, S.D., Ozyurek, A., & Maris, E
(
2010)
Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension.
Psychological Science, 21, 260-267.


Knoeferle, P
(
2007)
Comparing the time-course of processing initially ambiguous and unambiguous German SVO/OVS sentences in depicted events. In
R. van Gompel,
M. Fischer,
W. Murray, &
R. Hill (Eds.),
Eye movement research: Insights into mind and brain (pp. 517-533). Oxford: Elsevier.


Knoeferle, P
in pressa).
Language comprehension in rich non-linguistic contexts: Combining eye tracking and event-related brain potentials. In
Towards a cognitive neuroscience of natural language use. Cambridge: Cambridge University Press.

Knoeferle, P
in pressb).
Visually situated language comprehension in children and in adults. In
R.K. Mishra,
N. Srinivasan, &
F. Huettig (Eds.)
Attention and vision in language processing Springer Language and Cognition series

Knoeferle, P., Carminati, M.N., Abashidze, D., & Essig, K
(
2011)
Preferential inspection of recent real-world events over future events: Evidence from eye tracking during spoken sentence comprehension.
Frontiers in Psychology, 2, 376.


Knoeferle, P., & Crocker, M.W
(
2005)
Incremental effects of mismatch during picture-sentence integration: Evidence from eye-tracking. In
Proceedings of the 27th Annual Meeting of the Cognitive Science Conference (pp. 1166-1171). Mahwah, NJ: Erlbaum.

Knoeferle, P., & Crocker, M.W
(
2006)
The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking.
Cognitive Science, 30, 481-529.


Knoeferle, P., & Crocker, M.W
(
2007)
The influence of recent scene events on spoken comprehension: Evidence from eye-movements.
Journal of Memory and Language, 75, 519-543.


Knoeferle, P., Crocker, M.W., Scheepers, C., & Pickering, M.J
(
2005)
The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events.
Cognition, 95, 95-127.


Knoeferle, P., Habets, B., Crocker, M.W., & Munte, T.F
(
2008)
Visual scenes trigger immediate syntactic reanalysis: Evidence from ERPs during situated spoken comprehension.
Cerebral Cortex, 18, 789-795.


Knoeferle, P., & Kreysa, H
(
2012)
Effects of speaker gaze on syntactic structuring.
Frontiers in Psychology, 2, 376.


Knoeferle, P., Urbach, T.P., & Kutas, M
(
2011)
Comprehending how visual context influences incremental sentence comprehension: Insights from ERPs and picture-sentence verification.
Psychophysiology, 48, 495-506.


Knoeferle, P., Urbach, T., & Kutas, M
(
2014)
Different mechanisms for role relations versus verb-action congruence effects: Evidence from ERPs in picture-sentence verification.
Acta Psychologica, 152, 133-148.


Kolk, H., Chwilla, D., Van Herten, M., & Oor, P
(
2003)
Structure and limited capacity in verbal working memory: A study with event-related potentials.
Brain and Language, 85, 1-36.


Kreysa, H
(
2009)
Coordinating speech-related eye movements between comprehension and production. Unpublished doctoral dissertation. University of Edinburgh, UK.

Kreysa, H., & Knoeferle, P
(
2011a)
Effects of speaker gaze on spoken language comprehension: Task matters. In
L. Carlson,
C. Holscher, &
T. Shipley (Eds.),
Proceedings of the 33rd Annual Conference of the Cognitive Science Society. Cognitive Science Society.

Kreysa, H., & Knoeferle, P
(
2011b)
Peripheral speaker gaze facilitates spoken language comprehension: Syntactic structuring and thematic role assignment in German. In
B. Kokinov,
A. Karmiloff-Smith, &
N.J. Nersessian (Eds.),
Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press.

Kreysa, H., Knoeferle, P., & Nunnemann, E
(
2014)
Effects of speaker gaze versus depicted actions on visual attention during sentence comprehension. In
M. McShane,
B. Scassellati,
P. Bello, &
M. Guarini (Eds.),
Proceedings of the 36th Annual Conference of the Cognitive Science Society (pp. 2513-2518). Cognitive Science Society.

Kuperberg, G., Sitnikova, T., Caplan, D., & Holcomb, P
(
2003)
Electrophysiological distinctions in processing conceptual relationships within simple sentences.
Cognitive Brain Research, 17, 117-129.


Kutas, M., & Federmeier, K.D
(
2011)
Thirty years and counting: Finding meaning in the N400 component of the event related brain potential (ERP).
Annual Review of Psychology, 62, 621-647.


Kutas, M., & Hillyard, S.A
(
1980)
Reading senseless sentences: Brain potentials reflect semantic incongruity.
Science, 207, 203-205.


Kutas, M., & Hillyard, S.A
(
1984)
Brain potentials during reading reflect word expectancy and semantic association.
Nature, 307, 161-163.


Kutas, M., Van Petten, C., & Kluender, R
(
2006) In
M. Traxler &
M. Gerns-bacher (Eds.),
Handbook of psycholinguistics (2nd ed., pp. 659-724). New York: Elsevier.


Mayberry, M., Crocker, M.W., & Knoeferle, P
(
2009)
Learning to attend: A connectionist model of situated language comprehension.
Cognitive Science, 33, 449-496.


Nappa, R., & Arnold, J
(
2009)
Paying attention to intention: Effects of intention (but not egocentric attention) on pronoun resolution. In
Proceedings of the CUNY Conference
(p. 262).
Nation, K., & Altmann, C.M.M.G.T.M
(
2003)
Investigating individual differences in childrens real-time sentence comprehension using language-mediated eye movements.
Journal of Experimental Child Psychology, 86, 314-329.


Osterhout, L., & Holcomb, P.J
(
1992)
Event-related brain potentials elicited by syntactic anomaly.
Journal of Memory and Language, 31, 785-806.


Pulvermuller, F., Harle, M., & Hummel, F
(
2001)
Walking or talking?: Behavioral and neurophysiological correlates of action verb processing.
Brain and Language, 78, 143-168.


Richardson, D., & Matlock, T
(
2007)
The integration of figurative language and static depictions: An eye movement study of fictive motion.
Cognition, 102, 129-138.


Richardson, D.C., & Dale, R
(
2005)
Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension.
Cognitive Science, 29, 1045-1060.


Richardson, D.C., & Kirkham, N.Z
(
2004)
Multi-modal events and moving locations: Eye movements of adults and 6-month-olds reveal dynamic spatial indexing.
Journal of Experimental Psychology: General, 133, 46-62.


Salverda, A.P., Brown, M., & Tanenhaus, M.K
(
2011)
A goal-based perspective on eye movements in visual-world studies.
Acta Psychologica, 137, 172-180.


Scheepers, C., & Crocker, M.W
(
2004)
Constituent order priming from reading to listening: A visual-world study. In
M. Carreiras &
J.C. Clifton (Eds.),
The on-line study of sentence comprehension: Eyetracking, ERP, and beyond. United Kingdom: Psychology Press.

Sedivy, J.C., Tanenhaus, M.K., Chambers, C.G., & Carlson, G.N
(
1999)
Achieving incremental semantic interpretation through contextual representation.
Cognition, 71, 109-148.


Spivey, M.J., & Geng, J.J
(
2001)
Oculomotor mechanisms activated by imagery and memory: Eye movements to absent objects.
Psychological Research, 65, 235-241.


Spivey, M.J., Tanenhaus, M.K., Eberhard, K.M., & Sedivy, J.C
(
2002)
Eye-movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution.
Cognitive Psychology, 45, 447-481.


Staudte, M., & Crocker, M
(
2009)
The effect of robot gaze on processing robot utterances. In
N. Taatgen &
H. van Rijn (Eds.),
Proceedings of the Cognitive Science Conference (pp. 431–436). Cognitive Science Society, Inc.

Staudte, M., Crocker, M.W., Heloir, A., & Kipp, M
(
2014)
The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts.
Cognition, 133, 317-328.


Tanenhaus, M.K
(
2004)
That’s not it and "its" not "that": The role of conceptual composites in in-line reference resolution. In
M. Carreiras &
J. Charles Clifton (Eds.),
On-line sentence processing: ERPs, eye movements, and beyond (pp. 209-228). Psychology Press.

Tanenhaus, M.K., Magnuson, J.S., Dahan, D., & Chambers, C
(
2000)
Eye movements and lexical access in spoken- language comprehension: Evaluating a linking hypothesis between fixations and linguistic processing.
Journal of Psycholinguistic Research, 29, 557-580.


Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K., & Sedivy, J.C
(
1995)
Integration of visual and linguistic information in spoken language comprehension.
Science, 268, 632-634.


Tatler, B., Wade, N., Kwan, H., Findlay, J., & Velichkovsky, B
(
2010)
Yarbus, eye movements, and vision.
I-Perception, 1, 7-27.


Underwood, G., Jebbett, L., & Roberts, K
(
2004)
Inspecting pictures for information to verify a sentence: Eye movements in general encoding and in focused search.
The Quarterly Journal of Experimental Psychology, 56, 165-182.


Vissers, C., Kolk, H., van de Meerendonk, N., & Chwilla, D
(
2008)
Monitoring in language perception: Evidence from erps in a picture-sentence matching task.
Neuropsychologia, 967-982.


Wassenaar, M., & Hagoort, P
(
2007)
Thematic role assignment in patients with broca’s aphasia: Sentence-picture matching electrified.
Neuropsychologia, 45, 716-740.


Wu, Y.C., & Coulson, S
(
2005)
Meaningful gestures: Electrophysiological indices of iconic gesture comprehension.
Psychophysiology, 42, 654-667.


Yarbus, A.L
(
1967)
Eye movements and vision. New York: Plenum Press.


Zhang, L., & Knoeferle, P
(
2012)
Visual context effects on thematic role assignment in children versus adults: Evidence from eye tracking in German. In
R.P. Cooper,
N. Miyake, &
D. Peebles (Eds.),
Proceedings of the annual meeting of the cognitive science society (pp. 2593-2598). Boston, USA: The Cognitive Science Society.

Cited by
Cited by 1 other publications
Hernández-Gutiérrez, David, Francisco Muñoz, Jose Sánchez-García, Werner Sommer, Rasha Abdel Rahman, Pilar Casado, Laura Jiménez-Ortega, Javier Espuny, Sabela Fondevila & Manuel Martín-Loeches
2021.
Situating language in a minimal social context: how seeing a picture of the speaker’s face affects language comprehension.
Social Cognitive and Affective Neuroscience 16:5
► pp. 502 ff.

This list is based on CrossRef data as of 7 september 2023. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.