Cover not available
Part of
Visually Situated Language Comprehension
Edited by Pia Knoeferle, Pirita Pyykkönen-Klauck and Matthew W. Crocker
[Advances in Consciousness Research 93] 2016
► pp. 227–260

Characterising visual context effects

Active, pervasive, but resource-limited

ORCID logo

In this chapter, I will review recent research on visually situated language comprehension, and in doing so identify key characteristics of situated language comprehension. More specifically I will argue that both active visual context effects and the temporally coordinated interplay between visual attention and language comprehension are characteristic of situated comprehension, and are robust across a broad range of comprehension situations, spanning (a) different comprehension modalities (reading and spoken comprehension) and situations in which language is (versus isn’t) in accord with visual context; (b) different kinds of visual contexts (clipart depictions, photographs, and real-world objects and events); (c) speaker-based information such as eye-gaze and gestures; and (d) both concrete and abstract language. Because of their broad coverage ((a)–(d)), situated language comprehension paradigms are, in principle, well suited for developing a relatively comprehensive theory of situated language comprehension. One challenge in further specifying model predictions is the development of more detailed linking hypotheses between comprehension processes and one of the key measures used to examine situated comprehension (visual attention to objects across time).

Published online: 10 March 2016
DOI logo
https://doi.org/10.1075/aicr.93.09kno
References
Abashidze, D., Carminati, M.N., & Knoeferle, P
(2014) How robust is the recent-event preference? In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th Annual Meeting of the Cognitive Science Society (pp. 92-97). Cognitive Science Society.Google Scholar
Abashidze, D., Knoeferle, P., Carminati, M.N., & Essig, K
(2011) The role of recent real-world versus future events in the comprehension of referentially ambiguous sentences: Evidence from eye tracking. In B. Kokinov, A. Karmiloff-Smith, & N.J. Nersessian (Eds.), Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press.Google Scholar
Allopenna, P., Magnuson, J., & Tanenhaus, M
(1998) Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory & Language, 38, 419-439. DOI logoGoogle Scholar
Altmann, G.T.M
(2004) Language-mediated eye-movements in the absence of a visual world: The ‘blank screen paradigm’. Cognition, 93, B79-B87. DOI logoGoogle Scholar
Altmann, G.T.M., & Kamide, Y
(1999) Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73, 247-264. DOI logoGoogle Scholar
(2004) Now you see it, now you don’t: Mediating the mapping between language and the visual world. In J. Henderson & F. Ferreira (Eds.), The integration of language, vision and action (pp. 347-386). New York: Psychology Press.Google Scholar
(2007) The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language, 57, 502-518. DOI logoGoogle Scholar
(2009) Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation. Cognition, 111, 55-71. DOI logoGoogle Scholar
Andersson, R., Ferreira, F., & Henderson, J.M
(2011) I see what you’re saying: The integration of complex speech and scenes during language comprehension. Acta Psychologica, 137, 208-216. DOI logoGoogle Scholar
Arai, M., van Gompel, R., & Scheepers, C
(2007) Priming ditransitive structures in comprehension. Cognitive Psychology, 54, 218-250. DOI logoGoogle Scholar
Arbib, M
(2005) From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and Brain Sciences, 28, 105-167. DOI logoGoogle Scholar
Barsalou, L.W
(1999) Perceptual and symbol systems. Behavioral and Brain Sciences, 22, 577-609.Google Scholar
Brennan, S., Chen, X., Dickinson, C., Neider, M., & Zelinsky, G
(2007) Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition, 106, 1465-1477. DOI logoGoogle Scholar
Burigo, M., & Knoeferle, P
(2011) Visual attention during spatial language comprehension: Reference alone isn’t enough. In L. Carlson, C. Holscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society. Cognitive Science Society.Google Scholar
Carminati, M.N., & Knoeferle, P
(2013) Effects of speaker emotional facial expression and listener age on incremental sentence processing. PLoS ONE, 8, e72559. DOI logoGoogle Scholar
Carstensen, L.L., Fung, H.H., & Charles, S.T
(2003) Socioemotional selectivity theory and the regulation of emotion in the second half of life. Motivation and Emotion, 27, 103-123. DOI logoGoogle Scholar
Casasanto, C
(2008) Similarity and proximity: When does close in space mean close in mind? Memory & Cognition, 36, 1047-1056. DOI logoGoogle Scholar
Castelhano, M., Mack, M., & Henderson, J
(2009) Viewing task influences eye movement control during active scene perception. Journal of Vision, 9(3), 1-15. DOI logoGoogle Scholar
Chambers, C.G., & San Juan, V
(2008) Perception and presupposition in real-time language comprehension: Insights from anticipatory processing. Cognition, 108, 26-50. DOI logoGoogle Scholar
Chambers, C.G., Tanenhaus, M.K., Filip, H., & Carlson, G.N
(2002) Circumscribing referential domains during real time language comprehension. Journal of Memory and Language, 47, 30-49. DOI logoGoogle Scholar
Chambers, C.G., Tanenhaus, M.K., & Magnuson, J.S
(2004) Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 687-696. DOI logoGoogle Scholar
Cook, S.W., & Tanenhaus, M.K
(2009) Embodied communication: Speakers’ gestures affect listeners’ actions. Cognition, 113, 98-104. DOI logoGoogle Scholar
Cooper, R
(1974) The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84-107. DOI logoGoogle Scholar
Crocker, M.W., Knoeferle, P., & Mayberry, M
(2010) Situated sentence comprehension: The coordinated interplay account and a neurobehavioral model. Brain and Language, 112, 189-201. DOI logoGoogle Scholar
Dahan, D., & Tanenhaus, M
(2005) Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychonomic Bulletin & Review, 12, 453-459. DOI logoGoogle Scholar
Duñabeitia, J.A., Aviles, A., Afonso, O., Scheepers, C., & Carreiras, M
(2008) Qualitative differences in the representation of abstract versus concrete words: Evidence from the visual-world paradigm. Cognition, 110, 284-292. DOI logoGoogle Scholar
Engelhardt, P.E., Ferreira, F., & Patsenko, E.G
(2010) Pupillometry reveals processing load during spoken language comprehension. Quarterly Journal of Experimental Psychology, 63, 639-664. DOI logoGoogle Scholar
Ferreira, F., Foucart, A., & Engelhardt, P.E
(2013) Language processing in the visual world: Effects of preview, visual complexity, and prediction. Journal of Memory and Language, 69, 165-182. DOI logoGoogle Scholar
Fodor, J
(1983) Modularity of mind. Cambridge, MA: MIT Press.Google Scholar
Frazier, L., & Fodor, J.D
(1978) The sausage machine: A new two-stage parsing model. Cognition, 6, 291-325. DOI logoGoogle Scholar
Friederici, A.D
(2002) Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6, 78-84. DOI logoGoogle Scholar
Ganis, G., Kutas, M., & Sereno, M.I
(1996) The search for ”common sense”: An electrophysiological study of the comprehension of words and pictures in reading. Journal of Cognitive Neuroscience, 8, 89-106. DOI logoGoogle Scholar
Goolkasian, P
(1996) Picture-word differences in a sentence verification task. Memory & Cognition, 24, 584-594. DOI logoGoogle Scholar
Griffin, Z., & Bock, K
(2000) What the eyes say about speaking. Psychological Science, 11, 274-279. DOI logoGoogle Scholar
Guerra, E., & Knoeferle, P
(2014) Effects of object distance on incremental semantic interpretation: Similarity is closeness. Cognition, 133, 535-552. DOI logoGoogle Scholar
Habets, B., Kita, S., Shao, Z., Ozyurek, A., & Hagoort, P
(2011) The role of synchrony and ambiguity in speech-gesture integration during comprehension. Journal of Cognitive Neuroscience, 23, 1845-1854. DOI logoGoogle Scholar
Hagoort, P., Brown, C., & Groothusen, J
(1993) The syntactic positive shift (sps) as an ERP measure of syntactic processing. Language and Cognitive Processes, 8, 439-483. DOI logoGoogle Scholar
Hanna, J., & Brennan, S
(2007) Speakers’ eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language, 57, 596-615. DOI logoGoogle Scholar
Henderson, J.M., & Ferreira, F
(2004) Scene perception for psycholinguists. In J.M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (pp. 1-58). New York: Psychology Press.Google Scholar
Huang, Y.T., & Snedeker, J
(2009) Online interpretation of scalar quantifiers: Insight into the semantics –pragmatics interface. Cognitive Psychology, 58, 376-415. DOI logoGoogle Scholar
Huettig, F., & Altmann, G.T.M
(2005) Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition, 96, 23-32. DOI logoGoogle Scholar
Huettig, F., Rommers, J., & Meyer, A.S
(2011) Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica, 137, 151-171. DOI logoGoogle Scholar
Isaacowitz, D.M., Allard, E.S., Murphy, N.A., & Schlangel, M
(2009) The time course of age-related preferences toward positive and negative stimuli. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 64B, 188-192. DOI logoGoogle Scholar
Kelly, S.D., Barr, D.J., Church, R.B., & Lynch, K
(1999) Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Language, 40, 577-592. DOI logoGoogle Scholar
Kelly, S.D., Creigh, P., & Bartolotti, J
(2010) Integrating speech and iconic gestures in a stroop-like task: Evidence for automatic processing. Journal of Cognitive Neuroscience, 22, 683-694. DOI logoGoogle Scholar
Kelly, S.D., Ozyurek, A., & Maris, E
(2010) Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science, 21, 260-267. DOI logoGoogle Scholar
Knoeferle, P
(2007) Comparing the time-course of processing initially ambiguous and unambiguous German SVO/OVS sentences in depicted events. In R. van Gompel, M. Fischer, W. Murray, & R. Hill (Eds.), Eye movement research: Insights into mind and brain (pp. 517-533). Oxford: Elsevier. DOI logoGoogle Scholar
in pressa). Language comprehension in rich non-linguistic contexts: Combining eye tracking and event-related brain potentials. In Towards a cognitive neuroscience of natural language use. Cambridge: Cambridge University Press. DOI logo
in pressb). Visually situated language comprehension in children and in adults. In R.K. Mishra, N. Srinivasan, & F. Huettig (Eds.) Attention and vision in language processing Springer Language and Cognition series DOI logo
Knoeferle, P., Carminati, M.N., Abashidze, D., & Essig, K
(2011) Preferential inspection of recent real-world events over future events: Evidence from eye tracking during spoken sentence comprehension. Frontiers in Psychology, 2, 376. DOI logoGoogle Scholar
Knoeferle, P., & Crocker, M.W
(2005) Incremental effects of mismatch during picture-sentence integration: Evidence from eye-tracking. In Proceedings of the 27th Annual Meeting of the Cognitive Science Conference (pp. 1166-1171). Mahwah, NJ: Erlbaum.Google Scholar
(2006) The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking. Cognitive Science, 30, 481-529. DOI logoGoogle Scholar
(2007) The influence of recent scene events on spoken comprehension: Evidence from eye-movements. Journal of Memory and Language, 75, 519-543. DOI logoGoogle Scholar
Knoeferle, P., Crocker, M.W., Scheepers, C., & Pickering, M.J
(2005) The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition, 95, 95-127. DOI logoGoogle Scholar
Knoeferle, P., Habets, B., Crocker, M.W., & Munte, T.F
(2008) Visual scenes trigger immediate syntactic reanalysis: Evidence from ERPs during situated spoken comprehension. Cerebral Cortex, 18, 789-795. DOI logoGoogle Scholar
Knoeferle, P., & Kreysa, H
(2012) Effects of speaker gaze on syntactic structuring. Frontiers in Psychology, 2, 376. DOI logoGoogle Scholar
Knoeferle, P., Urbach, T.P., & Kutas, M
(2011) Comprehending how visual context influences incremental sentence comprehension: Insights from ERPs and picture-sentence verification. Psychophysiology, 48, 495-506. DOI logoGoogle Scholar
Knoeferle, P., Urbach, T., & Kutas, M
(2014) Different mechanisms for role relations versus verb-action congruence effects: Evidence from ERPs in picture-sentence verification. Acta Psychologica, 152, 133-148. DOI logoGoogle Scholar
Kolk, H., Chwilla, D., Van Herten, M., & Oor, P
(2003) Structure and limited capacity in verbal working memory: A study with event-related potentials. Brain and Language, 85, 1-36. DOI logoGoogle Scholar
Kreysa, H
(2009) Coordinating speech-related eye movements between comprehension and production. Unpublished doctoral dissertation. University of Edinburgh, UK.Google Scholar
Kreysa, H., & Knoeferle, P
(2011a) Effects of speaker gaze on spoken language comprehension: Task matters. In L. Carlson, C. Holscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society. Cognitive Science Society.Google Scholar
(2011b) Peripheral speaker gaze facilitates spoken language comprehension: Syntactic structuring and thematic role assignment in German. In B. Kokinov, A. Karmiloff-Smith, & N.J. Nersessian (Eds.), Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press.Google Scholar
Kreysa, H., Knoeferle, P., & Nunnemann, E
(2014) Effects of speaker gaze versus depicted actions on visual attention during sentence comprehension. In M. McShane, B. Scassellati, P. Bello, & M. Guarini (Eds.), Proceedings of the 36th Annual Conference of the Cognitive Science Society (pp. 2513-2518). Cognitive Science Society.Google Scholar
Kuperberg, G., Sitnikova, T., Caplan, D., & Holcomb, P
(2003) Electrophysiological distinctions in processing conceptual relationships within simple sentences. Cognitive Brain Research, 17, 117-129. DOI logoGoogle Scholar
Kutas, M., & Federmeier, K.D
(2011) Thirty years and counting: Finding meaning in the N400 component of the event related brain potential (ERP). Annual Review of Psychology, 62, 621-647. DOI logoGoogle Scholar
Kutas, M., & Hillyard, S.A
(1980) Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207, 203-205. DOI logoGoogle Scholar
(1984) Brain potentials during reading reflect word expectancy and semantic association. Nature, 307, 161-163. DOI logoGoogle Scholar
Kutas, M., Van Petten, C., & Kluender, R
(2006) In M. Traxler & M. Gerns-bacher (Eds.), Handbook of psycholinguistics (2nd ed., pp. 659-724). New York: Elsevier. DOI logoGoogle Scholar
Mayberry, M., Crocker, M.W., & Knoeferle, P
(2009) Learning to attend: A connectionist model of situated language comprehension. Cognitive Science, 33, 449-496. DOI logoGoogle Scholar
Nappa, R., & Arnold, J
(2009) Paying attention to intention: Effects of intention (but not egocentric attention) on pronoun resolution. In Proceedings of the CUNY Conference (p. 262).
Nation, K., & Altmann, C.M.M.G.T.M
(2003) Investigating individual differences in childrens real-time sentence comprehension using language-mediated eye movements. Journal of Experimental Child Psychology, 86, 314-329. DOI logoGoogle Scholar
Osterhout, L., & Holcomb, P.J
(1992) Event-related brain potentials elicited by syntactic anomaly. Journal of Memory and Language, 31, 785-806. DOI logoGoogle Scholar
Pulvermuller, F., Harle, M., & Hummel, F
(2001) Walking or talking?: Behavioral and neurophysiological correlates of action verb processing. Brain and Language, 78, 143-168. DOI logoGoogle Scholar
Richardson, D., & Matlock, T
(2007) The integration of figurative language and static depictions: An eye movement study of fictive motion. Cognition, 102, 129-138. DOI logoGoogle Scholar
Richardson, D.C., & Dale, R
(2005) Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cognitive Science, 29, 1045-1060. DOI logoGoogle Scholar
Richardson, D.C., & Kirkham, N.Z
(2004) Multi-modal events and moving locations: Eye movements of adults and 6-month-olds reveal dynamic spatial indexing. Journal of Experimental Psychology: General, 133, 46-62. DOI logoGoogle Scholar
Salverda, A.P., Brown, M., & Tanenhaus, M.K
(2011) A goal-based perspective on eye movements in visual-world studies. Acta Psychologica, 137, 172-180. DOI logoGoogle Scholar
Scheepers, C., & Crocker, M.W
(2004) Constituent order priming from reading to listening: A visual-world study. In M. Carreiras & J.C. Clifton (Eds.), The on-line study of sentence comprehension: Eyetracking, ERP, and beyond. United Kingdom: Psychology Press.Google Scholar
Sedivy, J.C., Tanenhaus, M.K., Chambers, C.G., & Carlson, G.N
(1999) Achieving incremental semantic interpretation through contextual representation. Cognition, 71, 109-148. DOI logoGoogle Scholar
Spivey, M.J., & Geng, J.J
(2001) Oculomotor mechanisms activated by imagery and memory: Eye movements to absent objects. Psychological Research, 65, 235-241. DOI logoGoogle Scholar
Spivey, M.J., Tanenhaus, M.K., Eberhard, K.M., & Sedivy, J.C
(2002) Eye-movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology, 45, 447-481. DOI logoGoogle Scholar
Staudte, M., & Crocker, M
(2009) The effect of robot gaze on processing robot utterances. In N. Taatgen & H. van Rijn (Eds.), Proceedings of the Cognitive Science Conference (pp. 431–436). Cognitive Science Society, Inc.Google Scholar
Staudte, M., Crocker, M.W., Heloir, A., & Kipp, M
(2014) The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts. Cognition, 133, 317-328. DOI logoGoogle Scholar
Tanenhaus, M.K
(2004) That’s not it and "its" not "that": The role of conceptual composites in in-line reference resolution. In M. Carreiras & J. Charles Clifton (Eds.), On-line sentence processing: ERPs, eye movements, and beyond (pp. 209-228). Psychology Press.Google Scholar
Tanenhaus, M.K., Magnuson, J.S., Dahan, D., & Chambers, C
(2000) Eye movements and lexical access in spoken- language comprehension: Evaluating a linking hypothesis between fixations and linguistic processing. Journal of Psycholinguistic Research, 29, 557-580. DOI logoGoogle Scholar
Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K., & Sedivy, J.C
(1995) Integration of visual and linguistic information in spoken language comprehension. Science, 268, 632-634. DOI logoGoogle Scholar
Tatler, B., Wade, N., Kwan, H., Findlay, J., & Velichkovsky, B
(2010) Yarbus, eye movements, and vision. I-Perception, 1, 7-27. DOI logoGoogle Scholar
Underwood, G., Jebbett, L., & Roberts, K
(2004) Inspecting pictures for information to verify a sentence: Eye movements in general encoding and in focused search. The Quarterly Journal of Experimental Psychology, 56, 165-182. DOI logoGoogle Scholar
Vissers, C., Kolk, H., van de Meerendonk, N., & Chwilla, D
(2008) Monitoring in language perception: Evidence from erps in a picture-sentence matching task. Neuropsychologia, 967-982. DOI logoGoogle Scholar
Wassenaar, M., & Hagoort, P
(2007) Thematic role assignment in patients with broca’s aphasia: Sentence-picture matching electrified. Neuropsychologia, 45, 716-740. DOI logoGoogle Scholar
Wu, Y.C., & Coulson, S
(2005) Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology, 42, 654-667. DOI logoGoogle Scholar
Yarbus, A.L
(1967) Eye movements and vision. New York: Plenum Press. DOI logoGoogle Scholar
Zhang, L., & Knoeferle, P
(2012) Visual context effects on thematic role assignment in children versus adults: Evidence from eye tracking in German. In R.P. Cooper, N. Miyake, & D. Peebles (Eds.), Proceedings of the annual meeting of the cognitive science society (pp. 2593-2598). Boston, USA: The Cognitive Science Society.Google Scholar
Cited by

Cited by 1 other publications

Hernández-Gutiérrez, David, Francisco Muñoz, Jose Sánchez-García, Werner Sommer, Rasha Abdel Rahman, Pilar Casado, Laura Jiménez-Ortega, Javier Espuny, Sabela Fondevila & Manuel Martín-Loeches
2021. Situating language in a minimal social context: how seeing a picture of the speaker’s face affects language comprehension. Social Cognitive and Affective Neuroscience 16:5 ► pp. 502 ff. DOI logo

This list is based on CrossRef data as of 7 september 2023. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.