Article published in:
Visually Situated Language Comprehension
Edited by Pia Knoeferle, Pirita Pyykkönen-Klauck and Matthew W. Crocker
[Advances in Consciousness Research 93] 2016
► pp. 227–260

Characterising visual context effects

Active, pervasive, but resource-limited

In this chapter, I will review recent research on visually situated language comprehension, and in doing so identify key characteristics of situated language comprehension. More specifically I will argue that both active visual context effects and the temporally coordinated interplay between visual attention and language comprehension are characteristic of situated comprehension, and are robust across a broad range of comprehension situations, spanning (a) different comprehension modalities (reading and spoken comprehension) and situations in which language is (versus isn’t) in accord with visual context; (b) different kinds of visual contexts (clipart depictions, photographs, and real-world objects and events); (c) speaker-based information such as eye-gaze and gestures; and (d) both concrete and abstract language. Because of their broad coverage ((a)–(d)), situated language comprehension paradigms are, in principle, well suited for developing a relatively comprehensive theory of situated language comprehension. One challenge in further specifying model predictions is the development of more detailed linking hypotheses between comprehension processes and one of the key measures used to examine situated comprehension (visual attention to objects across time).
Published online: 10 March 2016
https://doi.org/10.1075/aicr.93.09kno
Cited by

Cited by 1 other publications

Hernández-Gutiérrez, David, Francisco Muñoz, Jose Sánchez-García, Werner Sommer, Rasha Abdel Rahman, Pilar Casado, Laura Jiménez-Ortega, Javier Espuny, Sabela Fondevila & Manuel Martín-Loeches
2021. Situating language in a minimal social context: how seeing a picture of the speaker’s face affects language comprehension. Social Cognitive and Affective Neuroscience Crossref logo

This list is based on CrossRef data as of 02 march 2021. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

References

References

Abashidze, D., Carminati, M.N., & Knoeferle, P.
(2014) How robust is the recent-event preference? In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th Annual Meeting of the Cognitive Science Society (pp. 92-97). Cognitive Science Society.Google Scholar
Abashidze, D., Knoeferle, P., Carminati, M.N., & Essig, K.
(2011) The role of recent real-world versus future events in the comprehension of referentially ambiguous sentences: Evidence from eye tracking. In B. Kokinov, A. Karmiloff-Smith, & N.J. Nersessian (Eds.), Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press.Google Scholar
Allopenna, P., Magnuson, J., & Tanenhaus, M.
(1998) Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory & Language, 38, 419-439. CrossrefGoogle Scholar
Altmann, G.T.M.
(2004) Language-mediated eye-movements in the absence of a visual world: The ‘blank screen paradigm’. Cognition, 93, B79-B87. CrossrefGoogle Scholar
Altmann, G.T.M., & Kamide, Y.
(1999) Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73, 247-264. CrossrefGoogle Scholar
(2004) Now you see it, now you don’t: Mediating the mapping between language and the visual world. In J. Henderson & F. Ferreira (Eds.), The integration of language, vision and action (pp. 347-386). New York: Psychology Press.Google Scholar
(2007) The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language, 57, 502-518. CrossrefGoogle Scholar
(2009) Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation. Cognition, 111, 55-71. CrossrefGoogle Scholar
Andersson, R., Ferreira, F., & Henderson, J.M.
(2011) I see what you’re saying: The integration of complex speech and scenes during language comprehension. Acta Psychologica, 137, 208-216. CrossrefGoogle Scholar
Arai, M., van Gompel, R., & Scheepers, C.
(2007) Priming ditransitive structures in comprehension. Cognitive Psychology, 54, 218-250. CrossrefGoogle Scholar
Arbib, M.
(2005) From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and Brain Sciences, 28, 105-167. CrossrefGoogle Scholar
Barsalou, L.W.
(1999) Perceptual and symbol systems. Behavioral and Brain Sciences, 22, 577-609.Google Scholar
Brennan, S., Chen, X., Dickinson, C., Neider, M., & Zelinsky, G.
(2007) Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition, 106, 1465-1477. CrossrefGoogle Scholar
Burigo, M., & Knoeferle, P.
(2011) Visual attention during spatial language comprehension: Reference alone isn’t enough. In L. Carlson, C. Holscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society. Cognitive Science Society.Google Scholar
Carminati, M.N., & Knoeferle, P.
(2013) Effects of speaker emotional facial expression and listener age on incremental sentence processing. PLoS ONE, 8, e72559. CrossrefGoogle Scholar
Carstensen, L.L., Fung, H.H., & Charles, S.T.
(2003) Socioemotional selectivity theory and the regulation of emotion in the second half of life. Motivation and Emotion, 27, 103-123. CrossrefGoogle Scholar
Casasanto, C.
(2008) Similarity and proximity: When does close in space mean close in mind? Memory & Cognition, 36, 1047-1056. CrossrefGoogle Scholar
Castelhano, M., Mack, M., & Henderson, J.
(2009) Viewing task influences eye movement control during active scene perception. Journal of Vision, 9(3), 1-15. CrossrefGoogle Scholar
Chambers, C.G., & San Juan, V.
(2008) Perception and presupposition in real-time language comprehension: Insights from anticipatory processing. Cognition, 108, 26-50. CrossrefGoogle Scholar
Chambers, C.G., Tanenhaus, M.K., Filip, H., & Carlson, G.N.
(2002) Circumscribing referential domains during real time language comprehension. Journal of Memory and Language, 47, 30-49. CrossrefGoogle Scholar
Chambers, C.G., Tanenhaus, M.K., & Magnuson, J.S.
(2004) Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 687-696. CrossrefGoogle Scholar
Cook, S.W., & Tanenhaus, M.K.
(2009) Embodied communication: Speakers’ gestures affect listeners’ actions. Cognition, 113, 98-104. CrossrefGoogle Scholar
Cooper, R.
(1974) The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84-107. CrossrefGoogle Scholar
Crocker, M.W., Knoeferle, P., & Mayberry, M.
(2010) Situated sentence comprehension: The coordinated interplay account and a neurobehavioral model. Brain and Language, 112, 189-201. CrossrefGoogle Scholar
Dahan, D., & Tanenhaus, M.
(2005) Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychonomic Bulletin & Review, 12, 453-459. CrossrefGoogle Scholar
Duñabeitia, J.A., Aviles, A., Afonso, O., Scheepers, C., & Carreiras, M.
(2008) Qualitative differences in the representation of abstract versus concrete words: Evidence from the visual-world paradigm. Cognition, 110, 284-292. CrossrefGoogle Scholar
Engelhardt, P.E., Ferreira, F., & Patsenko, E.G.
(2010) Pupillometry reveals processing load during spoken language comprehension. Quarterly Journal of Experimental Psychology, 63, 639-664. CrossrefGoogle Scholar
Ferreira, F., Foucart, A., & Engelhardt, P.E.
(2013) Language processing in the visual world: Effects of preview, visual complexity, and prediction. Journal of Memory and Language, 69, 165-182. CrossrefGoogle Scholar
Fodor, J.
(1983) Modularity of mind. Cambridge, MA: MIT Press.Google Scholar
Frazier, L., & Fodor, J.D.
(1978) The sausage machine: A new two-stage parsing model. Cognition, 6, 291-325. CrossrefGoogle Scholar
Friederici, A.D.
(2002) Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6, 78-84. CrossrefGoogle Scholar
Ganis, G., Kutas, M., & Sereno, M.I.
(1996) The search for ”common sense”: An electrophysiological study of the comprehension of words and pictures in reading. Journal of Cognitive Neuroscience, 8, 89-106. CrossrefGoogle Scholar
Goolkasian, P.
(1996) Picture-word differences in a sentence verification task. Memory & Cognition, 24, 584-594. CrossrefGoogle Scholar
Griffin, Z., & Bock, K.
(2000) What the eyes say about speaking. Psychological Science, 11, 274-279. CrossrefGoogle Scholar
Guerra, E., & Knoeferle, P.
(2014) Effects of object distance on incremental semantic interpretation: Similarity is closeness. Cognition, 133, 535-552. CrossrefGoogle Scholar
Habets, B., Kita, S., Shao, Z., Ozyurek, A., & Hagoort, P.
(2011) The role of synchrony and ambiguity in speech-gesture integration during comprehension. Journal of Cognitive Neuroscience, 23, 1845-1854. CrossrefGoogle Scholar
Hagoort, P., Brown, C., & Groothusen, J.
(1993) The syntactic positive shift (sps) as an ERP measure of syntactic processing. Language and Cognitive Processes, 8, 439-483. CrossrefGoogle Scholar
Hanna, J., & Brennan, S.
(2007) Speakers’ eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language, 57, 596-615. CrossrefGoogle Scholar
Henderson, J.M., & Ferreira, F.
(2004) Scene perception for psycholinguists. In J.M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (pp. 1-58). New York: Psychology Press.Google Scholar
Huang, Y.T., & Snedeker, J.
(2009) Online interpretation of scalar quantifiers: Insight into the semantics –pragmatics interface. Cognitive Psychology, 58, 376-415. CrossrefGoogle Scholar
Huettig, F., & Altmann, G.T.M.
(2005) Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition, 96, 23-32. CrossrefGoogle Scholar
Huettig, F., Rommers, J., & Meyer, A.S.
(2011) Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica, 137, 151-171. CrossrefGoogle Scholar
Isaacowitz, D.M., Allard, E.S., Murphy, N.A., & Schlangel, M.
(2009) The time course of age-related preferences toward positive and negative stimuli. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 64B, 188-192. CrossrefGoogle Scholar
Kelly, S.D., Barr, D.J., Church, R.B., & Lynch, K.
(1999) Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Language, 40, 577-592. CrossrefGoogle Scholar
Kelly, S.D., Creigh, P., & Bartolotti, J.
(2010) Integrating speech and iconic gestures in a stroop-like task: Evidence for automatic processing. Journal of Cognitive Neuroscience, 22, 683-694. CrossrefGoogle Scholar
Kelly, S.D., Ozyurek, A., & Maris, E.
(2010) Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science, 21, 260-267. CrossrefGoogle Scholar
Knoeferle, P.
(2007) Comparing the time-course of processing initially ambiguous and unambiguous German SVO/OVS sentences in depicted events. In R. van Gompel, M. Fischer, W. Murray, & R. Hill (Eds.), Eye movement research: Insights into mind and brain (pp. 517-533). Oxford: Elsevier. CrossrefGoogle Scholar
in pressa). Language comprehension in rich non-linguistic contexts: Combining eye tracking and event-related brain potentials. In Towards a cognitive neuroscience of natural language use. Cambridge: Cambridge University Press. Crossref
in pressb). Visually situated language comprehension in children and in adults. In R.K. Mishra, N. Srinivasan, & F. Huettig (Eds.) Attention and vision in language processing Springer Language and Cognition series Crossref
Knoeferle, P., Carminati, M.N., Abashidze, D., & Essig, K.
(2011) Preferential inspection of recent real-world events over future events: Evidence from eye tracking during spoken sentence comprehension. Frontiers in Psychology, 2, 376. CrossrefGoogle Scholar
Knoeferle, P., & Crocker, M.W.
(2005) Incremental effects of mismatch during picture-sentence integration: Evidence from eye-tracking. In Proceedings of the 27th Annual Meeting of the Cognitive Science Conference (pp. 1166-1171). Mahwah, NJ: Erlbaum.Google Scholar
(2006) The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking. Cognitive Science, 30, 481-529. CrossrefGoogle Scholar
(2007) The influence of recent scene events on spoken comprehension: Evidence from eye-movements. Journal of Memory and Language, 75, 519-543. CrossrefGoogle Scholar
Knoeferle, P., Crocker, M.W., Scheepers, C., & Pickering, M.J.
(2005) The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition, 95, 95-127. CrossrefGoogle Scholar
Knoeferle, P., Habets, B., Crocker, M.W., & Munte, T.F.
(2008) Visual scenes trigger immediate syntactic reanalysis: Evidence from ERPs during situated spoken comprehension. Cerebral Cortex, 18, 789-795. CrossrefGoogle Scholar
Knoeferle, P., & Kreysa, H.
(2012) Effects of speaker gaze on syntactic structuring. Frontiers in Psychology, 2, 376. CrossrefGoogle Scholar
Knoeferle, P., Urbach, T.P., & Kutas, M.
(2011) Comprehending how visual context influences incremental sentence comprehension: Insights from ERPs and picture-sentence verification. Psychophysiology, 48, 495-506. CrossrefGoogle Scholar
Knoeferle, P., Urbach, T., & Kutas, M.
(2014) Different mechanisms for role relations versus verb-action congruence effects: Evidence from ERPs in picture-sentence verification. Acta Psychologica, 152, 133-148. CrossrefGoogle Scholar
Kolk, H., Chwilla, D., Van Herten, M., & Oor, P.
(2003) Structure and limited capacity in verbal working memory: A study with event-related potentials. Brain and Language, 85, 1-36. CrossrefGoogle Scholar
Kreysa, H.
(2009) Coordinating speech-related eye movements between comprehension and production. Unpublished doctoral dissertation. University of Edinburgh, UK.Google Scholar
Kreysa, H., & Knoeferle, P.
(2011a) Effects of speaker gaze on spoken language comprehension: Task matters. In L. Carlson, C. Holscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society. Cognitive Science Society.Google Scholar
(2011b) Peripheral speaker gaze facilitates spoken language comprehension: Syntactic structuring and thematic role assignment in German. In B. Kokinov, A. Karmiloff-Smith, & N.J. Nersessian (Eds.), Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press.Google Scholar
Kreysa, H., Knoeferle, P., & Nunnemann, E.
(2014) Effects of speaker gaze versus depicted actions on visual attention during sentence comprehension. In M. McShane, B. Scassellati, P. Bello, & M. Guarini (Eds.), Proceedings of the 36th Annual Conference of the Cognitive Science Society (pp. 2513-2518). Cognitive Science Society.Google Scholar
Kuperberg, G., Sitnikova, T., Caplan, D., & Holcomb, P.
(2003) Electrophysiological distinctions in processing conceptual relationships within simple sentences. Cognitive Brain Research, 17, 117-129. CrossrefGoogle Scholar
Kutas, M., & Federmeier, K.D.
(2011) Thirty years and counting: Finding meaning in the N400 component of the event related brain potential (ERP). Annual Review of Psychology, 62, 621-647. CrossrefGoogle Scholar
Kutas, M., & Hillyard, S.A.
(1980) Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207, 203-205. CrossrefGoogle Scholar
(1984) Brain potentials during reading reflect word expectancy and semantic association. Nature, 307, 161-163. CrossrefGoogle Scholar
Kutas, M., Van Petten, C., & Kluender, R.
(2006) In M. Traxler & M. Gerns-bacher (Eds.), Handbook of psycholinguistics (2nd ed., pp. 659-724). New York: Elsevier. CrossrefGoogle Scholar
Mayberry, M., Crocker, M.W., & Knoeferle, P.
(2009) Learning to attend: A connectionist model of situated language comprehension. Cognitive Science, 33, 449-496. CrossrefGoogle Scholar
Nappa, R., & Arnold, J.
(2009) Paying attention to intention: Effects of intention (but not egocentric attention) on pronoun resolution. In Proceedings of the CUNY Conference (p. 262).
Nation, K., & Altmann, C.M.M.G.T.M.
(2003) Investigating individual differences in childrens real-time sentence comprehension using language-mediated eye movements. Journal of Experimental Child Psychology, 86, 314-329. CrossrefGoogle Scholar
Osterhout, L., & Holcomb, P.J.
(1992) Event-related brain potentials elicited by syntactic anomaly. Journal of Memory and Language, 31, 785-806. CrossrefGoogle Scholar
Pulvermuller, F., Harle, M., & Hummel, F.
(2001) Walking or talking?: Behavioral and neurophysiological correlates of action verb processing. Brain and Language, 78, 143-168. CrossrefGoogle Scholar
Richardson, D., & Matlock, T.
(2007) The integration of figurative language and static depictions: An eye movement study of fictive motion. Cognition, 102, 129-138. CrossrefGoogle Scholar
Richardson, D.C., & Dale, R.
(2005) Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cognitive Science, 29, 1045-1060. CrossrefGoogle Scholar
Richardson, D.C., & Kirkham, N.Z.
(2004) Multi-modal events and moving locations: Eye movements of adults and 6-month-olds reveal dynamic spatial indexing. Journal of Experimental Psychology: General, 133, 46-62. CrossrefGoogle Scholar
Salverda, A.P., Brown, M., & Tanenhaus, M.K.
(2011) A goal-based perspective on eye movements in visual-world studies. Acta Psychologica, 137, 172-180. CrossrefGoogle Scholar
Scheepers, C., & Crocker, M.W.
(2004) Constituent order priming from reading to listening: A visual-world study. In M. Carreiras & J.C. Clifton (Eds.), The on-line study of sentence comprehension: Eyetracking, ERP, and beyond. United Kingdom: Psychology Press.Google Scholar
Sedivy, J.C., Tanenhaus, M.K., Chambers, C.G., & Carlson, G.N.
(1999) Achieving incremental semantic interpretation through contextual representation. Cognition, 71, 109-148. CrossrefGoogle Scholar
Spivey, M.J., & Geng, J.J.
(2001) Oculomotor mechanisms activated by imagery and memory: Eye movements to absent objects. Psychological Research, 65, 235-241. CrossrefGoogle Scholar
Spivey, M.J., Tanenhaus, M.K., Eberhard, K.M., & Sedivy, J.C.
(2002) Eye-movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology, 45, 447-481. CrossrefGoogle Scholar
Staudte, M., & Crocker, M.
(2009) The effect of robot gaze on processing robot utterances. In N. Taatgen & H. van Rijn (Eds.), Proceedings of the Cognitive Science Conference (pp. 431–436). Cognitive Science Society, Inc.Google Scholar
Staudte, M., Crocker, M.W., Heloir, A., & Kipp, M.
(2014) The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts. Cognition, 133, 317-328. CrossrefGoogle Scholar
Tanenhaus, M.K.
(2004) That’s not it and "its" not "that": The role of conceptual composites in in-line reference resolution. In M. Carreiras & J. Charles Clifton (Eds.), On-line sentence processing: ERPs, eye movements, and beyond (pp. 209-228). Psychology Press.Google Scholar
Tanenhaus, M.K., Magnuson, J.S., Dahan, D., & Chambers, C.
(2000) Eye movements and lexical access in spoken- language comprehension: Evaluating a linking hypothesis between fixations and linguistic processing. Journal of Psycholinguistic Research, 29, 557-580. CrossrefGoogle Scholar
Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K., & Sedivy, J.C.
(1995) Integration of visual and linguistic information in spoken language comprehension. Science, 268, 632-634. CrossrefGoogle Scholar
Tatler, B., Wade, N., Kwan, H., Findlay, J., & Velichkovsky, B.
(2010) Yarbus, eye movements, and vision. I-Perception, 1, 7-27. CrossrefGoogle Scholar
Underwood, G., Jebbett, L., & Roberts, K.
(2004) Inspecting pictures for information to verify a sentence: Eye movements in general encoding and in focused search. The Quarterly Journal of Experimental Psychology, 56, 165-182. CrossrefGoogle Scholar
Vissers, C., Kolk, H., van de Meerendonk, N., & Chwilla, D.
(2008) Monitoring in language perception: Evidence from erps in a picture-sentence matching task. Neuropsychologia, 967-982. CrossrefGoogle Scholar
Wassenaar, M., & Hagoort, P.
(2007) Thematic role assignment in patients with broca’s aphasia: Sentence-picture matching electrified. Neuropsychologia, 45, 716-740. CrossrefGoogle Scholar
Wu, Y.C., & Coulson, S.
(2005) Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology, 42, 654-667. CrossrefGoogle Scholar
Yarbus, A.L.
(1967) Eye movements and vision. New York: Plenum Press. CrossrefGoogle Scholar
Zhang, L., & Knoeferle, P.
(2012) Visual context effects on thematic role assignment in children versus adults: Evidence from eye tracking in German. In R.P. Cooper, N. Miyake, & D. Peebles (Eds.), Proceedings of the annual meeting of the cognitive science society (pp. 2593-2598). Boston, USA: The Cognitive Science Society.Google Scholar