Perception of the visual environment
The eyes are the front end to the vast majority of the human behavioural repertoire. The manner in which our eyes sample the environment places fundamental constraints upon the information that is available for subsequent processing in the brain: the small window of clear vision at the centre of gaze can only be directed at an average of about three locations in the environment in every second. We are largely unaware of these continual movements, making eye movements a valuable objective measure that can provide a window into the cognitive processes underlying many of our behaviours. The valuable resource of high quality vision must be allocated with care in order to provide the right information at the right time for the behaviours we engage in. However, the mechanisms that underlie the decisions about where and when to move the eyes remain to be fully understood. In this chapter I consider what has been learnt about targeting the eyes in a range of different experimental paradigms, from simple stimuli arrays of only a few isolated targets, to complex arrays and photographs of real environments, and finally to natural task settings. Much has been learnt about how we view photographs, and current models incorporate low-level image salience, motor biases to favour certain ways of moving the eyes, higher-level expectations of what objects look like and expectations about where we will find objects in a scene. Finally in this chapter I will consider the fate of information that has received overt visual attention. While much of the detailed information from what we look at is lost, some remains, yet our understanding of what we retain and the factors that govern what is remembered and what is forgotten are not well understood. It appears that our expectations about what we will need to know later in the task are important in determining what we represent and retain in visual memory, and that our representations are shaped by the interactions that we engage in with objects.
References (136)
References
Andersen, R., Snyder, L., Bradley, D., & Xing, J. (1997). Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annual Review of Neuroscience, 20, 303-330.
Ballard, D.H., & Hayhoe, M.M. (2009). Modelling the role of task in the control of gaze. Visual Cognition, 17(6-7), 1185-1204.
Ballard, D.H., Hayhoe, M.M., Li, F., & Whitehead, S. (1992). Hand-eye coordination during sequential tasks. Philosophical Transactions of the Royal Society of London Series B-Biological Sciences, 337(1281), 331-338; discussion 338.
Ballard, D.H., Hayhoe, M.M., & Pelz, J.B. (1995). Memory representations in natural tasks. Journal of Cognitive Neuroscience, 7(1), 66-80.
Birmingham, E., Bischof, W., & Kingstone, A. (2009). Get real! Resolving the debate about equivalent social stimuli. Visual Cognition, 17(6-7), 904-924.
Blackmore, S.J., Brelstaff, G., Nelson, K., & Troscianko, T. (1995). Is the richness of our visual world an illusion - transsaccadic memory for complex scenes. Perception, 24(9), 1075-1081.
Borji, A., & Itti, L. (2013). State-of-the-art in visual attention modeling. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(1), 185-207.
Brouwer, A. -M., & Knill, D. (2007). The role of memory in visually guided reaching. Journal of Vision, 7(5), 1-12.
Burgess, N. (2006). Spatial memory: How egocentric and allocentric combine. Trends in Cognitive Sciences, 10(12), 551-557.
Burgess, N. (2008). Spatial cognition and the brain. Annals of the New York Academy of Sciences, 1124, 77-97.
Buswell, G.T. (1920). An experimental study of the eye-voice span in reading. Chicago: Chicago University Press.
Buswell, G.T. (1935). How people look at pictures: A study of the psychology of perception in art. Chicago: University of Chicago Press.
Bylinskii, Z., Judd, T., Borji, A., Itti, L., Durand, F., Oliva, A., & Torralba, A. (n.d.). Mit saliency benchmark.
Cameron, E.H., & Steele, W.M. (1905). The Poggendorff illusion. Psychological Monographs, 7(1), 83-111.
Carmi, R., & Itti, L. (2006). Causal saliency effects during natural vision. In Proceedings of the eye tracking research & application symposium, ETRA 2006, San Diego, California, USA, March 27-29, 2006 (pp. 11-18).
Castelhano, M.S., & Henderson, J.M. (2007). Initial scene representations facilitate eye movement guidance in visual search. Journal Of Experimental Psychology-Human Perception and Performance, 33(4), 753-763.
Castelhano, M.S., Wieth, M., & Henderson, J.M. (2007). I see what you see: Eye movements in real-world scenes are affected by perceived direction of gaze. Attention in Cognitive Systems: Theories and Systems from an Interdisciplinary Viewpoint, 4840, 251-262.
Chang, S., Papadimitriou, C., & Snyder, L.H. (2009). Using a compound gain field to compute a reach plan. Neuron, 64(5), 744-755.
Chapman, P., & Underwood, G. (1998). Visual search of driving situations: Danger and experience. Perception, 27(8), 951-964.
Colby, C.L., & Goldberg, M.E. (1999). Space and attention in parietal cortex. Annual Review of Neuroscience, 22, 319-349.
Cristino, F., & Baddeley, R. (2009). The nature of the visual representations involved in eye movements when walking down the street. Visual Cognition, 17 (6-7), 880-903.
Deubel, H., & Schneider, W.X. (1996). Saccade target selection and object recognition: Evidence for a common attentional mechanism. Vision Research, 36(12), 1827-1837.
Dorr, M., Martinetz, T., Gegenfurtner, K.R., & Barth, E. (2010). Variability of eye movements when viewing dynamic natural scenes. Journal of Vision, 10(10), 28, 1-17.
Driver, J., Davies, M., Ricciardelli, P., Kidd, P., Maxwell, E., & Baron-Cohen, S. (1999). Gaze perception triggers reflective visuospatial orienting. Visual Cognition, 6(5), 509-540.
Droll, J.A., & Hayhoe, M.M. (2007). Trade-offs between gaze and working memory use. Journal of Experimental Psychology-Human Perception and Performance, 33(6), 1352-1365.
Duhamel, J.R., Colby, C.L., & Goldberg, M.E. (1992). The updating of the representation of visual space in parietal cortex by intended eye-movements. Science, 255(5040), 90-92.
Ehinger, K.A., Hidalgo-Sotelo, B., Torralba, A., & Oliva, A. (2009, August). Modeling Search for People in 900 Scenes: A combined source model of eye guidance. Visual Cognition, 17(6-7), 945.
Einhauser, W., Spain, M., & Perona, P. (2008). Objects predict fixations better than early saliency. Journal of Vision, 8(14), 18, 1-26.
Epelboim, J.L., Steinman, R.M., Kowler, E., Edwards, M., Pizlo, Z., Erkelens, C.J., & Collewijn, H. (1995). The function of visual search and memory in sequential looking tasks. Vision Research, 35(23-24), 3401-3422.
Epelboim, J.L., Steinman, R.M., Kowler, E., Pizlo, Z., Erkelens, C.J., & Collewijn, H. (1997). Gaze-shift dynamics in two kinds of sequential looking tasks. Vision Research, 37(18), 2597-2607.
Erdmann, B., & Dodge, R. (1898). sychologische Untersuchungen uber das Lesen auf experimenteller Grundlage. Halle: Niemeyer.
Fecteau, J., & Munoz, D. (2006). Salience, relevance, and firing: A priority map for target selection. Trends in Cognitive Sciences, 10(8), 382-390.
Fletcher-Watson, S., Findlay, J.M., Leekam, S.R., & Benson, V. (2008). Rapid detection of person information in a naturalistic scene. Perception, 37 (4), 571-583.
Foulsham, T., & Underwood, G. (2008). What can saliency models predict about eye movements? Spatial and sequential aspects of fixations during encoding and recognition. Journal of Vision, 8(2), 6.1-17.
Friesen, C., & Kingstone, A. (1998). The eyes have it! Reflexive orienting is triggered by nonpredictive gaze. Psychonomic Bulletin and Review, 5(3), 490-495.
Frintrop, S., Rome, E., & Christensen, H.I. (n.d.). Computational visual attention systems and their cognitive foundations: A survey. ACM Trans. on Applied Perception, 2010.
Furneaux, S., & Land, M.F. (1999). The effects of skill on the eye-hand span during musical sight- reading. Proceedings of the Royal Society of London Series B-Biological Sciences, 266(1436), 2435-2440.
Gallup, A.C., Chong, A., & Couzin, I.D. (2012). The directional flow of visual information transfer between pedestrians. Biology Letters, 8(4), 520-522.
Gibson, J.J. (1950). The Perception of the visual world (1st ed.). Boston: Houghton Mifflin.
Gibson, J.J. (1966). The Senses considered as perceptual systems. New York: Appleton-Century-Crofts.
Gibson, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.
Grimes, J. (1996). On the failure to detect changes in scenes across saccades. In K. Atkins (Ed.), Perception: Vancouver studies in cognitive science (pp. 89-110). New York: Oxford University Press.
Hayhoe, M.M., Shrivastava, A., Mruczek, R., & Pelz, J.B. (2003). Visual memory and motor planning in a natural task. Journal of Vision, 3(1), 49-63.
Henderson, J.M. (2003). Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7(11), 498-504.
Henderson, J.M., Brockmole, J.R., Castelhano, M.S., & Mack, M. (2007). Chapter 25 - visual saliency does not account for eye movements during visual search in real-world scenes. In R.L. Hill, R.P.V. Gompel, M.H. Fischer, & W.S. Murray (Eds.), Eye movements: A window on mind and brain (pp. 537-562). Oxford: Elsevier.
Hering, E. (1879). U ber Muskelgera¨usche des Auges. Sitzungsberichte der Akademie der Wissenschaften in Wien
. Mathematisch-naturwissenschaftliche Klasse. Abt. III, 79, 137-154.
Hollingworth, A. (2004). Constructing visual representations of natural scenes: The roles of short- and long-term visual memory. Journal Of Experimental Psychology-Human Perception and Performance, 30(3), 519-537.
Hollingworth, A. (2005). The relationship between online visual representation of a scene and long-term scene memory. Journal Of Experimental Psychology-Learning Memory and Cognition, 31(3), 396-411.
Hollingworth, A. (2007). Object-position binding in visual memory for natural scenes and object arrays. Journal of Experimental Psychology-Human Perception and Performance, 33(1), 31-47.
Hollingworth, A., & Henderson, J.M. (2002). Accurate visual memory for previously attended objects in natural scenes. Journal Of Experimental Psychology-Human Perception And Performance, 28(1), 113-136.
Hong, B., & Brady, M. (2003). A topographic representation for mammogram segmentation. In Medical image computing and computer-assisted intervention - miccai 2003, pt 2 (pp. 730-737). Oxford, England: Univ Oxford, Med Vis Lab.
Hooge, I., Over, E., Van Wezel, R., & Frens, M.A. (2005). Inhibition of return is not a foraging facilitator in saccadic search and free viewing. Vision Research, 45(14), 1901-1908.
Irwin, D.E. (1992). Visual memory within and across fixations. In K. Rayner (Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 146-165). New York: Springer-Verlag.
Irwin, D.E., & Andrews, R. (1996). Integration and accumulation of information across saccadic eye movements. In T. Inui & J.L. McClelland (Eds.), Attention and performance xvi: Information integration in perception and communication (pp. 125-155). Cambridge, MA: MIT Press.
Irwin, D.E., & Zelinsky, G.J. (2002). Eye movements and scene perception: Memory for things observed. Perception & psychophysics, 64(6), 882-895.
Itti, L. (2005). Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes. Visual Cognition, 12(6), 1093-1123.
Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40(10-12), 1489-1506.
Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 20(11), 1254-1259.
Jovancevic-Misic, J., & Hayhoe, M. (2009). Adaptive gaze control in natural environments. Journal of Neuroscience, 29(19), 6234-6238.
Judd, C.H. (1905). The Mu¨ller-Lyer illusion. Psychological Monographs, 7(1), 55-81.
Judd, T., Durand, F., & Torralba, A. (2012). A benchmark of computational models of saliency to predict human fixations
.
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009). Learning to predict where humans look. In
IEEE International Conference on Computer Vision (ICCV)
.
Kanan, C., Tong, M., Zhang, L., & Cottrell, G. (2009). SUN: Top-down saliency using natural statistics. Visual Cognition, 17(6-7), 979-1003.
Karn, K., Møller, P., & Hayhoe, M.M. (1997). Reference frames in saccadic targeting. Experimental Brain Research, 115(2), 267-282.
Koch, C., & Ullman, S. (1985). Shifts in selective visual-attention - towards the underlying neural circuitry. Human Neurobiology, 4 (4), 219-227.
Kuhn, G., & Tatler, B.W. (2005). Magic and fixation: Now you don’t see it, now you do. Perception, 34(9), 1155-1161.
Kuhn, G., Tatler, B.W., & Cole, G.G. (2009). You look where I look! Effect of gaze cues on overt and covert attention in misdirection. Visual Cognition, 17(6-7), 925-944.
Kuhn, G., Tatler, B.W., Findlay, J.M., & Cole, G.G. (2008). Misdirection in magic: Implications for the relationship between eye gaze and attention. Visual Cognition, 16(2/3), 391-405.
Laidlaw, K.E., Foulsham, T., Kuhn, G., & Kingstone, A. (2011). Potential social interactions are important to social attention.
Proceedings of the National Academy of Sciences of the United States of America
, 108, 5548-5553.
Land, M.F. (2004). The coordination of rotations of the eyes, head and trunk in saccadic turns produced in natural situations. Experimental Brain Research, 159 (2), 151-160.
Land, M.F. (2006). Eye movements and the control of actions in everyday life. Progress in Retinal and Eye Research, 25(3), 296-324.
Land, M.F., & Furneaux, S. (1997). The knowledge base of the oculomotor system. Philosophical Transactions of the Royal Society of London Series B-Biological Sciences, 352(1358), 1231-1239.
Land, M.F., & Horwood, J. (1995). Which parts of the road guide steering. Nature, 377, 339-340.
Land, M.F., & Lee, D.N. (1994). Where we look when we steer. Nature, 369(6483), 742-744.
Land, M.F., & McLeod, P. (2000). From eye movements to actions: How batsmen hit the ball. Nature Neuroscience, 3(12), 1340-1345.
Land, M.F., Mennie, N., & Rusted, J. (1999). The roles of vision and eye movements in the control of activities of daily living. Perception, 28(11), 1311-1328.
Land, M.F., & Tatler, B.W. (2001). Steering with the head: The visual strategy of a racing driver. Current Biology, 11(15), 1215-1220.
Land, M.F., & Tatler, B.W. (2009). Looking and acting: Vision and eye movements in natural behaviour. Oxford: OUP.
Macdonald, R.G., & Tatler, B.W. (2013). Do as eye say: Gaze cueing and language in a real-world social interaction. Journal of Vision, 13(4), 6,1-12.
McConkie, G.W., & Zola, D. (1979). Is visual information integrated across successive fixations in reading? Perception & Psychophysics, 25(3), 221-224.
Melcher, D. (2006). Accumulation and persistence of memory for natural scenes. Journal of Vision, 6(1), 8-17.
Montague, P., & Hyman, S. (2004). Computational roles for dopamine in behavioural control. Nature, 431, 760-767.
Najemnik, J., & Geisler, W.S. (2005). Optimal eye movement strategies in visual search. Nature, 434(7031), 387-391.
Najemnik, J., & Geisler, W.S. (2008). Eye movement statistics in humans are consistent with an optimal search strategy. Journal of Vision, 8(3), 4,1-14.
Navalpakkam, V., & Itti, L. (2005). Modeling the influence of task on attention. Vision Research, 45(2), 205-231.
Nuthmann, A., & Henderson, J.M. (2010). Object-based attentional selection in scene viewing. Journal of Vision, 10(8).
Nystr¨om, M., & Holmqvist, K. (2008). Semantic override of low-level features in image viewing-both initially and overall. Journal of Eye Movement Research, 2 (2), 1-11.
Oliva, A., Wolfe, J., & Arsenio, H. (2004). Panoramic search: The interaction of memory and vision in search through a familiar scene. Journal Of Experimental Psychology-Human Perception and Performance, 30(6), 1132-1146.
O’Regan, J.K., & Lévy-Schoen, A. (1983). Integrating Visual Information from Successive Fixations - Does Trans-Saccadic Fusion Exist. Vision Research, 23 (8), 765-768.
O’Regan, J.K., & Nöe, A. (2001). A sensorimotor account of vision and visual consciousness. The Behavioral and Brain Sciences, 24(5), 939-973; discussion 973–1031.
Parkhurst, D., Law, K., & Niebur, E. (2002). Modeling the role of salience in the allocation of overt visual attention. Vision Research, 42(1), 107-123.
Patla, A.E., & Vickers, J.N. (1997). Where and when do we look as we approach and step over an obstacle in the travel path? Neuroreport, 8(17), 3661-3665.
Patla, A.E., & Vickers, J.N. (2003). How far ahead do we look when required to step on specific locations in the travel path during locomotion? Experimental Brain Research, 148(1), 133-138.
Pelz, J.B., & Canosa, R. (2001). Oculomotor behavior and perceptual strategies in complex tasks. Vision Research, 41(25-26), 3587-3596.
Pertzov, Y., Avidan, G., & Zohary, E. (2009). Accumulation of visual information across multiple fixations. Journal of Vision, 9(10), 2.1-12.
Rayner, K. (1998). Eye Movements in reading and information processing: 20 Years of research. Psychological Bulletin, 124 (3), 372-422.
Reinagel, P., & Zador, A.M. (1999). Natural scene statistics at the centre of gaze. Network, 10 (4), 341-350.
Renninger, L.W., Verghese, P., & Coughlan, J. (2007). Where to look next? Eye movements reduce local uncertainty. Journal of Vision, 7 (3), 6, 1-17.
Rensink, R.A. (2000). The dynamic representation of scenes. Visual Cognition, 7(1-3), 17-42.
Rensink, R.A. (2002). Change detection. Annual Review Of Psychology, 53, 245-277.
Rensink, R.A., O’Regan, J.K., & Clark, J.J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8 (5), 368-373.
Rensink, R.A., O’Regan, J.K., & Clark, J.J. (2000). On the failure to detect changes in scenes across brief interruptions. Visual Cognition, 7(1-3), 127-145.
Ricciardelli, P., Bricolo, E., Aglioti, S.M., & Chelazzi, L. (2002). My eyes want to look where your eyes are looking: Exploring the tendency to imitate another individual’s gaze. Neuroreport, 13 (17), 2259-2264.
Rothkopf, C., & Ballard, D.H. (2009). Image statistics at the point of gaze during human navigation. Visual Neuroscience, 26 (1), 81-92.
Rothkopf, C.A., Ballard, D.H., & Hayhoe, M.M. (2007). Task and context determine where you look. Journal of Vision, 7(14), 16.1-20.
Sailer, U., Flanagan, J.R., & Johansson, R.S. (2005). Eye-hand coordination during learning of a novel visuomotor task. The Journal of Neuroscience, 25(39), 8833-8842.
Schultz, W. (2000). Multiple reward signals in the brain. Nature Reviews Neuroscience, 1(3), 199-207.
Siagian, C., & Itti, L. (2007). Biologically-inspired robotics vision monte-carlo localization in the outdoor environment. In
IEEE/RSJ Intelligent Robots and Systems
(pp. 1723-1730). San Diego, CA.
Smith, T., & Henderson, J.M. (2009). Facilitation of return during scene viewing. Visual Cognition, 17 (6-7), 1083-1108.
Sprague, N., Ballard, D.H., & Robinson, A. (2007). Modeling embodied visual behaviors. ACM Transactions on Applied Perception, 4, 11.
Stainer, M.J., Scott-Brown, K.C., & Tatler, B.W. (2013). Looking for trouble: A description of oculomotor search strategies during live CCTV operation. Frontiers in Human Neuroscience, 7, 615.
Steinman, R. (2003). Gaze control under natural conditions. The Visual Neurosciences.
Stratton, G.M. (1906). Symmetry, linear illusions, and the movements of the eye. Psychological Review, 13, 82-96.
Tatler, B.W. (2001). Characterising the visual buffer: Real-world evidence for overwriting early in each fixation. Perception, 30(8), 993-1006.
Tatler, B.W. (2007). The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions. Journal of Vision, 7(14), 4, 1-17.
Tatler, B.W. (Ed.). (2009). Eye guidance in natural scenes. Hove, UK: Psychology Press.
Tatler, B.W., Baddeley, R., & Gilchrist, I. (2005). Visual correlates of fixation selection: Effects of scale and time. Vision Research, 45(5), 643-659.
Tatler, B.W., Gilchrist, I., & Land, M. (2005). Visual memory for objects in natural scenes: From fixations to object files. Quarterly Journal of Experimental Psychology Section A-Human Experimental Psychology, 58(5), 931-960.
Tatler, B.W., Gilchrist, I.D., & Rusted, J. (2003). The time course of abstract visual representation. Perception, 32(5), 579-592.
Tatler, B.W., Hayhoe, M.M., Land, M.F., & Ballard, D.H. (2011). Eye guidance in natural vision: Reinterpreting salience. Journal of Vision, 11(5), 5, 1-23.
Tatler, B.W., Hirose, Y., Finnegan, S.K., Pievilainen, R., Kirtley, C., & Kennedy, A. (2013). Priorities for selection and representation in natural tasks. Philosophical Transactions of the Royal Society B, 368, 20130066.
Tatler, B.W., & Kuhn, G. (2007). Don’t look now: The magic of misdirection. In R.L. Hill, R.P.V. Gompel, M.H. Fischer, & W.S. Murray (Eds.), Eye movements: A window on mind and brain (pp. 697-714). Oxford: Elsevier.
Tatler, B.W., & Land, M.F. (2011). Vision and the representation of the surroundings in spatial memory. Philosophical Transactions of the Royal Society B-Biological Sciences, 366(1564), 596-610.
Tatler, B.W., & Vincent, B.T. (2008). Systematic tendencies in scene viewing. Journal of Eye Movement Research, 2(2), 5: 1-18.
t Hart, B., Vockeroth, J., Schumann, F., Bartl, K., Schneider, E., Konig, P., & Einhauser, W. (2009). Gaze allocation in natural stimuli: Comparing free exploration to head-fixed viewing conditions. Visual Cognition, 17(6-7), 1132-1158.
Torralba, A., Oliva, A., Castelhano, M.S., & Henderson, J.M. (2006). Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychological Review, 113(4), 766-786.
Treisman, A.M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97-136.
Triesch, J., Ballard, D., Hayhoe, M., & Sullivan, B. (2003). What you see is what you need. Journal of Vision, 3(1), 8694.
Wade, N.J., & Tatler, B.W. (2005). The moving tablet of the eye: The origins of modern eye movement research. Oxford: OUP.
Waller, D., & Hodgson, E. (2006). Transient and enduring spatial representations under disorientation and self-rotation. Journal Of Experimental Psychology-Learning Memory and Cognition, 32(4), 867-882.
Wischnewski, M., Belardinelli, A., & Schneider, W. (2010). Where to look next? Combining static and dynamic proto-objects in a TVA-based model of visual attention. Cognitive Computation, 2(4), 326-343.
Wischnewski, M., Steil, J., Kehrer, L., & Schneider, W. (2009). Integrating inhomogeneous processing and proto-object formation in a computational model of visual attention. Human Centered Robot Systems, 93-102.
Wolfe, J. (2007). Guided Search 4.0: Current Progress with a model of visual search. In W. Gray (Ed.), Integrated models of cognitive systems (pp. 99-119). New York: OUP.
Xu, T., Kuehnlenz, K., & Buss, M. (2010). Autonomous behavior-based switched top-down and bottom-up visual attention for mobile robots. IEEE Transactions on Robotics, 26(5), 947-954.
Yarbus, A.L. (1967). Eye movements and vision. New York: Plenum Press.
Zelinsky, G.J. (2008). A theory of eye movements during target acquisition. Psychological Review, 115(4), 787-835.