Ординатура / Офтальмология / Английские материалы / Eye Movements A Window on Mind and Brain_Van Gompel_2007

Figure 6. Scene during a single trial of the One Feature condition when brick color was task relevant. Fingertips are represented as small red spheres. In a single trial, a subject (a) selects a brick based on the pick-up cue,

(b) lifts the brick, (c) brings it towards themselves, (d) decides on which conveyor belt the brick belongs based on a put-down cue, (e) guides the brick to the conveyor belt, (f) sets the brick on the belt where the brick is carried off. In other trials, subjects may have used width, height, or stripes for the pick-up or put-down decision. Adapted from Droll et al. (2005). (See Color Plate 12.)

sort the bricks? Figure 7 shows the two possibilities. Either the subject can treat the brick as if it retained its old feature, or else he/she can sort it according to its current feature. Given that the subject invariably fixates the brick either before or during placement, one might expect that the new feature state would be clearly visible, and that the subjects would sort the brick on the basis of its current (changed) feature. Instead, they almost always sorted it by its old feature, despite the fact that they fixated the brick, with its new feature state, for an average of 750 ms after the change. This indicates that subjects are using

Figure 7. Two possible sorting decisions following a missed feature change. (a) Subjects may sort the brick by the old, pre-change, feature, in which case changes are missed due to a failure to update the new visual information. (b) Subjects may sort also the brick by the new, post-change, feature, in which case changes are missed due to a failure to maintain visual information. When subjects performed blocks of trials using the same feature for the put-down decision, missed changes were most often sorted by the old feature (85%). (See Color Plate 13.)

their memory of the brick feature, rather than its actual current state, to make the sorting decision. (Note that the put-down cue is not available until the block has been picked up, so subjects cannot plan the put-down movement immediately after pickup, but must fixate the put-down cue first.) This finding is important because failure to detect changes to items in scenes has traditionally been interpreted as a failure to retain information from the pre-change image in memory (O’Regan, 1992; Rensink, 2000). Although this is undoubtedly the case in many situations, in the current context, sorting by the old feature reveals that subjects do indeed maintain a memory representation of the previous state of the brick, but simply fail to update their representation with the new information following a change. Change blindness as a consequence of failure to update the internal representation is consistent with the suggestion of Henderson & Hollingworth (1999) and Simons & Rensink (2005). It seems likely that in many situations the information in the visual scene is stable. Changes like those in the Droll et al. experiment would be impossible in real scenes. Subjects presumably take advantage of prior experience about the stable properties of normal scenes to accumulate information about scenes in internal representations. Such a strategy makes sense given the limited bandwidth that is set by attention on the accrual of information. As long as the observer has accurate knowledge of the stable properties of scenes, there is no point in constantly updating the information in the image if it takes up attentional resources. It appears that even such low level information as color or size takes computational resources. The fact that subjects were fixating the brick while sorting onto the wrong conveyor belt suggests that attentional resources were devoted to the placement action, rather than to updating information about the brick. This phenomenon may be related to “inattentional blindness,” similar to the phenomena mentioned above (Mack & Rock, 1998). Note that the strategy of retaining information in memory in the brick sorting task is in contrast to the “just-in-time” strategy exhibited by observers in the block copying task (Ballard et al., 1995). In one case, observers opt to retain information in memory, and in the other they minimize memory load by fixating the block just at the point when the information is needed. Both are ways that the human cognitive system can deal with attentional and memory limitations. An important question for future research is: What determines which strategy is used?

In Droll et al.’s experiment, subjects treated the block properties as stable. An interesting observation in their experiment was subjects’ behavior in subsequent trials after a feature change was successfully detected. Figure 8 shows the total time spent fixating the brick on trials following a noticed change, and also the total duration of the hand movement between lifting and placement on the conveyor belt. Both fixations and hand movements were significantly longer on the trial immediately following a detected change, by as much as 400 ms. This effect fell off sharply over the next few trials. Thus subjects were taking more time to perform the task after they observed the unlikely event where a brick changed its properties. This suggests that they reallocated attentional resources to the brick on the next trial, when events violated their expectations. However, since the event was not repeated, they reverted to the prior strategy. This suggests that internal models of the environment are dynamic, and may often be more strongly influenced by the most

must learn where to look in a scene to get the information they need for component sub-tasks. They must learn the structure and dynamic properties of the world in order to fixate critical regions at the right time. They must learn how to allocate attention and gaze to satisfy competing demands in an optimal fashion and be sensitive to changes in those demands. There are many questions about the precise way that learning affects gaze patterns. Perhaps the most critical issue is understanding exactly how tasks exert their control on gaze. A growing understanding of the importance of reward in modulating the underlying neural mechanisms and theoretical developments using reinforcement learning models of complex behavior provides us with the tools to understanding how tasks are represented in the brain, and how they control acquisition of information through use of gaze.

Acknowledgements

This work was supported by NIH grants EY 05729 and RR 09283. The authors wish to thank Brian Sullivan, Keith Gorgos and Jennifer Semrau for assistance with the experiments and Keith Parkins for programming support.

References

Ballard, D. H., Hayhoe, M., & Pelz, J. B. (1995). Memory representations in natural tasks. Journal of Cognitive Neuroscience, 7(1), 66–80.

Ballard, D. H., Hayhoe, M. M., Pook, P. K., & Rao, R. P. N. (1997). Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 20, 723–767.

Basso, M., & Wurtz, R. (1998). Modulation of neuronal activity in superior colliculus by changes in target probability. Journal of Neuroscience, 18, 7519–7534.

Chapman, P., & Underwood, G. (1998). Visual search of dynamic scenes: Event types and the role of experience in viewing driving situations. In Underwood, G. (Ed.), Eye Guidance in Reading and Scene Perception 369–394). Oxford: Elsevier.

Chun, M., & Nakayama, K. (2000). On the functional role of implicit visual memory for the adaptive deployment of attention across scenes. Visual Cognition, 7, 65–82.

Dorris, M. C., & Glimcher, P. W. (2004). Activity in posterior parietal cortex is correlated with the subjective desirability of an action. Neuron, 44, 365–378.

Droll, J., Hayhoe, M., Triesch, J., & Sullivan, B. (2005). Task demands control acquisition and maintenance of visual information. Journal of Experimental Psychology: Human Perception and Performance, 31, 1416–1438.

Epelboim, J., Steinman, R., Kowler, E., Edwards, M., Pizlo, Z., Erkelens, C., & Collewijn, H. (1995). The function of visual search and memory in sequential looking tasks. Vision Research, 35, 3401–3422.

Findlay, J., & Gilchrist, I. (2003). Active Vision. Oxford: Oxford University Press.

Glimcher, P. (2003). The neurobiology of visual-saccadic decision making. Annual Review of Neuroscience, 26, 133–179.

Gordon, R. D., & Irwin, D. E. (1996). What’s in an object file? Evidence from priming studies. Perception & Psychophysics, 58(8), 1260–1277.

Hayhoe, M. (2000). Visual routines: a functional account of vision. Visual Cognition, 7, 43–64.

Hayhoe, M., Bensinger, D., & Ballard, D. (1998). Task constraints in visual working memory. Vision Research, 38, 125–137.

Hayhoe, M., Mennie, N., Sullivan, B. & Gorgos, K. (2005). The role of internal models and prediction in catching balls. Proceedings of AAAI Fall Symposium Series.

Hayhoe, M., Shrivastrava, A., Mruczek, R., & Pelz, J. (2003). Visual memory and motor planning in a natural task. Journal of Vision, 3, 49–63.

He, P., & Kowler, E. (1989). The role of location probability in the programming of saccades: Implications for center-of-gravity tendencies. Vision Research, 29, 1165–1181.

Henderson, J. (2003). Human gaze control during real-world scene perception. Trends in Cognitive Science, 7, 498–504.

Henderson, J. & Hollingworth, A. (1999). The role of fixation position in detecting scene changes across saccades. Psychological Science, 10, 438–443.

Hikosaka, O., Takikawa, Y., & Kawagoe, R. (2000). Role of the basal ganglia in the control of purposive saccadic eye movements. Physiological Review, 80, 953–978.

Hollingworth, A., & Henderson, J. (2002). Accurate visual memory for previously attended objects in natural scenes. Journal of Experimental Psychology: Human Perception and Performance, 28, 113–136.

Irwin, D. E., & Andrews, R. V. (Eds.). (1996). Integration and accumulation of information across saccadic eye movements. Cambridge: MIT Press.

Itti, L., & Koch, C. (2001). Computational modeling of visual attention. Nature Review Of Neuroscience, 2, 194–203.

Johansson, R., Westling, G., Bäckström, A., & Flanagan, J. R. (2001). Eye-hand coordination in object manipulation. Journal of Neuroscience, 21, 6917–6932.

Kahneman, D., Treisman, A., & Gibbs, B. J. (1992). The reviewing of object files: Object-specific integration of information. Cognitive Psychology, 25, 175–219.

Land, M. (2004). Eye movements in daily life. In The Visual Neurosciences (vol 2), Chalupa, L. & Werner, J. (Eds.), 1357–1368, MIT Press.

Land, M., & Furneaux, S. (1997). The knowledge base of the oculomotor system. Philosophical Transactions of the Royal Society of London B, 352, 1231–1239.

Land, M. F., & Lee, D. N. (1994). Where we look when we steer. Nature (London), 369, 742–744.

Land, M. F., & McLeod, P. (2000). From eye movements to actions: how batsmen hit the ball. Nature Neuroscience, 3, 1340–1345.

Land, M. F., Mennie, N., & Rusted, J. (1999). Eye movements and the roles of vision in activities of daily living: making a cup of tea. Perception, 28, 1311–1328.

Loomis, J., & Beall, A. (2004). Model-based control of perception/action. In Vaina, L. et al. (Eds.), Optic Flow and Beyond (pp. 421–441), Netherlands: Kluwer.

Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390 (6657), 279–281.

Mack, A., & Rock, I. (1998). Inattention blindness. Cambridge, MA: MIT Press.

Mannan, S. Ruddock, K. H. & Wooding, D. S., (1997). Fixation patterns made during brief examination of two-dimensional images. Perception, 26, 1059–1072.

Montague, P. R., Hyman, S. E., Cohen, J. D. (2004). Computational roles for dopamine in behavioral control.

Nature, 431, 760–767.

O’Regan, J. K. (1992). Solving the ‘real’ mysteries of visual perception: The world as an outside memory.

Canadian Journal of Psychology, 46, 461–488.

Parkhurst, D., Law, K., & Niebur, E.(2002). Modeling the role of salience in the allocation of overt visual attention. Vision Research, 42, 107–123.

Patla, A., & Vickers, J. (1997). Where do we look as we approach and step over an obstacle in the travel path?

NeuroReport, 8, 3661–3665.

Patla, A., & Vickers, J. (2003). How far ahead do we look when required to step on specific locations in the travel path during locomotion? Experimental Brain Research, 148, 133–138.

Pelz, J. B., & Canosa, R. (2001). Oculomotor Behavior and Perceptual Strategies in Complex Tasks, Vision Research, 41, 3587–3596.

Pelz, J. B., Canosa, R., Babcock, J., Kucharczyk, D., Silver, A., & Konno, D. (2000). Portable eyetracking: A study of natural eye movements. Proceedings of the SPIE: Vol. 3959. Human vision and electronic imaging (pp. 566–583). San Jose, CA: SPIE.

Platt, M. L., & Glimcher, P. W. (1999). Neural correlates of decision variables in parietal cortex. Nature, 400, 233–238.

Rensink, R. A. (2000). The dynamic representation of scenes. Visual Cognition, 7, 17–42.

Roelfsema, P., Lamme,V., & Spekreijse, H. (2000). The implementation of visual routines. Vision Research, 40, 1385–1411.

Rothkopf, C. A., & Pelz, J. B. (2004). Head movement estimation for wearable eye tracker. Proceedings ACM SIGCHI Eye tracking research & applications symposium (pp. 123–130). San Antonio, Texas.

Schultz, W. (2000). Multiple reward signals in the brain. Nature reviews: neuroscience, 1, 199–207. Schwartz, M., Reed, E., Montgomery, M., Palmer, C., & Mayer, N. (1991). The quantitative description of

action disorganization after brain damage: a case study. Cognitive Neuropsychology, 8, 381–414. Shinoda, H., Hayhoe, M. M., & Shrivastava, A. (2001). Attention in natural environments. Vision Research,

41, 3535–3546.

Simons, D. J. (2000). Change blindness and visual memory. A Special Issues of Visual Cognition. Hove, UK: Psychology Press.

Simons, D., & Rensink, R. (2005). Change blindness: Past, present, and future. Trends in Cognitive Science, 9, 16–20.

Sprague, N., & Ballard, D. (2003). Eye movements for reward maximization. In Advances in Neural Information Processing Systems, 16, MIT Press.

Stuphorn, V., Taylor, T., & Schall, J. (2000). Performance monitoring by the supplementary eye field. Nature, 408, 857–860.

Sugrue, L. P., Corrado, G., & Newsome, W. (2004). Matching behavior and the encoding of value in parietal cortex. Science, 304, 1782–1787.

Triesch, J., Ballard, D., Hayhoe, M., & Sullivan, B. (2003). What you see is what you need. Journal of Vision, 3, 86–94.

Turano, K., Geruschat, D., & Baker, F. (2003). Oculomotor strategies for the direction of gaze tested with a real-world activity. Vision Research, 43, 333–346.

Ullman, S. (1984).Visual Routines. Cognition, 18, 97–157.

Vogel, E. K., Woodman, G. F., & Luck, S. J. (2001). Storage of features, conjunctions and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 27(1), 92–114.

Watanabe, K., Lauwereyns, J., & Hikosaka, O. (2003). Neural correlates of rewarded and unrewarded movements in the primate caudate nucleus. Journal of Neuroscience, 23, 10052–10057.

Wolpert, D., Miall, C., & Kawato, M. (1998). Internal models in the cerebellum. Trends in Cognitive Science, 2, 338–347.

This page intentionally left blank

Abstract

Three subjects performed two tasks (Free-view and Walking) in two environments (Manmade and Wooded). Fixation duration and saccade size were extracted from eye movements monitored with a wearable eyetracker. Mean, median, and modal fixation durations were shorter during the Free-view task in the Man-made environment than in the Wooded environment. However, despite the difference in characteristics and predictability of the paths between the Man-made and the Wooded environments (paved walkway and dirt path, respectively), there was not a significant environment difference in the distributions of fixation duration during the Walking task. There were no significant differences in the distribution of saccade sizes across any of the conditions. While there was no significant environment effect on fixation duration and saccade-size distributions during the Walking task, subjects significantly increased the fraction of time gaze was directed to the path immediately before them from 35% on paved walkways to 62% on dirt paths.

Chapters 2, 3, and 4 in this book demonstrate a rich history of oculomotor research. From the earliest studies, eye-movement research has pursued two parallel goals: research on eye movements to understand the oculomotor system, and research with eye movements where the movements are used as externally visible markers of attention to probe perception and cognition. Delabarre’s (1898) first paper on recording eye movements began, “Many problems suggest themselves to the psychologist whose solution would be greatly furthered by an accurate method of recording the movements of the eye.” (Delabarre, 1898, p. 572). The amount of data available in the environment would overwhelm our processing capacity; cortical limitations require that only a small subset of the available information be selected for processing. Eye movements are used to select that subset, so monitoring those movements (which Buswell termed “symptoms of perception”) provides a pointer to the selected information and the strategies employed in performing that selection. These strategies are critical to daily perception and action, yet they are rarely consciously selected, so they are not available to self-report.

A given sequence of eye movements is the result of the information available in the environment, a subset of elements in the environment that are critical to ongoing tasks and are within the cortical limitations, and the motivation of an observer. The “visual stimulus” (typically an image of limited extent in laboratory experiments) is only one element in the equation. The results of Yarbus’s classical experiments (1961, 1967) demonstrated that the intended “independent variable” (the visual stimulus) can actually be less important than the explicit or implicit task. Land (this book) discusses oculomotor strategies during active behavior where, like in Yarbus’s classic work, the task can have at least as much influence as does the stimulus. This can be cast as the interaction of “bottom-up” (stimulus-driven) vs “top-down” (task-dependent) control. The bottom-up influence of an image has been proposed in which a “saliency map” is calculated based on the image properties (Itti & Koch, 2000, Parkhurst & Niebur, 2003). The goal of such models is to predict the likelihood that image elements will draw fixations, independent of top-down effects. Land (this book) discusses the issue of saliency vs task – top-down vs bottom-up in a number of domains. It is clear from many studies that at least an element of top-down, task-dependent control is necessary to reveal behavior (e.g., Findlay & Walker, 1999).

Oculomotor behavior can be examined at a number of different levels beyond isolated movements of the eye. Perhaps the simplest is to examine the relative frequency of the angular extent of individual saccadic eye movements, and the duration of the intervening fixations. The distributions of fixation duration have been shown to vary between tasks. Perhaps most studied is the overlearned task of reading. Eye movements related to reading are relatively easy to monitor because the subject is stationary, the stimulus is static, and only horizontal eye movements need to be recorded. Silent reading has a mean fixation duration of 225 ms, oral reading has a duration of 275 ms (Rayner, 1998). A wide range of other tasks have mean fixation durations of approximately 300 ms. Henderson & Hollingworth (1998) reported mean fixations of 330 ms for free-viewing images on a small display, the same value reported by Pelz & Canosa (2001) for subjects walking in a hallway. Considerably longer mean fixation durations have been reported for tasks