Ординатура / Офтальмология / Английские материалы / books.google.com / Visual Perception Fundamentals of awareness multi-sensory integration and high-order perception_Martinez-Conde_2006

Perrett, D.I., Rolls, E.T. and Caan, W. (1982) Visual neurons responsive to faces in the monkey temporal cortex. Exp. Brain Res., 47: 329–342.

Po¨ppel, E. (1985) Bridging a neuronal gap. Naturwissenschaften, 72: 599–600.

Po¨ppel, E. (1986) Long-range colour-generating interactions across the retina. Nature, 320: 523–525.

Po¨ppel, E., Held, R. and Frost, D. (1973) Residual visual function after brain wounds involving the central visual pathways in man. Nature, 243: 295–296.

Reese, B.E. and Cowey, A. (1988) Segregation of functionally distinct axons in the monkey’s optic tract. Nature, 331: 350–351.

Ress, D., Backus, B.T. and Heeger, D.J. (2000) Activity in primary visual cortex predicts performance in a visual detection task. Nat. Neurosci., 3: 940–945.

Riddoch, G. (1917) Dissociations of visual perception due to occipital injuries, with especial reference to appreciation of movement. Brain, 40: 15–57.

Rodman, H.R., Gross, C.G. and Albright, T.D. (1989) Afferent basis of visual response properties in area MT of the macaque: I. Effects of striate cortex removal. J. Neurosci., 9: 2033–2050.

Rodman, H.R., Gross, C.G. and Albright, T.D. (1990) Afferent basis of visual response properties in area MT of the macaque: II Effects of superior colliculus removal. J. Neurosci., 10: 1154–1164.

Sack, R.L., Lewy, A.J., Blood, M.L., Stevenson, J. and Keith, L.D. (1991) Melatonin administration to blind people: phase advances and entrainment. J. Biol. Rhythms, 6: 249–261.

Sahraie, A., Weiskrantz, L., Barbur, J.L., Simmons, A., Williams, S. and Brammer, M.J. (1997) Pattern of neuronal activity associated with conscious and unconscious processing of visual signals. Proc. Natl. Acad. Sci. USA, 94: 9406–9411.

Sanders, M.D., Warrington, E.K., Marshall, J. and Weiskrantz, L. (1974) ‘Blindsight’: vision in a field defect. Lancet, 1: 707–708.

Schoenfeld, M.A., Noesselt, T., Poggel, D., Tempelmann, C., Hopf, J.-M., Woldorff, M.G., Heinze, H.-J. and Hillyard, S.A. (2002) Analysis of pathways mediating preserved vision after striate cortex lesions. Ann. Neurol., 52: 814–824.

Siebler, M., Steinmetz, H. and Freund, H.-J. (1998) Therapeutic entrainment of circadian rhythm disorder by melatonin in a non-blind patient. J. Neurol., 245: 327–328.

Simpson, J.I. (1984) The accessory optic system. Annu. Rev. Neurosci., 7: 13–41.

Sprague, J.M. and Meikle, T.H. (1965) The role of the superior colliculus in visually guided behaviour. Exp. Neurol., 11: 115–146.

Stoerig, P. (1987) Chromaticity and achromaticity. Evidence for a functional differentiation in visual field defects. Brain, 110: 869–886.

Stoerig, P. (1993) Sources of blindsight. Science, 261: 493–494. Stoerig, P. (1998) Blindsight. In: Huber, A. and Koempf, D.

(Eds.), Klinische Neuroophthalmologie. Georg Thieme Verlag, Stuttgart, New York, pp. 375–377.

233

Stoerig, P. (1999) Blindsight. In: Wilson, R. and Keil, F. (Eds.), The MIT-Encyclopedia of the Cognitive Sciences. MITPress, Cambridge/MA, pp. 88–90.

Stoerig, P. (2001) The neuroanatomy of phenomenal vision. Ann. New York Acad. Sci., 929: 176–194.

Stoerig, P. and Barth, E. (2001) Low-level phenomenal vision despite unilateral destruction of primary visual cortex. Conscious Cogn., 10: 574–587.

Stoerig, P. and Cowey, A. (1997) Blindsight in man and monkey. Brain, 120: 535–559.

Stoerig, P. and Po¨ppel, E. (1986) Eccentricity-dependent residual target detection in visual field defects. Exp. Brain Res., 64: 469–475.

Stoerig, P., Hu¨bner, M. and Po¨ppel, E. (1985) Signal detection analysis of residual target detection in a visual field defect due to a post-geniculate lesion. Neuropsychologia, 23: 589–599.

Stoerig, P., Kleinschmidt, A. and Frahm, J. (1998) No visual responses in denervated V1: high-resolution functional magnetic resonance imaging of a blindsight patient. Neuroreport, 9: 21–25.

Supe´r, H., Spekreijse, H. and Lamme, V.A. (2001) Two distinct modes of sensory processing observed in monkey primary visual cortex (V1). Nat. Neurosci., 4: 304–310.

Ter Braak, J.W. and van Vliet, A.G.M. (1963) Subcortical nystagmus in the monkey. Psychiat. Neurol. Neurochir., 66: 277–283.

Tolias, A.S., Keliris, G.A., Smirnakis, S.M. and Logothetis, N.K. (2005) Neurons in macaque area V4 acquire directional tuning after adaptation to motion stimuli. Nat. Neurosci., 8: 591–593.

Ungerleider, L.G. and Mishkin, M. (1982) Two cortical visual systems. In: Ingle, D.J., Goodale, M.A. and Mansfield, R.J.W. (Eds.), Analysis of Visual Behavior. MIT-Press, Cambridge, MA, pp. 549–586.

Van Buren, J.M. (1963) Trans-synaptic retrograde degeneration in the visual system of primates. J. Neurol. Neurosurg. Psychiat., 26: 402–409.

van Essen, D.C. (1985) Functional organization of primate visual cortex. In: Peters, A. and Jones, E.G. (Eds.) Cerebral Cortex, Vol. 3. Plenum Press, New York, pp. 259–329.

Waitzman, D.M., Ma, T.P., Optican, L.M. and Wurtz, R.H. (1991) Superior colliculus neurons mediate the dynamic characteristics of saccades. J. Neurophysiol., 66: 1716–1737.

Weller, R.E. and Kaas, J.H. (1989) Parameters affecting the loss of ganglion cells of the retina following ablation of striate cortex in primates. Vis. Neurosci., 3: 327–342.

Weiskrantz, L. (1963) Contour discrimination in a young monkey with striate cortex ablation. Neuropsychologia, 1: 145–164.

Weiskrantz, L. (1986) Blindsight: A Case Study and Implications. Oxford University Press, Oxford.

Weiskrantz, L. (1990) Outlooks for blindsight: explicit methods for implicit processes [Review]. Proc. Roy. Soc. Lond. B Biol. Sci., 239: 247–278.

Weiskrantz, L. (1998) Consciousness and commentaries. In: Hameroff, S., Kaszniak, A. and Scott, A. (Eds.), Towards a Science of Consciousness II—The Second Tucson Discussion and Debates. MIT Press, Cambridge, MA, pp. 371–377.

234

Weiskrantz, L., Barbur, J.L. and Sahraie, A. (1995) Parameters affecting conscious versus unconscious visual discrimination with damage to the visual cortex (V1). Proc. Natl. Acad. Sci. USA, 92: 6122–6126.

Weiskrantz, L. and Cowey, A. (1963) Striate cortex lesions and visual acuity of the rhesus monkey. J. Comp. Physiol. Psychol., 56: 225–231.

Weiskrantz, L. and Cowey, A. (1967) Comparison of the effects of striate cortex and retinal lesions on visual acuity in the monkey. Science, 155: 104–106.

Weiskrantz, L., Warrington, E.K., Sanders, M.D. and Marshall, J. (1974) Visual capacity in the hemianopic field following restricted occipital ablation. Brain, 97: 709–728.

Yukie, M. and Iwai, E. (1981) Direct projection from the dorsal lateral geniculate nucleus to the prestriate cortex in macaque monkeys. J. Comp. Neurol., 201: 81–97.

Zeki, S. (1978) Functional specialisation in the visual cortex of the rhesus monkey. Nature, 274: 423.

Zeki, S. and ffytche, D.H. (1998) The Riddoch syndrome: insights into the neurobiology of conscious vision. Brain, 121: 25–45.

Zihl, J. (1980) ‘‘Blindsight’’: Improvement of visually guided eye movements by systematic practice in patients with cerebral lesions. Neuropsychologia, 18: 71–77.

Zihl, J. and von Cramon, D. (1980) Registration of light stimuli in the cortically blind hemifield and its effect on localization. Behav. Brain Res., 1: 287–298.

Zihl, J. and von Cramon, D. (1985) Visual field recovery from scotoma in patients with postgeniculate damage. A review of 55 cases. Brain, 108: 335–365.

Zihl, J., von Cramon, D. and Mai, N. (1983) Selective disturbance of movement vision after bilateral brain damage. Brain, 106: 313–340.

Zihl, J. and Werth, R. (1984) Contributions to the study of ‘‘blindsight’’. II. The role of specific practice for saccadic localization in patients with postgeniculate visual field defects. Neuropsychologia, 22: 13–22.

Introduction

When the monocular images cannot be fused, perception alternates between the two or more possible interpretations. This phenomenon (binocular rivalry) posits two basic questions, both related to the neural machinery of consciousness. One question is at what level within the visual pathways is rivalry resolved. Single unit recordings (reviewed in Logothetis, 1998) indicate that the proportion of neurons following the percept, instead of the sensory stimulation, increases from V1 (20% of recorded neurons correlated their firing rate with the perceptual report) to areas in the temporal cortex (80% of the recorded neurons). Early fMRI studies on binocular rivalry (Lumer et al., 1998; Tong et al., 1998; Lumer and Rees, 1999) also found neural activity related to consciousness in extraestriate cortex, and not in V1, in agreement with single unit recordings. More recent results,

Corresponding author. Tel.: +34-981-167000; Fax: +34-981- 167153; E-mail: fval@udc.es

however, indicate that blood oxygen level dependent (BOLD) oscillations in V1 (Polonsky et al. 2000; Tong and Engel, 2001) and also in LGN (Haynes et al., 2005; Wunderlich et al., 2005) are correlated with consciousness (see also the study by Kleinschmidt et al., 1998, with ambiguous figures). The reasons for the discrepancy between single unit and fMRI results might lie in the origins of the BOLD signal, more related to local field potentials than to spike firing (Logothetis et al., 2001; Logothetis and Wandell, 2004). In agreement with this interpretation, Gail et al. (2004) found that local field potential activity in V1 was correlated with consciousness, but firing rate was not. Taking all the studies together, it can be concluded that the neural correlates of consciousness are distributed along the ventral pathway, including V1 and quite possibly LGN, and overlap with the anatomy of the perceptual machinery.

The second question raised by binocular rivalry concerns the control of the perceptual alternation. It is well known that the alternation rate can be influenced by low-level factors (luminosity,

236

contrast, size, and motion), suggesting that perceptual alternance is controlled early in the visual pathway. Recent experiments also demonstrated that rivalry only appears when local (not global) elements are discordant (Carlson and He, 2004), and that the alternation rate depends on local neural adaptation (Blake et al., 2003; Chen and He, 2004). The lack of attentional or volitional effects (Meng and Tong, 2004) also supports that rivalry alternation is controlled at early stages of visual processing.

Neuroimaging studies, however, suggest a top-down control of perceptual changes. Lumer et al. (1998) found that activity in frontal, parietal, and extrastriate cortex in the right hemisphere was larger during rivalry than during the replay condition (a movie reproducing the perceptual transitions previously indicated by the observer in the binocular rivalry condition). Since the phenomenal changes were assumed to be similar in the two conditions, the authors proposed that the frontoparietal network was involved in the generation of the perceptual oscillation. In partial agreement with these results, Kleinschmidt et al. (1998) found bilateral activation in ventral prefrontal, frontal eye-fields, and parietal areas associated with perceptual reversal of ambiguous figures. Lumer and Rees (1999) compared rivalry with a stable-viewing condition and also found frontoparietal activation, presumably related to perceptual switching. In summary, neuroimaging results indicate that association areas generally involved in visual attention, become active during image transitions, and it has been proposed that the source of the perceptual oscillation lies in this frontoparietal network. Causality, however, cannot be claimed without lesion studies, and the few published experiments do not allow such strong claim. Bonneh et al. (2004) found that right hemisphere lesions slowed the alternation rate only when neglect was evident. O’Shea and Corballis (2003, 2005) showed that rivalry can be produced in left and right hemispheres in a split-brain patient, which contradicts a special role for the right hemisphere. Here we present the first neuropsychological study, to our knowledge, about the involvement of prefrontal cortex on binocular rivalry.

Methods

The experiment was approved by the ethics committee of the Institut Pere Mata. The patient was a man, 70 years old, who was diagnosed of schizophrenia and suffered a bilateral leucotomy (Friedman’s procedure) in 1956. A clinical MRI scanning (see Fig. 1) showed an enormous subcortical lesion that apparently isolated most of the prefrontal cortex from the rest of the brain. The dorsolateral prefrontal cortex was also damaged. The Wisconsin Card Sorting Test (WCST) evidenced a marked perseveration (93%), confirming that frontal functions were severely compromised. Despite the extensive lesion, and the years in the psychiatric institution, the patient was collaborative.

The stimuli were drawn in green and red shades and presented on a laptop screen viewed from about 70 cm. The patient wore red and green filters that were switched between the eyes at the beginning of each observation period. He was instructed to say aloud what he perceived and the experimenter (EG) held down one of two keys depending on the patient’s perception. Timing and stimulus presentation was controlled using Psychophysics Toolbox (Brainard, 1997; Pelli, 1997).

In the first experiment, the stimuli were two orthogonal gabor gratings (51 in diameter, spatial frequency 3 cpd). At the end of each 1 min observation period, one of the gratings was removed during 10 s. The purpose of this manipulation was to test the reliability of the patient’s perceptual reports. There were two sessions each comprising ten 1 min observation periods.

In the second experiment, the stimuli were a house in red shades and a face in green shades (Tong et al., 1998) presented in the center of the screen. The approximate size was 51 51. There were four experimental sessions on different days. Each session comprised ten 1 min observation periods.

Results

In Experiment 1, the patient always correctly reported the orientation of the grating presented at the end of each observation period, confirming the

accuracy of his perceptual reports. The mean duration of dominance phases (3.16 s), the percentage of time in dominance (28.12%), and the distribution of dominance phase durations (Fig. 2A), all of them closely corresponded with what could be expected for normal observers.

The results of Experiment 2 are summarized in Fig. 2B. Mean dominance duration was 3.97 s, the proportion of time in dominance was 82.29%, and the distribution of dominance phases were indistinguishable from control subjects (the two authors) depicted in Fig. 2C.

Discussion

Most long-range connections between prefrontal cortex and the rest of the brain were severed by the surgical procedure, and despite this extensive disconnection, binocular rivalry showed the typical perceptual alternation. This finding strongly suggests that prefrontal cortex does not play a causal role in the perceptual switching, in contradiction with previous interpretations of fMRI results implicating a frontoparietal network (Kleinschmidt et al., 1998; Lumer et al., 1998; Lumer and Rees,

1999). Although the lesion did not affect the right inferior frontal gyrus (the area identified in some of the previous fMRI studies), the subcortical connections in that area were damaged and consequently, the frontoparietal network should have been disrupted. It could also be argued that this patient’s brain had undergone a massive reorganization (he was operated in 1956) and the functions performed by the inferior frontal gyrus were assumed by other area(s). This interpretation, however, does not go well with the results on the WCST, that clearly shows an important deficit in frontal-lobe functioning (i.e., those functions tested by the WCST were not assumed by other areas).

It seems fair to assume that there is some contradiction between the present results and previous fMRI studies that adds up to an increasing number of contradictory findings in fMRI and lesion studies (Petersen et al., 1988; Rorden and Karnath, 2004; Fellows and Farah, 2005). A general explanation for these discrepancies would be that some fMRI activations might be epiphenomenal, or nonessential, for performing the task under study (see Rorden and Karnath, 2004, for an extended discussion).

A different explanation would be that there was some confounding variable in the fMRI experiments. For example, when rivalry was compared to a stable perception condition (Lumer and Rees, 1999), perceptual changes were present only in the rivalry condition, and change detection was accompanied by frontoparietal activation (Beck et al., 2001). Therefore, the frontoparietal activation identified in Lumer and Rees (1999) might not be related to rivalry, but to detection of changes in the environment. When rivalry condition is compared to a movie mimicking subject’s perception during rivalry (replay condition), as in Lumer et al. (1998), there are two other possible confoundings. The replay condition is run after the binocular rivalry task, thus if practice reduced activation in frontal lobes (Maccotta and Buckner, 2004; Tomasi et al., 2004; Kelly and Garavan, 2005) frontal activation might be larger during rivalry than during the replay condition. Still another possible source of confusion would be that the two conditions were

different in difficulty, being perceptual changes during rivalry much more difficult to track than during the replay condition (O’Shea and Corballis, 2005b), and task difficulty is known to correlate with frontoparietal activations (Giesbrecht et al., 2003).

Our results, together with those of O’Shea and Corballis (2003, 2005a) and Bonneh et al. (2004) cast serious doubts on the special role of the right hemisphere during binocular rivalry, and the role of prefrontal areas. This conclusion may not apply to other multistable phenomena, such as ambiguous figures, that are much more prone to attention (Meng and Tong, 2004) and are affected by prefrontal lesions (Ricci and Blundo, 1990; Meenan and Miller, 1994).

Acknowledgements

The research was financed by the Spanish Ministry of Education (BS02001-0415/PSCE). We thank

the Institut Pere Mata (Ctra. Institut Pere Mata, Reus, Tarragona 43206, Spain) for the facilities provided for this study.

References

Beck, D.M., Rees, G., Frith, C.D. and Lavie, N. (2001) Neural correlates of change detection and change blindness. Nat. Neurosci., 4: 645–650.

Blake, R., Sobel, K.V. and Gilroy, L. (2003) Visual motion retards alternations between conflicting perceptual interpretations. Neuron, 39: 869–878.

Bonneh, Y.S., Pavlovskaya, M., Ring, H. and Soroker, N. (2004) Abnormal binocular rivalry in unilateral neglect: evidence for a non-spatial mechanism of extinction. Neuroreport, 15: 473–477.

Brainard, D.H. (1997) The psychophysics toolbox. Spatial Vis., 10: 433–436.

Carlson, T.A. and He, S. (2004) Competing global representations fail to initiate binocular rivalry. Neuron, 43: 907–914.

Chen, X. and He, S. (2004) Local factors determine the stabilization of monocular ambiguous and binocular rivalry stimuli. Curr. Biol., 14: 1013–1017.

Fellows, L.K. and Farah, M.J. (2005) Is anterior cingulate cortex necessary for cognitive control? Brain, 128: 788–796.

Gail, A., Brinksmeyer, H.J. and Eckhorn, R. (2004) Perceptionrelated modulations of local field potential power and coherence in primary visual cortex of awake monkey during binocular rivalry. Cereb. Cortex, 14: 300–313.

Giesbrecht, B., Woldorff, M.G., Song, A.W. and Mangun, G.R. (2003) Neural mechanism of top-down control during spatial and feature attention. NeuroImage, 19: 496–512.

Haynes, J.D., Deichman, R. and Rees, G. (2005) Eye-specific effects of binocular rivalry in the human lateral geniculate nucleus. Nature, 438: 496–499.

Kelly, A.M. and Garavan, H. (2005) Human functional neuroimaging of brain changes associated with practice. Cereb. Cortex, doi: 10.1093/cercor/bhi005.

Kleinschmidt, A., Bu¨chel, C., Zeki, S. and Frackowiak, R.S.J. (1998) Human brain activity during spontaneously reversing perception of ambiguous figures. Philos. Trans. R. Soc. Lond. B Biol. Sci., 265: 2427–2433.

Logothetis, N.K. (1998) Single units and conscious vision. Philos. Trans. R. Soc. Lond. B Biol. Sci., 353: 1801–1818.

Logothetis, N.K., Pauls, J., Augath, M., Trinath, T. and Oeltermann, A. (2001) Neurophysiological investigation of the basis of fMRI signal. Nature, 412: 150–157.

Logothetis, N.K. and Wandell, B.A. (2004) Interpreting the BOLD signal. Annu. Rev. Physiol., 66: 735–769.

239

Lumer, E.D., Friston, K.J. and Rees, G. (1998) Neural correlates of perceptual rivalry in the human brain. Science, 280: 1930–1934.

Lumer, E.D. and Rees, G. (1999) Covariation of activity in visual and prefrontal cortex associated with subjective visual perception. Proc. Natl. Acad. Sci. USA, 96: 1669–1673.

Maccotta, L. and Buckner, R.L. (2004) Evidence for neural effects of repetition that directly correlate with behavioral priming. J. Cogn. Neurosci., 16: 1625–1632.

Meenan, J.P. and Miller, L.A. (1994) Perceptual flexibility after frontal or temporal lobectomy. Neuropsychologia, 32: 1145–1149.

Meng, M. and Tong, F. (2004) Can attention selectively bias bistable perception? Differences between binocular rivalry and ambiguous figures. J. Vis., 4: 539–551.

O’Shea, R.P. and Corballis, P.M. (2003) Binocular rivalry in split brain observers. J. Vis., 3: 610–615.

O’Shea, R.P. and Corballis, P.M. (2005a) Visual grouping on binocular rivalry in a split-brain observer. Vision Res., 45: 247–261.

O’Shea, R.P. and Corballis, P.M. (2005b). In: Alais, D. and Blake, R. (Eds.), Binocular rivalry in the divided brain. MIT Press, Cambridge, MA, pp. 301–316.

Pelli, D.G. (1997) The video toolbox software for visual psychophysics: transforming numbers into movies. Spatial Vis., 10: 437–442.

Petersen, S.E., Fox, P.T., Posner, M.I., Mintun, M. and Raichle, M.E. (1988) Positron emission tomographic studies of the cortical anatomy of single word processing. Nature, 331: 585–589.

Polonsky, A., Blake, R., Braun, J. and Heeger, D.J. (2000) Neuronal activity in human primary visual cortex correlates with perception during binocular rivalry. Nat. Neurosci., 3: 1153–1159.

Ricci, C. and Blundo, C. (1990) Perception of ambiguous figures after focal brain lesions. Neuropsychologia, 28: 1163–1173.

Rorden, C. and Karnath, H-O. (2004) Using human brain lesions to infer function: a relic from a past era in the fMRI age? Nat. Rev. Neurosci., 5: 813–819.

Tomasi, D., Ernst, T., Caparelli, E.C. and Chang, L. (2004) Practice-induced changes of brain function during visual attention: a parametric fMRI study at 4 Tesla. Neuroimage, 23: 1414–1421.

Tong, F. and Engel, S.A. (2001) Interocular rivalry revealed in the human cortical blind-spot representation. Nature, 411: 195–199.

Tong, F., Nakayama, K., Vaughan, J.T. and Kanwisher, N. (1998) Binocular rivalry and visual awareness in human extrastriate cortex. Neuron, 21: 753–759.

Wunderlich, K., Schneider, K.A. and Kastner, S. (2005) Neural correlates of binocular rivalry in the human lateral geniculate nucleus. Nat. Neurosci., 8: 1595–1602.

Our global sensory experience is not generally compartmentalized to rigid perceptual categories corresponding to the different sensory domains. However, traditional research in perception has not often crossed sensory modality boundaries. The last couple of decades have changed this landscape, and crossmodal research has now become one of the richest fields of neuroscientific enquiry.

This final section explores how visual perception is influenced by the other senses, and how our brain integrates visual information with information from the other sensory modalities. The five chapters herein attack the problem of crossmodal perception from multiple fronts, ranging from intracellular recordings in visually deprived animals to fMRI studies in human synesthetes.

David Burr and David Alais discuss how visual and auditory signals are combined, how they are aligned in time, and how attentional resources are allocated to these two modalities.

Noam Sagiv and Jamie Ward review several types of crossmodal interactions in synesthesia and discuss how they may relate to crossmodal interactions in normal perception.

Salvador Soto-Faraco and colleagues find that crossmodal dynamic capture (whereby the perceived direction of motion in the auditory modality is influenced by visual motion) is a relatively automatic process, robust to various top-down factors.

Maria V. Sanchez-Vives and colleagues present new physiological and anatomical data concerning audio-visual interactions in the primary visual cortex of the visually deprived cat.

Kristin Porter and Jennifer Groh discuss their recent findings that sound location in the primate inferior colliculus is encoded by a rate code, which is a different format from the place code used to represent the location of visual stimuli.

Susana Martinez-Conde

As Ernst and Bu¨lthoff (2004) point out in their excellent review, the key to robust perception is the efficient combination and integration of multiple sources of sensory information. How the brain achieves this integration — both within and between sensory modalities — to form coherent perceptions of the external environment is one of the more challenging questions of sensory and cognitive neuroscience. Neurophysiologically, sensory interactions have become well documented over several decades. More recently, perceptual research combined with solid modeling is beginning to complement the neurophysiology. This chapter summarizes some recent psychophysical work on audiovisual interactions from our laboratories.

Corresponding author. Tel.: +39-050-3153175; Fax: +39-050-315-3210; E-mail: dave@in.cnr.it

Pitting sight against sound: the ventriloquist effect

Ventriloquism is the ancient art of making one’s voice appear to come from elsewhere, exploited by the Greek and Roman oracles, and possibly earlier (Connor, 2000). We regularly experience the effect when watching television and movies, where the voices seem to emanate from the actors’ lips rather than from the actual sound source. The original explanations for ventriloquism (dating back to the post-Newtonian scientific efforts of early 18th century) assumed that it was based on the physical properties of sound, that performers somehow projected sound waves in a way to appear to emanate from their puppets, using special techniques (Connor, 2000). Only relatively recently has the alternative been considered, that ventriloquism is a sensory illusion created by our neural systems. These explanations assume that vision

244

predominates over sound, and somehow captures it (Pick et al., 1969; Warren et al., 1981; Mateeff et al., 1985; Caclin et al., 2002).

More recently, another approach has been suggested for combination of information. Several authors (Clarke and Yuille, 1990; Ghahramani et al., 1997; Jacobs, 1999; Ernst and Banks, 2002; Battaglia et al., 2003) have suggested and shown that multimodal information may be combined in an optimal way by summing the independent stimulus estimates from each modality according to an appropriate weighting scheme. The weights are given by the inverse of the variance (s2) of the underlying noise distribution (which can be assessed separately from the width of the psychometric function). For auditory and visual combination this can be expressed as

where S is the optimal estimate, SA and SV are

the independent estimates for audition. wA and wV are the weights by which the unimodal estimates are scaled, and are inversely proportional to the auditory and visual variances s2A and s2V

This model is ‘‘optimal’’ in that it combines the unimodal information to produce a multimodal stimulus estimate with the lowest possible variance (i.e., with the greatest reliability see Clarke and Yuille, 1990).

We (Alais and Burr, 2004b) tested the predictions of Eq. (1) directly by asking observers to localize in space brief light ‘‘blobs’’ or sound ‘‘clicks,’’ presented first separately (unimodally) and then together (bimodally). The purpose of the unimodal presentation was to measure the precision of these judgments under various conditions to provide estimates of variances s2A and s2V: Fig. 1A shows typical results for four different stimuli: visual blobs of various degrees of blur and auditory tones. The data are fitted by cumulative Gaussian curves from which one can extract two parameters: the best estimate of perceived position

^ (often also referred to as the ‘‘point of subjective

S

A

0.5

0.0

Displacement of probe (deg)

Fig. 1. (A) Unimodal psychometric functions for localization of an auditory stimulus (green), and visual Gaussian blobs of variable size. Localization for fine blobs is very good (as indicated by the steep psychometric functions), but is far poorer for very blurred blobs. Auditory localization is in between, similar to visual localization with 321 blobs. The curves are best-fitting cumulative Gaussian functions. (Reproduced with permission from Alais and Burr, 2004b.) (B) Bimodal psychometric functions for dual auditory and visual presentations. In the ‘‘conflict’’ presentation, the visual stimulus was displaced rightward by 51 and the auditory stimulus leftward by the same amount (as indicated by vertical lines). The 41 stimulus (black symbols) tend to follow the visual standard, the 641 stimulus (blue symbols) the auditory standard and the 321 stimulus (red symbols) falls in between. The curves are not best fits to the data, but predictions from the Bayesian model described in Eqs. (1)–(4). Modified from Alais and Burr (2004b, p. 258), Copyright, with permission from Elsevier.

equality’’ or PSE), given by the point where the curves crosses 50%, and the threshold for making