1Introduction

back and forth across the retina without in any way undermining our ability to orient ourselves and to move around.

In this introduction I shall call attention to some fundamental elements pertaining to the ‘act of seeing’ and construct a mosaic where, at first, the parts may seem only loosely connected. Nevertheless, all elements are important at one level or another of our visual encounters with the external world. Many aspects of vision are enigmatic, and we shall, for a moment, dwell on some philosophical questions. I hope that this broad background may cause some reflection about the active role we play when engaged in ‘seeing with the mind’ as opposed to ‘seeing with the eyes’, and that these considerations may help us to better understand Virgil’s destiny and similar stories.

The electromagnetic spectrum elicits an orderly sensation of colors, and vibration frequencies of strings give rise to a sound spectrum and to music. These are examples of how physical stimulation is transformed by sense organs to give the physical dimension a quality and a structure specific for each sense. Sensory organs and neural processes exhibit many physical and logical structures. However, in order to comprehend vision, we need to take a broader perspective. We need to go beyond the idea of objects eliciting sensory processes and to also ask what are the important perceived qualities of each stimulus in the image. As subjects, we experience qualitative properties, like for instance colors, in a direct and immediate way, but are they a property of the external world, of ‘reality’, or constructs of our brains? Maybe they are some intriguing attribute of the signals that the brain receives from the eyes? Is it at all possible to find an answer to such questions by studying, for instance, the responses of visual neurons to optical stimulation of the eye?

The eye and the visual sense have always interested philosophers and scientists. One reason for this interest, and especially that of many physicists, lies close at hand. Vision is our most important tool for understanding the organization of objects and events in the external world in which we are moving, thus it is an important means of obtaining experience and knowledge. Relying on their senses, physicists in earlier times had a direct approach to the natural phenomena they studied. The impression of order mediated by the senses was crucial in experimental investigations. The famous Scottish physicist James Clerk Maxwell (1831–1879) said that

‘if the sensation which we call color has any laws, it must be something in our own nature which determines the form of these laws’ (Maxwell, 1872).

Today, neuroscientists and biophysicists are attempting to elucidate these laws at the level of sensory cells in the retina, in rods and cones, and in nerve cells in the brain. In addition to studying the chemical and electrical phenomena that follow the stimulation of photoreceptors with light, and the neural processing thereafter, modern science also focuses on the structuring of perceptual attributes which, according to earlier beliefs, follows from the functioning of the peripheral sense organs. However, as we shall see later, these relationships are not that simple.

In order to arrive at an understanding of the external world, we rely on a variety of sensory experiences, of light, space, gravity, sound, taste, smell, etc. Scientists from

many different fields study sensory processes, and they are often integrated in an interdisciplinary environment. Vision science has become an advanced area of research, and it is a key example of how progress in science depends on collaboration of many disciplines.

Visual perception

A naive realist would perhaps tell us that ‘the way the world appears to us is the way it really is’. However, you need not have studied vision to discover that this statement is problematic. Our conscious perception of, for instance, a place or a situation, is different from a simple copy, like a photograph. Euclid (Greek philosopher; ca 300 BC), for instance, thought that visual experience was mediated by ‘rays of vision’ projected from the eyes to touch the surfaces of objects like a million invisible fingers (Zajonc, 1993). It is not clear to what extent these rays of vision were considered dependent on learning and interpretation, being ‘rays of thought and comprehension’ leading to mental images, or if they were regarded as carriers of the immediate and subjective elements, the qualitative sensory attributes of external objects (like color, sound, etc.).

The poetic idea of ‘rays of vision’ may appear strange to us, but according to more modern views it is also too simple to think that the visual scenes one perceive originate solely from images, i.e. from light reflected off objects and imaged optically on the retina, like in a camera. Between these opposing views one finds other theories that take neural mechanisms into account to a variable degree. Such theories allow for different levels of processing, from the lowest retinal neurosensory level to the higher cognitive level(s) of the brain. There is no clear dividing line between a ‘bottom-up’ explanation and a cognitive interpretation (‘top-down’ view) of visual perception. It is a border that will be moved in this or that direction as our knowledge about the brain increases. Maybe some day a synthesis will be reached where the various approaches are not exclusive, but complement each other (Spillmann and Dresp, 1995). From Oliver Sack’s account of Virgil’s story, and from modern neuroscience, we know that vision is not a passive process – it has an active mental component.

In the seventeenth century Rene Descartes (French mathematician and philosopher; 1596–1650) was already wondering why we see the objects so vividly ‘out there’, in three-dimensional space, when the brain only has two-dimensional images to start with, which are relatively diffuse and distorted. In addition, the images on the retinas are inverted (Figure 1.1). One might explain this apparent paradox by saying that our perception of the things around us is not congruent with the things themselves, but more like a hypothesis, or a model. According to Richard L. Gregory and the German physiologist Hermann von Helmholtz, this model is a mental construct that goes beyond what is mediated directly by the retinal image and the sensory organs, building on earlier experience and on our memory. Consider how we, as children, establish a relationship with our environment. We crawl around touching things, weighing them in our hands and putting them in our mouth. As children we are curious and explore

Figure 1.1 Rene Descartes and the retinal image. Light rays from the object points VXY form an inverted image RST on the retina. The drawing describes an experiment where Descartes made a little hole in the back of a bull’s eye and covered it with a thin paper. When the front of the eye was directed towards a well-lit scene, he could see a distorted image of this scene on the paper. The drawing is from La Dioptrique (1637).

the environment with all our senses. In this way we build a relationship between the outer world and ourselves, giving the objects an identity.

It is tempting, for a moment, to interpret Euclid’s rays of vision as being the qualitative attributes that the individual projects upon the things ‘out there’ in front of us, for example the form and the red color of an apple. Red is a qualitative sensory experience that is normally associated with long-wavelength light being reflected from the apple’s surface. However, it does not require much thinking to realize that our experience of the redness of an apple cannot be deduced from this physical

they are not, or you feel that the train moves backwards. The phenomenon is a physiological consequence of having been ‘adapted to’ movement in one direction. One interpretation of the motion aftereffect in terms of a neural correlate is that your movement-sensitive neurons need a re-calibration after having been exposed and adapted to movement in one direction for some time. Another example of apparent movement is shown in Figure 1.17.

The motion after-effect is a phenomenon of which we can say: ‘that which moves, does not move at the position where it is, nor at the position where it is not’, in analogy with the Greek philosopher Zenon’s (about 460 BC) statement about physical movement.

Like the perception of color and depth, perceiving motion is thus also a qualitative experience. If you are still in doubt, you may think about the phenomenon of ‘motion blindness’, a strange disability that has been shown to occur in persons with certain brain injuries. Such a person sees the moving object, like a moving car, in different positions, but he or she cannot observe the movement from one position to the next (Zihl et al., 1983; see also p. 400).

This discussion may have made clear why we suggest it is a good idea to reinterpret and revitalize Euclid’s ‘rays of vision’. We can choose to view them as the immediate, qualitative attributes of our visual experience (like color, lightness, depth, movement, etc.) that we so effortless project out onto the objects of the outside world. These and other acquired properties – collection of elementary perceptions that together make up our model of the world – contribute, together with our interpretation and our understanding, to the mental image in front of us. The importance of interpretation and meaning is strikingly demonstrated in so-called cognitive, reversible ‘visual illusions’, where we can choose among alternative meanings. Figure 1.3 shows some examples. Figure 1.3(a) is called ‘young woman/ old woman’, and in Figure 1.3(b) we can more or less decide if we want to see a vase or two faces in profile, facing each other. Figure 1.3(c) and (d) are also ambiguous; that is, our perception is multistable and alters between two perspectives, one from above and the other from below.

Experience and qualia

The physical entity that ‘elicits’ a perception of, for instance, ‘red’, or the chemical agent eliciting a subjective taste-experience, is called a stimulus (one stimulus, many stimuli). The qualitative experience itself may be called qualia. One of the most difficult problems to account for scientifically is the subjective experience of sensory qualities (qualia) and their relationship to the physical and chemical stimulation of a sense organ. Even if the perception of ‘red’ may arise when the retina is stimulated by long wavelength light, this is neither a sufficient nor a necessary condition.

Figure 1.3 Visual illusions in which you can choose what you want to see. You see either a young girl or an old lady (a); a vase or the profiles of two people facing each other (b); a staircase and a cube that can be seen in different perspectives and from different directions – from above or from beneath (c, d). We see only one of these interpretations, never both at the same time. With some practice you can choose which one you want to see.

The perceptions resulting from sensory stimulation – the pain from a prick of a needle, the sour taste of a lemon or the coldness of ice – are subjective and private experiences, even though the feeling that something is ‘cold’ usually requires the stimulation of specific cold receptors in the skin. Although the receptors’ reactions upon stimulation are measurable, my experiences of qualities like pain, smell, taste, sound, color, etc. are not directly available for observation by another person. However, under certain circumstances they can be described indirectly by means of observing the subject’s reaction to stimulation. For instance, it is not possible to know if my experience of ‘redness’ or ‘greenness’ is the same as yours, but this has not prevented us from agreeing on ways to arrange our color perceptions in a color space. In color vision, phenomenological–geometrical systems, like color circles and color scales, have been devised to organize our qualitative experiences (see Figures 5.1–5.3).

The response of sensory cells to stimulation, in the form of electrical potentials and chemical processes, is observable and measurable in an objective and scientific manner. Yet how can our conscious subjective feelings of ‘coldness’, or of a bitter taste, or of the redness of an object, arise from changes in electrical potentials of nerve cells? Are these realms of subjective and objective realities compatible? A common answer to this question is to point to a chain of stages and reactions,

beginning with the peripheral event (the physical or chemical stimulation), continuing with the intermediate nerve reactions, and ending with conscious awareness, or the whole organism’s response. This chain consists of transducer mechanisms (for instance the rods and cone receptors in the primate retina), signal pathways (the nerve cells and channels in the retina and the pathways from the eye to the brain), specialized receiver stations in the brain (which are different, for example, for lightness contrast, for movement, for color, etc.), and binding mechanisms that join appropriate elements together in conscious experience. Strictly speaking, however, nobody has yet identified or even localized the brain processes that lead to the ‘redness’ or the ‘greenness’ of, for instance, a pepper, or to the ‘sourness’ of a lemon.

Although it has been made plausible that qualities and feelings are mediated by, or originate in, the brain, they are, as we have seen, often related to – or projected upon – objects and events out there in the physical world. Following the amputation of a limb, for example, people are left with perceptions associated with the non-existing (phantom) body part. We see the redness of the pepper lying on the table, and not inside our heads, and we consider the sourness of the lemon as a property of its pulp. The red percept is projected onto the pepper and sourness is also something that obviously belongs to the external object. One is tempted, like Hermann von Helmholtz, to regard such qualitative attributes as natural symbols – as potential carriers of meaning – open to fast, direct and non-verbal interpretations of the state of a fruit, for example, to determine if it is ripe, fresh or foul.

Experience and language

Every scientific endeavor requires a specialized language. It is necessary to identify the subject of scientific inquiry and to find a line of demarcation relative to related fields. In vision it is necessary to distinguish stimuli from percepts, i.e. to separate the abstract physical description of measurable quantities from a symbolic representation of subjective phenomena. To meet this end, nomenclatures are continuously being developed and refined. Let us take a look at a few misconceptions that originate in a less precise use of language.

We have already pointed out that the word ‘color’ belongs to the realm of subjective experience. Related terms, denoting correlated physical entities, are spectral distribution, wavelength, chromaticity, or something else, depending on the topic of discussion. Lightness and brightness refer to the subjective impressions of intensity of a surface (of being more or less bright), whereas the related physical terms may be luminance, luminance ratio or luminance contrast. Light also refers to our experience, whereas electromagnetic radiation refers to a physical phenomenon. In many other cases, our language does not (yet) have separate words for the subjective and the physical. This is still the case when referring to size, movement, texture, contours, depth, etc.

However, even with the most refined language, there will always be something left out when talking about subjective experiences. Some of our perceptions and emotions, and some of our thinking, can be expressed symbolically by words, but not all. Language can build a bridge between ‘subjective’ inner experiences and ‘objective’ outer events, but of course the immediacy and the quality of the real experience is lost in the symbolic, verbal descriptions – although something else is gained. The written word, for example ‘red’, or the sound of the word when we say it aloud, are substitutes for ‘the real thing’. Language uses written or spoken symbols to express or point to feelings and qualities. Language symbols point to something meaningful for those who, through their own experience, know what is meant by, for instance, a ‘yellow banana’ and a ‘green banana’. Communication presupposes that one has such abilities.

The limitations of language also become clear when you try to describe or explain the experience of ‘red’ or ‘green’ to a color-defective person who cannot distinguish between them. This task is utterly frustrating, but it is, in fact, not easy either to use common language to explain a color that you saw yesterday for a person with normal color vision, or even to ourselves. Usually, therefore, we are content with using words only as pointers to an implicit meaning.

For the especially interested, a thorough discussion of this philosophical topic can be found in the article by B. A. Farrell, in particular, in The Philosophy of Mind (1962), and in Ludwig Wittgenstein’s Remarks on Color (1977).

The concept of consciousness normally includes our ability to perceive – to be aware of – and to reflect on what we experience, and to eventually give verbal expression to it. If one is unable to distinguish colors like red and green, it has some consequences for one’s awareness and view of the world. This applies to about 8 percent of all men and 0.45 percent of all women whose color vision deviate from the normal. The most common form of deficiency is a reduced ability to distinguish between red and green. It is assumed that people with normal color vision have the same qualitative impressions as the color-defective, but that their color palette is richer.

When a verbal account of a common experience is missing, i.e. when a person has problems with putting certain relations into words in his or her mother tongue, one may conclude that there is a conceptual deficiency, or some kind of ‘concept blindness’. When important concepts are missing, one often gets the impression that conscious experience is also suffering; we have a tendency to see what we expect to see, or what we are used to. What then is the more fundamental process, sensation or reflection? Is Aristotle’s statement still true that nothing can exist in consciousness without first passing through the senses, or is this like asking what comes first, ‘the chicken or the egg?’

This question about the role of sensation and reflection, from early on considered to be of great importance, has divided philosophers into two groups: the empiricists

against the rationalists. The empiricist holds that everything we can know about the world is based on sensory experience, whereas the rationalist thinks that we can make true statements about the world independently of our experience, and uses logic and mathematics to strengthen his case.

To the child, the phenomena are very much present, and seeing and hearing come before words. The intellectual mind will find it hard to separate what it sees from what it knows or believes. This lack of ‘direct access’ can sometimes be an obstacle to understanding, such as when confronted with another culture or with art. A painting, for instance, may represent a content and a meaning that cannot be fully described in words. This other reality – if it is impressionistic, like that of Claude Monet, or expressionistic, like that of Edward Munch – can in fact be overshadowed by the use of common language (Gombrich, 1972; Grenness and Magnussen, 1989).

We have said that our sensations, thinking and emotions are private and not directly accessible for others. Yet this does not prevent us from trying over and over again ‘to point to’ our feelings, by gestures and signs, or by expressing their contents in a verbal language. This language can, of course, be developed and refined to create new symbols of experiences and knowledge. Neuroscience has reminded us that learning, including visual learning and recognition, depends on the active use of our sensory organs, and that use and experience bring about a certain structuring of the nervous system. As described below, Thorsten Wiesel and David Hubel (Nobel Prize Laureates, 1981) have shown, for example, that adult cats are only able to distinguish types of visual stimuli they have been exposed to in their first weeks with open eyes. These stimuli leave lasting traces in the nervous system (Hubel and Wiesel, 1963).

It appears that normal sensitivity to a visual stimulus depends on the organism receiving adequate stimulation in an early phase of development. If such stimulation is wanting, or if it is too restricted, this can influence the nervous system’s structure and functioning later in life.

A classical example from Thorsten Wiesel and David Hubel’s research is the effect of isolating kittens in a cage and allowing them to see only vertical black and white stripes in the first few weeks after they have opened their eyes. These kittens will later be unable to distinguish horizontal black and white stripes. Hubel and Wiesel (1963) found that the cells in the cat’s visual cortex were particularly selective for and could only discriminate between stimuli that had been presented to them during the first few weeks of vision. Primate cells also possess this property, but in higher vertebrates the critical period lasts longer. If cells are deprived of adequate visual stimulation during the critical period, these cells will not die, but will be recruited for other functions.

This may throw some light on the case of Virgil, the blind man who regained his eye sight. It may be that that neurons in his brain which were originally designed to deal with and interpret visual information had been allocated for other purposes. Supernormal tactile capabilities may have originated through frequent use. The reports of von Senden (1960) seem to support this view.

This conditioning or habituation of the nervous system seems like a ‘bottom-up’ process of adjusting and adapting the sensitivity of the nervous system to certain features of the external stimulus. This activity leads the nervous system to acquire a structure and hence ‘knowledge’ that can be used later to interpret sensory impressions in a ‘top-down’ process where new sensory information is compared with previous information. This combination of ‘bottom-up’ and ‘top-down’ processes may appear as a closed loop with no beginning or end, except that it is limited to a particularly plastic period of the animal’s life. It may be difficult to decide where one process begins and the other one ends during the plastic learning period. Perhaps it is easier to assume that both processes are active at the same time.

In this context it would be of great interest to identify the predispositions (heredity, instinct, etc.) that newborns carry with them and which allow them to develop more complicated perceptions. Studies of sensory experience, and particularly sensory deprivations in animals (when stimulation is too weak or absent), can contribute to a better understanding of the complex interplay between heredity and environmental influences.

Through modern non-invasive techniques for monitoring brain activities, like functional magnetic resonance imaging (fMRI), it has become possible to study some of the problems mentioned above experimentally. Modern neuroscience has opened new gates to the understanding of cognitive functions, and visual science is one of the most important sources for such knowledge.

‘The sweet smell of purple’

There are many unanswered questions related to how ‘the hypothesis of the world’ – i.e. our understanding of reality – comes into being. For example, how are sensory data treated by the nervous system before they become conscious perceptions? We shall repeatedly come back to this difficult problem throughout this book, but let us here just take a brief look at an interesting phenomenon that might be of relevance.

Normally, it is assumed that the dimensions of what we might call the sensory spaces of different sensory modalities are separated at birth, and that they gradually become related to each other as a result of experience. For instance, vision is separated from hearing, and both are separated from touch. Nevertheless, one may ask whether sensory structures emerge during development (this is the standpoint of the empiricists), or if they already exist as abstract representations in the newborn. In the latter case (which resembles the standpoint of the rationalist), what is left for the child is to further develop, modify and refine an already existing program through active engagement and experience.

One may wonder if our understanding of the world around us depends more on abstract properties that are extracted from sensory data than on the sensory data themselves. One theory claims that babies are born with an undifferentiated sensory system. A newborn does not ‘know’ if it sees, hears or touches something. Everything