belongs to the same sensory space. A baby a few months old can apparently distinguish the forms of objects equally well by tactile or visual manipulation. Eye– hand coordination may already exist at birth, but it is not stable and can be influenced. Since vision develops rapidly, cross-modal organizations changes during the first halfyear of life, and afterwards the hand becomes a fascinating and important tool for grasping things and an instrument for the eye (Streri, 1993).

According to this theory there is a kind of primitive unity of the senses at birth, and it is the repeated interaction with the environment that leads to differentiation and autonomy of separate sensory systems. The more specific representations of object properties, such as color, form, size, etc., may develop gradually. As a consequence of differentiation, the coordination of information about an object from different sensory systems requires integration at a higher brain level.

This theory suggests the possibility that we enter this world with a form of synesthesia, and that we grow away from it. Synesthesia refers to a strange phenomenon that is sometimes encountered in adults, namely that words or letters evoke an impression of color, or that sounds have a taste. In an adult light activates the visual center, and this activity can be measured by non-invasive methods like PET (positron emission tomography) or fMRI. A flash of light on the eye of a newborn does not cause a reaction in the part of the brain where the visual center is later located. According to this theory, the specific contacts between the eye and the visual areas in the brain are formed gradually as the brain matures. When some adults with synesthesia feel, for example, that ‘purple smells sweet’, this may be because something went wrong in establishing the normal connections and some of the original cross-connections between modalities have been maintained. In this way the senses ‘mix’ to produce the strangest phenomena, which are said to be of great pleasure to the persons who have them.

Observations also suggest that some connections between sensory modalities develop during adulthood. The purpose here may be that information that belongs together, for instance in the auditory and visual systems, leads to amplification when both occur simultaneously (e.g. looking at the mouth while listening to somebody talking). Such processes may also be crucial for learning.

Vision and natural science

Psychophysics is the name of a discipline that studies the responses of organisms to stimulation. The safe mechanistic approach is to study small responses at or near threshold. In humans one can, for instance, measure how much light energy is necessary to produce an impression of light and color. Such psychophysical measurements are important for our understanding of the functioning of the nervous system, and they consequently supplement neurophysiological studies.

A comparison of the information capacity of the auditory and visual systems shows that vision has been allocated the most resources. This dominance of the visual system is anatomically reflected in that about 60 percent of all nerve fibers from a sensory organ to the brain come from the eyes. From the ear about 30 000 nerve fibers transmit acoustic information to the brain, but from each eye it is between 1 and 2 million. In the cortex there are about 800 000 nerve cells in the auditory center compared with about 500 million in the visual centers.

Below is a simplified ‘bottom-up’ characterization of the main physical, neural and mental processes that lead to vision:

1.Imaging – this process accounts for the spatial light distribution on the retina by the optical eye media (cornea, pupil, lens, vitreous, etc.). In recent years, the development of adaptive optics has allowed the quality of the images projected on the retina, as well as those taken of the retina, to be improved.

2.Detection and discrimination – detection refers to the transformation of the energy in the light quanta absorbed by the photoreceptors (rods and cones) to electrical potentials (transduction) and to the neural activity that follows. This includes the chemistry and genetics of visual pigments. A comparison of the weakest stimulus (the minimum energy) required to elicit a sensory impression – the threshold value – with that required to elicit a response in nerve cells makes it possible to correlate psychophysics with molecular biology and neurophysiological processes. There may be a difference between detection (‘seeing something’) and discrimination of what that something is.

3.Neural encoding and signal transmission – the input from about 100 million receptors in the retina converges on somewhere between 1 and 2 million retinal ganglion cells. After reception there are four functional cell types that organize the information from a retinal image, decomposing it and encoding it before the signals are transmitted to higher brain centers. Already at the retinal level it is possible to suggest hypotheses that connect neurophysiology and psychophysics (e.g. about spectral sensitivity and receptors).

4.Adaptation – the eye can adjust to changing light levels, from moonlight to the brightest sunlight, as well as to the changing colors of light. Light adaptation covers an intensity range greater than 1 1012 and thus cannot be due to regulation by the pupil (which accounts for a factor of only about 10 in intensity).

5.Differentiation and structure – we can imagine a cooperation of diverging, converging and parallel pathways in the retina and in the cortex that all receive input from a common set of receptors. Processing of this information may take place in a hierarchy of functional units, or in cell types that treat different components of the image separately. Today, about 40 areas in the cortex that deal with different aspects of vision have been identified.

6.Identification, recognition and interpretation – this represent the central brain processes which, in addition to processing in the visual cortex, include the other senses and higher mental activities, like memory, context, will, etc. Although it has been claimed that mental activity is ‘nothing more’ than complex and highly organized neural activity, throughout this book we shall use the concept of mental activity (qualia) to mean what we intuitively know as such, through personal experience and introspection. The gap in our understanding of mental and neural activities is still too large for us to use a common language for both.

The emotions and the actions of an individual (e.g. facial expressions, speech, gestures, movement, etc.), that follow a chain of physical and neural processes, reflect the personality and the cultural background of the individual. They are organized at a level that cannot, for reasons mentioned earlier, currently be explained by natural science, within a detailed scheme of cause and effect.

As we have seen, the visual process can be studied at many different levels and by many disciplines. The list above, which is not complete, shows a progression from physical stimulation towards ‘higher’ functions that are governed by complex mental activities. The imaging under point (1) is satisfactorily explained by physical and physiological optics. Biophysics, molecular biology and neuroscience deal primarily with points (2)–(5), investigating processes in single cells, the interactions between individual cells and the interplay of many cells in neural networks.

Signals from about 100 million rods and 6 million cones converge on between 1 and 2 million ganglion cells and an equal number of nerve fibers. In the brain we see divergence of these inputs as these ganglion cells successively establish contact with roughly 500 million nerve cells. The perception of contrasts, contours, colors, textures, three-dimensional space, movement and orientation, to mention a few elements, depends on connections and interaction between many cells in a network. Later, we will look into these processes in more detail.

Interpretation, understanding and behavior are subject to scientific investigation within the fields of neuropsychology and cognitive psychology. The visual process is, due to its complexity, studied by many overlapping disciplines.

Decomposition of ‘the optical image’

When an organism is adapted to the prevailing lighting conditions, the most important features of the visual environment are differences in lightness and color – what we normally call contrast. Contrast relative to a background is a prerequisite for seeing anything at all. Without contrast, neither the form of an object nor its movement can be detected (identification of the object usually requires even larger contrasts than detection).

Contrary to what is often believed, the projection of the optical image on the retina to the brain is not a simple transmission – such as, for instance, a TV broadcast.

Rather, different types of retinal cells seem to split up an image into a set of different, fundamental components. The pattern of light and shadow in a scene imaged on the retina is transformed in complicated ways by the interaction of retinal cells, and the resulting signals that are transmitted to the higher brain areas are subject to further decomposition. This process involves several representations or abstract ‘maps’ of the optical image.

What do these ‘maps’ look like? What is the structure and meaning to the seemingly chaotic activity evoked in millions of nerve cells by even the simplest stimulation? First of all, we know that already in the retina cells respond only to stimuli that are situated within a small, limited area of the visual field. In the central part of the eye, the fovea, this territory of a particular cell – called its receptive field – comprises a limited number of photoreceptors. According to the traditional view, the retinal cell is marginally affected by what happens outside its receptive field. Recent research, however, has demonstrated that more remote areas of the retina may affect the cell’s response. Taken together, a million ganglion cells (each of which has direct contact with a nerve fiber) cover the whole visual field, each cell being dedicated to processing input from a small, fixed position on the retina. Some transient ganglion cell types are specialized for signaling temporal or spatial changes at their particular location in the optical image, whereas others respond in a sustained fashion to stationary, colored surfaces.

Higher-level brain cells are more specialized. Some cells respond only to a narrow range of wavelengths while other cells are activated only when their receptive fields are exposed to a contour of a certain angular orientation, e.g. to an oblique line drawn on a piece of paper. If this line were rotated 10 away from its orientation, these cells would stop responding, and others would take over. Some cells respond in a selective fashion to size and contrast. While some respond to broad stripes, others prefer narrow stripes of the proper orientation. Still other cells signal movement, but usually only if it occurs in one particular direction, for instance from left to right within their territory of visual space.

From the earliest retinal levels to the visual cortex there thus seems to be a parametric decomposition of physical properties into different components, or dimensions. The extraction of visual form, for instance, may be achieved in many ways and from several visual inputs. It may be constructed by means of light, color, movement, contours or even depth. Information derived about these attributes at an early level is sent to the higher brain for further processing by way of diverging and converging visual pathways, in the magno-, parvoand koniocellular cells. We shall return to these important cell systems later.

Different aspects of an image seem to be distributed to different areas of the brain for further structuring before the information is integrated to prepare for a course of action. One can imagine a letter, for instance the letter ‘B’, which to the cells is composed of many contour segments of many different orientations – short quasistraight contours that appear curved when taken together, with filled-in areas of even black contrast in between. Bits and pieces of the letter are detected separately before