Structure of the Document

First we will focus our attention on the human perceptual process, its input channels (HIC) and the characteristics of computer output media (COM) (Chapter 2 ).

Then, we will approach the process of human control and manipulation, dwelling on the characteristics of the human output channels (HOC) and the computer input modalities (CIM) (Chapter 3 ).

In the next chapter ( 4 ) the issue of the interaction process will be dealt with, addressing aspects of input/output coupling and temporal and spatial synchronization.

Gradually moving from basic processes to more abstract levels, we will describe aspects of cognition and learning, its representational architecture in machine and man, as far as possible, and on the issue of interacting agents (Chapter 5 ).

Finally, in the last chapter, a number of scenarios, or rather ``dreams'' will be depicted, elucidating fundamental aspects put forward in this taxonomy.

Perception

 

In this chapter we will deal with perception, i.e., the process of transforming sensorial information to higher-level representations which can be used in associative processes (memory access) and cognitive processes such as reasoning. In the perceptual process, two basic components can be identified as described earlier (figure 1.1 , right part). First there is the human sensory system with its typical characteristics and constraints, providing the ``human input channel'' (HIC) processing functions. Second, there are those components of a computer system which provide a multidimensional signal for the human user: ``computer output media'' (COM).

According to our basic model introduced in section 1 , we use the term perception to describe the communication from a machine to a human. First, we will show the different kinds of human input channels through which perception becomes possible and give a short overview on these channels at the neurophysiological level ( 2.1 ). The second part of this section ( 2.2 ) deals with devices and methods used by computers to address these human senses. Finally, in 2.3 we will present our results of bi- and multimodal machine-to-man communication.

Human Input Channels

 

In this introductory section on perception, we will review the concept of modality from the neurobiological point of view [153,308], gradually narrowing the scope to those modalities relevant to research within .

As remarked by Shepherd [308], the notion of sensory modality can be traced back to the 1830s, in particular to the monumental ``Handbook of Human Physiology'', published in Berlin by Johannes Muller, who promulgated the ``law of specific nerve energies''. This states that we are aware not of objects themselves but of signals about them transmitted through our nerves, and that there are different kinds of nerves, each nerve having its own ``specific nerve energy''. In particular, Muller adopted the five primary senses that Aristotle had recognized: seeing, hearing, touch, smell, taste. The specific nerve energy, according to Muller, represented the sensory modality that each type of nerve transmitted.

The modern notion, beyond a great degree of terminological confusion, is not very much different: we recognize that there are specific receptor cells, tuned to be sensitive to different forms of physical energy in the environment and that they serve as stimuli for the receptor cells. A table in Shepherd's book illustrates the point. The table can be simplified and re-written in our framework as follows (Table 2.1 ).

   Table 2.1 : An overview of input channels at the neurophysiological level

The different sensory modalities used by human beings are not processed in isolation. Multimodal areas exist in cortical and sub-cortical areas, such as the posterior parietal cortex (area 5 and 7) and the superior culliculus. The integration of the different channels is essential, among other things, for allowing the brain to reconstruct an internal body model and an internal representation of external Euclidean space [228].