A Taxonomy of Multimodal Interaction in the Human Information Processing System
A Report of the ESPRIT PROJECT 8579 -- WP 1 --
February, 1995
Written by: L. Schomaker, J. Nijtmans (NICI) A. Camurri, F. Lavagetto, P. Morasso (DIST) C. Benoît, T. Guiard-Marigny, B. Le Goff, J. Robert-Ribes, A. Adjoudani (ICP) I. Defée (RIIT) S. Münch (UKA) K. Hartung, J. Blauert (RUB)
Esprit Project 8579/MIAMI (Schomaker et al., '95) Thu May 18 16:00:17 MET DST 1995
Introduction
In this chapter we will introduce our underlying model of human-computer interaction which will influence the whole structure of this document as well as our research in . The next step will be the definition of basic terms (more specific terms will follow in later sections) and a statement of what will be included and excluded in the project, respectively. Some 'philosophical' considerations will follow in section 1.2 , and finally we will give an overview on the structure of this report.
Definitions of Basic Terms
One thing most publications on multimodal systems have in common is each author's own usage of basic terms on this topic. Therefore, we want to state in the beginning of this report what our basic model looks like, how we are going to use several terms within this document, and which levels of perception and control we are considering in .
The basic model for human-computer interaction
Levels of observation
(Multi-) Modality
Multimodal vs. multimedia vs. virtual reality system
Communication channels
The Psychophysical Law
Uncertainty and discrimination
The basic model for human-computer interaction
In order to depict a taxonomy of multimodal human-computer interaction we will have to clarify a number of concepts and issues. The first assumption is that there are minimally two separate agents involved, one human and one machine. They are physically separated, but are able to exchange information through a number of information channels. As schematically shown in figure 1.1 , we will make the following definitions.
Figure
1.1 :
A model for the identification of basic processes in human-computer
interaction. Note that in fact two loops exists: the intrinsic
feedback as in eye-hand coordination, and the extrinsic loop, imposed
by the computer.
There are two basic processes involved on the side of the human user: Perception and Control. Note that we take the perspective of the human process throughout this document. With respect to the Perceptive process, we can make a distinction between:
Human Input Channels (HIC) and
Computer Output Media (COM)
Within the Control process, we can make a distinction between:
Human Output Channels (HOC) and
Computer Input Modalities (CIM)
Then, within both of the agents, a cognitive or computational component can be identified, which processes the incoming input information and prepares the output. Also, at this intermediate cognitive level, intentional parameters will influence the processing, either implicitly, such as by design, in the case of non-intelligent agents, or explicitly, as in humans or in more sophisticated agents containing an explicit representation of goals and ``beliefs''. With respect to the machine, it should be noted that here the design is known, whereas for the human cognitive apparatus, the architecture must be inferred and cannot be observed directly.
Instead
of the word modality at the human input side, and the word media at
the human output side, we have chosen for the word channel,
which also allows for a more clear distinction between the
abbreviations (HOC
CIM
COM
HIC
HOC
...)
which can also be pronounced as:
HOC: human output to computer
CIM: computer input from man
COM: computer output to man
HIC: human input from computer
The two halves of the loop will be further investigated in sections 2 , Perception, and 3 , Control and Manipulation, respectively. The interaction process between the operator (man) and the computer (machine) will be considered in section 4 , Interaction, whereas the internal processing is described in 5 , Cognition.