(Multi-) Modality

 

According to [67], modality is defined as follows:

`` Perception via one of the three perception-channels. You can distinguish the three modalities: visual, auditive, and tactile (physiology of senses).''

visual:

concerned with, used in seeing (comp. against optical)

auditive:

related to the sense of hearing (comp. against acoustical)

tactile:

experienced by the sense of touch

haptic:

most authors are using `` tactile'' and `` haptic'' as synonyms. However, in [120] tactile as perception-modality is distinguished from haptic as output manner.

optics/optical:

optics is the theory on light as well as the infrared and ultraviolet radiation. Thus the attribute `` optical'' refers to physical quantities and laws rather than to physiological ones. (comp. against visual)

acoustics/acoustical:

acoustics is the theory on vibrations and oscillations in elastic mediums, especially of sound, its generation, spreading, and its reception. Thus the attribute `` acoustical'' refers to physical rather than to physiological quantities and laws. (comp. against auditive)

The author of [67] defines only three modalities and associates them with three of the human senses. Although they will be the three ones considered in , there are some more senses as defined by physiology:

   Table 1.1 : Different senses and their corresponding modalities (taken from: [311])

In our opinion, the sense of smell and the sense of taste are not very interesting for our concerns (see 2.1.1 for a deeper discussion). However, the sense of balance seems to become more and more interesting with respect to virtual reality environments (see 4.4 ). Presently it is already used in flight simulators for example.

Whenever more than two of these modalities are involved, we will speak of multimodality. To be more precise, in some cases we will also use the term bimodal (or bimodality) to denote the usage of exactly two different modalities. In this sense, every human-computer interaction has to be considered as multimodal, because the user looks at the monitor, types in some commands or moves the mouse (or some other device) and clicks at certain positions, hears the reaction (beeps, key clicks, etc.) and so on.

Therefore, in our understanding of multimodality is restricted to those interactions which comprise more than one modality on either the input (i.e., perception) or the output (i.e., control) side of the loop and the use of more than one device on either side. Thus, the combination of, e.g., visual, auditive, and tactile feedback which is experienced by typing on a keyboard is explicitly excluded, whereas the combination of visual and auditive output produced by the monitor and a loudspeaker when an error occurred is a 'real' multimodal (or --- in this case --- bimodal) event.

Multimodal vs. Multimedia vs. Virtual reality system

 

Notice the following paragraph in [243]:

`` Multimodal System: A Definition In the general sense, a multimodal system supports communication with the user through different modalities such as voice, gesture, and typing. Literally, `multi' refers to `more than one' and the term `modal' may cover the notion of `modality' as well as that of `mode'.

• Modality refers to the type of communication channel used to convey or acquire information. It also covers the way an idea is expressed or perceived, or the manner an action is performed.

• Mode refers to a state that determines the way information is interpreted to extract or convey meaning.

In a communication act, whether it be between humans or between a computer system and a user, both the modality and the mode come into play. The modality defines the type of data exchanged whereas the mode determines the context in which the data is interpreted. Thus, if we take a system-centered view, multimodality is the capacity of the system to communicate with a user along different types of communication channels and to extract and convey meaning automatically. We observe that both multimedia and multimodal systems use multiple communication channels. But in addition, a multimodal system is able to automatically model the content of the information at a high level of abstraction. A multimodal system strives for meaning.''

In , we decided to agree with this perspective. Especially the common usage of a multimedia system, which is often merely a PC with a sound card, a loudspeaker, and a CD ROM (if the hardware is considered) or a kind of hypertext/hypermedia environment with a combination of text, video, and sound (if the software is considered) is not what we want to address in this project. Such a kind of explanation of the term multimedia is given by Steinmetz:

`` Multimedia, from the user's point of view, means that information can also be represented as audio signals or moved images.'' [322, page 1,] (translation by the authors)

A definition given by Buxton states the following:

``[...] 'Multimedia' focuses on the medium or technology rather than the application or user.'' [195, page 2,]

The second aspect, namely to concentrate on the user, is what we want to do in this project. The first aspect, the concentration on the application, is characteristic for research performed in the VR domain. Therefore, a distinction between a multimodal and a Virtual Reality (VR) system has to be made. A 'minimal' definition of a VR system is provided by Gigante:

``The illusion of participation in a synthetic environment rather than external observation of such an environment. VR relies on three-dimensional (3D), stereoscopic, head-tracked displays, hand/body tracking and binaural sound. VR is an immersive, multisensory experience.'' [86, page 3,]

In this project, we understand that the main difference is the intention behind both research directions: VR aims at the imitation of reality for establishing immersive audio-visual illusions, whereas multimodality attempts to enhance the throughput and the naturalism of man-machine communication. The audio-visual illusion of VR is just a trick for triggering the natural synergy among sensory-motor channels which is apparent in our brain but is not the only possibility of efficient exploitation of the parallelism and associative nature of human perceptual-motor processes. From this point of view, VR research is a subset of multimodality research. For example, in a following section we suggest two views of multimodal systems, computer-as-tool and computer-as-dialogue-partner, which are a further alternatives to the computer-as-audiovisual-illusion which is typical of VR systems. A discussion of VR will follow in section 4.4 .

Having in mind our basic model as well as Nigay and Coutaz' statement that ``A multimodal system strives for meaning.'', one aspect that distinguishes our research from both, multimedia and VR research, is the concentration on the internal processing steps, known as cognition. Whereas synchronization of different modalities (or media) plays a key role in a multimedia system, this won't be enough for a multimodal system. The fusion of modalities as well as of cognitive processes has to be considered in order to find the meaning or the intention of the user's actions (see also 1.1.2 and 1.2.2 ). This aspect of multimodality will be further investigated in section 5 .