7.5 Test Procedures |
249 |
appropriate. Speech output, for instance, may be better for visually demanding tasks, whereas a visual display may be better for illustrating spatial relations. Also, contextual information may be better transmitted when spanned across more than one sensory channel, such as through hearing and touch. Ross and Blasch (2000) argue that the best interface for visually impaired people is a combination of speech output and tactile cues using a tapping interface. More research is needed, though, to understand (i) which types of contextual information should be represented using different forms of output technologies, and (ii) individual user preferences, especially for people with different visual impairments (Bradley and Dunlop 2003).
Social implications. One of the ideas behind context-aware computing is to facilitate groups of people in sharing contextual information. There are, however, some important social implications surrounding this notion, with regards to how this is information is being shared and distributed to others (Bellotti and Edwards 2001). What rights should be given to other people when they send information of potential interest to the user? For instance, the user’s perception of what is useful and important may be entirely different to the sender. Ideally, context-aware computing research needs to address how different social relationships and situations affect how contextual information is shared and disseminated.
7.5 Test Procedures
The design and evaluation of electronic navigation aids brings together two very different research domains:
•Cognitive mapping research has a long history of investigating peoples’ understanding of the geographical environment they live in and how they build cognitive maps of that world.
•Since the seventies, human computer interaction (HCI) researchers have been developing methodologies and approaches to the design, implementation and evaluation of easier-to-use interactive computer systems.
This section will give a very brief overview of these approaches, but will focus mainly on HCI approaches and their application to the design and evaluation of electronic navigation aids.
7.5.1 Human Computer Interaction (HCI)
User-centred design is the core to much good practice in human-computer interaction and much of modern software engineering. This approach to interactive system design puts users at the heart of the design processes for the systems that they will use. Having users involved in the design and development phases of applications, particularly at the start, overcomes the difficulties in traditional requirements analysis of modelling users’ experience, understanding, working practices and desires. User-centred design has been proven to not only lead to
250 7 Navigation AT: Context-aware Computing
more usable systems but to save development time in large projects. Building on the premise that we often cannot describe what we want but can critique what we have, user-centred design processes often make heavy use of prototypes to aid the dialogue between designers and end-users.
Prototypes are partial implementations of the system that can be used to engage users and to run initial usability trials by getting users to perform tasks with the prototype system. These prototypes can be of varying quality and completeness: in the early stages of product development simple hand drawn paper prototypes are used as the basis for discussions with users about their requirements. When designs are more developed, semi-functional systems can be implemented—these often lack features and the robustness of the final system but still give an impression of using that system. So-called Wizard of Oz experiments can be carried out using completely impersonated system functionality—that functionality being provided by a human acting as if they were the system. For example, an audiobased navigation system could be impersonated by a designer by simply reading navigation instructions to a user wearing a hands-free mobile phone headset. The Wizard of Oz technique can be very useful where considerable development is needed to assess the value of a real system.
If the intended users of a system cannot be central to the design team, one solution is often to substitute them with personas, shown in Figure 7.4. A persona is a description of a fictitious intended user of the system. Personas are given names and often embellished with personal information to make the fictitious person feel like a real person to the design team.
Evaluation of how people perform with systems is fundamental to user-centred design. Traditionally this evaluation was of desktop computer systems intended for office use; as such, the evaluation focussed heavily on users performing tasks sitting at a computer in an office environment. Many of the traditional techniques have been adapted for use in evaluating more modern systems, such as mobile computing devices (including palmtop computers, Personal digital assistants and mobile phones) and context-aware systems. There are too many techniques for interactive system evaluation to discuss here and they are well documented in HCI text books (e.g. Dix et al. 2003; Preece et al. 2002) and usability evaluation texts (e.g. Rubin 1994). Two traditional interactive evaluation techniques, think-alouds
Figure 7.4. Sample persona
7.5 Test Procedures |
251 |
and controlled experiments, and one evaluation technique used in HCI exclusively for mobile system evaluation, percentage preferred walking pace, are worthy of further discussion here.
A think-aloud is a very simple, cost-effective, evaluation approach that is both easy to run and can produce significant feedback to designers in a very short time. To run a think-aloud the designer needs:
1.An interactive system to test
2.Normal support materials for first-time system use, such as manuals, tutorials etc.
3.A user (one per think-aloud session—normally a trial will compose several sessions)
4.A list of tasks for the user to do with the system
The user is given the system and asked to work through the task sheet “thinking aloud”, i.e. they are asked to verbalise their thoughts as they work through the set of tasks using the system. In an office environment, the experimenter sits behind the user taking notes and occasionally giving neutral prompts such as “what are you thinking now?”. A video recording is normally also made for later analysis— ideally this captures multiple camera angles covering the user’s face, hands and the screen plus audio from the user and the system. The video can also be used after the formal session to discuss design problems with the user and engage him/her in possible redesigns.
When well designed, the approach provides considerable scope for users to act unexpectedly and use the system in ways that the designers never intended, never mind tested. Despite being designed for desktop office applications, thinkalouds can easily be adapted to mobile devices such as computer-based navigation aids and context-aware systems. The only problem is recording the events, but the experimenter can still learn many lessons by simply noting them down in a notepad as they walk behind the user.
While valuable lessons can be learned by designers on how to make their system more usable using a few think-aloud sessions, concrete numbers and comparisons are hard to gain from this approach: the very nature of think-alouds results in them impacting on most numeric measures that could be used for concrete comparison, e.g. the time it takes a user to complete the tasks. Controlled experiments complement think-alouds where either a research hypothesis is being tested or two systems are being compared. Controlled experiments are often run in a similar environment to think-alouds but with more focussed tasks, no interaction between users and experimenter once the study is underway and no need for the user to verbalise their thoughts. After an initial training period, the experimenter stays silent and simply records the user’s behaviour—most often focussing on numeric information such as the time to complete the tasks and the number of task-related errors the user makes in the process. When correctly designed, this form of experiment leads to statistically testable numeric results, thus it is important for scientific studies or system comparisons of the form system A is better/faster/less-error-prone/safer than system B. However, such experiments are often less helpful in finding design
252 7 Navigation AT: Context-aware Computing
problems than think-alouds: the environment can be so constrained that unexpected user behaviour is very rare. Controlled experiments must reduce as many environmental variations as possible in order to get statistically significant results with a reasonable number of users. It is thus difficult, but not impossible, to run valid controlled experiments on mobile devices in mobile settings: for example, the effects of, say, traffic conditions are likely to swamp any difference between two systems when measuring how long it takes a person to navigate across a city. The variation in conditions has to be taken into account in the experimental design and, often, many more users are required to achieve statistically significant results. As a compromise between realistic settings and the desire for solid numerical results, some researchers have carried out trials in semi-constrained environments, for example Brewster (2002) asked users to complete tasks on a hand-held computer while walking laps of a quiet outdoor route.
Adults tend to have a preferred walking speed (PWS) that they will walk at if allowed to walk freely. A person’s walking pace is, however, often affected by their concentration on other activities, such as navigating in an unknown environment or walking while using a mobile phone or a navigation system. This reduced speed can be expressed as a percentage of the user’s preferred walking speed (PPWS) and can be used as an estimate of how much, say, the mobile computing device is interfering with the user’s walking. An ideal navigation system would allow the user to walk at their particular PWS. If this pace is known, then real systems can be compared on what the PPWS is while using the systems. This approach has been used both to test hand-held computers (e.g. Brewster et al. 2003) and navigation systems for visually impaired people (e.g. Petrie et al. 1998; Soong et al. 2000).
7.5.2 Cognitive Mapping
Kitchin and Blades (2002) provide a comprehensive literature study of cognitive mapping methods and categorise methods for individual studies as follows.
•Unidimensional data generation. Studies in this category involve studying cognitive models a single dimension at a time, focusing either on distance or angle. Various approaches to distance analysis can be used that attempt to overcome different problems in getting people to express what their internal model is. These include simple magnitude estimates, e.g. “if Glasgow–London is 100 units, how far is Glasgow–Dublin?” and rating distances into different categories, e.g. very near, near, medium, far, very far, etc. Direction estimates usually involve either standing (or imagining oneself standing) at a location and pointing to another or drawing the direction from one location to another on paper.
•Two-dimensional data generation. Simple graphic approaches to 2D studies involve getting the experimental subject to draw maps of an area. Unfortunately, these drawings can be affected not only by the subject’s cognitive map but by their ability to express that through drawing—Kitchin and Blades discuss alternatives to try to reduce this problem. Completion tasks are one solution where subjects complete a partial map—either free hand drawing additional information on pre-prepared maps or filling in blank spaces on the map.