Questionnaire Design

Questionnaires are an inexpensive way to gather data from a potentially large number of respondents. Often they are the only feasible way to reach a number of reviewers large enough to allow statistical analysis of the results. A well-designed questionnaire that is used effectively can gather information on both the overall performance of the test system as well as information on specific components of the system. If the questionnaire includes demographic questions on the participants, they can be used to correlate performance and satisfaction with the test system among different groups of users.

It is important to remember that a questionnaire should be viewed as a multi-stage process beginning with definition of the aspects to be examined and ending with interpretation of the results. Every step needs to be designed carefully because the final results are only as good as the weakest link in the questionnaire process. Although questionnaires may be cheap to administer compared to other data collection methods, they are every bit as expensive in terms of design time and interpretation.

The steps required to design and administer a questionnaire include:

1. Defining the Objectives of the survey

2. Determining the Sampling Group

3. Writing the Questionnaire

4. Administering the Questionnaire

5. Interpretation of the Results

This document will concentrate on how to formulate objectives and write the questionnaire. Before these steps are examined in detail, it is good to consider what questionnaires are good at measuring and when it is appropriate to use questionnaires.

What can questionnaires measure?

Questionnaires are quite flexible in what they can measure, however they are not equally suited to measuring all types of data. We can classify data in two ways, Subjective vs. Objective and Quantitative vs. Qualitative.

When a questionnaire is administered, the researchers control over the environment will be somewhat limited. This is why questionnaires are inexpensive to administer. This loss of control means the validity of the results are more reliant on the honesty of the respondent. Consequently, it is more difficult to claim complete objectivity with questionnaire data then with results of a tightly controlled lab test. For example, if a group of participants are asked on a questionnaire how long it took them to learn a particular function on a piece of software, it is likely that they will be biased towards themselves and answer, on average, with a lower than actual time. A more objective usability test of the same function with a similar group of participants may return a significantly higher learning time. More elaborate questionnaire design or administration may provide slightly better objective data, but the cost of such a questionnaire can be much higher and offset their economic advantage. In general, questionnaires are better suited to gathering reliable subjective measures, such as user satisfaction, of the system or interface in question.

Questions may be designed to gather either qualitative or quantitative data. By their very nature, quantitative questions are more exact then qualitative. For example, the word "easy" and "difficult" can mean radically different things to different people. Any question must be carefully crafted, but in particular questions that assess a qualitative measure must be phrased to avoid ambiguity. Qualitative questions may also require more thought on the part of the participant and may cause them to become bored with the questionnaire sooner. In general, we can say that questionnaires can measure both qualitative and quantitative data well, but that qualitative questions require more care in design, administration, and interpretation.