Testing questionnaires

Ross Woods. Rev. 2018

All questionnaires need testing with people before you use them widely. You might understand your questions perfectly, but you can't know how others might interpret them. Time spent in testing is well-spent. Imagine the frustration of having to start over from scratch because your questionnaire was found to have a serious flaw late in the fieldwork.

Expect to be surprised that some people will not understand even the most obvious (to you) of questions. Then, when you ask, they produce the most logical and obvious responses that had never occurred to you. Your questions could:

Note how much extra help you give people. Even an unwitting use of body language or perceptible attitude may alter people's responses. Don't help people if they must be able to do the questionnaire with no help at all.

Questionnaires with closed questions need thorough, extensive testing because respondents must choose between the answers you provide. Consequently, your list of answers must allow all possibilities. Don't even think of using the questionnaire until you know it works very well.

Questionnaires with open questions are much more flexible, because you tend to use them as a basis for interviews to which you might spontaneously add follow-up questions. It might even be possible to let the questionnaire evolve during fieldwork so that you can follow up on emerging trends.

Field-testing steps

After proof-reading, test your first draft with colleagues to eliminate the most obvious glitches.

Then do your testing with persons of the target population who have not yet seen the materials nor been asked your questions in any other form. People who already know the subject matter or saw previous drafts of materials won't be tripped up as if they were doing the questionnaire from scratch.

If possible, observe them for non-verbal clues, anything that makes them hesitate because they are confused. Obviously, you will also ask for verbal feedback, but the non-verbal feedback can be more useful. If something is unclear, some will stop and re-read it, which is quite observable. In some cases, they will be clearly frustrated. Others will just decide to skip that bit.

The sources of confusion will probably be:

You should see how long it takes them. Respondents normally should be able to do a written survey questionnaire in 15-30 minutes. Some are less.

Then make corrections and test your questionnaire again on a new group of people from the target population who have never seen the questions before, and collate suggestions for improvement. You need to use different people each time, because they need to be doing the questionnaire for the first time; they can no longer give their first impression of the meaning of the questions. If necessary, you can repeat this kind of test more widely with other new groups of people from the target population.

Decide when to stop. You can keep improving forever, so stop when your questionnaire is good enough, that is, when the incoming suggestions are trivial and you are satisfied with it.

A note on terminology

Terminology is not standardized, and some researchers use the terms field test and pilot study as synonymous. Others make the following differentiation:
Field test: expert review
Pilot study: tested with members of the target population.