A Population Survey

Why do this problem?

One aspect of questionnaire writing that is often overlooked is the effect of primer questions, that is, questions which have an impact on respondents' answers to later questions.  Sometimes this is used to deliberately manipulate people's views, but usually it simply biases the results of the survey in an unintended way.

This problem provides a minimal example of this phenomenon: a one or two question questionnaire.

Possible approach

One approach is to ask all of your students to visit this problem before the lesson or at the start of the lesson, click on the questionnaire link and answer the survey.  This has the additional benefit of providing more data for other users of this problem to see.  You could, if you wish, ask your students to copy down the question or questions asked and their responses so that you can discuss the class's responses in addition to the globally collected responses.  If the class do answer the questionnaires at the start of a lesson in a computer room or using tablets, they will be able to see their results immediately on the Google doc (though it is not possible to isolate their responses precisely), but the box plots are only updated hourly.

An alternative approach is to gather the class data with pencil and paper.  One way is give everyone a primer question.  Give half the class one of the primer questions and the other half of the class the other primer question, and ask them to note their answer.  (For example, this could be printed on differently coloured pieces of paper, or half the class could close their eyes while the question is asked to the other half by displaying it on a projector.)  Then ask the whole class the second question and for them to write down their answers.  Then the class can collect their responses and work together to analyse them.  Another way is to split the class into three, giving primer questions to two of the groups and a relatively irrelevant question to the third group, for example "What are people who come from the Philippines called?", "Which continent is the Philippines in?" or "What is the capital of the Philippines?"

Once the data is collected, the class can be told about the primer questions, and asked what they expect the numerical answers to the population question to be: how would they differ for each group?  After a short discussion, their collected data can be plotted in some way, or students can be shown the box plots from the data collected on the site.  To what extent does the data support their earlier thoughts or challenge it?

Moving beyond this specific case, students could then be asked why survey designers might be careful to avoid this effect, or why they might deliberately want to use it.

Key questions

• Why do we have to think carefully about the whole sequence of questions that we ask in a questionnaire?
• What effects might be caused by having related questions on a questionnaire?