The Surveyor Who Came to Tea

It was a sunny Monday afternoon, and I was sitting at home writing a statistics resource for NRICH when there was a knock on the door.  A young man by the name of Pranav* was standing there with a folder, a computer and a clipboard, and politely asked if I would be willing to participate in a survey for a large national polling company.  (He was probably a university student working during his summer holidays.)  Why yes, I answered - and what a great opportunity it will be to find out first hand how surveys are performed!  He kindly spent time with me after we had finished the questionnaire discussing how he picked my house, why he was happy to speak with me rather than asking if there was anyone else in the house, and some of the issues around how their questionnaires are constructed.

These surveys help to gauge the views of the population on all sorts of matters, so that it is not just the vocal minority whose voices are heard.  They allow public bodies to plan more effectively and be more effective in their work.  They also allow commercial companies to gather information to improve their marketing.

The survey questions

Right at the beginning, I was asked my age, as this affects which questions they will ask.  For example, if I had been under 18, they might have asked for parental consent to interview me, and they would not have asked me about smoking or alcohol consumption.  After that, there were questions on a wide variety of topics.  Some of these are for the survey company themselves, whereas others are commissioned by specific clients.  The topics included:

• Do I collect stamps?  This was a series of survey questions commissioned by the Royal Mail to find out about people's stamp-collecting behaviour.
• Politics: a series of questions on my political views, including: If there were a General Election tomorrow, who would I vote for?  Which party do I feel most closely aligns with my views?  What do I think is the most important issue facing Britain today?  And what other issues face Britain today?  These are regularly asked in this company's polls (at least in the UK), so they can have an up-to-date picture of people's political opinions and also record the changes in these opinions over time.
• Marketing: I was not asked about this, but some people are asked marketing questions instead of politics questions; the computer chooses which to ask at random.
• Alcohol and smoking: these questions were commissioned by the NHS to monitor national trends.
• Height and weight questions: What are my height and weight?  Am I trying to lose or gain weight?  Have I been advised by my GP to lose weight?  And do I think it is appropriate for GPs to raise the issue of weight with patients?  Again, these were commissioned by the NHS.
• Technology and internet: a series of questions asking about what sort of hi-tech devices we have in our home, including questions about TVs, computers, tablets, phones and so on, followed by questions on internet usage ("Which of these have you used the internet for recently?").  I'm not sure who was interested in these questions.
• Newspaper subscriptions: do we subscribe to any (print) newspapers?  Again, I'm not sure who was interested in these questions.
• Finally, some standard demographic questions: do we own our home (with or without a mortgage) or are we renting; what is our household income band, and then what are my ethnicity, gender, sexuality, religion and nationality.

Why might they have wanted to ask me demographic questions?

 

Why me?

I asked Pranav why he had chosen my home.  He explained (and I have paraphrased what he said):

I am given a small area to work on.  I am given a list of all of the addresses in the area, which is typically about 250-400 addresses.  I then go round them, knocking on every door that I can, until I find enough people willing to answer the survey.  Many homes I come to are empty, and even when I find people in, they usually refuse to take part.  My target is 12 interviews in two days.  Sometimes I knock on doors for over two hours without anyone agreeing to take part, while other times, I have a whole string of interviews with very few refusals.

It turned out that I visited every address in the list that I could yesterday, but most of them were in a big tower block that I couldn't get into.  So I had run out of addresses without doing enough interviews.  My supervisor gave me permission to go to the roads just outside the specified area to knock on more doors.  And that's how I ended up at your house.

I also have targets related to the type of interviewee.  For example, yesterday and today, one of my targets is to interview at least 6 males but at most 6 females.  Another target is related to the age groups that people are in.  I will only interview one person in a house, so if I am meant to be interviewing lots of young people for some particular survey, I might ask the person who answers the door if there is someone younger in the house whom I could interview instead.

What issues might be caused with this method of sampling?
How could the company deal with these difficulties?


Pranav read out most of the questions from his small laptop computer, and typed in my answers.  He was happy for me to look at the screen most of the time, but for some questions, such as "What is the most important issue facing Britain today?", he was instructed to make sure I couldn't see the screen before the question appeared.

Why might I not have been allowed to see the screen for a question like this?


There was also a series of questions about my weight, which were roughly like this:

• What is your height?
• What is your weight?
• In the last three months, have you being trying to change your weight?  (Options included things like: maintain the same weight, lose weight, gain weight, not thought about it)
• Have you discussed your weight with your GP (family doctor) in the last three months?
• Do you think a GP should bring up the subject of weight if they think a patient is overweight?

For these questions, Pranav gave me the laptop and asked me to read the questions and answer them myself.  At the end of this series, it said "please give the computer back to the interviewer".

Why might the company have decided that I should read and answer these questions myself?
What issues might be addressed or caused by asking an interviewee to read and answer these questions themselves with no input from the interviewer?


Near the end of the interview, there was a question about nationality, which asked something like: "What would you say is your nationality? British or Mixed British / English / Scottish / Welsh / Irish / Other"

What do you think this question is trying to find out?  What issues might arise with this question?



There was another possible issue with this question.  A few questions earlier, there was a standard ethnicity question: "To which of these ethnic groups would you say you belong?" and the options were the standard list (which I'm only quoting part of):

• White
  • White British
  • White Irish
  • ...
• Mixed
  • White and Black Caribbean
  • White and Black African
  • ...
• Asian or Asian British
• Black or Black British
• Other

Why might asking this question a few questions earlier affect people's answer to the nationality question?


How could the questionnaire be improved to address this issue?


Another possibly problematic question was a political one: "If there were a General Election tomorrow, which party would you vote for?" and the choices given were Conservative / Labour / Liberal Democrats / Other (with the emblems of the first three shown as well).  Would someone perhaps be more likely to say one of the first three if their unprompted answer would have been a different party, say the Greens?  One would have to do a much larger survey, asking people the question in two different ways, to really find out an answer to this question!

So next time you read or hear a figure in the news, something like "a recent survey showed that 17% of people think that ...", spare a thought for those people who spend hours after hours knocking on doors and trying to collect this data!


* Name changed


This resource is part of the collection Statistics - Maths of Real Life