Pilot Surveys (Higher)
A pilot survey tests a questionnaire on a small group first, so that its findings can be used to revise the questions before the main data collection. It is a small-scale trial run that checks the questions are clear, unambiguous and produce usable answers, and spots practical problems such as poor timing or missing response options, before the questionnaire is used on the full sample in GCSE Statistics.
Before you start
Make sure you're comfortable with these topics first:
Method
- Design a draft version of the questionnaire, including all planned questions and response options.
- Select a small sample of people, ideally similar to those who will take part in the main survey, and give them the draft survey.
- Record how respondents interpret each question, how long the survey takes to complete, and any questions they leave blank, misunderstand or find confusing.
- Check the results can actually be analysed as planned, for example that response categories cover all likely answers without overlapping.
- Revise the questionnaire based on the pilot's findings: reword unclear questions, add missing response options, remove leading or biased questions, and adjust timing if needed.
- Only proceed to the full-scale survey once the pilot has shown the revised questionnaire works well in practice.
Common mistakes
MistakeChanging every question after the pilot because a single respondent had a problem with it.
InsteadLook for patterns across the pilot group. Several people blanking or misreading the same question is a signal to change it, while one odd answer may need only a note, although one person's confusion can still reveal a real ambiguity.
MistakeTreating the pilot answers as findings about the topic, or adding them to the main results after the questionnaire has been changed.
InsteadA pilot tests the questionnaire, not the population: the group is small and answered an earlier version of the questions. Collect the main data afresh with the improved questionnaire.
MistakeSolving a problem by deleting the question that caused it, without checking whether the survey still collects the data it needs for its aim.
InsteadReword the question, split it or add response options first. Remove a question only if the aim of the survey can still be answered without it.
Worked examples
Example 1
A researcher is planning a survey about how students in a school travel to school. Before running the main survey of 300 students, they trial the questionnaire on 15 students. One question asks: how do you travel to school - walk, car, or bus? Three of the 15 pilot respondents wrote in bike as their answer, since it was not listed. Explain what this pilot survey reveals and how the researcher should respond.
- Identify the problem revealed by the pilot: the response options for the travel question are not exhaustive, since 3 out of 15 pilot respondents (20%) needed to write in bike as it was not listed.
- Consider the consequence of not fixing this: in the main survey of 300 students, these responses would be lost, forced into an inaccurate category, or left as unusable free text.
- Decide on the fix: add bike as a listed response option, and consider adding a general other, please state option to catch any remaining travel methods.
- Confirm the value of piloting: this problem was caught using a small trial of 15 students, before it could affect the responses of the full 300 students.
- Final answer: the pilot reveals the response options are incomplete; the researcher should add a bike option, plus an other catch-all option, before running the main survey.
Example 2
A pilot of a survey about part-time jobs is completed by 25 students. Question 1 is 'How many hours a week do you work at your part-time job?' Nine students left it blank. When asked why, eight said they do not have a job and one said they skipped it by accident. Every other question was answered by all 25 students without comment. Use the pilot to decide what to change.
- Work out the blanks: 9 of the 25 pilot students, 36%, left Question 1 blank. That is far more than one accidental skip, so it points to a problem with the question.
- The follow-up shows that 8 of the 9 do not have a job. The wording was clear, but the question was put to people it does not apply to, and it gave them no way to say so.
- The remaining blank (1 of 25, 4%) was an accident, so it is not evidence of a fault.
- Fix Question 1 by adding a filter question first: 'Do you have a part-time job? Yes / No', with an instruction that students who tick No skip to the next section.
- Leave the questions that every student answered without comment unchanged, since the pilot gave no reason to change them.
- Final answer: the pilot shows that Question 1 was asked of people it does not apply to, so add a 'Do you have a part-time job?' filter question, leave the questions that worked as they are, and treat the single accidental blank as no reason to change anything.
Practice questions
Q1[1 mark]Give one reason why a researcher would run a pilot survey before the main survey.Show answer
Answer: To check that questions are clear and easy to understand, or to spot problems with the questionnaire before it is used on the full sample (any one valid reason)
Q2[1 mark]A pilot survey shows a question takes respondents an average of 4 minutes to answer, but the researcher only allowed 1 minute per question in their timing plan for a 20-question survey. What should the researcher do?Show answer
Answer: Revise the time allowed for the survey, or shorten/simplify the questionnaire, since the actual time needed is much longer than planned
Q3[2 marks]In a pilot of 20 people, 6 people left a question about income blank. What does this suggest, and what might the researcher change?Show answer
Answer: The question may be too sensitive or unclear; the researcher could reword it, offer income bands instead of an exact figure, or make it optional
Q4[1 mark]State the difference between a pilot survey and the main survey.Show answer
Answer: A pilot survey is a small-scale trial on a small sample to test and refine the questionnaire; the main survey is the full-scale data collection carried out afterwards, usually with a revised questionnaire, on a much larger sample
Q5[2 marks]A pilot survey question asks: do you agree that the new park is a great improvement? Every pilot respondent answered yes. Identify the flaw and rewrite the question so it would work better in the main survey.Show answer
Answer: The question is leading/biased, pushing respondents towards yes; an improved version could offer balanced options, e.g. what is your opinion of the new park: great improvement, some improvement, no improvement, or worse than before
Q6[2 marks]A researcher pilots a survey on 10 friends before sending it to 500 randomly selected customers. Explain one weakness of this choice of pilot sample, beyond its small size.Show answer
Answer: The 10 friends are not representative of the target population of customers, since they were not selected using the same method and may share similar views to the researcher, so problems that real customers would raise might not be revealed
Exam-style questions
Written in the style of a GCSE Statistics exam paper, with a full mark scheme.
Exam tips
- Link every change to evidence from the pilot, such as 'X of Y respondents left it blank, so I would ...'. A change with no evidence behind it is only an opinion.
- Say that a pilot reduces the chance of problems in the main survey rather than removing it: a small group can miss a problem that only some respondents would have.
Q1[2 marks]State two purposes of carrying out a pilot survey before the main data collection.Show mark scheme
Tick each line you got.
Nothing ticked yet - 2 available
Q2[3 marks]A pilot survey of 30 people found that 12 people gave the same don't know response to a question about their spending habits. Explain what this might indicate about the question, and suggest one improvement.Show mark scheme
Tick each line you got.
Nothing ticked yet - 3 available
Q3[4 marks]A company is designing a customer satisfaction survey to send to 4000 customers. Explain, using an example, how a pilot survey could be used to check for: (a) an ambiguous question, and (b) an unsuitable question order. (c) State one reason a pilot survey saves the company time and money overall.Show mark scheme
Tick each line you got.
Nothing ticked yet - 4 available
See real GCSE Statistics past-paper questions, with official mark schemes
Free printable worksheet
Want more practice on paper? Download the Pilot Surveys (Higher) worksheet pack - 26 pages of exam-style questions with a full mark scheme. Free to download, no card or account. Print it for personal and classroom use.
This topic is chapter 4 of GCSE Statistics Higher Essentials 1, the whole course as one free printable PDF. Its worksheet is also chapter 4 of GCSE Statistics Higher Topic Workbook 1 and chapter 4 of GCSE Statistics Higher 10-Minute Tests 1, free printable practice books.
Other versions of this worksheet: