Revision Library

revisionlibrary.co.uk

HIGHER

H23 Pilot Surveys

EDEXCEL 1ST0 · Calculator allowed · about 80 minutes

Total Marks

Name: _______________________________ Date: ____ / ____ / ______

Answer ALL questions. Show all your working.

A pilot survey is carried out before a main statistical investigation.

(a)Define the term 'pilot survey'.(1)

(b)State one purpose of carrying out a pilot survey.(1)

(c)State one way in which a pilot survey is different from the main survey it is testing.(1)

(Total for Question 1 is 3 marks)

For each statement about pilot surveys, state whether it is True or False.

(i)A pilot survey should always take place before the main survey, not after it.(1)

(ii)A pilot survey is normally carried out on the entire population that the main survey will target.(1)

(iii)One purpose of a pilot survey is to estimate how long the main survey's questionnaire will take people to complete.(1)

(Total for Question 2 is 3 marks)

A train operating company is planning a nationwide survey of passengers about a new ticketing app. Before this, it plans to run a pilot survey on a small number of passengers.

Give four different reasons why the company might do this.

(a)Give one reason for the pilot, relating to how clear the questions about the app are.(1)

(b)Give a different reason, relating to how long the questionnaire takes to complete.(1)

(c)Give a different reason, relating to the response rate expected for the main survey.(1)

(d)Give a different reason, relating to the practical method used to collect responses.(1)

(Total for Question 3 is 4 marks)

A university student is investigating whether students support a proposed new bus route to campus. Before her main survey of the whole student population, she pilots her questionnaire on 8 course-mates who study on her own course.

(a)Identify a problem with how the student chose her pilot sample, and explain why it is a problem.(2)

(b)Suggest a better way for the student to select her pilot sample.(2)

(Total for Question 4 is 4 marks)

A leisure centre is designing a questionnaire to find out what members think of its facilities. One draft question is shown below.

(a)State the type of problem with this question.(1)

(b)Explain why this is a problem to identify during a pilot survey.(1)

(c)Rewrite the question so that it is no longer leading.(2)

(Total for Question 5 is 4 marks)

As part of a pilot survey, a museum asks visitors two draft questions.

Question 1: "Did you enjoy your visit today?" Response options: Yes / No
Question 2: "What could we do to improve your visit?" (visitors write their own answer, with no response options given)

(a)The pilot shows that almost every visitor ticks 'Yes' to Question 1, but the museum learns very little about why. Suggest a change to Question 1 that would gather more useful information, and explain why it would help.(2)

(b)The pilot shows that Question 2 produces very varied answers that are difficult for staff to compare and summarise. Suggest a change to Question 2 that would make responses easier to analyse, and explain why it would help.(2)

(Total for Question 6 is 4 marks)

An online retailer pilots a customer questionnaire that asks shoppers to tick their annual household income, using the response options shown below.

(a)A pilot respondent with a household income of exactly £40,000 is unsure which box to tick. Identify the problem with the response options, and explain why it occurs.(2)

(b)Rewrite the response options so that they no longer overlap, and still cover every possible income.(2)

(Total for Question 7 is 4 marks)

A clothing retailer posts a pilot questionnaire to 60 customers. 51 completed questionnaires are returned.

(a)Work out the response rate for this pilot survey, as a percentage.(2)

(b)State whether a response rate of 85% would generally be considered high or low for a survey, giving a reason.(1)

(Total for Question 8 is 3 marks)

An energy supplier is planning a main survey of 8000 randomly selected customers, to find out their views on a new smart meter service. Before this, the supplier will run a pilot survey.

(a)Suggest a suitable percentage of the main survey's sample size to use for the pilot survey, and explain why this percentage is sensible.(2)

(b)Using a pilot size of 5% of the main survey's sample, work out the number of customers who should be given the pilot questionnaire.(2)

(c)State one problem that could occur if the supplier chose a pilot size that was much too small, for example only 10 customers.(1)

(Total for Question 9 is 5 marks)

An ecologist is developing a method for estimating the number of ladybirds in a meadow, using a timed sweep-net count within a marked 5-metre square quadrat. Before her main survey of several meadows, she pilots this counting method on one afternoon in June.

(a)State one practical problem, unrelated to question wording, that this pilot might reveal about the counting method.(1)

(b)State a different practical problem, relating to the equipment or amount of time used.(1)

(c)Explain how identifying problems like these during the pilot could improve the main survey.(2)

(Total for Question 10 is 4 marks)

A dentist's surgery emails a link to a short feedback survey to a pilot group of 25 patients. Only 9 patients complete it.

(a)Give one reason why some patients might not have responded to this pilot survey.(1)

(b)Explain why a response rate this low, found during the pilot, could be a problem if the same method were used, unchanged, for the main survey.(2)

(Total for Question 11 is 3 marks)

A college has 750 students, made up of 450 in Year 12 and 300 in Year 13, as shown in the table.

Year group	Number of students
Year 12	450
Year 13	300

Before running a full survey about a proposed new common room, the college pilots its questionnaire using a stratified sample of 50 students.

(a)Work out how many Year 12 students and how many Year 13 students should be included in this stratified pilot sample.(3)

(b)Of the 30 Year 12 students given the pilot questionnaire, 27 respond. Of the 20 Year 13 students, 12 respond. Work out the overall response rate for the pilot survey, as a percentage.(2)

(Total for Question 12 is 5 marks)

A sports club is investigating how satisfied members are with a new training schedule. The investigation follows the statistical enquiry cycle: plan, collect, process, interpret.

(a)Which stage of the statistical enquiry cycle would a pilot survey normally be part of?(1)

A) Collect
B) Plan
C) Process
D) Interpret

(b)The pilot reveals that several members misunderstood a question about training times. Explain how this finding should affect the collect stage, before the main survey of all members is carried out.(2)

(Total for Question 13 is 3 marks)

A hospital pilots a patient-experience questionnaire on 20 patients before sending an improved version to every patient treated that year. Three of the draft questions are shown below.

(a)Identify the problem with Question 1, and rewrite it so the problem is fixed.(2)

(b)Identify the problem with Question 2, and explain why it is difficult for a patient to answer accurately. Rewrite it as two separate questions.(2)

(c)Identify the problem with the response options for Question 3, and rewrite them so that they no longer overlap.(2)

(Total for Question 14 is 6 marks)

A supermarket wants to pilot a new customer satisfaction questionnaire before sending it to its 20,000 loyalty-card holders nationally. Two possible pilot designs are suggested.

Design A: give the questionnaire to 15 staff members' friends and family to complete.
Design B: select a random sample of 200 loyalty-card holders from across the country and post the questionnaire to them.

Compare Design A and Design B, and explain which would give a more useful pilot for testing the questionnaire before the main national survey.

(Total for Question 15 is 4 marks)

A phone network plans to survey all 2,000,000 of its customers about a new tariff. Before this, it pilots the survey on 150 customers.

Discuss the benefits and limitations of using a pilot survey in this situation.

(Total for Question 16 is 3 marks)

A regional airport pilots a passenger feedback survey by emailing a questionnaire to 60 passengers who were all on the same early-morning flight. Only 15 of the 60 passengers respond (a 25% response rate). The pilot also shows that one question was leading.

Recommend, with reasons, what the airport should do at the plan stage and the collect stage before running its main survey of all passengers.

(Total for Question 17 is 6 marks)

A researcher claims: "The bigger a pilot survey is, the better it always is."

Evaluate this claim.

(Total for Question 18 is 3 marks)

A town council is planning a main survey of a sample of 3000 residents, out of the 45,000 residents in the town, about a proposed new park. It is deciding how large to make its pilot survey.

(a)The council suggests using a pilot size equal to 6% of the main survey's sample. Work out this suggested pilot sample size.(2)

(b)Justify, with reference to both reliability and practicality, why a pilot size of 180 residents might be more appropriate than a pilot of just 15 residents, or a pilot of 2500 residents.(3)

(Total for Question 19 is 5 marks)

Mark scheme · H23 Pilot Surveys

Question 1

(a) B1 a small-scale trial run of a survey or questionnaire, carried out before the main survey, oe
(a) Answer: A pilot survey is a small-scale trial run of a survey or data collection method, carried out on a small group before the main survey takes place.
(b) B1 any valid purpose, e.g. checks the wording of questions is clear, oe checks how long the survey takes, oe estimates the likely response rate
(b) Answer: To check that the questions are clear and easy for people to understand before the main survey.
(c) B1 the pilot uses a much smaller sample size than the main survey, oe it happens earlier/before the main survey
(c) Answer: A pilot survey is carried out on a much smaller number of people than the main survey.

Question 2

(i) B1 True cao
(i) Answer: True
(ii) B1 False cao
(ii) Answer: False
(iii) B1 True cao
(iii) Answer: True

Question 3

(a) B1 checks that the wording of the questions about the app is clear and not confusing for passengers, oe
(a) Answer: To check that the questions about the app are worded clearly and are not confusing.
(b) B1 checks how long the questionnaire takes passengers to complete, oe
(b) Answer: To find out how long the questionnaire takes passengers to complete.
(c) B1 gives an estimate of the likely response rate, so the company can plan how many passengers to contact for the main survey, oe
(c) Answer: To estimate the response rate, so the company knows roughly how many passengers to contact in the main survey.
(d) B1 checks that the method of collecting responses (e.g. an app link or paper form) works properly in practice before it is used on a large scale, oe
(d) Answer: To check that the practical method of collecting responses, such as a link sent through the app, actually works before it is used nationwide.

Question 4

(a) B1 identifies that all 8 pilot respondents share the same course, so are not representative of the whole student population
(a) B1 explains that problems found (or not found) in this pilot might not reflect problems that would occur with the wider student population, e.g. students on other courses may have different travel patterns/views
(a) Answer: All 8 respondents are on her own course, so they are not representative of the whole student population; problems that would affect students on other courses might not show up in this pilot.
(b) B1 suggests selecting students from a range of courses, year groups or halls of residence, e.g. a random or stratified sample
(b) B1 explains this would make the pilot sample more representative of the population being surveyed in the main survey
(b) Answer: She should select pilot respondents from a range of different courses and year groups, for example using a random sample, so the pilot is more representative of the whole student population.

Question 5

(a) B1 it is a leading question cao
(a) Answer: It is a leading question.
(b) B1 it pushes/encourages members towards agreeing, so responses would be biased and not reflect members' true opinions, oe
(b) Answer: The wording pushes members towards agreeing, so the results would be biased and would not show what members really think.
(c) B1 no longer contains wording that suggests an expected answer
(c) B1 remains clear and answerable, e.g. offers a genuine scale of response
(c) Answer: e.g. "What do you think of the new gym equipment?" with response options Much better, Better, About the same, Worse, Much worse.

Question 6

(a) B1 suggests changing the closed Yes/No question to one with a wider range of options, e.g. a rating scale such as Excellent, Good, Average, Poor, or adding a follow-up question asking visitors to explain their answer
(a) B1 explains this would reveal more detail about visitors' opinions than a simple Yes/No response, oe
(a) Answer: Replace the Yes/No question with a rating scale (e.g. Excellent, Good, Average, Poor), or add a follow-up asking visitors to explain their answer, so the museum learns more detail than a simple Yes/No gives.
(b) B1 suggests changing the open question to a closed question with a set of response categories/tick boxes covering common areas, e.g. signage, cafe, exhibits, staff
(b) B1 explains fixed categories are quicker/easier for staff to count and compare across all visitors than free-text answers, oe
(b) Answer: Give visitors a closed list of tick-box categories (e.g. signage, cafe, exhibits, staff) instead of a blank space, because fixed categories are much easier for staff to count and compare across every visitor than free-text answers.

Question 7

(a) B1 the response categories overlap
(a) B1 £40,000 appears in both the '£20,000-£40,000' box and the '£40,000-£60,000' box, so it is unclear which one to tick
(a) Answer: The categories overlap. A household income of exactly £40,000 could be ticked in either the '£20,000-£40,000' box or the '£40,000-£60,000' box, so it is not clear which one to choose.
(b) B1 boundaries no longer shared between adjacent categories, e.g. Under £20,000, £20,000-£39,999, £40,000-£59,999
(b) B1 categories cover every income with no gaps, including an open-ended top category, e.g. £60,000 and over
(b) Answer: e.g. Under £20,000, £20,000-£39,999, £40,000-£59,999, £60,000 and over

Question 8

(a) M1 51/60, oe
(a) A1 85% cao
(a) Answer: 85%
(b) B1 high, with an acceptable reason, e.g. most people who received a questionnaire returned it (51 out of 60)
(b) Answer: High. Most of the customers who were sent a questionnaire (51 out of 60) returned it.

Question 9

(a) B1 gives a reasonable percentage, e.g. between 5% and 10% (accept any sensible value in this range), oe
(a) B1 justifies the choice: large enough to reveal problems with the questionnaire and give a reasonable estimate of the response rate, but small enough not to use up too much of the time/budget intended for the main survey
(a) Answer: e.g. Around 5% to 10% of the main sample. This is large enough to reveal wording or logistical problems and to estimate the likely response rate, but small enough to avoid using up too much of the time and budget meant for the main survey.
(b) M1 8000 x 0.05, oe
(b) A1 400 cao
(b) Answer: 400
(c) B1 with so few responses, any problems found (or the response rate estimate) could occur by chance and might not reliably represent what would happen with the full 8000 customers, oe
(c) Answer: With only 10 customers, any problems found (or not found) could just be due to chance, and might not reliably predict what happens across all 8000 customers.

Question 10

(a) B1 e.g. weather conditions on the day (wind, rain or temperature) affect how many ladybirds are active/visible, so counts vary for reasons unrelated to the true population, oe
(a) Answer: Weather conditions on the day, such as wind or rain, affect how many ladybirds are active and visible, so the count could vary for reasons that have nothing to do with the true population size.
(b) B1 e.g. the sweep net may be difficult to use accurately in the time allowed, or small ladybird larvae may be hard to identify and count quickly, oe
(b) Answer: The time allowed may not be long enough to sweep the whole quadrat accurately, or small ladybird larvae may be hard to identify and count quickly enough.
(c) B1 problems found can be corrected (e.g. choosing calmer days, allowing more time, using clearer identification guides) before the main data collection takes place
(c) B1 this makes the counts from the main survey more consistent/reliable and comparable between different meadows, oe
(c) Answer: Problems found in the pilot, such as poor weather or rushed identification, can be fixed before the main survey, for example by choosing calmer days or allowing more time. This makes the counts from the main survey more consistent and fair to compare between meadows.

Question 11

(a) B1 any valid reason, e.g. the email was missed/went to a spam folder, oe patients did not have time, oe patients were not comfortable giving feedback online
(a) Answer: The email might have gone to a spam folder, or patients simply did not have time to complete it.
(b) B1 identifies that too few responses would likely be collected overall if the same low rate continued at a larger scale
(b) B1 identifies that patients who do respond might not be typical of all patients (e.g. only those with strong opinions reply), so conclusions from the main survey could be biased
(b) Answer: Too few patients would respond to give a reliable sample size in the main survey, and the small number who do respond might not be typical of all patients, for example only those with strong opinions might reply, so the conclusions could be biased.

Question 12

(a) M1 450/750 x 50, oe (or equivalent ratio method)
(a) A1 30 cao (Year 12)
(a) A1 20 cao (Year 13)
(a) Answer: 30 Year 12 students and 20 Year 13 students
(b) M1 (27 + 12)/50, oe
(b) A1 78% cao
(b) Answer: 78%

Question 13

(a) B1 B cao
(a) Answer: B) Plan
(b) B1 identifies that the confusing question should be reworded/clarified at the plan stage before data collection begins
(b) B1 explains that the collect stage will then use the improved/corrected questionnaire on the full membership, producing clearer, more valid data
(b) Answer: The confusing question should be reworded and clarified before the collect stage starts. The collect stage will then use the improved questionnaire on all members, producing clearer, more valid responses than if the original wording had been used.

Question 14

(a) B1 identifies Question 1 as a leading question, e.g. 'caring, attentive' and 'Don't you agree' suggest the expected answer
(a) B1 gives an improved, unbiased version, e.g. 'What did you think of the nurses on this ward?' with a genuine scale of response options
(a) Answer: Question 1 is leading, because 'caring, attentive' and 'Don't you agree' suggest the expected answer. Improved: 'What did you think of the nurses on this ward?' with options such as Excellent, Good, Average, Poor.
(b) B1 identifies that Question 2 asks about two different things at once (doctors and food), so a patient who is happy with one but not the other cannot answer clearly
(b) B1 rewrites as two separate questions, e.g. 'What did you think of the doctors?' and 'What did you think of the food?'
(b) Answer: Question 2 asks about two different things (doctors and food) at once, so a patient satisfied with one but not the other cannot give one clear answer. Rewrite as: 'What did you think of the doctors?' and 'What did you think of the food?'
(c) B1 identifies that the waiting-time categories overlap, e.g. 15 minutes appears in both '0-15 minutes' and '15-30 minutes', and 30 minutes appears in both '15-30 minutes' and '30-60 minutes'
(c) B1 gives improved, non-overlapping categories that still cover every waiting time, e.g. 0-14 minutes, 15-29 minutes, 30-59 minutes, 1 hour or more
(c) Answer: The categories overlap at 15 and 30 minutes. Improved: 0-14 minutes, 15-29 minutes, 30-59 minutes, 1 hour or more.

Question 15

B1 an overall judgement that Design B is more useful/appropriate
B1 explains that Design A's sample (staff friends/family) is unlikely to be representative of loyalty-card holders nationally, e.g. they may share similar ages, locations or views to staff, so problems affecting the wider population might not be revealed
B1 explains that Design B's random national sample is more representative of the population targeted by the main survey, so issues found (e.g. with wording or response rate) are more likely to also apply to the main survey
B1 notes that Design B's larger sample size (200 compared with 15) also gives a more reliable early estimate of things such as response rate, before the expense of the full national rollout
Answer: Design B is more useful. A random national sample of 200 loyalty-card holders is far more representative of the people who will receive the main survey than 15 staff friends and family, who are likely to share similar characteristics to staff. Design B's larger, representative sample also gives a more reliable early estimate of the response rate before the costly national rollout.

Question 16

B1 a clear overall judgement is given (e.g. the pilot is worthwhile, even though it has some limitations), rather than only listing points
B1 a benefit is explained, e.g. testing on 150 customers is quick and low-cost, and can reveal problems with the questionnaire before it is sent to all 2,000,000 customers
B1 a limitation is explained, e.g. 150 is a tiny fraction of 2,000,000 customers (0.0075%), so the pilot might not reveal rarer problems that only affect a small number of customers, such as those with unusual tariffs or devices
Answer: Running a pilot survey is worthwhile. Testing on 150 customers is quick and low-cost, and lets the phone network fix problems with the questionnaire before spending time and money contacting all 2,000,000 customers. However, 150 is a tiny fraction (0.0075%) of 2,000,000, so rarer problems, such as issues affecting customers with unusual tariffs or devices, might not be revealed by the pilot.

Question 17

B1 a clear overall recommendation is given, e.g. revise both the questionnaire and the sampling method before proceeding to the main survey
B1 plan-stage point: reword the leading question identified in the pilot so it no longer suggests an expected answer
B1 a different plan-stage point: redesign the sampling method so future samples are drawn from passengers on a range of flights, times and destinations, not just one early-morning flight
B1 collect-stage point: take steps to improve the response rate for the main survey (e.g. sending reminder emails, or offering a paper option at the gate), since a 25% pilot response rate is low
B1 a different collect-stage point: use a broader/more representative method of contacting passengers for the main data collection, so as not to under-represent passengers on other flights or those less likely to check email
B1 explicitly links recommendations to the pilot evidence given (the 25% figure and the single-flight sample), rather than giving generic advice
Answer: The airport should reword the leading question and, at the plan stage, redesign the sampling method to draw future samples from a range of flights and times, not just one early-morning flight. At the collect stage, it should take steps to improve on the low 25% pilot response rate, for example by sending reminder emails or offering a paper option at the gate, and should contact passengers by more than one method so that those on other flights, or less likely to check email, are not under-represented in the main survey.

Question 18

B1 the claim is correctly judged to be false/not always true
B1 a valid justification: beyond a certain size, a much bigger pilot reveals little extra information, since most wording or logistical problems would already show up in a smaller trial
B1 a different valid justification: a very large pilot uses up time, cost and potential respondents that were intended for the main survey, which works against the purpose of piloting cheaply first
Answer: The claim is false. Beyond a certain size, a much bigger pilot reveals little extra useful information, because most wording or logistical problems would already have shown up in a smaller trial. A very large pilot also uses up time, cost and potential respondents meant for the main survey, which defeats the point of piloting cheaply first.

Question 19

(a) M1 3000 x 0.06, oe
(a) A1 180 cao
(a) Answer: 180
(b) B1 explains that 15 is too small: too few responses to reliably reveal problems or estimate a response rate, since results could easily happen by chance
(b) B1 explains that 2500 is too large relative to the 3000 target: it would leave very few new residents for the main survey and would use up much of the time/cost that piloting is meant to save
(b) B1 explains that 180 gives a reasonable balance: large enough to reveal wording/logistical problems and give a reasonably reliable response-rate estimate, while still leaving the great majority of the 3000 target sample available for the main survey
(b) Answer: A pilot of 15 is too small to reliably reveal problems or estimate a response rate, since results could easily be due to chance. A pilot of 2500 is too large, since it would leave very few new residents out of the 3000 target for the main survey and would waste much of the time and cost piloting is meant to save. A pilot of 180 strikes a reasonable balance: large enough to reveal genuine problems and give a fairly reliable response-rate estimate, while leaving most of the 3000 target sample for the main survey.

Pilot Surveys - Worksheets, Questions and Revision