Worked question 9 - Further Statistics: Hypothesis Testing and Chi-Squared Depth
Key facts: chi-squared tests, in more depth
Original content written for Revision Library.
A chi-squared test compares observed frequencies O with the expected frequencies E predicted by a model, using X^2 = sum of (O-E)^2/E over all classes or cells, after any pooling. For a goodness-of-fit test to a fully specified distribution, df = (number of classes) - 1; if a parameter is estimated from the same sample (for example a Poisson mean, a binomial p, or a geometric p), one further degree of freedom is lost for each parameter estimated, because the estimation process artificially improves the apparent fit. For continuous data grouped into classes, expected frequencies are found from probabilities calculated using the standard Normal distribution, via z = (x - mu)/sigma and the function Phi. For an r by c contingency table testing independence between two categorical variables, E = (row total * column total)/grand total (this follows directly from the multiplication rule P(row and column) = P(row) * P(column) applied to the sample), and df = (r-1)(c-1). Classes or cells with an expected frequency below 5 should be pooled with a neighbouring class or column before the test statistic is calculated, and the degrees of freedom adjusted accordingly. Yates' continuity correction, X^2 = sum of (|O-E| - 0.5)^2/E, is applied only when df = 1 (every 2 by 2 contingency table), since this is the only case where the discrete test statistic is poorly approximated by the continuous chi-squared distribution. The signed standardised residual for a cell, e = (O-E)/sqrt(E), shows the direction as well as the size of a departure from the model, and the sum of its squares reproduces X^2. Because X^2 is never negative, only large (upper-tail) values ever provide evidence against H0; the calculated statistic is compared with the critical value from chi-squared tables at the chosen significance level and the stated degrees of freedom, and H0 is rejected if the statistic exceeds this value. As with any hypothesis test, the significance level equals P(Type I error), reducing it increases P(Type II error) for a fixed sample size, and increasing the sample size generally increases the power of the test; a chi-squared statistic also scales with sample size, so a very large sample can make even a small, unimportant deviation from a model 'statistically significant'. Combining ('pooling') data from distinct groups before testing can also mask or even reverse a genuine association found within each group, an effect known as Simpson's paradox.
Branch 1: took sessions - passed 54, failed 66, row total 120. Did not take sessions - passed 10, failed 20, row total 30.
Branch 2: took sessions - passed 24, failed 6, row total 30. Did not take sessions - passed 72, failed 48, row total 120.
Answer and mark scheme
Question 9
| Credit | Answer or working |
|---|---|
| M1 | (a) correct method for at least one percentage |
| A1 | (a) all four percentages correct: Branch 1: 45.0% and 33.3%; Branch 2: 80.0% and 60.0% |
| A1 | (a) comment: in both branches, students who took the extra hazard sessions had a higher pass rate than those who did not |
| M1 | (b) combines the two branches' figures first: sessions 78 passed out of 150; no sessions 82 passed out of 150 |
| A1 | (b) combined percentages correct: 52.0% (sessions) and 54.7% (no sessions) |
| A1 | (b) comment: remarkably, when the branches are combined, students who did NOT take the extra sessions now have the higher pass rate; the direction of the association has reversed compared with each individual branch |
| M1 | (c) expected frequencies: 80, 70, 80, 70 (e.g. E(sessions, pass) = 150*160/300 = 80) |
| M1 | (c) uses X2 = sum of (O-E)2/E |
| A1 | (c) cao: X2 = awrt 0.214 |
| A1 | (c) 0.214 < 3.841, do not reject H0: no significant evidence of an association between taking the sessions and passing, in the combined data |
| M1 | (d) expected frequencies: 19.2, 10.8, 76.8, 43.2 |
| A1 | (d) cao: X2 = awrt 4.17 |
| A1 | (d) 4.17 > 3.841, reject H0: significant evidence of an association between taking the sessions and passing, at Branch 2 |
| B1 | (e) pooling can hide, distort, or even reverse a genuine association that is present within each branch, because the branches differ substantially both in their overall pass rates and in how many students take the extra sessions (branch acts as a lurking/confounding variable). This effect is known as Simpson's paradox |
| B1 | (e) different branches can also show different strengths (or even different conclusions) from the same test, as seen here: Branch 2 shows a significant association, while the combined data does not |
| B1 | (e) the company should analyse (and test) each branch separately rather than relying only on the combined totals; if a single overall comparison is genuinely needed, a method that accounts for branch (such as testing within each branch and combining the evidence, or matching/stratifying by branch) should be used rather than a naive pooled total |
- (a) Answer: Branch 1: 45.0% (sessions) vs 33.3% (no sessions). Branch 2: 80.0% (sessions) vs 60.0% (no sessions). Sessions associated with a higher pass rate in both branches.
- (b) Answer: Combined: 52.0% (sessions) vs 54.7% (no sessions) - the direction of association reverses compared with each branch separately.
- (c) Answer: X2 = 0.214 < 3.841; not significant, no evidence of association in the combined data.
- (d) Answer: X2 = 4.17 > 3.841; significant evidence of association at Branch 2.
- (e) Answer: Pooling across branches can mask or reverse a real association (Simpson's paradox) because branch is a confounding variable; the company should test each branch separately, or use a method that adjusts for branch, rather than pooling the raw totals.





