Next month I'm taking a day out of work to give a workshop to A-level maths students in east London. A colleague and I will try to convince 120 young people that sticking at their mathematical studies will do them good in later life. (And how could it not? They could end up like me. Well. Perhaps best to stick to theoretical material on the day).
Rachel (my colleague) suggested we use our workshop to introduce the students to the concept of "regression to the mean". We're going to simulate a drug trial by asking the students to each roll a dice and record the score four times. These numbers will represent the number of headaches which four patients present with when they visit their GP. Aha! says the GP, I know about a clinical trial for a new medicine to cure headaches. I'll send those patients who had five or six headaches - the most serious cases - to take part in the trial. We then get the students to roll the dice again, for those severely ill 'patients' only. Surprise surprise - across the class of 120 students - we can expect with extremely high probability to show that the number of headaches after the 'drug' is less than the number before. Of course this has nothing to do with the 'drug' at all (as I hope is obvious, unless there is some hitherto unsuspected therapeutic benefit to be gained from rolling a dice - perhaps this is the basis of holistic medicine?) but is a simple example of regression to the mean. (The phenomenon has the - for statistics - poetic name, because Francis Galton first noticed that the sons of tall fathers had heights closer to the population average, and vice versa).
Now I've known this for decades (oh, so many decades) but it's only this morning that I made some sort of political connection. I do reason very slowly, so I apologise if this has always seemed obvious to everyone else. But the same problem will apply to the evaluation of a couple of important government policies - and potentially those of a new Tory government also.
Suppose you were interested in spending extra money on the educational opportunities for the most disadvantaged students. (Good). If you selected schools which had the lowest outcomes - say the bottom 5% in the distribution of SAT scores - and spent £X million pounds on them - and returned to the school to measure the 'impact' of that money 5 years later - you would actually be astonished if you didn't find an increase in the performance of the school. But nothing on earth could permit the inference that it was the money you had spent which had delivered the improvement. (Of course, you can protect against the faulty inference by ensuring a proper control group - best to split the low-achieving schools into two groups, at random, and make the financial intervention in one group only, and then compare results).
There's another striking example going on just now. GB Athletes are doing fantastic in the Olympics, and some political commentators are making all sorts of ludicrous claims - see Steve Richards in the Independent, who says that their success 'proves' that government intervention lifts athletic performance, and hence spending more public money will make anything better (the ever-astute Guido noticed this too). Leave aside the non sequitur of the subordinate clause in that last sentence (Richards' claim is a great example of the Yes,Minister fallacy: my cat has four legs, your dog has four legs, my cat is your dog); leave aside Mr Richards' definition of 'public' expenditure (the money came from the Lottery, not taxation) and leave aside the slightly more plausible hypothesis that the new cohort of athletes have had better training plans than their predecessors. Assume that Richards is right, and that nothing has changed between previous British Olympic sportspeople and the current cohort, other than that more money was spent on them. Then because the previous medal tally was so low (I know not sporting league tables, but I believe the tally was extremely low in the last few Games) it would be bizarre in the extreme had the tally not returned to something like its historical temporal mean. Richards' reasoning is faulty. The Left have not 'proven' that increased expenditure has delivered greater Olympic returns, and, by "Steve Richards style" deduction, have therefore shown that not increasing public expenditure need not lead to poorer public services.
