Timandra Harkness’s verdict, writing in Unherd is coruscating; and boy, when she coruscates it stings. Her view is unequivocal, she states that OFQUAL’s mathematical model for allocating A level grades was a prejudice machine.
“Any one individual’s achievements so far, or their potential in the view of teachers who know them, had less influence on their eventual results than the attainments of others who attended the same school in past years. It’s closer to buying car insurance than taking an exam for which you have worked for nearly two years. Just enter postcode, make, and model and we will predict your likelihood of making a claim, and hence your premium.”
The current hiatus was not supposed to happen. The Royal Statistical Society (RSS) had volunteered to help with what is a complex multivariate analysis. Getting it right depends on clear recognition of the difference between correlation and causation. Just because two numbers move in the same direction it does not mean that they are directly-linked. It is the difference between A happens at the same time as B and A happens because of B. This is known in the trade as a “nonsense” or “spurious” correlation. There is a rigorous process for rooting out such anomalies.
(If you have time to spare it is worthwhile watching Professor Alex Edmond’s masterclass on the dangers of spurious correlation www.ted.com/talks/alex_edmans_what_to_trust_in_a_post_truth_world#t-443345 )
The RSS proposed two distinguished practitioners but found themselves rebuffed by OFQUAL who raised a requirement to sign Non-Disclosure Agreements which would have prevented the experts from commenting on the results for five years. The RSS felt this to be an unreasonable constraint on the dissemination of the broader key lessons that might emerge.
QFQUAL said in response “Throughout the process we have had an expert advisory group in place, first meeting with them in early April. The group includes independent members drawn from the statistical and assessment communities. The advisory group has provided advice, guidance, insight and expertise as we developed the detail of our standardisation approach”.
Based upon the evidence of the last few days would OFQUAL now care to name its advisers and advice etc. from the statistical and assessment community? In exams – this is known as “please show your workings”.
This has been a massive Systems Crash. What we to talk about here is akin to a plane crash enquiry – nothing can ever put a crashed plane back in the air but assembling the pieces bit by bit and evaluating the black box results means that such a disastrous event will not be replicated. Although it might be considered expedient to move the (not yet immune) herd along – this is not “a nothing to see here/one off” situation. Pending the arrival of an effective vaccine, we must be prepared for the medium- and long-term effects of Coronavirus on the education and examinations systems. We cannot assume that this month’s exam results are the only ones that might be impacted. The development of publicly accepted, statistically robust, evaluation systems must be a priority.
Everybody understood the need not to let the Coronavirus overwhelm the NHS – we cannot permit splash back into other critical systems. If only somebody, somewhere could have a light bulb moment and say to both OFQUAL and the RSS – can you please get together and fix this?