Trevor Chow Blog Archive Newsletter About The Blog

Back-of-the-envelope low-quality musings: #takes

Longform stuff I've put time and thought into is at #effortposts.

To hear from academics and policymakers, #interviews.

Compiled scribbles on what I read can be found in #notes.

Topic-related tags include #economics, #coronavirus and #miscellaneous.

Ofqual and Offerholders

15 Aug 2020

It has become abundantly clear that there are disparities in the A-Level results received by students from different schools or different areas - but the exact details are still murky. I think there are a few things we can clear up regarding what happened and what can be done.

Type Geo

What went wrong?

Let’s first take a look at the Ofqual meso-standardisation modelling approach of “Direct Centre-level Performance”. Firstly, Ofqual looked at the prior attainment (before A-Levels) and grade achieved (at A-Levels) of historical cohorts - this produced a relationship between the two. This relationship was applied for the prior attainment of this cohort to produce an expected distribution of this cohort. They then populated that distribution based on the rankings provided by teachers.

Notice that I haven’t used the phrase “Centre Assessment Grade” at all - this is because the model by and large doesn’t take that into account. In fact, the only time CAGs come into play is for centres with a small cohort within a subject - because the unreliability of predictions increases when there is relatively little data, Ofqual decided to weigh CAGs as part of their expected distribution for these smaller groups.

This procedure has a two effects. Firstly, its very nature means that it prevents outliers - that is, people who overperform or underperform relative to their school’s historical results are brought back within the distribution. While the central tendency may be all that matters in most statistical analyses, this is abundantly not the case when it comes to determining people’s path into further education - indeed, this benefits underperforming students at historically good schools and limits overperforming students at historically worse ones. Secondly, to the extent to which CAGs systematically overstate the results students get, this creates a bias towards those smaller cohorts - these may often be from independent schools and exacerbate the existing inequality between educational outcomes.

What should have occurred?

The statistical testing done by Ofqual suggests that the DCP meso-standardisation was the best statistical approach possible - that is, it had the best predictive accuracy and minimised the effect on inequalities. But that is not the standard we should be holding our educational process, because it was perfectly possible to avoid both problems.

One possible solution (h/t Jonny) was for Ofqual to do its statistical modelling on the basis of school data rather than subject-specific school data. Then, it would hand off the total number of A*s, As and so on for the school to distribute. This has several benefits. Firstly, the school’s cohort size is always going to be larger than its cohort size per subject - this reduces the variance, which has been shown to be reasonably large. Secondly, the larger cohort sizes mean that Ofqual wouldn’t need to rely on CAGs, which skew the results in favour of certain schools. Thirdly, schools are likely better placed to make decisions about how to distribute those grades across departments and students than Ofqual. Fourthly, this would entails handing grades over to schools much earlier and giving them time to make appeals if they think they have faced systematic issues.

What can the government do now?

Unfortunately, it is far too late for the above to occur. But given we know that there are issues with the results that have been produced, we need to consider the policy responses that exist now. There are basically two considerations here - level effects and distributional effects.

The level effect is as follows. Suppose a simplified world where there is 1 university and 5 students, with each taking one A-Level. Since the university is able to take 3 students and expects that those 5 will get A B C D E, they set the grade requirement at C or above. A shift to where those 5 get A* A B C D or B C D E F is irrelevant for ranking them. Systematically over-grading students in the former shift leads to a situation where the university is forced to take 4 people, more than they anticipated. By contrast, systematically under-grading students in the latter shift gives the university the option of letting the 3rd student in. So the level effect suggests that ceteris paribus, we should prefer under-grading.

The distribution effect matters too - we want the right people to be getting the right grades, and we don’t want a system that randomly allocates grades or structurally hands better grades to those in specific types of schools. We’ve seen how the current mechanism causes distributional problems that may lead to favouring independent schools. There is a normative claim to suggesting that it is worth mitigating the distributional harms, even if we force universities to take more students than they would like. This is the argument given by some for giving everyone their CAGs as Scotland has, and it is premised on the idea that universities can take these extra students, as evidenced by Worcester College, Oxford.

However, I am less persuaded by this than most. In part, I think Worcester is the exception rather than the rule, with most universities being unable to accommodate a significantly higher student body. This is because of their deteriorating finances (with fewer international students) and because of the social distancing demands that make finding accommodations difficult. I’m also unpersuaded that giving CAGs doesn’t itself lead to other distributional problems, due to the sorts of schools that are likely to give overly generous CAGs. As such, I do not see the distributional gains swamping the level harms.

Instead, I think that the most direct way of dealing with the distributional problems is through a wide range of mitigatory policies. That means providing financial support for universities to expand accommodation, and making those funds conditional upon increased leniency for those who are most likely to have been punished by the modelling approach. It also means making appeals and retakes free, as well as getting universities to allow offerholders to defer their offers for the following academic year. There is plenty more that can be done, such as providing financial support to those who cannot afford to take a gap year or expanding the triple lock to using CAGs for appeals.

It is clear that the government’s approach to this exam season has been problematic - it has disadvantaged some of the most vulnerable and engaged in a deterministic policy. It is only appropriate they put forward policies that can correct for this while minimising the disruptions of the level effect.

[ miscellaneous  takes  ]