The former TES journalist writes for NAHT on current education issues. The views expressed do not necessarily reflect those of NAHT
Some implications of Ofqual’s moves to tackle “grade inflation”
A-level results, to be published on Thursday, will see little or no overall improvement in either the pass rate or the proportion of higher grades awarded, this blog can confidently forecast.
In previous years, this might have seemed like a bold prediction, as the percentage of high grades in particular followed a seemingly remorseless upward trajectory.
But this year, things are different (note 1), because of the way that the overall grade statistics are being calculated. And this has implications, I think, not just for the results of individual pupils at A-level – and those announced at GCSE next week – but for other aspects of school-by-school performance statistics, accountability and management.
As ever, many schools’ futures will hang on the technicalities of results calculations, and the details of this have big ramifications. And, also as ever, those technicalities come in a lot of detail, so as usual please bear with me.
OK, as has been reported elsewhere, Thursday’s results are going to be particularly interesting because of a recent change in the way A-level and GCSE calculations are regulated by the exams watchdog, Ofqual.
Quietly in 2010 and last year, and with more media coverage this year, the regulator has brought in reforms designed to halt what it now is happy to describe as “grade inflation”. Now, this could be seen as a quasi-political move, with Ofqual simply responding to concerns about years of improving exam statistics by insisting the exam boards don’t allow it to happen any more, but the argument is a bit more technical than this.
In a paper first published in May 2011 and updated this May (http://bit.ly/NAPJ4Q) , the regulator sets out one reason why – for perfectly justifiable reasons – grades at both A-level and GCSE has shown that constant upward trend in recent years.
The argument runs as follows. When an exam syllabus changes – and, as readers of this blog who work in secondary schools will need no reminding, there have been many changes over the past couple of decades – something fairly obvious can happen in terms of the performance of exam candidates.
This is that the standard of work they produce in the exam hall will tend, on average, to be slightly worse than that of their predecessors the year before. The reason is that the pupils from the year before (let’s call them year A) will have been working towards an exam which their teachers will have grown used to (Note 2), having already had several “goes” at it and also having past papers to practice on, whereas the ones in the first year of a new syllabus (year B) will not benefit from this prior experience on the part of their teachers.
Exam boards then have two options, on observing this phenomenon. First, they could say to the year B pupils “tough: we have our standards and it’s not our fault that teachers – through no fault of their own – haven’t prepared you quite as effectively this year as they did last”.
In other words, the boards would look at the quality of students’ work needed to achieve a particular grade, and hold it constant. If this means, then, that in the first year of a new syllabus, the number of students achieving good grades goes down as pupils and teachers struggle with the new demands of a course, the boards would say “so be it”.
Or, as a second option, the boards could compensate. They could say to the Year B pupils: “We know it’s not your fault that this is the first year of a new syllabus; we know that, if all other things were equal, without the fact of this new syllabus, we would expect you to do at least as well as your predecessors, so we’re going to recognise this in our grading system. Assuming that we think that the standard of work produced has fallen slightly this year, we are, then, going to lower the grade boundaries – the minimum number of marks needed for a particular grade - to make sure that two candidates from the two successive years who are roughly the same quality end up with the same grades.”
Exam boards, so far at least, have tended to take the latter approach because they want candidates of a similar underlying quality from one year to the next to emerge with the same grades, and those in the latter year not, effectively, to be penalised for being in the first year of a new course. While perhaps a tough sell to a public sceptical about grade inflation, it doesn’t seem an unreasonable approach to me.
But what happens the year after the new syllabus is introduced, in year C? In recent years, the boards have then reverted to, effectively, the first option above. With no syllabus changes coming in for the first time for year C students, they simply compare the work of Year C with that of Year B and judge whether they think more or fewer pupils should get good grades based on the relative quality of the work produced. In other words, there is no compensatory move to lower grade boundaries to take account of a new syllabus, because there is no new syllabus.
But can you spot how this process can – and is now said by Ofsted – lead to a gradual fall in the standard of work needed to produce a grade over time?
The argument is this: if a particular standard of work is needed to produce a particular grade under the old syllabus, at year A, and then you drop the standard required for a particular grade in year B, but then hold that new standard constant in year C and in subsequent years where the syllabus does not change, you have a problem. By the time we get to years E and F, say, pupils will have had the benefit of teachers with several years’ experience of the new syllabus, and might therefore be just as well prepared as their predecessors from year A were to do well in their version of this exam. But the demands made of pupils in years E and F, in terms of the standard of work expected for a particular grade, will only be what they were in year B, when the boards were compensating for this being a new syllabus.
Then, when the syllabus changes again, in year G say, the boards will compensate again, with boundaries set yet lower than they would have been in year F, to reflect the fact that this is a “harder” year for pupils, who would be on the first year of a new course. And so, sceptics or critics of this system would argue, there is a gradual stepping-down in standards.
So Ofqual has now moved to a different approach.
A-levels went through their last major change in exams for which results were first produced in 2010. In that year, the A* grade was introduced alongside other reforms including, in most subjects, a reduction in the number of papers from six to four.
Ofqual responded by allowing the boards to react as they would have done in previous years: to ensure they set grade boundaries to reflect the fact that, while the changes might have made the exam more challenging for that year’s cohort, the overall percentage of pupils getting each grade would remain broadly unchanged. In other words, if the changes meant that pupils this year actually found it harder to produce good work, the boards would have been allowed to compensate by lowering grade boundaries.
The key difference, though, has been what has happened in the second year of the reformed A-levels. Ofqual has said that it does not want the statistics to rise, as would normally happen as teachers and pupils get used to a new syllabus. Instead of the boards expecting the same standard of work in 2011 to achieve the same grade as it did in 2010, then, and then watching as more pupils get good grades over the years, they must now hold the proportions of grades more or less constant unless they can demonstrate that underlying standards of pupil ability have been going up.(I’ve put this bit in italics because it is important to my later argument). There should be, then, no gradual improvement in grades as pupils and teachers get more used to a syllabus. And, by this argument, if the reality is that pupils are helped by their teachers to produce better work as the years go by and the syllabus beds in, the logic here is that the boards should compensate for this process by raising the grade boundaries: increasing the number of marks needed for each grade.
Once you get past all the technical detail above, then, this is radical stuff. It’s not surprising that, in the reporting of this so far, there has been an acknowledgement that it will be controversial, not least with teachers, I’m guessing, who may wonder why their cumulative efforts to get their pupils better prepared for exams every year may no longer be rewarded. Against that, the argument above about an incremental reduction in standards if Ofqual’s regulation of the system had not been changed in this way will have to be balanced.
Ofqual seems happy with how the approach has worked at A-level so far, though. And, interestingly – and more significantly for school-by-school accountability - its paper sets out how it is now pursuing the same methods at GCSE.
Again, at GCSE, secondary schools will need no reminding that we have had reform after reform in recent years. Last summer, pupils took new exams in most subjects. Ofqual responded by insisting, it says(Note 3), that new GCSE subjects produced roughly the same average results as those under the old regime. Crucially, it says it is now going to use the same approach – expecting results to be roughly the same from year to year, all other things being equal – at GCSE for this year and in future years. So expect results at GCSE now not to change much from year to year.
I say “all other things being equal” because Ofqual is not saying that this means that results statistics have to be exactly the same from one year to the next now, nor that we are never going to see major gains in the proportions of pupils getting good grades. That would be, effectively, a return to the system of norm referencing that operated until the 1980s, whereby the proportions gaining each grade was fixed in advance.
However, the boards will now have to demonstrate to Ofqual that, if they do want to increase (or decrease) the percentages gaining a certain grade, that they have good reasons for doing so. In practice, this means that the boards will have to have other data showing that – for example – a particular year group is of higher, or lower, ability – for want of a better word - than its predecessors and therefore that more, or fewer, pupils from that year deserve good grades. Boards have, in recent years, already been using measures of pupils’ prior ability in trying to reach standards judgements, but this new regime seems to be standardising and formalising the approach.
At A-level, the boards use the previously-achieved GCSE results of each pupil to assess whether the cohort for a particular A-level is particularly strong, or weak, that year. A rise in the number of top grades, then, might be proposed by the board – and accepted by Ofqual – if it had statistics showing that there were more able candidates taking this subject this year.
At GCSE, similar methods suggest that, under Ofqual’s rules, there could be an increase in the proportions gaining top grades, but only if the boards’ statistics on the underlying ability levels of the candidates suggest the cohort is more able this year.
Interestingly, however, Ofqual’s paper shows that the main method for calculating the prior ability of the cohort is key stage 2 results for that year group.
And we already know, of course, the key stage 2 results for pupils up to 2011. Since these show no major improvements, it is difficult to see how GCSE results are going to increase much between now and when the 2011 year six cohort take their GCSEs, in 2016.
In other words, Ofqual’s rules seem to be saying: GCSE results will stay roughly the same each year, unless key stage 2 results indicate there has been an improvement in the cohort. But the KS2 results for the years to 2016 don’t seem to indicate any major change. The implication is not to expect any major change in GCSE results in the years to 2016.
This, of course, has major knock-on effects. First, it will mean that ministers will not be able to take a rise in exam scores back to a supposedly grateful electorate as evidence of their own success. This, though, may not be a major problem for the coalition, as the targets system – whereby Government performance was judged in part by rising numbers of pupils doing well at GCSE and A-level – has been dropped under Michael Gove. ( I think this has been a good thing, despite my natural aversion as a journalist to anything which would appear to make politicians less accountable to the electorate, but that’s something for another day…)
But the second point is far more significant, I think. If GCSE success rates really are not going to keep rising much, then a second set of targets, imposed on schools last year by Mr Gove, will very much be affected.
In June 2011 (http://bit.ly/kBtXok), the Education Secretary put forward expectations that there would be a huge change in GCSE performance over the coming years. “The current average performance will become the new ‘floor’ for secondaries- all schools should have at least 50 per cent of pupils getting five good GCSEs including English and maths”.
Given the above, it is hard to see any way in which this can happen. Given that the latest average national figure on this measure is 58 per cent, and the large current variation in exam results between schools, either large numbers of schools achieving the 50 per cent score by 2015 will mean many schools currently above that benchmark start seeing their results fall substantially, which isn’t going to happen; or the national average figure will have to rise significantly, which given the arguments above, also isn’t going to happen.
And, given that schools failing to achieve these “floor standards” (otherwise known as targets) are said by the Government to face takeover as academies, some will say ministers have deliberately made this new benchmark seemingly almost impossible to hit for many schools.
I don’t have the evidence to know that the Government is being quite so cynical, however. It may just be that these targets were set without a detailed consideration of whether the exam results on which they sit are technically capable of producing such a major change. If I were a betting man, I would wager that that would be the explanation, rather than a calculated move to set many schools impossible goals. However, whether he would have known that these were virtually impossible statistical expectations because of the way national exam standards are set, Mr Gove would have been aware that that these were highly challenging targets and therefore that enforced takeovers would be likely in many cases.
All of this, of course, may be slightly redundant as a discussion as, bewilderingly perhaps, only a year after these new GCSE targets were set, plans with Mr Gove at their heart were leaked which would, seemingly, scrap the GCSE system altogether in favour of a return to O-levels.
Since that leak was published in the Daily Mail in June, however, we have heard little from the Department for Education on its exams proposals, with a paper now just promised for the autumn. But several questions follow. Where will the GCSE targets set last year be after this paper has been published; will Ofqual’s calculations be taken into account; will the O-level plan still be alive then; and how will it interact with yet more reforms already on the cards for future years, covering not just GCSEs, but A-levels, too?
It is very difficult to know the answers to any of these questions. Keeping up with the complexities and seeming inconsistencies of Government policy, as well as the technicalities of GCSE and A-level reform, is a full-time business, and certainly not for the faint-hearted.
Note 1: Indeed, at time of writing, articles in the Independent and the Sunday Telegraph have included similar results predictions.
Note 2: Providing this syllabus itself has not changed for a few years.
Note 3: This is what Ofqual says in its paper setting out its new methodology (the one which I linked to at the start of the piece). However, comparing the average UK results of a range of “new syllabus” subjects in 2011 – French, German, geography and history – with those for their old syllabuses 2009 does show that the figures were generally better in 2011 than they were in 2009, so make of that what you will.