Warwick Mansell's Blog

A change in the way science tests for 11-year-olds were carried out this year, and the consequent effect on results, has led to national headlines that standards are now at their lowest level for more than 10 years.

I am really struggling to get my head around how such negative stories can be generated when the data on which they are based clearly do not support this inference, even though I warned in the spring on this blog that this misinterpretation could happen.

The basis for the reporting was that the proportion of pupils achieving level 4 in science dropped from 88 per cent last year to 81 per cent this summer. The proportion reaching level 5 fell even more steeply, from 43 to 28 per cent.

The reporting is fair enough, then, you might think. Except that the two years’ results were generated under hugely different conditions. Trying, then, to use the scores created by the two systems as evidence that standards have slipped is just not valid.

Of course, last year’s science tests were taken by the whole 600,00-pupil cohort in England. The results were used to generate league tables, as well as being important to Ofsted judgements, targets and the panoply of mechanisms which is England’s school-by-school accountability system.

Last year, the last government wisely accepted a recommendation from its “expert group” on assessment which highlighted the downsides of this approach, including the negative effect on science teaching of over-extensive test preparation. It is also the case that, perhaps paradoxically, more insightful judgements on overall standards can be achieved by testing fewer pupils.

So this year we have a new system. Only five per cent of year six children nationally were tested in science. And, crucially, the results were “low-stakes” for their schools: league tables would not be constructed according to them and they would form no other part in the accountability system for individual institutions.

It should surprise no-one, then, that results fell back in these circumstances. Schools which were encouraged by the previous system to spend months preparing children for these tests, including extensive teaching to the test, no longer had any incentive to do so. Also, pupils may well have felt less of a need to try hard at the new tests, since their schools would have had no reason to tell them that they were important.

To state what should be obvious, then, the change in results would appear to say much more about the alteration in the testing system than about any underlying change in standards of science education.

To be fair, I cannot be sure about this. It is just a hunch. But clearly, neither are the results evidence of changing underlying educational standards, when the measurement system has also changed. It is, surely, basic science that if you want to measure change, you do not change the measurement mechanism.

To use a comparable example from economics, you can imagine the scenario if the official means of measuring inflation was changed, leading to a rise in the number which was generated at the end of the process. Would we really expect to be reading that inflation was at its highest level for years, based only on a comparison with numbers generated under the old measurement system? This would just not be credible.

The error here is not the Government’s, however: the lack of comparability of the figures is made clear in its official statement on the results.

It is ridiculous, then, to say that this is evidence that standards of science education are lower now than they were in previous years.

That does not mean, though, that the new figures are not revealing in their own way. While they do not show that standards have slipped, they may offer some sign, as other reports said, that the previous figures generated under Sats were inflated by teaching to the test.

There is much evidence, of course, that teaching to the test is going on in a lot of schools. The gap between the new figures and the old would appear to suggest that when incentives for schools to go in for close test coaching are taken away, we get a more realistic picture of performance.

I would, though, be a bit cautious even before reaching that conclusion. The absence of teaching to the test in science, and perhaps schools’ decision to focus even more attention on the remaining “high-stakes” Sats subjects of English and maths will, I think, explain a good part of the “fall” in results this year. But it may also be the case that children who are no longer being told by their schools that these results are important to them have simply not tried as hard as their predecessors did.

That does not mean they have learnt less science as a result, by the way. It just means they might have put in less effort for the tests themselves.

There may be quite a large element of silver lining in these stories, though, for heads who would like to see the back of Sats. If the media do come to believe or imply – rightly, in a sense, although I am not sure there will ever be any one “true” measure of national education standards – that sample tests are revealing a truth about underlying standards which have remained hidden by Sats and their tendency to promote teaching to the test, perhaps the move towards sample tests will have been strengthened.

Sample tests, as I have written earlier on this blog and many others have also argued for a while now, are potentially a much better way of measuring national education standards, without educational side-effects, than Sats. But let us wait until they have been in place for more than one year before we start trying to use them to judge trends in national education performance.

Supporters of teacher assessment will have been cheered by the following paragraph on the “research and statistics” section of the Department for Education’s website. It reads:

“From 2010, Key stage 2 science at school level will therefore be assessed by teacher assessment only, which takes greater account of pupils' practical grasp of the subject and is based on their attainment throughout the academic year across the full programme of study.”

If this is true of science, many would say, is it not also true of English and maths?