Maths scores in world education rankings inflated for England and Wales – study

England’s recent improvement in international tests comparing the educational performance of 15-year-olds across the globe is based on “flawed” data which, if corrected, could result in a sharp drop down the rankings, according to new research.

The government hailed the most recent round of results in the OECD’s Programme for International Student Assessment (Pisa) in 2018, which saw a marked improvement in England’s maths results, as a vindication for policy changes driven by Michael Gove when he was education secretary.

Research by UCL Institute of Education suggests, however, the results and subsequent rankings were distorted because of low levels of participation in the test and a disproportionate underrepresentation of lower achieving students. If the data was adjusted on the basis of a more representative sample, it suggests England could plummet 11 places in the rankings for maths.

The study, to be published in the Review of Education, claims there is a similar story in Wales, where Pisa maths scores could be inflated by as much as 15 points. If adjusted, Wales would fall seven places in the world rankings, taking it below the international average and on a par with scores in countries such as Malta and Belarus.

The Pisa tests, which are held every three years by the Organisation for Economic Co-operation and Development (OECD), assess reading, science and mathematics among 15-year-olds in about 80 countries. They have become increasingly high-profile, their results keenly awaited by governments as a report card on their time in office, and their findings shaping education policy.

Although the UCL paper focuses on Pisa data for the UK in the 2018 round of tests, it points out that similar issues may have affected previous rounds and other countries, particularly those with similarly low overall participation rates, such as Canada, Sweden and New Zealand.

The author of the report, John Jerrim, a professor of education and social statistics, said: “Pisa is meant to be a representative study of 15-year-olds across the UK. But there are serious flaws with some children being excluded from the study, schools being unwilling to participate, and some pupils not showing up for the test.

“The reported average Pisa mathematics score for England was 504 points – significantly above the average across industrialised (OECD) countries. But, had a truly representative sample been drawn, I estimate England’s Pisa mathematics score would have been about 494 – roughly the same as the OECD average.”

Jerrim also claims key limitations with the data was not transparently reported and has urged the Office for Statistics Regulation to conduct an urgent review of how the data is reported. “Clear guidelines need to be put in place to ensure more transparent reporting in the future,” he said.

He called on the OECD to reconsider its technical standards and its data adjudication processes. “The processes currently in place are nowhere near robust enough to support the OECD’s claims that Pisa provides truly representative and cross-nationally comparable data.”

An OECD spokesperson said it could not rule out that exclusions and non-responses at student level had resulted in a small upward bias in results in England and Wales. “In fact, we cannot exclude that a similar level of upward bias was present in past Pisa administrations or in other countries for which no such exercise was conducted.

“The OECD will ensure that this evidence is considered in future discussions about the appropriate sampling standards and supports efforts to assess the quality of Pisa even beyond the publication of the datasets.”

A Department for Education spokesperson said: “The PISA 2018 data for England met the standards of the OECD and was fully accepted as valid and suitable for its international reports. Our data is subject to rigorous scrutiny processes and the OECD was satisfied that the school data was representative of the sample schools chosen.”

Comments are closed.