In the latter months of 2019, much was made in the media of Australia’s PISA results. There was significant criticism that Australia’s results were falling and that this was a significant issue.
PISA is the Program for International Student Assessment (PISA). It is a worldwide study by the Organisation for Economic Co-operation and Development (OECD) in member and non-member nations intended to evaluate educational systems by measuring 15-year-old school pupils’ scholastic performance on mathematics, science, and reading.
To date, I have been a supporter of PISA testing as it purports to test the application of knowledge in reading, scientific literacy and mathematical literacy by 15 year-olds, rather than just what they know. However, the words that follow are well worth reading and reflecting upon. From a credible and long-standing academic, this article brings into question the global judgements made from this set of data.
Whilst the claims made for PISA data now raise some doubts in my mind, any good data set is useful if the conclusions drawn are kept limited to what the data actually says. It remains a worry that the scores for the sample of students who represent all of Australia’s students have dropped in an absolute sense. That is a cause for concern and at The Knox School, we work hard to ensure each student maximises their outcomes, no matter the measure.
What is reassuring, is that the sample of students from Australia’s independent schools rank highly: third in reading literacy; fourth in scientific literacy; and eighth in mathematical literacy.
This article was originally published in the Washington Post on December 3rd 2019. It was adapted from part one of an article to appear in Journal of Educational Change. The longer version discusses three illusions PISA has created and relied on for its undeserved impact on global education: the illusion of excellence, the illusion of science, and the illusion of progress.
PISA is a masterful magician. It has successfully created an illusion of education quality and marketed it to the world. In 2018, 79 countries took part in this magic show out of the belief that this triennial test accurately measures the quality of their education systems, the effectiveness of their teachers, the ability of their students, and the future prosperity of their society.
PISA’s magical power in the education universe stems from its bold claims and successful marketing. It starts by tapping into the universal anxiety about the future. Humans are naturally concerned about the future and have a strong desire to know if tomorrow is better than, or at least as good as, today. Parents want to know if their children will have a good life; politicians want to know if their nations have the people to build a more prosperous economy; the public wants to know if the young will become successful and contributing members of the society.
PISA brilliantly exploits the anxiety and desire of parents, politicians, and the public with three questions:
How well are young adults prepared to meet the challenges of the future? Are they able to analyse, reason and communicate their ideas effectively? Do they have the capacity to continue learning throughout life? (OECD, 1999, p. 7).
These words begin the document that introduced PISA to the world in 1999 and have been repeated in virtually all PISA reports ever since. The document then states the obvious: “Parents, students, the public and those who run education systems need to know” (OECD, 1999, p. 7). And as can be expected, PISA offers itself as the fortuneteller by claiming that:
PISA assesses the extent to which 15-year-old students, near the end of their compulsory education, have acquired key knowledge and skills that are essential for full participation in modern societies. … The assessment does not just ascertain whether students can reproduce knowledge; it also examines how well students can extrapolate from what they have learned and can apply that knowledge in unfamiliar settings, both in and outside of school. This approach reflects the fact that modern economies reward individuals not for what they know, but for what they can do with what they know. (OECD, 2016, p. 25).
This claim not only offers PISA as a tool to sooth anxiety but also, and perhaps more importantly, makes it the tool for such purpose because it helps to knock out its competitors. As an international education assessment, PISA came late. Prior to PISA, the International Association for the Evaluation of Educational Achievement (IEA) had already been operating international assessments since the 1960s, offering influential programs such as TIMSS and PIRLS. For a start-up to beat the establishment, it must offer something different and better. That’s exactly what PISA promised: a different and better assessment.
The IEA “surveys have concentrated on outcomes linked directly to the curriculum and then only to those parts of the curriculum that are essentially common across the participating countries” (OECD, 1999, p. 10) and that’s a problem according to PISA because:
School curricula are traditionally constructed largely in terms of bodies of information and techniques to be mastered. They traditionally focus less, within curriculum areas, on the skills to be developed in each domain for use generally in adult life. They focus even less on more general competencies, developed across the curriculum, to solve problems and apply one’s ideas and understanding to situations encountered in life. (OECD, 1999, p. 10).
PISA overcomes the limitations by assessing “what skills are deemed to be essential for future life,” which may or may not be covered by school curriculum. So it claims. In other words, PISA asserts that other international surveys measures how well students have mastered the intended school curriculum of education systems, but the school curriculum could be misaligned with what is needed for future life.
To make the offer even better, PISA makes another seductive claim to education policy makers: “By directly testing for knowledge and skills close to the end of basic schooling, OECD/PISA examines the degree of preparedness of young people for adult life and, to some extent, the effectiveness of education systems,” (OECD, 1999, p. 11). To paraphrase, PISA not only tells you if your children are prepared for future life, but also tells you that you have control over it through improving “the effectiveness of education.” Thus, “if schools and education systems are to be encouraged to focus on modern challenges,” PISA is needed.
However, the claim, the foundation upon which PISA has built its success, has been seriously challenged. First, there is no evidence to justify, let alone prove, the claim that PISA indeed measures skills that are essential for life in modern economies. Second, the claim is an imposition of a monolithic and West-centric view of societies on the rest of the world. Third, the claim distorts the purpose of education.
The claim that PISA measures knowledge and skills essential for the modern society or the future world is not based on any empirical evidence. Professor Stefan Hopmann of the University of Vienna writes:
There is no research available that proves this assertion beyond the point that knowing something is always good and knowing more is better. There is not even research showing that PISA covers enough to be representative of the school subjects involved or the general knowledge-base. PISA items are based on the practical reasoning of its researchers and on pre-tests of what works in most or all settings – and not on systematic research on current or future knowledge structures and needs. (Hopmann, 2008, p. 438).
In other words, the claim was just a fantasy, an illusion, entirely made up by the PISA team. But PISA keeps repeating its assertion that measures skills needed for the future. The strategy worked. PISA successfully convinced people through repetition.
Furthermore, there is empirical evidence that suggests what PISA measures is not significantly different from other international assessments or intelligence tests. For example, despite its claim to measure something different from studies such as TIMSS, performance on PISA is significantly correlated with TIMSS.
And ironically, the PISA project used results from other studies to support its case. PISA published an influential report aimed at demonstrating the importance of what it measures for economic development (Hanushek & Woessmann, 2010). The report made a number of stunning claims about the long term economic impact of improving PISA outcomes, including, for example, “having all OECD countries boost their average PISA scores by 25 points over the next 20 years … implies an aggregate gain of OECD GDP of USD 115 trillion over the lifetime of the generation born in 2010” (Hanushek & Woessmann, 2010, p. 6).
The report has been challenged by a number of scholars (Kamens, 2015; Klees, 2016; Komatsu & Rappleye, 2017; Stromquist, 2016). One of the most devastating problems with the conclusion of significant relationship between test scores and economic growth is the logic underlying the analysis utilized to reach the conclusion. The report compared test scores in a given period (1964-2003) with economic growth during roughly the same period (1960-2000), which is logically flawed because the students who took the test were not in the workforce at the time. It takes time for the students to enter the workforce and make up a significant portion of the workforce. Thus “test scores of students in any given period should be compared with economic growth in a subsequent period” (Komatsu & Rappleye, 2017, p. 170). Studies that compared test scores with economic growth in the subsequent periods using the same dataset and method found no “consistently strong nor strongly consistent” relationship between test scores and economic growth and “that the relationship between changes in test scores in one period and changes in economic growth for subsequent periods were unclear at best, doubtful at worst (Komatsu & Rappleye, 2017, p. 183), essentially invalidating the claims made in the report.
Even if the claims were valid, they primarily relied on results of international assessments besides PISA. While the report states that it “uses recent economic modeling to relate cognitive skills – as measured by PISA and other international instruments – to economic growth” (Hanushek & Woessmann, 2010, p. 6), the fact is that results from PISA constituted a very small portion of the data used in the modeling. Only three rounds of PISA had been offered by the time the report was released. Moreover, the economic data covered the period of 1960 to 2000, the year when PISA was first implemented. Only one round of PISA data was included but the report relied on “data from international tests given over the past 45 years in order to develop a single comparable measure of skills for each country that can be used to index skills of individuals in the labour force” (Hanushek & Woessmann, 2010, p. 14).
Hanushek and others (Hanushek, 2013; Hanushek & Woessmann, 2008; Hanushek & Woessmann, 2012) have repeated similar claims about the economic impact of improving PISA. Whether the conclusions are correct is a different matter. The point is that PISA’s claim to measure something different from other international assessments is a lie. It indeed measures the same construct as others. The claim to better measure what matters in the modern economy or the future world than other tests that had been in existence prior to the invention of PISA is but a made-up illusion.
A Monolithic View of Education
Underlying PISA’s claim is the assumption that there is a set of skills and knowledge that are universally valuable in all societies, regardless of their history and future. “A fundamental premise for the PISA project is that it is indeed possible to ?measure the quality of a country‘s education by indicators that are common, i.e. universal, independent of school systems, social structure, traditions, culture, natural conditions, ways of living, modes of production etc.” (Sjøberg, 2015, p. 116). But this assumption is problematic.
The first problem is that there is more than one society in the world and societies are different from each other. For all sorts of reasons—cultural, political, religious, and economical—different societies operate differently and present different challenges. Meeting different challenges requires different knowledge and skills. As a result, “one can hardly assume that the 15-year olds in e.g. USA, Japan, Turkey, Mexico and Norway are preparing for the same challenges and that they need identical life skills and competencies” (Sjøberg, 2015, p. 116).
The second and a bigger problem with PISA’s assumption of a universal set of valuable skills and knowledge for all countries is its imposition of a monolithic, primarily Western view of societies. PISA was first and foremost developed to serve member states of OECD, most of which are the world’s most advanced economies with only a few exceptions such as Mexico, Chile and Turkey. The 35 OECD members in no way represent the full spectrum of diversity across the nearly 200 countries in the world today. The assumptions supporting PISA are primarily based on the economic and education reality of OECD members. Not surprisingly, “the PISA framework and its test are meant for the relatively rich and modernized OECD-countries. When this instrument is used as a ‘benchmark’ standard in the 30+ non-OECD countries that take part in PISA, the mismatch of the PISA test with the needs of the nation and its youth may become even more obvious” (Sjøberg, 2015, p. 116).
Distorted View of Education
Although PISA claims that it does not assess according to national curricula or school knowledge, its results have been interpreted as a valid measure of the quality of educational systems. But the view of education promoted by PISA is a distorted and extremely narrow one (Berliner, 2011; Sjøberg, 2015; Uljens, 2007). PISA treats economic growth and competitiveness as the sole purpose of education. Thus it only assesses subjects—reading, math, science, financial literacy, and problem solving—that are generally viewed as important for boosting competitiveness in the global economy driven by science and technology. PISA shows little interest in other subjects that have occupied the curricula of many countries such as the humanities, arts and music, physical education, social sciences, world languages, history, and geography (Sjøberg, 2015).
While preparing children for economic participation is certainly part of the responsibility of educational institutions, it cannot and should not be the only responsibility (Labaree, 1997; Sjøberg, 2015; Zhao, 2014, 2016). The purpose of education in many countries includes a lot more than preparing economic beings. Citizenship, solidarity, equity, curiosity and engagement, compassion, empathy, curiosity, cultural values, physical and mental health, and many others are some of the frequently mentioned purposes in national education goal states. But these aspects of purpose of education “are often forgotten or ignored when discussions about the quality of the school is based on PISA scores and rankings” (Sjøberg, 2015, p. 113).
The distorted and narrow definition of the purpose of education is one of the major reasons for some of the peculiar and seemingly surprising discoveries associated with PISA. There is the persistent pattern of negative correlation between PISA scores and students’ interest and attitude. Many researchers have found that higher PISA scoring countries seem to have students with lower interest in and less positive attitude toward the tested subject (Bybee & McCrae, 2011; Zhao, 2012, 2014, 2016). For example, PISA science score has a significant negative correlation with future science orientation and with future science jobs (Kjærnsli & Lie, 2011). High PISA scores have also been found to be associated with lower entrepreneurship confidence and capabilities (Campbell, 2013; Zhao, 2012). Moreover, high PISA scoring education systems seemed to have a more authoritarian orientation (Shirley, 2017; Zhao, 2014, 2016). Additionally, PISA scores have been found to have a negative correlation with student wellbeing (Shirley, 2017; Zhao, 2014, 2016), a finding that was finally openly acknowledged by PISA in a 2017 report (OECD, 2017). These findings basically suggest that PISA only measures a very narrow aspect of education and neglects to pay attention to the broader responsibilities of educational systems. Furthermore, pursuing the narrowly defined purpose of education may come at the cost of the broader purpose of education (Zhao, 2017, 2018). “There are very few things you can summarise with a number and yet Pisa claims to be able to capture a country’s entire education system in just three of them. It can’t be possible. It is madness” (Morrison, 2013).
In summary, PISA successfully marketed itself as a measure of educational quality with the claim to measure skills and knowledge that matters in modern economies and in the future world. Upon closer examination, the excellence defined by PISA is but an illusion, a manufactured claim without any empirical evidence. Furthermore, PISA implies a monolithic and espouses a distorted and narrow view of purpose for all education systems in the world. The consequence is a trend of global homogenization of education and celebration of authoritarian education systems for their high PISA scores, while ignoring the negative consequences on important human attributes and local cultures of such systems.
Yong Zhao is a Foundation Distinguished Professor in the School of Education at the University of Kansas. He previously served as the Presidential Chair and Director of the Institute for Global and Online Education in the College of Education, University of Oregon, where he was also a Professor in the Department of Educational Measurement, Policy, and Leadership. Prior to Oregon, Yong Zhao was University Distinguished Professor at the College of Education, Michigan State University, where he also served as the founding director of the Centre for Teaching and Technology, as well as the US-China Centre for Research on Educational Excellence. He is a regular visitor to Australia and his works focus on the implications of globalization and technology on education. He has published over 100 articles and 30 books.