Friday July 19, 2024

Because TIMSS says so

By Dr Ayesha Razzaque
December 21, 2020

The writer is an independent education researcher and consultant. She has a PhD in Education from Michigan State University.

In 2019, somewhere in Pakistan a group of fourth graders was shown a picture depicting a scene in a desert. It had a camel walking through the sand, a reptile perched on a rock, a spider on a cactus and a bug on a plant, with the sun and some clouds in the sky above a mountain range rising in the background. Students were asked to name two living and two non-living things in the picture. Only 34 percent of them were able to answer the question correctly.

The IEA (International Association for the Evaluation of Educational Achievement) at Boston College’s Lynch School of Education publishes the results of the Trends in International Mathematics and Science Study (TIMSS). TIMSS has been monitoring trends in mathematics and science achievement every four years, at grade 4 and grade 8 levels over the last 25 years by conducting sample-based testing in countries around the world.

The latest TIMSS assessment of grades 4 and 8 for mathematics and science took place in 2019. It was the first time Pakistan participated and we entered only for grade 4 assessments. According to a government official, participation cost us Rs24.7 million. The assessment report just came out. The test question described above was one of many that students were asked in the TIMSS science assessment for grade 4. Among the 64 participating countries, Pakistan stands second from the bottom in both grade 4 science and mathematics learning.

It is not that students were given questions that were too challenging and beyond their level of understanding. Indeed, according to the National Curriculum 2006, the question is considered appropriate for grade 2. Interestingly, though, when grade 4 science achievement scores are analyzed for gender disparity, Pakistan has the fourth largest skew towards girls. Analysis of test scores in all areas of mathematics (numbers, measurement & geometry, data) and cognitive domains (knowing, applying, reasoning) also all skew favorably towards girls.

Every four years, the TIMSS report reiterates the same lessons: Supportive household environment, household wealth (other research suggests per-capita GDP), emphasis on literacy and numeracy, pre-primary education, school principal’s years of education and experience, school emphasis on academic success, school safety, discipline and no-tolerance towards bullying – these are all factors positively correlated with academic achievement. Absenteeism, hunger, tiredness, dislike of the subject, and lack of confidence are all negatively correlated with academic achievement.

None of this is or should be news. We have known this for years from locally conducted studies of contextual data. TIMSS measures Pakistani students’ achievement by a standardized yardstick and gives us a reliable rating relative to students in other participating countries, but there are no new earth-shattering revelations. The only reason I can think of why these same findings will cause amazement and receive breathless coverage in the development sector for a few weeks is because it comes in the form of a shiny report from an international, high quality assessment that compares us to other countries.

In the coming years, continued participation in TIMSS will yield a longitudinal picture of trends in the academic achievement of Pakistani fourth graders. That makes one wonder: what are the numerous government departments, both at the federal and provincial levels, that are tasked with assessing, monitoring and tracking student learning doing?

TIMSS has put the database files from every round of assessments on its website for public download and ready for use. Why does it take a multinational study for us to get access to the same, maybe even less detailed, kind of data our own public sector departments have? At the federal level, there is the National Education Assessment System (NEAS) – while in provinces there is an alphabet soup of departments and authorities doing the same at various grade levels.

Across provinces, children are assessed – in fact over-assessed – sometimes as frequently as six times in twelve years of school education. This is in addition to regular annual school exams. Visit the websites of these various testing commissions, authorities, and units though, and you will find them light on data but heavy on photographs covering speeches, events and workshops. The most you can hope for is a badly written report here or there.

This is what passes for transparency in government departments. Surely, the reason to keep data from studies and assessments so inaccessible cannot be for ‘national security’. This opaqueness serves only one purpose – to block analysis by independent third-parties and / or conceal years of incompetence and lack of progress.

The reporting (if funds allow) is bland and straightforward and devoid of exploratory analysis that attempts to expose new relationships in the data. This is not surprising, since very few eyeballs have access to the data. This is where publication of data sets can help. Making data available to hackathons, hobbyist data scientists, researchers, and graduate students can raise this number to tens or even hundreds of thousands of (possibly better trained) eyeballs. If public sector departments stuffed to the brim with employees lack the expertise to analyze and disseminate data, it might be worth considering outsourcing this function to the private sector.

Most critically though, there is no evidence that the conclusions from foreign or local studies are ever used to systematically inform any program or policy level decisions. The problems the TIMSS report has identified, yet again, have been known for decades. The latest intervention, the SNC and the follow-up conversation around its implementation, does not address any of them. Right now, the people developing data driven recommendations and the people setting the education policy agenda are talking past each other. Broadly speaking, even beyond the education sector, the public sector has failed to base policy decisions on data.

In the wake of the release of the 2019 TIMSS report, we can expect the usual chorus of take-aways: remodel our schools and education according to Finland / South Korea / Singapore / insert any other country’s name near the top of the TIMSS ranking – regardless of the fact that we already know, have known for years, what needs to be done. All of our problems, all interventions we have attempted continue to fail due to the one same weakness in our education system: Teaching is a least desirable profession, and teachers are unmotivated, underqualified and under-resourced. There is no way around fixing this problem without making significant investments.

Unfortunately, the emphasis remains on creating reports and documents, like the Single National Curriculum (SNC), which cost almost nothing. Even if public schools were to adopt Singapore's, Finland’s, Cambridge’s or anyone else’s curriculum wholesale tomorrow, it would not improve student learning one bit, as long as that is not backed by matching requisite resources.

Cultural differences between societies means that there is not one single way to get education right. Finland and South Korea, both among the global top-scorers, have famously different approaches. Finland spends a modest share of its GDP and pays its teachers decent, but not extraordinary, salaries. One could argue that the Finnish approach relies on tight social cohesion and a shared sense of responsibility to achieve its performance. South Korea on the other hand, spends generously on teachers, with the best among them achieving a kind of popular rock star status.

To demonstrate that merely throwing money at the problem is not a solution, Saudi Arabia also spends a great deal of its GDP on education, but its performance on the TIMSS is only a few ranks above Pakistan. Anecdotally speaking, while the country is spending a lot on hiring expat teachers who may well be capable of better teaching, workplace and classroom dynamics leave them unempowered and prevent them from challenging students, and so (for fear of student complaints) most decide to not rock the boat. The end result is that Saudi Arabia’s average achievement scores are only marginally better than Pakistan’s, whose schools operate on a shoestring budget in comparison.

Whether you look at Finland, South Korea, Singapore or any other high scoring country, the one commonality they all share is talented, capable and motivated teachers. Without this essential building block, no SNC, no new National Education Policy, no amount of money will make a difference. This conclusion is not a new one, but this time it comes packaged in the form of an international report of repute, which shows us our non-performance relative to the rest of the world. Maybe that will help convey this lesson to the ears that need to hear it – because TIMSS says so.