Aiming for a ‘great buy’

The writer is an independent education researcher and consultant. She has a PhD in Education from Michigan State University.

News coverage of Covid vaccine development has introduced the world to the use of Randomized Controlled Trials, or RCTs, where one group of participants are given the vaccine, and another group given a placebo. At the end of the experiment, researchers have quantitative evidence of the effect the vaccine had on participants of the vaccine group versus those of the placebo group.

The 2019 Nobel prize in Economics was awarded to Abhijit Banerjee, Esther Duflo and Michael Kremer for bringing this experimental approach, borrowed from the foundations of the scientific method, to global poverty alleviation and economics more broadly. The traditional approach in economics relies on the assumption that people are rational actors and applies economic theory to make deductions.

Conducting RCTs can be more expensive but nowhere near as expensive as a nation- or province-wide roll-out of an untested policy without evidence that ends up failing. Ever-limited resources make it even more crucial to first collect quantitative evidence on a variety of possible interventions. Once evidence is available, a decision can be made to choose which interventions deliver the greatest bang for the proverbial buck available. There are a million good ideas out there. The questions are: What goals do we need/ want to accomplish? What programs have quantitatively proven to aid in that goal? Which ones are the best-buys and worth paying for with a limited budget?

In October, the World Bank issued a report titled ‘Cost-effective approaches to improve global learning’ – that are transferable and heavily rely on evidence-based approaches to ranking programs and interventions based on their cost-benefit. ‘Great buys’ are highly cost-effective interventions supported by strong evidence – like providing parents information on benefits, cost and quality of education. ‘Good buys’ are cost-effective interventions with good evidence – structured yet flexible lesson plans with linked material, teacher monitoring and training, targeted and remedial teaching, reducing travel time to schools, merit-based scholarships, adaptive learning software, early childhood learning.

‘Promising but low-evidence buys’ are interventions that are cost-effective but supported by only a small yet credible evidence base, e.g., teacher accountability and incentives, community involvement in school management. Finally, straight up ‘Bad buys’ are programs for which there is strong evidence available that they do not work – like buying computer hardware alone, cash transfers, infrastructure alone, more teachers (without improved training), free textbooks.

A big item on the PTI’s education agenda for next year is the formulation of the new National Education Policy, or NEP. Work on the NEP has just begun and will pick up pace in 2021. An evidence of the ad-hoc nature of policy formulation is that the Single National Curriculum (SNC) and accompanying books, which should have been informed by the NEP and logically followed it, were developed first.

The Ministry of Federal Education and Professional Training has invited the public to email its contributions to the education policy. While this is an improvement in terms of openness over the process used to develop the SNC, it is still a far cry from the kind of systematic, inclusive and bottom-up consultative processes that are successfully used with increasing frequency around the world. Going by some of the internal deadlines that have been set for developing the policy, there simply will not be enough time for that kind of stakeholder consultation. More likely, it will be formulated in typical top-down fashion, a hurriedly thrown together list of existing programs, and unburdened by the weight of evidence of what does or does not work. The policy formulation process, however well-intentioned, is ad hoc rather than being guided by evidence and an institutionalized policy formulation framework.

The good news is that employing evidence in the development of the new National Education Policy does not require starting entirely from scratch. Over the years, several development sector pilot projects in education have collected evidence to assess the impacts of interventions. While the reliability of those studies may vary, benefiting from their existence is only a matter of reviewing their reports.

However, the first point of resistance to this evidence-based approach to selecting education programs is the local political culture. A review of pet projects that receive support from political quarters reads like the list of ‘Bad buys’ from the World Bank report I mentioned earlier – laptop schemes, blended learning, distance education by TV and radio schools, etc.

RCTs and other means of collecting evidence of efficacy only get in the way of quickly earning positive publicity and headlines. For example, I recently spoke with a head teacher and asked if meal plans in her school would help retain girls to which she responded, ‘Who sees what has gone into one’s stomach? Parents are more interested in tangibles like free school bags, stationery, etc.' Yet, counter to her intuitive judgment, experimental research from elsewhere has reported the opposite. Nevertheless, development plans of provincial governments are filled with programs that a large body of research has classified as 'Bad buys' (in terms of their impact on learning outcomes).

So, what priorities should we be spending our money on? The last year was defined by Covid-19 and was a major factor that kept education in the international headlines. Academically, the year was marred by repeated school closures and extended summer and winter vacations. For most students, these prolonged closures resulted in disengagement from learning with long-term effects. According to some estimates, this has set back their learning by months and is projected to significantly reduce their lifetime incomes.

Our schooling system was failing children even before the pandemic. Results from TIMSS 2019, an internationally comparable test of math and science. placed Pakistani fourth graders second last among the 64 participating countries. A big focus of NEP 2021 should be strengthening foundational numeracy and literacy even if it means watering down other subjects.

A proven solution to learning losses is targeted instruction which blurs grade level boundaries, and groups and teaches students according to their current level of understanding to bring them at grade level. The year 2021 may be the year we see the federal and provincial governments step up and meet this challenge with the resources it demands, and get students back on track. The alternative will be to ignore the problem and act like everything is ‘normal’ and see an even greater than usual number of students fall behind, get frustrated and ultimately drop out of school earlier than they need to. Incidentally, providing targeted instruction also happens to be on the World Bank’s list of ‘Good buys.’

Some priorities are so basic they do not need an RCT. We have repeatedly seen that a lot of interventions fail because of the same bottlenecks – poorly motivated and under-educated teachers and schools lacking the most basic infrastructure. That means making every school a ‘good school’ and a commitment to providing children going to damaged or under-resourced schools decent school buildings, with access to clean water and toilets to allow them to spend hours at school, electricity to run fans to allow them to bear the heat, and desks to work on. Secondary and higher secondary schools in particular need functional and accessible science labs, computer labs and libraries. Providing these essential facilities in schools is low-hanging fruit and is an easy way to raise enrollment rates for boys and girls. In fact, evidence (not yet published) from a study of private schools in Khyber Pakhtunkhwa suggests that improvement in government schools has a positive spillover effect on nearby private schools that feel compelled to lift standards to keep their enrollment.

Differences between how high-end private schools and all other schools weathered school closures exposed the digital divide. The government is developing a distance learning strategy but, like all policies, this one too will only do any good if the words put on paper are backed by necessary investments to expand broadband coverage of schools and homes, access to devices and effective delivery of online and broadcast instruction. Yet, instead of hearing more about how the government plans to accomplish this hard task, we continue to hear about the television and radio school transmissions, whose reach and efficacy is quite low, according to a recent study.

Without giving careful consideration to evidence, cost, resource requirements, value for money and implementation mechanisms, the new policy document will remain a checkbox exercise. A policy document should contain a coherent list of objectives linked to a clean list of unambiguous, quantified baseline levels of achievement indicators and future targets by a set deadline.

Then, select and support programs that are proven to aid those objectives – do what needs to be done, rather than what can be done. Instead of selecting programs that can be implemented quickly (TV school) or will generate positive publicity (laptop scheme) without considering evidence of efficacy, we must deploy the limited resources available on programs that have proven to be effective – the ‘Great’ and ‘Good buys’.