Graveyard of reform ideas

Important decisions for effective education policies should be taken only after a thorough public debate

Graveyard of reform ideas

In my last column, I used the example of a recent reform in teacher education programmes in Pakistan to highlight the need for policy analysis before a particular policy is adopted. Important and resource intensive alternatives are adopted, more often than not, without any analysis or debate.

Pakistan has become a graveyard of reform ideas largely because of lack of a culture of public debate. We have been using ‘local ownership’ and ‘sustainability’ as buzzwords without acknowledging that we can achieve neither without analysis and debate on policy alternatives.

Some scholars of international education policy have used the term ‘travelling reforms’ to talk about the process of policy borrowing from a vast international field of education policy solutions. Notable international development stakeholders, both multilateral and bi-lateral funders of educational development in developing countries, have been funding production of knowledge about solutions to education problems in developing contexts. This knowledge underpins their development assistance strategies and business cases as they go about defining their development assistance agendas.

Notwithstanding the participation of southern researchers in production of knowledge about education reforms in developing countries, the studies are largely framed, designed and led by the researchers located in developed countries. While that should not be seen as necessarily a problem, it becomes so when the knowledge so produced is not sufficiently interrogated in the developing contexts.

The power dynamics between the north and south are such that this, more often than not, doesn’t happen. Many reform ideas, such as child-centered learning, continuous assessment of student achievement, continuous professional development, teaching, standards, use of tests for accountability, have been produced and circulated against the backdrop of north-south power relations. As an illustration, let me describe at some length how Punjab stumbled upon the idea of testing all children at grade level 5 and 8 through an autonomous body, the Punjab Examination Commission (PEC).

Nearly 12 years ago, the government of Punjab replaced what was a system of formal yearly examinations with what it called Continuous Assessment System (CAS). CAS was announced as part of a no pass/no fail policy in Punjab. Prior to this policy, students who did not get a passing score were forced to repeat the class. CAS required the teachers to continuously assess students learning achievements and take steps to improve them. It also provided specific guidelines on how this was to be done. Teachers were never consulted or prepared for this shift. According to some reports, it was during one of their trips to a foreign country that some very senior officials in the Ministry of Education became charmed with the idea of continuous assessment.

Some officials, who were serving in the Punjab’s school education department at the time of enforcement of CAS, told me that it was implemented in an extremely ‘high-handed’ manner by the then government. Since the practices and implications of continuous assessment, which is not a bad idea per se, were not adequately debated, it assumed little ownership amongst the teachers and other stakeholders. They widely perceived it as resulting from the influence of a few individuals within the government. Its implementation elicited sharp reaction from the teachers and head teachers, who as one retired official of the education department told me, "were mad at the government for making them work extra without any visible benefit".

CAS introduced assessment practices for which the teachers were not prepared. A great deal of capacity-building and sensitisation was needed for its smooth implementation for which the professional development institutions were not prepared either. All in all, CAS demanded substantive changes in the culture and language of assessment, without which it was bound to fail.

Soon after the enforcements of CAS, the senior officials who supported CAS were transferred and those who came in their place looked at it as a problem they inherited from their predecessors. Exactly one year after its introduction, the government of Punjab constituted a committee to review its effectiveness and suitability. While apparently, the committee seemed to be set up for merely a review, the cards were already stacked against CAS. The committee recommended for it to be eliminated from the schools of Punjab and be replaced with the summative formal examinations of pupils on completion of grades 5 and 8, as was the case before CAS was enforced in 2002.

The examinations prior to introduction of CAS were simply used to declare the students pass or fail and fit [or not] for promotion to the next class. They were not designed and administered by a central and autonomous examination authority. The student achievement data was not a tool of accountability of teachers, schools, and districts. But by the time this shift occurred, a new discourse about the use of large scale testing (LST) for accountability of schools was also gaining hold globally. The knowledge producers in northern institutions were favouring the use of such terms standards, testing, and accountability. It is in this context that the new system of examinations was to be founded.

So while the CAS was abandoned due to the reaction of the system against it, the system of examinations that replaced it was formed within the framework of a wide-ranging set of education reforms in Punjab, which were supported by the multilateral and bilateral donors. These reforms required rigorous monitoring of the student achievement and the use of this data for accountability purposes. The provisions of agreements that the government of Punjab reached with its multilateral and bilateral donors included reforms in its assessment systems. The international donors played a substantial role in Punjab’s decision to adopt LST of students in grades 5 and 8.

But here it is important to note that several assessment programmes, such as National Education Assessment Project (NEAS) and its provincial counterparts, were already in place when the shift toward LST happened in Punjab. They had similar development objectives. But they were gradually phased out without any public debate on their performance and potential usefulness in supporting education reforms. Regardless of the benefits that may have accrued from LST after it was put in place, there is no evidence of any ex ante cost-benefit analysis or public debate before it was adopted. The justification for LST in Punjab was not derived from a comparative cost/benefit analysis, but on a continued emphasis on testing and providing results to individual students.

Why did the NEAS and its provincial counterparts not suffice to meet the assessment needs of the education system? I have been raising this question to friends and colleagues involved in testing and the answers they supply point toward the peculiar history of the assessment policy in Punjab. The CAS, which is described in the previous section, had already targeted all students in Punjab. Its failure did not take the emphasis away from measuring the performance of all students. Reaching all students with a testing service remained an important goal but it became intertwined with the donors’ emphasis on using testing as a tool for measuring the effectiveness of education reforms.

Arguably, the sample-based assessments, if properly conducted, can also provide valid and reliable evidence about the performance of the education system. They are also relatively less disruptive, without high stakes for students and teachers, and less resource intensive. The results produced by them can also be used to identify and address the problems of teaching and learning. In fact, sample-based assessments can meet nearly all objectives of assessment except providing a result card to each student. The latter can only be done by testing each student.

The benefit of providing a result card to each student, which is what PEC does these days, needed to be weighed against the costs of doing so. Was it worthwhile to use the resources to conduct large-scale tests? Could the same resources be used to better disseminate and make use of the results of sample-based assessment to inform teachers’ professional development and other aspects of improvements in teaching and learning? It may be that the benefits of LST exceeded its costs or it may be that they didn’t. But we do not know because no such analysis was ever conducted.

I caution my readers not to read the above as an argument against PEC or LST. What I have been trying to do in this and the last few articles is to argue for deliberation and public debate before adopting cost-intensive policies and not taking critical positions against particular policies.

The point is that we should not be dazzled into accepting solutions offered by any policy entrepreneur, individuals, think tanks, or donors. All of them are important actors. But important policy decisions should not be taken on the insistence of any particular stakeholder but only after a thorough public debate. Doing so will only enhance the health, longevity, and effectiveness of education policies.

Graveyard of reform ideas