DiscussionJishnu Das(DECRG)Science of Delivery in EducationThis discussion is based on “The Good news from Pakistan”, by Sir Michael Barber and hispresentation at The World Bank on 6th June 2013. The material for the discussion is partlyfrom the LEAPS project with Tahir Andrabi (Pomona College) and Asim Khwaja (Harvard). Ialso thank the South Asia Education Team, and in particular, Dhushyanth Raju, for theirvaluable input.
Everyone agrees that somethingmust be done Children face “double-burden” of low schooling attainment andvery low learning Primary Enrollments increased in recent years But test-scores remain very low (1sd below OECD averages) Teachers big part of the solution Solving problems of high teacher absence rates (15-30%), poor time-on-task, low attention can significantlyimprove scores Poor infrastructure may be a contributing factor
What I agree with Managing teachers is responsibility of managers, not parents School Principals in private schools Schools Principals and the educational administration inpublic schools Contrast: Not the job of parents to ensure that teachers cometo work. No private firm forces customers to ensure that staffcome to work—that’s the work of management Quote: “Today you ask us why we don’t get teachers to come.The NGO says that we should file an RTI, meet the districtofficer, hold up traffic and force the teachers to come. We haveto do this for the schools, for the electricity, for the doctors, forthe roads, for the garbage and for anything at all. You tell me—when should I work in my fields? Do you see that shop sellingPepsi? I go there and buy the Pepsi. If it’s not there, I don’thave to do anything. Yet, it’s always there.”
What I also agree with Some states have lost (or maybe never had) theability to govern frontline workers Lant Pritchett calls this “The Flailing State” This is hard and has to be learnt We don’t know what it takes, but know it can bedoneSource: Pritchett, Lant. 2009. “Is India a Flailing State: Detours on the Four Lane Highway to Modernization.”, HKS FacultyResearch WP, RWP09-013
An Indian Example: Bihar Re-election platform of better service delivery plus Critical coalition building and politics to obtain the legitimacyto use sticks and carrots to govern That’s what good politicians doOne of India’s worstgoverned states: Securityissues, low growth andunchecked corruption.Absence rates fordoctors/nurses 70%, forteachers 40%Nitish Kumarelected CM200320052010Complete turn-around: Greatersecurity, high growth.Absence rates fordoctors/nurses30%, for teachers28%Source: Muralidharan, Karthik, Jishnu Das, Alaka Holla, Michael Kremer and Aakash Mohpal, 2013. “The Fiscal Cost of WeakGovernance: Evidence from Primary Education in India.” In Process.
An Indian Example High benefit-cost ratios of top-down monitoring Increasing inspection in last 3 months by 10 p.p. Costs Rs.450 million Saves Rs.4,048 million in reduced teacher absence To produce same effective student-teacher ratio through extrateacher hiring costs Rs.5,204 million This is a longitudinal, nationwide study Gathering evidence that top-down monitoring when there is politicalbackup works in improving services Theory: James Wilson, Paul Grout’s work on Bureaucracies Changes at the top can have large cascading effects at the bottomSource: Muralidharan, Karthik, Jishnu Das, Alaka Holla, Michael Kremer and Aakash Mohpal, 2013. “The Fiscal Cost of Weak Governance: Evidencefrom Primary Education in India.” In Process.Wilson, James Q. 1991.“What Governments Do, and Why They Do it.” Basic Books Classics.Grout, Paul . 2013. “Collapsing Morale in Bureaucratic Environments.” University of Bristol
The Service Delivery Problematic: What doesdeliverology contribute?Problem #1: No Political Will to improve education outcomes: Top-down monitoring can failbadly (Example: Banerjee and others: Nurses break time-stamp machines when wages aretied to attendance). Building political will for better services is the holy grail.Deliverology: Could be that deliverology helps build that political willNot claimed: Political will a pre-condition (Pakistan, but also Malaysia and Sierra Leone)Problem #2: Political Will, but inability to govern the frontlineDeliverology: Could be that deliverology provides the governabilityClaimed: This would be an important step forward in these situationsProblem #3: Political Will and ability to governDeliverology: Deliverology is one among several paths that the top manager chooses, butany number of methods could workSource: Banerjee, Abhijit, Rachel Glennerster and Esther Duflo. 2008. “Putting a Band-Aid on a Corpse: Incentives for Nurses in the Indian Public HealthCare System. Journal of the European Economic Association
Reported resultsBetter infrastructureGreaterenrollment andlearningBetween August 2011 and January 2013, Sir Michael’s and the Punjabeducation leaders’ efforts achieved the following results (pp. 12-29):Approaching one and a half million extra children enrolled in school.Facilities with functioning electricity, drinking water, toilet and boundarywalls increased from 69 per cent to 91 per cent.Student attendance increased from 83 per cent to 92 per cent.Teacher attendance increased from 81 per cent to 91 per cent.81,000 new teachers hired on merit.Simple, easy-to-use lesson plans for every teacher and new textbooks forevery studentGrowing evidence that learning outcomes are rising too; whereas twoyears ago Punjab-India and Punjab-Pakistan were roughly on a par, nowPunjab-Pakistan is out in front.A voucher scheme which enables over 140,000 out-of- school children ofpoor families to attend private schools; this is the fastest growing voucherschemes in the world.New ProgramsSource: Barber, Sir Michael. 2013. “The Good News from Pakistan.” Content accessed fromhttp://www.reform.co.uk/content/20419/research/education/the_good_news_from_pakistan
What can the data tell us? Problems of counterfactuals (program is province-wide) Researchers very good at dealing with this using innovative methods More seriously, critical lack of data Enrollment: Consistent government data till 2010/11, subsequent rounds notreleased Nielsen household survey 2011-2013, commissioned by DFID Data not publicly available, not compared to other data sources Government data through annual census of schools Monitoring data may have problems; covers only public schools LEAPS dataset, selected villages in 3 districts: 2003-2010 School Inputs: Projects own monitoring database Not publicly available nor triangulated with other data sources LEAPS dataset, selected villages in 3 districts: 2003-2010 Learning: No consistent, comparable data over time Annual Survey of Education Report (ASER): Volunteer-army model, widecoverage, very short test, home-based tests. Good for overall picture in a year, but not for year-on-year changes LEAPS dataset: Consistent, comparable data for 2003-2010 for 3 districts At this point, what can the data tell us?
Enrollment: Household Surveys50607080901995 2000 2005 2010 2015year1All Children Rural OnlyAll and Rural Enrollment: Punjab, PakistanEnrollment rate (primary) 2003 2004 2005 2006 2010LEAPS Villages 54 61 68 69 71Government HH survey54(2002) 66 64 70 70Complete stagnation fromverified, single datasourceDeliverology begins:Data source changes;15 p.p jump from 70 to85% questionableAge of Deliverology: Smallincreases relative toprevious experience; slowdown between 2011 and2012. 85% in 2011, 86% in2012, 86.8% in 2013. Nostandard-errors in report.Age of largeincreases, fromverified, single datasource. Identical inLEAPS dataLarge WB programin Punjab New release ofgovernment data shows74% NER
Enrollment: TriangulateGovernment Monitoring Data•Annual School census based on the monitoring division: Same data used to show input improvements (butnot enrollment in report)•Decreasing number of schools due to consolidation and merger policy•No change in public school enrollment at all: Insignificant decline of 4000 children•Note: Student-teacher ratio declined: Good if it increased learning, bad if it only increased costs•Parallel work from India shows very small impacts of decreasing STR on learning, partly due to increasedabsence!•Triangulation does not provide consistent evidence. Effect of deliverology could have been positive, nothingor negative depending on counterfactual; in all likelihood effects small in either directionNielsen Household Survey: Enrollment increased, share of public schoolsincreased. This implies that public enrollment numbers must have increasedVariable 2009/10 2010/11 2011/12 2012/13Schools 61,474 60,494 59,461 57,425FunctionalSchools59,029 58,187 56,867 54,392Teachers 165,086 161,947 158,386 165,600Total Enrollment 10,644,408 10,619,575 10,632,546 10,640,159
Enrollment: Triangulate (Slide added post-discussion) Post discussion, I became aware of the release of the Government’s householdsurvey, the PSLM for 2011-2012 According to PSLM 2011-2012 The Net Enrollment Rate (NER) for children 6-10 in primary education was74% in 2011-2012 Much of the increase comes from an increase in enrollment rates in privateschools 42% in 2007/08; 37% in 2010/2011 and 44% in 2011/2012 This much smaller increase is completely consistent with the constantgovernment enrollment numbers in the monitoring data Severe discrepancy between PSLM and Nielsen for 2011/2012: Againemphasizes need to understand how data were collected!Source: PLSM 2011/2012: Accessed from http://www.pbs.gov.pk/sites/default/files/pslm/publications/pslm2011-12/complete_report_pslm11_12.pdf
InputsUsing Government project monitoring data, there have been improvements in inputsTriangulation: ASER, same storyAvailability forfunctionalschools2009/10 2010/11 2011/12 2012/13Electricity 61.8 66.2 69.5 78.3Drinking Water 87.3 88.9 92.4 97.6Boundary wall 82.2 84.7 87.9 89.7Toilets 80.3 84.8 89.6 95.4All Four 49.7 54.5 60.3 71.5
InputsContinues long-standing upwards trend, dating back to 2003, data based on LEAPS villages2003 2004 2005 2006 2010Number of permanent classrooms 3.89 4.05 3.81 3.41 5.67Number of semi-permanentclassrooms 0.55 0.47 0.99 1.19 0.04Number of staff rooms 0.22 0.29 0.20 0.32 0.22Number of boys toilets 0.71 0.77 0.53 0.58 1.25Number of girls toilets 0.41 0.42 0.89 0.66 1.23
LearningChanges in Grade 4 test-scores, 2004-2011. Using Equated tests inLEAPS villages2004 2006 2011Math -0.29 -0.21 -0.44English -0.55 -0.22 -0.12Urdu -0.41 -0.20 -0.34Math scores worsened, English improved, Urdu remained the same.There are currently no data from after 2011 in LEAPS, turn to ASER in 2011and 2012
LearningASER data from Punjab show large improvements in single year: may be as large as 1 standard-deviation equivalents: No comparable results from any other countries!Note: Identical results from province of KP, without deliverology. Mostly likely hypothesis: surveychanges, rather than real learning. Needs to be independently verified!
Innovative new programs Vouchers and Teacher merit hiring are extensions of pre-existingprograms and discussions Punjab Education Foundation was set up around 2007/08 andhas been leading innovative methods for private schoolfinancing Teacher hiring has been under discussion for a long time. Merit hiring may or may not be a good thing; needs to beevaluated Why? Because teacher value-added has a very smallcorrelation with observed teacher attributes
Does deliverology deliver? Initially very excited about the good news from Pakistan Perhaps this new method could improve governance when there waspolitical will Reassessment Very little support for enrollment increases. Story does nottriangulate, trend not supportive of break Inputs have improved, and continue a trend started back in 2003 withPadha-Likha Punjab (similar to Education for All) No consistent data on learning: Improvements in ASER very large toanything we have ever seen, and identical in KP and Punjab;potentially survey changes Reassessment: No consistent story of large improvements that areoff-trend. Deliverology one of many potential methods that are beingtried, each with (potentially) some effects Reassessment: Punjab made huge strides between 2003 and2010, both in enrollment and learning
Next Steps Biggest issue is lack of consistent, credible data to evaluate andunderstand what deliverology does Key components currently missing are legislated dataindependence, regular public scrutiny of improvements, third-party evaluations Contrast with evidence from several rigorous randomizedcontrol evaluations in Pakistan, that are now entering thescale-up phase The lack of data in support is surprising given the emphasis inthe method on data