1. Evaluation Evidence on English
Ed i P li iEducation Policies:
What Can be Learnt?What Can be Learnt?
Sandra McNally
University of Surrey and
Centre for Economic Performance, ,
London School of Economics

2. OutlineOutline
• School Resources
• Pedagogy
• Choice and competition• Choice and competition
• School Autonomy
• Some conclusions

3. What works (or not) in schools: CEP research
Policy Sector Project Positive Little or
no effect
Additional comments
Increasing
overall school
Primary Holmlund, McNally,Viarengo (2009)
√
Holmlund et al. find modest effects.
overall school
expenditure Gibbons, McNally,Viarengo (2011)
√
Estimated effects (for urban schools) much larger in
Gibbons et al.
Excellence in
Cities
Secondary Machin, McNally and Meghir (2010) √ Moderate effects but cost-effective; Highest effect for
most able pupils in most disadvantaged schools
Li dLiteracy and
Numeracy
Strategies
Primary Machin and McNally (2008; 2010) √ Moderate effects at very low cost.
Academies Secondary Machin and Wilson (2009)
Machin and Vernoit (2011) √
Machin and Wilson (2009) found no effect in short-term
Machin and Vernoit (2011) find positive effects forMachin and Vernoit (2011)
Machin and Silva (2013)
√ Machin and Vernoit (2011) find positive effects for
schools that have been in programme for some time.
But Machin and Silva (2013) find no effects in the tail
of the distribution
Choice and
competition
Primary Gibbons, Machin, Silva (2006, 2009)
√
Suggestion that small effect might be possible in schools
with greater autonomy
Special
Educational
Primary Keslair, Maurin, McNally (2012) √ No effect (or even a negative effect)
Needs

5. Evidence on the effects of school expenditure
(Gibbons, McNally, Viarengo)
E id f l ff t f h l di hi t f• Evidence for causal effects of school spending on achievement from
LA‐income differentials and Area Cost Adjustments
• £1000 per student school spending (per year) raised achievement by
~0.25 s.d. age 11
• Spending more effective in schools with more ‘disadvantaged’
students

6. Ch i P dChanging Pedagogy:
The Literacy Houry
• National Literacy Strategy introduced
nationwide in September 1998
(so first year of testing in Summer 1999)(so first year of testing in Summer 1999).
• Sets out a framework for teaching: termly
hi bj i fteaching objectives for 5-11 age range;
provides a practical structure of time and
class management for a daily literacy hour.

8. Some results
• Policy seemed to raise performance: About
0.091 of a standard deviation.
• Compare the per pupil costs of the policy
i h h i b fi fl d iwith the economic benefits, as reflected in
predicted labour market earnings.

9. Cost-Benefit Calculations
• Translate test gains to earnings impact using
h d Th di d l fanother data set. The present discounted value of
the cumulative effect of the literacy hour is
b £2 103 d £5 476between £2,103 and £5,476
• The main costs were 14 local centres and literacy
l Th l hconsultants. The total cost per annum was thus
£2.5 million (covering 222,261 pupils in
aggregate) The cost per pupil is £25 52 peraggregate). The cost per pupil is £25.52 per
annum.

10. Choice and Competition:
Rationale (in economic terms)
hi f d d h l• Better matching of students and schools
(based on the tastes and needs of the
/student/family and what the school provides)
• Facilitates competition between schools as a
mechanism to raise standards (i e parentsmechanism to raise standards (i.e. parents
given freedom to choose school ‐ puts
pressure on schools to attract students if theirpressure on schools to attract students if their
funding is linked to student numbers).

11. School choice and other policiesSchool choice and other policies
• Usually school choice is linked to competition.y p
• Can also be linked to school accountability (in
England, the publication of ‘league tables’ of school g , p g
performance).
• Recently, in many countries, school choice has y, y ,
increased as a result of the creation of new (publicly
funded) schools that depart from the traditional
model (characterised by their autonomy).
• Effects of school choice may be different depending
on polices relating to accountability and school
autonomy.

12. Competition choice and attainmentCompetition, choice and attainment
Gibbons, Machin and Silva (2008)
• Pupil Choice: Do children who with more choice show
faster educational progress?faster educational progress?
S h l C titi b t h l D h l th t• School Competition between schools: Do schools that
enrol children who have more choices perform better?
• They study London and surrounding area using pupil
census data for 2002 and 2003 on tests in primary school
at age 7 and age 11: ‘Value Added’

13. 1. Geocode school and pupil home addresses
Pupil homesp
School

14. 2. Calculate median commuting distance (straight line)
Pupil homesp
School
‘Travel area’: circle
at median distance

15. 3 Count how many schools each pupil can reach if they3. Count how many schools each pupil can reach if they
travel no more than the median commute to each school
2
3
2
2
1
= pupil choice index1
0
1
0

16. 4. Calculate school competition index = mean of thep
pupils’ choice index by school attended
2
3
2
2
1
00
1
0
0
0
1
0
0
Non-competitive: =0Competitive: =1.25

17. School Competition index
4 5
5-6
6-7
1-2
2-3
3-4
4-5
0-1

18. The strategy
• Local Authority (LA) boundaries create discontinuities in
school choice:school choice:
– Pupils living near LA boundaries have less choice than pupils
in the centrein the centre
– Schools near LA boundaries face less competition from other
schools than schools in the centre
• Use distance from LA boundary to predict choice (home)• Use distance from LA boundary to predict choice (home)
and competition (school)
(B i f I t t l V i bl St t )(Basis for Instrumental Variable Strategy)

19. FindingsFindings
• Pupils with more choices do no better than pupils with
restricted choicesrestricted choices
• Pupil achievement is marginally higher in schools that
i iappear more competitive
– Correlation attributable to Faith schools only
• Isolated schools close to LA boundaries face less
competition, but perform (insignificantly) better than more
central schools

20. Choice and Competition – Summary
• The evidence from this English study (like many others)
i it i diffi lt t d t t ff t f h i dis it is difficult to detect effects of choice and
competition on achievement, but their may be an effect
on inequalityon inequality.
• See also the large literature on house prices and school• See also the large literature on house prices and school
quality, which shows that parents are willing to pay
significant amounts more to buy houses in catchmentg y
areas with better performing schools. This willingness
to pay shows a significant parental demand for places
i i l h lin particular schools.

21. School Types – Autonomy and Governance
• More recent reforms have increased the variety ofMore recent reforms have increased the variety of
education through different school types.
• Examples are:Examples are:
- Charter schools in the US;Charter schools in the US;
- Free schools in Sweden;
- Academies in England- Academies in England.

22. School Types – Autonomy and Governance
These school types have much more autonomy thanThese school types have much more autonomy than
standard state/public schools.
Their governance structures are also more varied , with
less role from the state and more from the privatep
sector.
They were introduced recently, so there is not a large
body of research, but there is significant research and
policy interest in whether different school types can
enhance performance.

23. School Types – Some Selected Studies
Study Question and Approach Context Findings
1). Abdulkadiroğlu et al
(2011)a
Look at the impact of charter school
attendance on achievement.
Identification is from student
US - charter schools
in Boston.
i) Large, significant gains in test
scores for pupils in charter schools.
ii) Larger effects where assignmentIdentification is from student
assignment lotteries.
ii) Larger effects where assignment
lotteries are binding.
2) Bohlmark and Did the infl of free schools S eden i) Moderate positi e impact of free2). Bohlmark and
Lindahl (2007)b
Did the influx of free schools
enhance academic achievement.
Adopt a differences-in-differences
approach before and after the reforms
Sweden -
municipality panel
data, 1988/9 to
2002/3.
i) Moderate positive impact of free
school growth on municiplaity
academic performance.
ii) Bigger effects for children from
to compare areas with large growth in
free schools to those with less
growth.
highly educated families; no effects
for low education families and
immigrants.
3). Machin and Vernoit
(2011)c
The introduction of Academy schools
to English education. Uses future
Academies as control group.
England - secondary
schools in the 2000s
i) Quality of pupil intake improves
relative to predecessor schools.
ii) Modest positive effect ong p ) p
achievement.
iii) Bigger effects for schools making
bigger autonomy movements.
Notes: a - Abdulkadiroglu, A., J. Angrist, S. Dynarksi, T. Kane and P. Pathak (2011) Accountability and Flexibility in Public Schools: Evidence From Boston's Chaters and
Pilots, Quarterly Journal of Economics. b - Bohlmark, A. and M. Lindahl (2007) The Impact of School Choice on Pupil Achievement, Segregation and Costs: Swedish
Evidence, IZA Discussion Paper. c - Machin, S. and J. Vernoit (2011) Changing School Autonomy: Academy Schools and their Introduction to England's Education, CEE
Discussion Paper.

24. Chapter by Steve Machin and Olmo Silva in book by Paul
Marshall (Google eBook): The Tail: How England's schoolsMarshall (Google eBook): The Tail: How England s schools
fail one child in five ? and what can be done
• Machin and Silva (2013): School Structure,( )
School Autonomy and the Tail
• Look at the effects of ‘school academies’ acrossLook at the effects of school academies across
the distribution of pupil prior attainment.
Outcome is exam at age 16 (GCSE).Outcome is exam at age 16 (GCSE).
• Find no beneficial effects of school academies in
the tail of the distribution (all benefits are furtherthe tail of the distribution (all benefits are further
up the distribution)

25. The Effect of Academy Conversion on Pupils of Different Abilities
(Machin and Silva, 2012)

26. The Effect of Academy Conversion on Pupils of Different Abilities
(Machin and Silva, 2012)

27. School Types – Summary
• Early evidence suggests that in some circumstances,
there may be beneficial effects from school types that
offer more autonomy and more diverse governanceoffer more autonomy and more diverse governance
structures.
• But it is early days and evaluations are so far small
scale and so the generalisability of the findings isscale and so the generalisability of the findings is
somewhat limited.

28. ConclusionConclusion
• Resources matter! There is justification for protecting• Resources matter! There is justification for protecting
education budgets
• Interventions to affect pedagogy can be very costInterventions to affect pedagogy can be very cost
effective (important to evaluate properly in advance)
• Choice and competition have been a blunt instrument toChoice and competition have been a blunt instrument to
increase achievement – but this could change in an
environment where schools have more autonomy.
• Evidence to date suggests more autonomy could have
good effects – but not necessarily for ‘hard to reach’
il D ’t t thi t l ti i ltpupils. Don’t expect this to revolutionise results.