Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Young Lives 2016-17 School Survey: Value-added analysis and school effectiveness


Published on

This slidedeck is from the Young Lives classroom observation sub-study dissemination event held in India on 1 June 2018. The event showcased learnings from the sub-study, and sought to answer questions such as 'where is value added in the classroom?', and 'who is taught by the most effective teachers?'.

A related blog reflecting on this event, written by Rhiannon Moore, is available here:

Published in: Education
  • Be the first to comment

  • Be the first to like this

Young Lives 2016-17 School Survey: Value-added analysis and school effectiveness

  1. 1. @yloxford @younglivesindia Young Lives 2016-17 School Survey Value-added analysis and school effectiveness Caine Rolleston (Institute of Education, UCL) Rhiannon Moore (Young Lives, University of Oxford) @caine_rolleston @rhi_moore
  2. 2. WHY EDUCATIONAL ‘EFFECTIVENESS’? • During lives of YL cohorts large increases in enrolments towards universal access • Much interest in system expansion & reform, dilemmas of access, quality, equity, efficiency • YL survey data showed wide disparities in learning levels between countries suggesting differences of system quality/equity (quality for whom?) • Within country gaps e.g. urban/rural, public/private • Unique opportunity to study educational effectiveness comparatively across 4 developing countries
  3. 3. • Two major themes (1) educational effectiveness (2) equity • Began with question of ‘what makes for an effective system/school’ • Designed value-added school surveys & assessments (repeated measures) • AIM to identify more and less effective schools in context • BUT this evidence is highly contingent, v limited generalisability • i.e. not deterministic, many paths dependent on stage/route • BUT important lessons are possible across systems, though not usually about ‘inputs’ • Four YL countries represent very different approaches to the distribution of educational opportunities, key questions: – Who goes to more effective schools and why? – Do all pupils benefit equally from school effectiveness? SCHOOL EFFECTIVENESS RESEARCH
  4. 4. • Value-added is a measure of student progress over a defined period of time – Aim of ‘controlling’ as much as possible for differences in student outcomes which are outside the control of the school • Progress relative to students in other schools with a similar starting score – Anticipate all students will make some progress over one year. – Value-added measures if students in a particular school make more or less progress than others. • Designed to compare ‘like for like’ • Provides a summary measure of school quality WHAT DO WE MEAN BY ‘VALUE-ADDED’?
  5. 5. • Unconditional value-added estimated using student attainment scores at the beginning and end of a defined period of time – Possible critique – sets lower expectations for schools serving disadvantaged children? • Conditional value-added estimates include student background factors – Recognises that students are not randomly allocated to schools. – Takes into account that it may be harder for students from disadvantaged backgrounds to make progress. CONDITIONAL OR UNCONDITIONAL?
  6. 6. Young Lives 2016-17 School Survey: India
  7. 7. SCHOOL SURVEY RESEARCH DESIGN School effectiveness design: ➢ Student performance in terms of progress (rather than cross-sectional measure) ➢ The teaching and learning processes that affect student progress ➢ The ‘value-added’ of one year of school To do this, we administered: ➢ Cognitive tests at beginning and end of one school year ➢ Background instruments and psychosocial measures to contextualise learning progress
  8. 8. YOUNG LIVES SCHOOL SURVEYS: INDIA India Secondary School Survey (2016-17) • Class 9 students • Different types of school • Progress in Maths and English in Class 9 • Tests at the beginning & end of the school year
  9. 9. SECONDARY SCHOOL SURVEY: SAMPLE Sample design to explore school choice available in each of the 20 Young Lives sites Sample stratified by 4 school types: •State government schools •Tribal/Social Welfare schools •Private Aided schools •Private Unaided schools Number of schools sampled in each site proportional to the total number of schools in that site: Total number of schools in a site Proportion sampled > 80 schools 10% sampled 21-80 20% sampled 8-20 schools 50% sampled <8 schools 100% sampled (exception: less prevalent school types are oversampled)
  10. 10. INDIA SCHOOL SURVEY: SCHOOLS ➢ 205 schools ➢ School type: • 14% Private Aided; • 27% Private Unaided; • 41% State Government; • 18% Tribal/Social Welfare ➢ 64% located in rural areas ➢ 59% have one Class 9 section ➢ Medium of instruction: • 41% Telugu • 38% English • 18% Telugu & English • 2% Urdu
  11. 11. Findings: Variation in school value-added
  12. 12. UNCONDITIONAL VALUE-ADDED - MATHS -100 -50 0 50 100 0 50 100 150 200 School by Value-Added rank Private Aided Private Unaided State Govt TSW • Clear patterns when we look at unconditional value-added – private unaided schools add more value, govt schools more varied
  13. 13. CONDITIONAL VALUE-ADDED - MATHS -100 -50 0 50 100 0 50 100 150 200 School by Value-Added rank Private Aided Private Unaided State Govt TSW • Pattern becomes less clear when look at conditional value-added – govt and TSW schools move up the distribution
  14. 14. GROWING INEQUALITIES – STARTING SCORES AND VA -100 -50 0 50 100 300 400 500 600 700 Wave 1 maths test score (school average) School Fitted line • Schools with higher scores at the start of Class 9 add more value over the school year – suggests inequalities will continue to widen over time
  15. 15. Findings: Which schools add more value?
  16. 16. DIFFERENCES BY SCHOOL MANAGEMENT • Private unaided schools appear to add considerably more value on average, even when we control for the more advantaged background of their students. -20-10 0 102030 Private Aided Private Unaided State Govt Tribal Social Welfare Mean School VA (uncon) Mean School VA (con)
  17. 17. LARGER SCHOOLS ADD MORE VALUE… • Are there too many small schools? Findings suggest that smaller schools (those with fewer sections) add less value than larger schools -5 05 1015 1 section 2 sections 3 or more sections Mean school VA (uncon) Mean school VA (con)
  18. 18. AND THIS IS TRUE ACROSS SCHOOL TYPES • The same pattern is seen across govt and private schools • Those a larger number of sections in Class 9 add more value than those with just one section • Efficiency, competition, location? 0 1020304050 1 section 2 sections 3 or more sections Private Unaided schools only Mean school VA (uncon) Mean school VA (con) -5 05 MeanMathsvalue-added 1 section 2 sections 3 or more sections State Govt schools only Mean school VA (uncon) Mean school VA (con)
  19. 19. MINIMAL URBAN – RURAL GAP IN MATHS • Findings suggest an urban-rural gap in value-added – but this decreases considerably when we control for differences in student background (for maths – gap remains large for English) -5 05 10 Rural Urban Mean school VA (uncon) Mean school VA (con)
  20. 20. Findings: Who attends schools adding more value?
  21. 21. WEALTHIER STUDENTS ATTEND BETTER SCHOOLS -10 0 1020 Q 1 (poorest) Q 2 Q 3 Q 4 Q 5 (least poor) Mean school VA (uncon) Mean school VA (con) • Considerable evidence of ‘sorting’ by student background - students from wealthier households attend schools which add a lot more value
  22. 22. AND SO DO BOYS… • On average boys are attending better schools than girls (although the gap is relatively small) 012345 Female Male Mean school VA (uncon) Mean school VA (con)
  23. 23. AND THOSE WITH MORE EDUCATED MOTHERS • Children with more educated mothers attend schools which add more value than those whose mothers have not been to school 0 10203040 Never been to schoolPrimary Secondary Upper Secondary Higher ed. Mean school VA (uncon) Mean school VA (con)
  24. 24. WE ALSO SEE SORTING WITHIN SCHOOLS ➢ Children from less wealthy households are ‘sorted’ into less effective sections within schools as well. -5 05 1015 Q 1 (poorest) Q 2 Q 3 Q 4 Q 5 (least poor) Mean school VA (uncon) Mean school VA (con) Mean class VA (uncon) Mean class VA (con)
  25. 25. Discussion and implications
  26. 26. SUMMARY OF FINDINGS • Schools in this sample are relatively homogeneous in intake, with larger differences found between schools • Suggests ‘sorting’ of children into schools – characteristic of a school system with extensive range of ‘school choices’ • More advantaged students appear to be ‘sorted’ into more effective schools • This has implications for equality of opportunities during and after secondary school • There are considerable gaps at the start of Class 9, and these widen over time as schools with higher starting schools also add more value • Part of differences between schools and school types comes from differences in student background – but a sizable gap in effectiveness remains when this is controlled for
  27. 27. POLICY IMPLICATIONS & DISCUSSION • Need for govt policy to mitigate negative effects of ‘sorting’ for children from disadvantaged backgrounds • Starting scores are already much lower for these children – need for action much earlier than Class 9 • Also need remedial action to counter lower progress made in Class 9 • Findings indicate that very small schools face particular challenges in effectiveness • There is a real need for policy to address this, given the large number of very small schools • Although on average private schools add more value, our findings show that many government schools are equally or more effective – but the sector is much more heterogeneous • Need to improve consistency of govt school sector by ‘raising the floor’ of achievement and progress through setting minimum standards and increasing quality assurance measures
  28. 28. @yloxford @younglivesindia Young Lives 2017-18 Classroom Observation Sub-study Teacher-student interactions, equity and learning Caine Rolleston (Institute of Education, UCL) Rhiannon Moore & Ana Grijalva (Young Lives, University of Oxford) @caine_rolleston @rhi_moore
  29. 29. WHY CLASSROOM OBSERVATION? • From the school survey, we can identify certain teachers as more or less ‘effective’ • But we don’t know what is happening in those classrooms to explain this • Aim to unlock the ‘black box’ of the classroom • The Classroom Observation study aimed to collect data which can be used to answer RQs such as: • To what extent do teacher-student classroom interactions explain differences in student learning attainment in secondary classrooms? • What in terms of observed interactions in the classroom explains higher and lower effectiveness (value-added)? • What are the characteristics of classroom environments where students learn more? • How do teacher-student interactions vary between different types of schools, and between schools in different localities?
  30. 30. Classroom Observation Study 2017-18
  31. 31. CLASSROOM OBSERVATION STUDY DESIGN • 3 components of the study: • Classroom observation using CLASS • Semi-structured teacher questionnaires • Classroom videos, coded using CLASS • 45 maths/English teachers in 23 schools • In 4 districts (2 in AP, 2 in Telangana) • Teachers purposively sampled using data from 2016-17 school effectiveness survey, based on the following criteria: • Mixture of teachers with high / low / average VA • Different school management types • Urban / rural areas
  32. 32. OBSERVATION USING CLASS METHOD • Classroom Observation was undertaken using the Classroom Assessment Scoring System-Secondary (CLASS-S) method • Designed by Robert Pianta at University of Virginia • CLASS aims to measure teacher-student interactions identified as being important to how students learn • Several studies have found that higher scores on CLASS are positively associated with student academic performance and positive academic attitudes • The CLASS method was designed for use in the USA, but has also been used in many other countries in South America, Europe, Africa, and East Asia • But it has never been used in India before this study
  33. 33. CLASS-S DIMENSIONS • CLASS-S is based on 3 domains of teacher-student interaction, split into 11 dimensions Domain Dimension Emotional Support Positive climate Teacher sensitivity Regard for student perspectives Classroom organisation Behaviour management Productivity Negative climate Instructional Support Instructional learning formats Content understanding Analysis and inquiry Quality of feedback Instructional dialogue Student engagement
  34. 34. OBSERVING TEACHERS USING CLASS • Observers trained and certified in use of the CLASS method • Observers work in pairs to observe each teacher • This ‘double assessment’ increases score validity • Teachers are observed for 4 cycles: • 1 cycle = 30 mins (15 mins observing, 15 mins coding) • So each teacher was observed for 2 lessons • Each teacher is giving a score for each dimension using the CLASS rubric • Scores range between 1-7 • 1-2 is a ‘low score’, 6-7 is a ‘high score’ • These dimension scores are then averaged to give a score for each domain
  35. 35. Classroom Observation Findings
  36. 36. VARIATION IN CLASSROOM PRACTICES • Considerable variation in teacher CLASS scores • Scores are highest in the ‘Classroom Organisation’ domain • Similar pattern seen across all school types, districts and rural / urban areas Subject Emotional Support Classroom Organisation Instructional Support Mean score SD Range Mean score SD Range Mean score SD Range Maths 4.5 0.84 2.5 – 5.75 5.6 0.61 4.33 - 6.33 4.3 0.88 3 - 5.95 English 4.2 0.92 2.42 – 6.33 5.4 0.71 3.75 – 6.75 3.8 1.11 2.05 - 6.5
  37. 37. CLASSIFYING TEACHERS BY CLASS SCORE • Teachers can then be classified by their CLASS score into low / medium / close to high • No teachers scored 6-7 so we have classified any teachers achieving over 4.55 as being ‘close to high scoring’ Maths English Number of teachers Range of scores Number of teachers Range of scores Close to high CLASS score 6 5.25 - 5.75 2 5.92 – 6.33 Medium CLASS score 16 3.33 – 5.08 13 3.17 – 5.17 Low CLASS score 1 2.5-2.5 7 2.42 – 4.17
  38. 38. WHO IS TAUGHT BY HIGH SCORING TEACHERS? • Patterns in the characteristics of students taught by high / average / low ranked teachers • Students from more disadvantaged groups (e.g. SES, parental background) are more likely to be taught by lower ranked teachers 0 20406080 100 Close to High Scoring teacher Medium Scoring teacher Low Scoring teacher Never been to school Primary Secondary Upper Secondary Higher ed.
  39. 39. DO HIGH SCORING TEACHERS ADD MORE VALUE? • There is a positive association between CLASS ranking and teacher contextual value-added score • Stronger relationship for English than for maths -40-20 0 2040 Close to high CLASS score Medium CLASS score Low CLASS score Mean English VA Mean maths VA
  40. 40. CLASS SCORE & VALUE-ADDED • English and maths teachers both score more highly in Classroom Organisation than any other domain • This is true for teachers who have above average VA and below average VA • Classroom Organisation scores also more consistent – less variation 234567 Above average VA Below average VA Emotional Support Classroom Organisation Instructional Support 23456 MeanCLASSscore(maths) Above average VA Below average VA Emotional Support Classroom Organisation Instructional Support
  41. 41. Classroom Observation Video Clips
  42. 42. CLASSROOM OBSERVATION VIDEO: AIMS • In addition to live observation we also filmed the lessons of 6 teachers • Teachers were selected based on high CLASS scores • Also wanted a mixture of maths/English teachers and teachers from government and private schools • These videos were then coded using CLASS • These videos provide some examples of the types of teacher-student interactions taking place in the classrooms where our classroom observation study took place
  43. 43. Discussion
  44. 44. DISCUSSION • Data from the Classroom Observation study provides further evidence that children from disadvantaged backgrounds in India are sorted into less effective schools • Subject to a ‘double disadvantage’ in terms of home background and school quality • High scores in Classroom Organisation domain compared to other domains – positive? • No single route to good teaching – need for structured ways for teachers to improve • Findings suggest that CLASS is predictive of teacher effectiveness in the Indian context