02.21.2013 - Petra Todd

271 views

Published on

Aligning Learning Incentives of Students and Teachers

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
271
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

02.21.2013 - Petra Todd

  1. 1. Aligning Learning Incentives of Students andTeachers: Results from a Social Experiment in Mexican High Schools Jere Behrman, Susan Parker, Petra E, Todd, Kenneth I. Wolpin
  2. 2. OverviewStudy the effects of a performance incentive program aimed atimproving mathematics knowledge in Mexican high schools.
  3. 3. OverviewStudy the effects of a performance incentive program aimed atimproving mathematics knowledge in Mexican high schools. Mexico ranked last of 34 OECD countries in the 2009 rankings of PISA test sores in mathematics.
  4. 4. OverviewStudy the effects of a performance incentive program aimed atimproving mathematics knowledge in Mexican high schools. Mexico ranked last of 34 OECD countries in the 2009 rankings of PISA test sores in mathematics. Less than 10 % of students score at or above the Proficient level on the 2008 national 9th grade Mathematics test and over 50% score at the Pre-Basic level.
  5. 5. OverviewStudy the effects of a performance incentive program aimed atimproving mathematics knowledge in Mexican high schools. Mexico ranked last of 34 OECD countries in the 2009 rankings of PISA test sores in mathematics. Less than 10 % of students score at or above the Proficient level on the 2008 national 9th grade Mathematics test and over 50% score at the Pre-Basic level.ALI program designed to promote mathematics achievementthrough monetary incentives for performance on curriculum-basedtests.
  6. 6. Empirical Literature on School Performance Incentives1. Teacher Incentives: Glewwe et. Al. (2003) in Kenya,Springer et. al. (2010) in Tennessee, Muralidharan andSundararaman (2011) in India.
  7. 7. Empirical Literature on School Performance Incentives1. Teacher Incentives: Glewwe et. Al. (2003) in Kenya,Springer et. al. (2010) in Tennessee, Muralidharan andSundararaman (2011) in India.2. Student Incentives: Angrist and Lavy (2009) in Israel,Kremer et. al. (2009) in Kenya, Fryer (2010) in Chicago,Dallas, New York and Washington D.C., Levitt et. al.(2010) in a Chicago suburb.
  8. 8. Empirical Literature on School Performance Incentives1. Teacher Incentives: Glewwe et. Al. (2003) in Kenya,Springer et. al. (2010) in Tennessee, Muralidharan andSundararaman (2011) in India.2. Student Incentives: Angrist and Lavy (2009) in Israel,Kremer et. al. (2009) in Kenya, Fryer (2010) in Chicago,Dallas, New York and Washington D.C., Levitt et. al.(2010) in a Chicago suburb.3. Student and Teacher Incentives: Jackson (2010) inDallas.
  9. 9. Empirical Literature on School Performance Incentives 1. Teacher Incentives: Glewwe et. Al. (2003) in Kenya, Springer et. al. (2010) in Tennessee, Muralidharan and Sundararaman (2011) in India. 2. Student Incentives: Angrist and Lavy (2009) in Israel, Kremer et. al. (2009) in Kenya, Fryer (2010) in Chicago, Dallas, New York and Washington D.C., Levitt et. al. (2010) in a Chicago suburb. 3. Student and Teacher Incentives: Jackson (2010) in Dallas. Effect sizes on test scores generally .10-.25 sd.
  10. 10. Some Facts About Education in MexicoSchool Completion Rates (1996 1st grade entry cohort): 87% complete 6th grade – 82% enter 7th grade 65% complete 9th grade – 62% enter 10th grade 47% complete 10th grade 39% complete 11th grade 38% complete 12th grade 28% enter college
  11. 11. Some Facts About Education in MexicoSchool Completion Rates (1996 1st grade entry cohort): 87% complete 6th grade – 82% enter 7th grade 65% complete 9th grade – 62% enter 10th grade 47% complete 10th grade 39% complete 11th grade 38% complete 12th grade 28% enter collegeFederal high schools (1,000 schools): Per-pupil expenditure - 21,000 pesos Average teacher monthly salary - 20,000 pesos Pct. of high school students attending - 25% Average annual tuition – 1,200 pesos
  12. 12. ALI ProgramPilot program period: AY 2008/09, 2009/10 and 2010/11.Program participants: all students in 88 Federal high schools in Mexico – 24 of 31 states.Overall design: Random assignment to three treatment groups of 20 schools each and 28 control schools
  13. 13. TreatmentsTreatments: T1 (20 schools) Payment provided to students related to their individual performance.
  14. 14. TreatmentsTreatments: T1 (20 schools) Payment provided to students related to their individual performance. T2 (20 schools) Payment provided to mathematics teachers based on the performance of the students in their classes.
  15. 15. TreatmentsT3 (20 schools) 1. Payment to students based on individualperformance and on performance of classmates.
  16. 16. TreatmentsT3 (20 schools) 1. Payment to students based on individualperformance and on performance of classmates. 2. Payment to mathematics teachers based onperformance of students in their classes and of studentsin all other mathematics classes.
  17. 17. TreatmentsT3 (20 schools) 1. Payment to students based on individualperformance and on performance of classmates. 2. Payment to mathematics teachers based onperformance of students in their classes and of studentsin all other mathematics classes. 3. Payment to non-mathematics teachers based onperformance of students in all mathematics classes.
  18. 18. TreatmentsT3 (20 schools) 1. Payment to students based on individualperformance and on performance of classmates. 2. Payment to mathematics teachers based onperformance of students in their classes and of studentsin all other mathematics classes. 3. Payment to non-mathematics teachers based onperformance of students in all mathematics classes. 4. Payment to principals and other administratorsbased on performance of students in all mathematicsclasses.
  19. 19. RandomizationSchool-based block randomization design.Nine blocks characterized by school size and graduationrates prior to the initiation of the program.Within each block, schools are allocated randomly to thethree treatment groups and the control group.
  20. 20. ENLACE scores are reported both standardized (mean=500, sd=100) andin four categories.National figures include students who never attended high school.
  21. 21. ALI TestsThe tests are based on the standardized curriculum foreach grade and were produced especially for this projectby a Mexican educational testing service (CENEVAL).
  22. 22. ALI TestsThe tests are based on the standardized curriculum foreach grade and were produced especially for this projectby a Mexican educational testing service (CENEVAL).Grade 10: Algebra, Geometry and Trigonometry (classhours - 4hrs/wk) – 2.5 hour ALI examination
  23. 23. ALI TestsThe tests are based on the standardized curriculum foreach grade and were produced especially for this projectby a Mexican educational testing service (CENEVAL).Grade 10: Algebra, Geometry and Trigonometry (classhours - 4hrs/wk) – 2.5 hour ALI examinationGrade 11: Analytical Geometry, Calculus (class hours -4hrs/wk) – 2.5 hour ALI examination
  24. 24. ALI TestsThe tests are based on the standardized curriculum foreach grade and were produced especially for this projectby a Mexican educational testing service (CENEVAL).Grade 10: Algebra, Geometry and Trigonometry (classhours - 4hrs/wk) – 2.5 hour ALI examinationGrade 11: Analytical Geometry, Calculus (class hours -4hrs/wk) – 2.5 hour ALI examinationGrade 12: Probability and Statistics, Applied Statistics(class hours - 5hrs/wk) – 2.5 hour examination on 12thgrade material, 1.25 hours each on 10th and 11th gradematerial.
  25. 25. Incentive Schedules : Teachers (T2, T3)
  26. 26. Incentive SchedulesIncentive schedules are based on the categorical scoreson an initial test (grades 10 and 11) and on the end-of-year ALI test (grades 10,11,12).
  27. 27. Incentive SchedulesIncentive schedules are based on the categorical scoreson an initial test (grades 10 and 11) and on the end-of-year ALI test (grades 10,11,12). The initial test score for the tenth grade is the national 9th year mathematics ENLACE (curriculum- based test).
  28. 28. Incentive SchedulesIncentive schedules are based on the categorical scoreson an initial test (grades 10 and 11) and on the end-of-year ALI test (grades 10,11,12). The initial test score for the tenth grade is the national 9th year mathematics ENLACE (curriculum- based test). The initial test score for the eleventh grade is the 10th grade ALI curriculum test (except in first year – 9th grade ENLACE).
  29. 29. Incentive SchedulesThe 10th grade test score cutoffs mimic the controlgroup’s distribution of categorical scores on the 9th grademathematics ENLACE.
  30. 30. Incentive SchedulesThe 10th grade test score cutoffs mimic the controlgroup’s distribution of categorical scores on the 9th grademathematics ENLACE.The 11th grade test score cutoffs mimic the controlgroup’s distribution on the 9th grade ENLACE in year 1and on the 10th grade ALI test in years 2 and 3.
  31. 31. Incentive SchedulesThe 10th grade test score cutoffs mimic the controlgroup’s distribution of categorical scores on the 9th grademathematics ENLACE.The 11th grade test score cutoffs mimic the controlgroup’s distribution on the 9th grade ENLACE in year 1and on the 10th grade ALI test in years 2 and 3.The 12th grade test score cutoffs mimic the controlgroup’s distribution on the 12th grade mathematicsENLACE.
  32. 32. Incentive Schedules: Students (T1, T3)
  33. 33. Incentive Schedules: Students (T1, T3)
  34. 34. Incentive Schedules : Students (T1, T3)
  35. 35. Incentive Schedules : Students (T1, T3)
  36. 36. Incentive Schedules : Students (T1, T3)
  37. 37. Incentive Schedules : Students (T1, T3)
  38. 38. Incentive Schedules : Teachers (T2, T3)
  39. 39. Incentive Schedules : Teachers (T2, T3)
  40. 40. Incentive Schedules : Teachers (T2, T3)
  41. 41. Incentive Schedules: Performance of Others (T3)In addition to the incentives based on own performance, Students receive an additional payment of one percent of the total amount received by all of the students in their class.
  42. 42. Incentive Schedules: Performance of Others (T3)In addition to the incentives based on own performance, Students receive an additional payment of one percent of the total amount received by all of the students in their class. FTE mathematics teachers receives an additional payment of 25 percent of the average (FTE) amount earned by the other mathematics teachers.
  43. 43. Incentive Schedules: Performance of Others (T3)In addition to the incentives based on own performance, Students receive an additional payment of one percent of the total amount received by all of the students in their class. FTE mathematics teachers receives an additional payment of 25 percent of the average (FTE) amount earned by the other mathematics teachers. A FTE non-mathematics teacher receives a payment of 25 percent of the average (FTE) amount earned by the mathematics teachers.
  44. 44. Incentive Schedules: Performance of Others (T3)In addition to the incentives based on own performance, Students receive an additional payment of one percent of the total amount received by all of the students in their class. FTE mathematics teachers receives an additional payment of 25 percent of the average (FTE) amount earned by the other mathematics teachers. A FTE non-mathematics teacher receives a payment of 25 percent of the average (FTE) amount earned by the mathematics teachers. The principal of the school receives a payment of 50 percent of the average (FTE) amount earned by the mathematics teachers.
  45. 45. AttritionThere are existing incentive programs that pay students for attendanceand the bonus from the ALI program is uncertain.
  46. 46. Testing Protocol One external monitor per classroom – one overall external supervisor in school.
  47. 47. Testing Protocol One external monitor per classroom – one overall external supervisor in school. Teachers not present during test administration.
  48. 48. Testing Protocol One external monitor per classroom – one overall external supervisor in school. Teachers not present during test administration. Test answer sheets and test booklets collected by monitors at the end of the exam and returned to testing agency for scoring.
  49. 49. Testing Protocol One external monitor per classroom – one overall external supervisor in school. Teachers not present during test administration. Test answer sheets and test booklets collected by monitors at the end of the exam and returned to testing agency for scoring. Despite these measures, we found evidence that led to a suspicion of student cheating.  In some treatment schools, students and teachers received unusually high levels of incentive payments.  Some answer sheets of students within the same classroom exhibited strings of matching correct and incorrect answers.
  50. 50. Analysis of Student CopyingAnalysis performed by George Wesolowsky (professor emeritus,McMaster University) – uses method described in his J. of AppliedStatistics (2000) article.1. Statistical model determining probability that student i answers multiplechoice question j incorrectly Incorporates a parametric function of the “difficulty” of the question and the “ability” of the student.2. Determine for every pair of students and for each question, the probabilitythat the two students will have the same answer (assume, e.g., that all wronganswers are equally likely).3. The probability distribution of the number of matches is a compoundbinomial; approximated as normal.4. Choose a critical value for the number of observed matches. Reject the nullof no copying if the number of matches exceeds the critical value. A Bonferronicorrection is used with a critical value such that the probability is one that atleast one pair of students is falsely accused.
  51. 51. A Caveat: Lack of Test-Taking Effort by Control StudentsAssumption 1: a. test-taking effort of T1 students no less than that of T3 students b. T1 effect is zero in all yearsLower Bound Estimate of Treatment Effect in Year 3:a T3: 31.1 standardized points for 10th grade 16.9 for 11th grade 34.0 for 12th gradea. Adjusted for copying.
  52. 52. A Caveat: Lack of Test-Taking Effort by Control StudentsAssumption 2: a. test-taking effort of C students same in all years. b. T1 effect is zero in year one onlyLower Bound Estimate of Treatment Effect in Year 3a: T3: 46.5 standardized points for 10th grade 28.5 for 11th grade 47.1 for 12th grade T1: 15.4 standardized points for 10th grade 11.6 for 11th grade 13.1 for 12th gradea. Uses treatment effects adjusted for copying.
  53. 53. Payment Outcomes
  54. 54. Payment Outcomes
  55. 55. Payment Outcomes
  56. 56. Conclusions• Evaluated effects of student and teacher incentive programs, in isolation and combined• Student incentives alone increased mathematics test scores by 0.2-0.3 std. dev. (adjusted estimates)• Teacher incentives alone had no effect• Combined student and teacher incentives increased scores by 0.3-0.6 std. dev.• Positive impacts over entire baseline test score distribution, with slightly larger impacts at higher quantiles.
  57. 57. TESTIMONIOS ALUMNOS Sulem Hernández González Alfajayucan, Hidalgo “Estudio para ser Técnico en Informática y con el dinero que me gané, pude comprar mi computadora”
  58. 58. TESTIMONIOS ALUMNOS Sulem Hernández González Erika del Carmen Olán Magaña Alfajayucan, Hidalgo Comalcalco, Tabasco “Estudio para ser Técnico en Informáticaque me dieron en el “Con lo y con el dinero que programagané, pude comprar mi me apoyé a mi familia para computadora” construir los cimientos de mi casa”
  59. 59. TESTIMONIOS ALUMNOS Cristian Pérez Pérez Santiago Papasquiaro,Guanajuato “Con lo que me gané compré una vaca y hace poco tuvo un becerro. Con la venta de la leche que da la vaca, me ayudo para mis útiles de la escuela”

×