Estimating the Effect of Leaders onPublic Sector Productivity: The Case        of School Principals    Eric Hanushek, Stev...
Questions• Does the quality of leadership explain a substantial share  of the variation in organization outcomes?• Does th...
Methodological challenges• Difficult to separate contributions of  leaders from other factors  – Observed characteristics ...
Growing Literature on Principals• Early work focused on observed  characteristics  – Clark, Martorell and Rockoff also con...
Direct estimation of Variance in    Principal Effectiveness• Builds on Rivkin, Hanushek and Kain  (2005) research on teach...
Our Approaches• Use value-added model with school fixed  effects to estimate principal fixed effects  – Compute variance o...
• Examine relationship between estimates  of principal quality and changes in teacher  quality during a principal’s tenure...
UTD Texas Schools Project• Stacked panels of students and staff• Annual testing• Student demographic characteristics  – Di...
Estimation of Variation in Principal Quality-Broad Issues• Non-random selection of principals and students  into schools c...
First empirical approach• Principal by spell fixed effects based  on first three years at a school  – Regress math score o...
Alternative Approach• Ignore issue of tenure and use all  spells• Control for school fixed effects• Potential bias if ther...
Test Measurement Issues• Random error inflates estimates of  variation in principal quality  – Use Bayesian shrinkage esti...
• Tests focus on basic skills, so initial  achievement differences may  influence translation of principal quality  into t...
Table 3. Distribution of Principal by Spell Fixed           Effects by Low Income ShareShare      Standard    10th    25th...
Sensitivity checks• Shrink and reweight• School fixed effects included in  specification estimated over sample of all  sch...
Alternative, Test-Measurement Error Adjusted        Estimates of the Variance in Principal                    Effectivenes...
Deficiencies of fixed effects • Unobservables, even if orthogonal to   principal quality, inflate variance estimate   • Th...
Principal turnover based        variance estimates• Derive variance estimates from  relationship between year-to-year chan...
• Variance in principal effectiveness equals  additional year-to-year variation in  transition years over non-transition y...
• If it is actually caused by principal quality,  differences in school quality should be  larger in non-adjacent years du...
specificationprincipal quality (θ) in cohort y,quality of other school factors not under the controlof the principal (γ)
Taking ExpectationE (∆ Asy − ∆ Asy  ) 2 = 2(σ θ2s − σ θ2yθ y  ) + 2(σ θ2s − σ θ2yθ y  ) + E (es )                         ...
• Regress squared year-to-year difference  in school average test score gains on  indicator for principal change• Assumpti...
Within School Covariance• Covariance between principal quality in  adjacent years with same principal in both  years equal...
Sensitivity checks• Add squared differences in demographic  characteristics in some specifications• Use non-adjacent years...
Results for entire sampleTiming of Comparison               Adjacent year     one year inbetweenStudent Demographicand Mob...
Estimated Standard Deviation     by Poverty QuartileQuartile   Lowest   2nd     3rd      highestAdjacent   0.029    0.037 ...
Principal quality and teacher             turnover• Principal may have limited control over  entrants  – Job security an i...
Figure 1. Teacher Transitions by PrincipalEffectiveness and School Poverty Rate             Lowest Quartile Disadvantaged ...
Estimation• Argument is that better principals are  more likely to “dismiss” least effective  teachers• Data does not link...
Campus by year fixed effect            regressions• Regress share of teachers that exit grade g in  school s in year y on ...
Table 8. Coefficients on Principal Quality Quartile-Grade AverageValue-Added Interactions Using First Three Years Sample b...
Principal Transitions and Value                Added• Transitions categorized by new role and destination• New role   – Pr...
Probability Principal Remains in Same Position following 3rd  Year in a School, by Quartile of Estimated Quality and      ...
Probability Principal with Fewer than 25 Years of ExperienceBecomes Principal in Different School following 3rd Year in aS...
Future Work• Estimate very flexible model with school  by year fixed effects• Use these estimates to examine whether  vari...
Summary• Purposeful sorting and unobserved factors  complicate estimates of leadership quality  distribution• We find subs...
• Details  – Turnover based estimates ignore any between school    variation in principal quality  – Find a higher quality...
Principal turnover• Least effective principals least likely to remain in  a school   – Often transition to other schools, ...
INEE. Ponencia Profesor Rivkin. Universidad Illinois. Estimating the Effect of Leaders on Public Sector Productivity: The ...
Upcoming SlideShare
Loading in …5
×

INEE. Ponencia Profesor Rivkin. Universidad Illinois. Estimating the Effect of Leaders on Public Sector Productivity: The Case of School Principals

2,033 views

Published on

Aunque mucho se ha escrito sobre la importancia del liderazgo en la determinación del éxito de la organización, hay poca evidencia cuantitativa debido a la dificultad de separar el impacto de los líderes de otros componentes de la organización - particularmente en el sector público. Las escuelas proporcionan un entorno especialmente rico para el estudio del impacto de la gestión del sector público, no sólo por la hipótesis de la importancia del liderazgo, sino también debido a los abundantes datos de rendimiento que proporcionan información sobre los resultados institucionales. Estimaciones basadas en los resultados del valor añadido del director en el rendimiento del estudiante revelan una variación significativa en la calidad del director que parece ser mayor para las escuelas más pobres. Valoraciones alternativas del límite inferior basadas en la estimación directa de la varianza producen estimaciones más pequeñas de la variación de la productividad del director, no obstante, son igualmente importantes, sobre todo para las escuelas más pobres. Los patrones de las salidas de los profesores por decisión del director validan la noción de que la gestión del personal docente es un canal importante de influencia del director. Por último, echando un vistazo a la movilidad del director por razones de calidad, se revela poca evidencia sistemática de que los líderes más eficaces tienen una mayor probabilidad de dejar las escuelas más pobres.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,033
On SlideShare
0
From Embeds
0
Number of Embeds
744
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Substantial variation in teacher quality as measured by the contribution to student achievement. Observable characteristics of teachers explain little of the variation in value added to learning Salary and other factors affect teacher transition probabilities Limited evidence on the link between salaries and teacher quality
  • Estimate the variation in teacher quality as measured by value added to student achievement Account for systematic differences in rate of gain by initial achievement Use regression to account for sorting of students and teachers Compare variance components across different levels of aggregation to estimate an upper bound on contribution of measurement error
  • Estimate the variation in teacher quality as measured by value added to student achievement Account for systematic differences in rate of gain by initial achievement Use regression to account for sorting of students and teachers Compare variance components across different levels of aggregation to estimate an upper bound on contribution of measurement error
  • Estimate the variation in teacher quality as measured by value added to student achievement Account for systematic differences in rate of gain by initial achievement Use regression to account for sorting of students and teachers Compare variance components across different levels of aggregation to estimate an upper bound on contribution of measurement error
  • Estimate the variation in teacher quality as measured by value added to student achievement Account for systematic differences in rate of gain by initial achievement Use regression to account for sorting of students and teachers Compare variance components across different levels of aggregation to estimate an upper bound on contribution of measurement error
  • Estimate the variation in teacher quality as measured by value added to student achievement Account for systematic differences in rate of gain by initial achievement Use regression to account for sorting of students and teachers Compare variance components across different levels of aggregation to estimate an upper bound on contribution of measurement error
  • Estimate differences in teacher quality by transition status Remain in same school Change school within district Change district Exit Texas public school Control for student fixed effects Control for school by year fixed effects-within school quality differences Adjust for maternity exits Examine quality variation over time
  • Substantial variation in teacher quality Most within schools/not systematic Observed characteristics have little explanatory power Sizeable differences by race/ethnicity and income Little evidence that urban district loses best teachers Exits significantly worse, though interpretation complicated Younger district switchers may be slightly better
  • INEE. Ponencia Profesor Rivkin. Universidad Illinois. Estimating the Effect of Leaders on Public Sector Productivity: The Case of School Principals

    1. 1. Estimating the Effect of Leaders onPublic Sector Productivity: The Case of School Principals Eric Hanushek, Steven Rivkin, and Greg Branch January 2013
    2. 2. Questions• Does the quality of leadership explain a substantial share of the variation in organization outcomes?• Does the variation in principal effectiveness differ by the share of low income students in a school?• Is the pattern of teacher turnover consistent with the notion that raising the quality of teachers constitutes an important mechanism through which principals influence school quality?• Are “effective” principals more likely to leave high poverty schools?
    3. 3. Methodological challenges• Difficult to separate contributions of leaders from other factors – Observed characteristics explain little of the variation in student performance – Typically behaviors cannot be observed or directly related to outcomes – Semi-parametric analyses infer effectiveness from contributions to student outcomes • Unlike the case for teachers, principal actions often affect quality of school in future periods
    4. 4. Growing Literature on Principals• Early work focused on observed characteristics – Clark, Martorell and Rockoff also consider experience• Other work builds on the panel data methods used by Bertrand and Schoar – raise methodological concerns • Grissom and Loeb; Miller (2009)
    5. 5. Direct estimation of Variance in Principal Effectiveness• Builds on Rivkin, Hanushek and Kain (2005) research on teacher quality – Variation in achievement increases with a principal change• Coelli and Green (2012) use this approach for Canada
    6. 6. Our Approaches• Use value-added model with school fixed effects to estimate principal fixed effects – Compute variance of principal effectiveness• Infer variance in principal effectiveness from relationship between year-to-year fluctuations in school value-added and principal turnover – Carefully examining sensitivity to timing
    7. 7. • Examine relationship between estimates of principal quality and changes in teacher quality during a principal’s tenure• Describe principal quality differences by transition status• Consider differences by poverty share throughout analysis
    8. 8. UTD Texas Schools Project• Stacked panels of students and staff• Annual testing• Student demographic characteristics – Divide schools by student poverty rate• Information on staff – Role – Experience – school• Can follow principals and students who switch schools and roles within Texas public schools
    9. 9. Estimation of Variation in Principal Quality-Broad Issues• Non-random selection of principals and students into schools complicates analysis• Tenure-quality relationship complex – Length of tenure unlikely to be monotonically related to effectiveness • Principals likely learn from experience – Skills – Behavior rewarded in school and district • Principal effects on school quality likely grow in magnitude over time • May be positive or negative
    10. 10. First empirical approach• Principal by spell fixed effects based on first three years at a school – Regress math score on lagged math score, student demographic variables, and grade by year fixed effects using aggregate data – bias potentially introduced by unobserved school factors
    11. 11. Alternative Approach• Ignore issue of tenure and use all spells• Control for school fixed effects• Potential bias if there are time-varying school factors not accounted for• Estimate of variance includes differences due to tenure
    12. 12. Test Measurement Issues• Random error inflates estimates of variation in principal quality – Use Bayesian shrinkage estimator to mitigate effects of random error – Unlike the case with the estimation of teacher quality, it is not a serious problem given adequacy of sample sizes even in small schools
    13. 13. • Tests focus on basic skills, so initial achievement differences may influence translation of principal quality into test score growth• Create Z scores and re-weight observations such that average achievement in all schools aggregates over the same test distribution in terms of the share of students in each of ten deciles of the pre-test distribution
    14. 14. Table 3. Distribution of Principal by Spell Fixed Effects by Low Income ShareShare Standard 10th 25th 75th 90th deviationlow incquartileall -0.29 -0.15 0.11 0.22 0.21lowest 0.16 -0.18 -0.06 0.13 0.222nd 0.18 -0.24 -0.14 0.09 0.193rd 0.21 -0.30 -0.16 0.10 0.21highest 0.26 -0.38 -0.24 0.11 0.29
    15. 15. Sensitivity checks• Shrink and reweight• School fixed effects included in specification estimated over sample of all schools with multiple principals during period
    16. 16. Alternative, Test-Measurement Error Adjusted Estimates of the Variance in Principal EffectivenessAdjustment Neither Shrunk Reweighted Shrunk and Shrunk nor Reweighted ReweightedStandard 0.207 0.200 0.270 0.241deviation
    17. 17. Deficiencies of fixed effects • Unobservables, even if orthogonal to principal quality, inflate variance estimate • These include changes over time in student cohort quality and district curricula • Not accounted for with shrinkage
    18. 18. Principal turnover based variance estimates• Derive variance estimates from relationship between year-to-year changes in school average achievement and principal turnover – If principal quality matters, changes should be larger in years in which there is a change in principal – Builds on Rivkin et al (2005) estimates of variance in teacher effectiveness
    19. 19. • Variance in principal effectiveness equals additional year-to-year variation in transition years over non-transition years – Fluctuations between non-transition years provide valid counterfactual for what would have taken place in transition years in the absence of a change in leadership – Not valid if there is additional turbulence during transition years (e.g. Ashenfelter dip)
    20. 20. • If it is actually caused by principal quality, differences in school quality should be larger in non-adjacent years due to an increase over time in principal effects – Compare non-adjacent years around transitions in order to investigate source of additional variation
    21. 21. specificationprincipal quality (θ) in cohort y,quality of other school factors not under the controlof the principal (γ)
    22. 22. Taking ExpectationE (∆ Asy − ∆ Asy ) 2 = 2(σ θ2s − σ θ2yθ y ) + 2(σ θ2s − σ θ2yθ y ) + E (es ) s s s s Assume cov(principal quality) = var(principal quality) if principal same Assume cov(principal quality) = 0 if principal different
    23. 23. • Regress squared year-to-year difference in school average test score gains on indicator for principal change• Assumptions to identify within school variance in principal quality from turnover coefficient – Principal turnover orthogonal to other unobserved changes that affect achievement – Schools draw principals from common distributions during this period
    24. 24. Within School Covariance• Covariance between principal quality in adjacent years with same principal in both years equals variance in principal quality• Covariance between principal quality in adjacent years equals zero in schools that change principals• Coefficient on the principal turnover indicator equals 2 times variance in principal quality
    25. 25. Sensitivity checks• Add squared differences in demographic characteristics in some specifications• Use non-adjacent years in some specifications
    26. 26. Results for entire sampleTiming of Comparison Adjacent year one year inbetweenStudent Demographicand Mobility controls no yes no yesDifferent Principal 0.0052 0.0048 0.0058 0.0056Coefficient (3.41) (3.16) (4.35) (4.28)estimated standard deviation 0.051 0.049 0.054 0.053of principal quality(square root of 0.5*coefficient)
    27. 27. Estimated Standard Deviation by Poverty QuartileQuartile Lowest 2nd 3rd highestAdjacent 0.029 0.037 0.049* 0.067yearsNon- 0.027 0.035 0.057* 0.064adjacentyears
    28. 28. Principal quality and teacher turnover• Principal may have limited control over entrants – Job security an issue, but can still exert influence over who remains • Desirability of school for high quality teachers • Decision to move out lower performers• Focus on effectiveness of exiting teachers and rate of turnover
    29. 29. Figure 1. Teacher Transitions by PrincipalEffectiveness and School Poverty Rate Lowest Quartile Disadvantaged 2nd Quartile DisadvantagedBottom Bottom 2nd 2nd 3rd 3rd Top Top 3rd Quartile Disadvantaged Highest Quartile DisadvantagedBottom Bottom 2nd 2nd 3rd 3rd Top Top 0 .1 .2 .3 0 .1 .2 .3 Quartiles Principle Effectiveness Change School Change District Exit Sample
    30. 30. Estimation• Argument is that better principals are more likely to “dismiss” least effective teachers• Data does not link students and teachers – Focus on differences in grade average value- added within schools• Grade with lower mean value-added is more likely to have a teacher below the “dismissal” threshold
    31. 31. Campus by year fixed effect regressions• Regress share of teachers that exit grade g in school s in year y on controls and grade average value added interacted with principal quality quartile indicators• Control for – student demographics – grade by year fixed effects – School by year fixed effects• Potential problem of a mechanical relationship – In future plan to sever time periods • Measure quality with data in 2nd year of spell • Examine link with teacher turnover in subsequent years
    32. 32. Table 8. Coefficients on Principal Quality Quartile-Grade AverageValue-Added Interactions Using First Three Years Sample bySchool Poverty Poverty Quartiles all highest Grade average gain*2nd quartile -0.018 -0.065 principal quality (0.89) (1.79) Grade average gain*3rd quartile -0.029 -0.025 principal quality (1.35) (0.65) Grade average gain*4th quartile -0.079 -0.102 principal quality (3.68) (3.16)
    33. 33. Principal Transitions and Value Added• Transitions categorized by new role and destination• New role – Principal – Other position in school – Central office administrator• Destination – same school – New school-same district – Central office-same district – New school-New district – Central office-new district – Exit Texas public schools
    34. 34. Probability Principal Remains in Same Position following 3rd Year in a School, by Quartile of Estimated Quality and School Poverty Rate (<25 yrs ex) Principal Quality Quartile Lowest 2nd 3rd highestSchool PovertyQuartileLowest 59% 68% 73% 76%2nd 52% 70% 81% 72%3rd 44% 55% 64% 58%Highest 63% 73% 72% 67%
    35. 35. Probability Principal with Fewer than 25 Years of ExperienceBecomes Principal in Different School following 3rd Year in aSchool, by Quartile of Estimated Quality and School Poverty Rate (total probability of changing position)Principal Quality Quartile lowest 2nd 3rd highestSchool PovertyQuartilelowest 7%(41) 6%(32) 8%(27) 9%(24)2nd 5%(48) 8%(30) 3%(19) 12%(28)3rd 12%(56) 11%(45) 11%(36) 15%(42)highest 12%(37) 12%(27) 10%(28) 9%(33)
    36. 36. Future Work• Estimate very flexible model with school by year fixed effects• Use these estimates to examine whether variance in estimates of principal quality rises with tenure in a school• Account for Ashenfelter dip in direct estimates of variance in principal quality• Modify teacher turnover analysis
    37. 37. Summary• Purposeful sorting and unobserved factors complicate estimates of leadership quality distribution• We find substantial variation in estimates of principal quality – A one standard deviation increase in principal quality would increase school average achievement by roughly 0.05 standard deviations (roughly half as much as a one std dev increase in teacher quality• Least effective principals least likely to remain in a school – Often transition to other schools, particularly from a high poverty school
    38. 38. • Details – Turnover based estimates ignore any between school variation in principal quality – Find a higher quality variance in high poverty schools – Direct estimates of principal quality appear to overstate variance, even in specifications that include school fixed effects• Evidence is consistent with notion that the management of teacher composition is one channel through which principals influence school quality
    39. 39. Principal turnover• Least effective principals least likely to remain in a school – Often transition to other schools, particularly from a high poverty school• Little or no evidence that the most effective principals are disproportionately likely to leave even high poverty schools

    ×