A COMPARISSON OF TEACHER AND TEST BASED ASSESSMENT FOR SPANISH PRIMARY AND SECONDARY EDUCATION póster de Óscar D. Marcenaro-Gutiérrez y Anna Vignoles Congreso FES grupo Sociología de la educación Madrid 2013

1. A comparisson of teacher and test based
assessment for Spanish primary and
secondary education
Óscar D. Marcenaro-Gutiérrez*
Anna Vignoles**
*Departamento de Estadística y Econometría (UMA) y Fundación Pública
Centro de Estudios Andaluces (FCEA)
** University of Cambridge
XI Conference FES
Madrid, 10-12th
July 2013
G13
1

2. Outline:
Motivation
Aim
Data
Methodology
FES Madrid
 Conclusions

3. 3
FES Madrid

4. Motivation:
1. Needless to say: importance of education as an
engine of socio-economic change.
4
The profitability of investment in education is higher
that investment in capital goods
(Psacharopoulos, 1985)
FES Madrid

5. Importance of training as an engine of socio-
economic change: Budgetary effort.
Motivation:
FES Madrid

6. Motivation:
Central to Economics of Education: Do we have
precise measures of students’ outcomes?
 To what extent does the nature of the assessment criteria
produce social class inequalities in academic outcomes?
FES Madrid

7. Motivation:
2. Teacher assessment matters, because:
- Affects how students perceive their own academic
ability (self-esteem, etc.).
- It is used to determine whether a child can proceed to
the next grade.
1. Virtual lack of datasets allowing the comparison
“Teachers’ actual assesment-external test assesment”;
novelty of the paper.
7
It is crucial to determine whether such assessment is
a good guide to pupils’ actual level of achievement:
PISA? Actual exams?....
FES Madrid

8. Aim:
 Are assesment tests (test-based assessment –TestA-) and
official marks (teachers-based assessment –TeachA-)
telling different things? We explore this empirically.
 Do these differences vary systematically with pupil
characteristics?
8
gender
ethnicity
socio-economic background
FES Madrid

9. Aim:
 Are assesment tests (test-based assessment –TestA-) and
official marks (teachers-based assessment –TeachA-)
telling different things? We explore this empirically.
 Do these differences vary systematically with pupil
characteristics?
 Differences in achievement levels of pupils in public (100%)
schools and semi-private schools.
 Do this potential gap discourage further study, in general or
in any particular subject area (such as science, arts, etc.), after
completing compulsory education? (i.e. future success) 9
FES Madrid

10. 10
Literature:
Gender or Ethnicity of the teacher might interact with that of the
student to produce systematic differences in pupil achievement
(Dee, 2005 & 2007): teachers over grade students of the same gender
to themselves.
Comparing teacher assessments of pupil achievement and test
scores: Reeves et al (2001) , Gibbons and Chevalier (2008) –UK-,
TestA-TeachA consistent.
Gutierrez and Adserá (2012) –as opposed to Calero and Waisgrais
(2008)- public school students higher grades: might be explained
by differences in grading practices between public and private
school teachers.
FES Madrid

11. Data:
 PISA: 15-years-old students -4th
year ESO-(OECD countries).
 Dependent variables:
- Math Test scores (normalised).
- Reading Test scores (normalised).
 Independent variables:
- Student characteristics
- Family background 26 variables
- School characteristics
 PISA-2009:
Spain: 25.887 students within 889 schools.
Andalusia: 1700 students within 51 schools 11
Data (1st
stage):
FES Madrid

12. Linked databases (novelty):
- Andalusian Social Survey: Education and Homes in
Andalusia (Encuesta social: Educación y Hogares en Andalucía,
ESOC10); includes diagnostic assesment tests.
- Administrative marking records for those in the survey
sample.
Data (2nd stage):
FES Madrid

13.  ASS10:
- Students born in 1994 + students born in 1998.
15 years 11 years
- Pseudo-panel: past and future outcomes.
- Very important: Includes interviews aimed at parents
and children.
13
Crossing information
(consistency)
Data (2nd stage):
FES Madrid

14.  ASS10:
- ADDITIONAL added value: link ASS10 with information from
an administrative information system.
14
Information entered by the educators from
public and semi-private schools
• Repeaters.
• Disabled (or with special educational requirements).
• Private schools (only 2.5%) –data non avaialble-.
Data (2nd stage):
Excluding
FES Madrid

15. Andalusia 2009, huge differences with
the rest of Spain?
16
• Differences between natives and immigrants men more substantial in
Andalusia as compared to the rest of Spain
• Public school Vs Private school: same differences as for the rest of Spain.
• Mother educational level: variables even more relevant.
• Once again…. IMPORTANCE OF HUMAN CAPITAL
ACCUMULATION OF THE PROGENITOR AS A WAY OF
ACADEMIC SUCCESS FOR CHILDREN, PARTICULARLY FOR
WOMEN.
A) School failure:
B) Performance determinants:
FES Madrid

16. 17
¿Consistency? (PISA-ASS)0.1.2.3.4
Density
-4 -2 0 2 4
Normalized scores
PISA reading TestA
ASS10 reading TestA
Reading
0.1.2.3.4
Density
-4 -2 0 2 4
Normalized scores
PISA math TestA
ASS10 math TestA
Math
Substantial
overlap
FES Madrid

17.  High grade of consistency between PISA-2009 y ASS10-SEN
conditional effects:
18
Reading
TestA
(ASS10)
TestA
(PISA09)
Women 25,93*** 6,73***
Inmigrant -36,07** -34,73**
Semi-private 40,10*** 8,40*
Mother ed.: primary -9,53 14,01
Mother ed.: lower secondary 1,42 8,73
Mother ed.: upper secondary 25,54** 20,88*
Mother ed.: Higher ed. 35,77*** 41,83***
Constant 504,70*** 477,37**
Observations 1287 843
R2
0,12 0.07
¿Consistency?
FES Madrid

18. 0.1.2.3.4
Density
-4 -2 0 2 4
Normalized scores
Difference in assesments (reading)
0.1.2.3.4
Density
-4 -2 0 2 4
Normalized scores
Difference TestA-TeachA (female students)
Difference TestA-TeachA (male studnets)
Difference in assesments (maths)
women men
Differences TestA-TeachA:
FES Madrid

19. Multivariate analysis
Multinomial Probit
Probability of:
 Dropping out or
repeating.
 Enrol in Vocational
training.
 Academic track (Sciences,
etc.)
OLS regressions/FE/RE
Effects of variables on the
difference TestA-TeachA
(ASS-SEN)
20
FES Madrid

20. Table 2.a. Discrepancies TestA - TeachA (1994 cohort).
21
Specification V
Reading Maths
Female -0.03 -0.50***
Immigrant 0.10 0.26
Semi-private school (=1) 0.49*** 0.20**
Education level of father:
Degree or higher -0.06 -0.05
Education level of mother:
Degree or higher -0.11 -0.11
Constant -0.14** 0.12*
Number of observations 1,087 1,051
R-squared 0.04 0.06
Differences TestA-TeachA: (conditional analyses)
FES Madrid

21. Table 2.b. Discrepancies TestA - TeachA (1994 and 1998 cohorts).
22
Differences TestA-TeachA: (conditional analyses)
Specification V
Reading Maths
Cohort 1994 (=1) -0.05 -0.06
Female -0.06 -0.34***
Immigrant -0.12 -0.02
Semi-private school (=1) 0.37*** 0.22***
Education level of father:
Degree or higher -0.07 -0.05
Education level of mother:
Degree or higher -0.01 -0.03
Constant -0.07* 0.09**
Number of observations 2,778 2,725
R-squared 0.03 0.04
Teachers over estimating
achievement of girls?
(preconceived ideas about ability’?
Teachers under estimating
achievement of students at
Semi-private schools.
(Do teachers measure the child’s
achievement relative to other????)RE: Virtually identical results
(2-3 students from each school).
FES Madrid

22. 23
¿Difference TestA-TeachA varies across the
ability distribution?
Differences TestA-TeachA: (conditional analyses)
FES Madrid

23. 24
Note: Including controls for immigrant, father education level and mother education level.
Table 4: Controlling for quintile in the previous year
Specification V
Reading Maths
Prior achievement -Reading- (TeachAt-1):
2nd
quintile -0.11
3rd
quintile -0.64***
4th
quintile -0.33***
5th
quintile (top) -0.74***
Prior achievement -Math- (TeachAt-1):
2nd
quintile -0.02
3rd
quintile -0.65***
4th
quintile -0.49***
5th
quintile (top) -0.60***
Female 0.08 -0.40***
Semi-private school (=1) 0.54*** 0.22***
Constant 0.21 0.44***
Number of observations 995 1,001
Number of centres 447 451
Differences TestA-TeachA: (conditional analyses)
pupils TeachAt-1
narrows gap
FES Madrid

24. 25
Note: Including controls for immigrant, father education level and mother education level.
Table 5: Controlling for TeachAt-1
Differences TestA-TeachA: (conditional analyses)
Specification V
Reading Maths
Prior achievement -Reading- (TeachAt-1): -0.26***
Prior achievement -Math- (TeachAt-1): -0.20***
Female 0.03 -0.50***
Semi-private school (=1) 0.48*** 0.17**
Household cultural index level:
Average cultural index -0.01 0.07
Higher cultural index 0.15* 0.18*
Constant -0.18*** 0.11
Number of observations 1,057 1,030
Number of centres 466 460
≈
“teachers’ views about the ability of
pupils are persistent”
FES Madrid

25. 26
Table 6: By school type
Differences TestA-TeachA: (conditional analyses)
≈
Specification V
Semi-private school Public school
Reading Maths Reading Maths
Female -0.12 -0.50*** -0.00 -0.49***
Constant 0.49*** 0.42*** -0.16** 0.11
Number of observations 265 248 822 803
Number of centres 148 137 334 331
FES Madrid

26. 27
¿ To what extent TeachA-TestA at 15 predict education outcomes at age 16?
Choice at 16:
1) Repeating or dropping out of school at age 16 (ref.).
2) Undertaking vocational study;
3) For those staying on the academic track, enrolled in:
Sciences, Arts & Humanities or Social Sciences)
FES Madrid

27. 28
Table 7:
Exit to:
Vocational
Training
Arts Sciences
Social
Sciences
TestA-TeachA (reading) -0. 49*** -0.33** -0.60*** -0.47***
Female 0.05 0.25 0.07 0.41***
Immigrant -0.05 0.50 -0.34 -0.27
Semi-private school (=1) 0.41 0.19 0.50*** 0.41**
Constant -0.71*** -1.31*** 0.97*** 0.90***
Number of observations 1164
Χ2
90.39***
“pupils who are under estimated by teachers
in terms of their achievement are more likely
to drop out”
Choice at 16:
FES Madrid

28. Conclusions
 High relevance of parents human capital (particularly
mothers); this is deeper in Andalusia, relevant to economic
growth and labour market opportunities of women!!!
(higher performance and lower failure)
29
 PISA09 and ASS10 similar results, TestA consistent.
FES Madrid

29. Conclusions
 Systematic differences between teacher assessments
and actual test scores.
 Girls’ achievement is over estimated by teacher assessments
relative to test scores (preconceived ideas?????).
 Teachers under estimating achievement of students at Semi-
private schools (measuing relative acievements –peer
effect-?????).
 Pupils who are under estimated by teachers in terms of their
achievement are more likely to drop out→ TestA-TeachA has
a longer term impact on pupil outcomes.
30
FES Madrid

30. Thank you!
31
FES Madrid

31. A comparisson of teacher and test based
assessment for Spanish primary and
secondary education
Óscar D. Marcenaro-Gutiérrez*
Anna Vignoles**
*Departamento de Estadística y Econometría (UMA) y Fundación Pública
Centro de Estudios Andaluces (FCEA)
** University of Cambridge
XI Conference FES
Madrid, 10-12th
July 2013
G13
32