School surveys were introduced into the Young Lives research study in 2010 in order to capture detailed information about children’s experiences of schooling, and to improve our understanding of: - the relationships between learning outcomes, and children's home backgrounds, gender, work, schools, teachers and class and school peer-groups. - school effectiveness, by analysing factors explaining the development of cognitive and non-cognitive skills in school, including value-added analysis of schooling and comparative analysis of school-systems. - equity issues (including gender) in relation to learning outcomes and the evolution of inequalities within education This presentation gives details of the 2016 Survey.

1. Young Lives School Surveys
Research Update, August 2016
Padmini Iyer, Rhiannon Moore & Jack Rossiter
Young Lives, University of Oxford

2. PRESENTATION OVERVIEW
 Overview of YL school surveys
 School surveys 2016-17: research
questions
 School effectiveness research design
 Survey timeline
 Sampling approach
 School survey instruments
• Background questionnaires
• Cognitive tests

3. YOUNG LIVES SCHOOL SURVEYS, 2010 – 2017

4. SCHOOL SURVEYS 2016-17: RESEARCH QUESTIONS
Benchmarking levels of attainment
& progress in key domains
 What are students’ levels of
performance in core curricular and
transferable skills domains (Maths,
Functional English and Problem
Solving), and how do levels compare
across and between contexts?
 How do students progress over time
(in one academic year & since
previous surveys)?
 What is the nature of inequalities in
attainment & progress, and how are
these shaped over time?

5. School & teacher quality and
effectiveness
 What is the role of key dimensions of
school quality (incl. inputs,
leadership, organisation &
management) in shaping educational
outcomes over time?
 How is access to educational quality
distributed (in terms of equity) and
which policies explain or may be
used to improve this position?
 Which teacher characteristics &
practices are associated with
improved learning outcomes?
SCHOOL SURVEYS 2016-17: RESEARCH QUESTIONS

6. Transitions
 Which factors and characteristics at
student and school levels explain timely
progression through educational careers,
and specifically transition into secondary
schooling?
SCHOOL SURVEYS 2016-17: RESEARCH QUESTIONS

7. SCHOOL EFFECTIVENESS RESEARCH DESIGN
Student outcome measures Background instruments
Maths test
Repeated measures; testing mathematical
knowledge, application and reasoning skills
Principal questionnaire
Background data on the principal; school-level
information
Functional English test
Repeated measures; testing reading and
comprehension skills
Teacher questionnaire
Background data on Maths & English teachers;
class-level information
Transferable Skills test
Cross-sectional measure; testing problem-
solving and critical thinking skills
Student questionnaire
Background data on students; psycho-social
measurements
School facilities observation
Data on school infrastructure
School rosters
School, teacher, class and student rosters to
ensure identification of students and linking of
data across levels

8. SCHOOL SURVEYS: TIMELINE
INDIA VIETNAM ETHIOPIA
2016
June
Item selection: Maths and English
Fieldworker training
July
Wave 1 data collection
Fieldworker trainingAugust
September
Wave 1 data collection
Fieldworker training
October
Wave 1 data collection
November Wave 2 pilot
December
2017
January Fieldworker training Wave 2 pilot
February
Wave 2 data collection
Wave 2 pilot
March Fieldworker training &
Wave 2 data collectionApril
Fieldworker training &
Wave 2 data collectionMay

9. SAMPLING: ETHIOPIA
 30 sites
• 20 YL
• 10 more from Round 2
school survey
 School census in each site
 Includes 62 primary
schools offering G7/G8
 Approx. 12,000 students
 Around 3,000 students
linked from Round 2
school survey in 2012/13
 100-200 YL children

10.  Sample design to explore school
choice available in each of the 20
Young Lives sites
 Sample stratified by 4 school types:
• State government schools
• Tribal/Social Welfare schools
• Private Aided schools
• Private Unaided schools
 Number of schools sampled in each
site proportional to the total
number of schools in that site:
Total number of schools in a site Proportion sampled
> 80 schools 10% sampled
21-80 20% sampled
8-20 schools 50% sampled
<8 schools 100% sampled
(Exception: less prevalent school types are oversampled)
SAMPLING: INDIA

11. SAMPLING: INDIA
 20 Young Lives sites
 212 secondary schools
• 83 State Government
• 41 Tribal/Welfare
• 31 Private aided
• 57 Private unaided
 All Class 9 students in sampled
schools: around 12,000 students
• 7,856 in Telugu medium schools
• 4,164 in English medium schools
• 200 in Urdu medium schools
 119 Young Lives children
expected to be in Class 9 at our
sampled schools
• Attending 60 schools
• 18 sites (none in Site 3 or 16)

12. SAMPLING: VIETNAM
 20 Young Lives districts
 55 upper secondary schools
 Maximum of five Grade 10 classes per
school
• Random selection process for schools with
six or more Grade 10 classes
 ~9,000 Grade 10 students
 ~1,000 Young Lives children expected to
be in Grade 10 in 2016-17
• …but maximum number of classes means
we may not include all of them
• Any primary school survey partcipants
(non-YL) to be identified retrospectively

13. BACKGROUND QUESTIONNAIRES
Wave 1 background questionnaires
1. Head Teacher Questionnaire:
 HT background: experience, qualifications & training
 School background: School type, management, fees & funding
 Information on teachers: numbers; absence; means of
rewarding/disciplining teachers
2. Student:
 Student background: ethnic background/language; meals per day;
health problems
 Home environment: asset index; family size; parents’ literacy and
education; student work; parental support
 School: travel to school & residence during school term; use of
toilets at school; use of computer at school; school meals; school
choice; education history and repetition of grades; educational
aspirations; exposure to learning English

14. COGNITIVE TESTS: MATHS
Assessing Maths knowledge and skills
 Country-specific content domains, led by
curriculum
 E.g. Ethiopia’s Minimum Learning
Competencies for Grades 5-8
 Cross-country cognitive domains
 TIMSS 2015 framework
• Knowledge
• Application
• Reasoning
basic number
competency
integers,
rational
numbers,
powers and
bases
fractions,
decimals,
ratios and
percentages
area,
perimeter,
volume and
surface area
geometry and
shapes
algebra
measurement
charts and
graphs
reasoning and
problem
solving

15. COGNITIVE TESTS: MATHS
Priority
 Finely graded test scores in each country
Option
 Common scale – taking advantage of
substantial content-domain overlap
between countries
Bethlehem has balanced some bags of
marbles. All the marbles are of the same
weight. The number of marbles in each bag is
written on it.
How many marbles are there in the bag
marked M?
A. 13
B. 17
C. 33
D. 49

16. COGNITIVE TESTS: ENGLISH
The English test is intended to…
 Measure the construct of ‘functional’
English, based on the tasks which learners
need to complete or may need to complete
in their future;
 Focus on the types of English language
knowledge and skills which students may
be exposed to:
• in school;
• at home and in their wider community;
• in the future.
 Be suitable for children studying in
different mediums of instruction
 Be suitable for use across different
contexts
 Link to the CEFR international framework
for language learning, with clear ‘can-do’
indicators

17.  Developed in partnership with Educational Initiatives
 Items benchmarked against other Indian studies
 CEFR grades A1 (basic user) – C2 (proficient user)
COGNITIVE TESTS: ENGLISH
Areas covered by the English test:
word identification
and vocabulary
word meaning and
contextual vocabulary
sentence
comprehension and
construction
reading
comprehension
Identifying the
meaning of words
relating to a range
of topics, for
example school and
employment
For example, questions
requiring children to
identify the meaning of
an unfamiliar word
based on the context in
which it is used
For example completing
a sentence using an
appropriate word
Different types of
text, for example
posters, stories,
non-fiction texts

18. CEFR level: A2
Sentence construction and comprehension
COGNITIVE TESTS: ENGLISH
Common items across Ethiopia, India and Vietnam
 Around 30 common items across the three countries for Wave 1
• More common items between Ethiopia and Vietnam
• Fewer ‘easy’ items and additional ‘harder’ items in India
For example:
India pilot:
81% correct
Ethiopia pilot:
62% correct
Vietnam pilot:
57% correct
India pilot:
41% correct
Ethiopia pilot:
31% correct
Vietnam pilot:
39% correct
CEFR level: A2
Sentence construction and comprehension

19. COGNITIVE TESTS: TRANSFERABLE SKILLS
Assessing Transferable Skills
 Problem solving, critical thinking,
communication, teamwork (World Bank
2014)
 Transferable skills test at Wave 2
 Cross-curricular problem solving:
• ‘An individual’s capacity to use cognitive
processes to resolve real, cross-
disciplinary situations where the solution
path is not immediately obvious’ (PISA
2003)
 PISA (2003) problem-solving assessments
• Suitable for 15-year-olds
• Paper & pen tests
 Pre-piloted in Ethiopia, India, Vietnam

20. STUDENT TESTS: TRANSFERABLE SKILLS
Level 2 Problem Solving (‘Reasoning, decision-making problem solvers’)
VN pre-pilot results (N = 57): 51%
ET pre-pilot results (N = 24): 31%
PISA 2003 results: 52%

21. COGNITIVE TESTS: TRANSFERABLE SKILLS
Next steps
 Exploring critical thinking assessments
 Alternative problem-solving assessments
 Full-scale pilot of problem-solving &
critical thinking items

22. Any questions / comments?