Rhiannon Moore (Young Lives) presenting at UKFIET Conference 2017 on functional English skills and findings from Young Lives school survey data

1. Functional English skills in Ethiopia, India and
Vietnam: comparing English ability and use
amongst 15-year olds in three countries
Gayatri Vaidya (Educational Initiatives)
Rhiannon Moore (Young Lives, University of Oxford)
UKFIET Conference 2017

2.  Young Lives school survey overview
 Functional English & its importance
 Development of repeated measures
cross-country English tests
 Findings:
 A common scale across three countries:
how much progress do children make?
 What functional English skills do children
have?
 What are the characteristics of children
with higher or lower levels of English?
 Discussion and next steps
PRESENTATION OUTLINE

3. YOUNG LIVES & LINKED SCHOOL SURVEYS
 Longitudinal survey of children,
their households, schools and
communities running for 15 years
in 4 countries
 Young Lives school surveys:
introduced in 2010 with a sub-
sample of YL children and peers
 2016-17 school surveys: school
effectiveness in Ethiopia, India
and Vietnam
• Ethiopia: upper primary (Grades 7-8)
• India: lower secondary (Grade 9)
• Vietnam: upper secondary (Grade 10)

4. School effectiveness design:
 Student performance in terms of progress (rather
than cross-sectional measure)
 Teaching and learning processes affecting progress
 The ‘value-added’ of one year of school
To do this, we administered:
 Cognitive tests at beginning and end of one school
year:
 Functional English
 Maths
 Transferable Skills (end of year only)
 Background instruments and psychosocial measures
to contextualise learning progress
SCHOOL SURVEY DESIGN

5. The 2016-17 school survey is the first time Young Lives has
tested English across multiple countries, due to:
• Policy demand
• Global relevance
• Transferable skill
Conceptualising our area of interest as functional English:
“application of […] skills in purposeful contexts and
scenarios that reflect real-life situations” (OFQUAL 2011: 10)
Different purposes for learning English – what does success
look like?
Conceptualising ‘functional’ EnglishWHY TEST FUNCTIONAL ENGLISH?

6. Perceptions
of English
Exposure to
English
… as an
aspirational
language
… as a
language of
social
mobility
… as an
equaliser
…
exacerbating
existing
inequalities
… as a
neutral
language
… as a
global
language
Medium of
instruction School
subject
Teacher
proficiency
Private
tuition
Urban/ruralFamily and
community
ENGLISH: PERCEPTION & EXPOSURE

7. SKILL STRUCTURE & BLUEPRINT DESIGN
Sounds &
Letters
Words &
meanings
Sentence
structure &
comprehension
Reading and
comprehension
 Weightage to different level of Language acquisition
 Focus on ‘usability’ and ‘working understanding’ than
standardness
 Test items focussed on basic knowledge of words, ability to derive
meaning and being able to understand simple dialogues
 Reading- Comprehension included simple stories, Scientific text &
posters

8. CROSS-COUNTRY TEST DESIGN
 The assessments were
designed for Wave 1 &
Wave 2 for all three
countries
 Depending on the pilot
results, items were
selected for each
country
 There were common
items within the wave
and between the wave
to create comparative
data points
 The item selection
ensured presence of
items at different
difficulty levels

9. WAVE 1: SKILL-WISE PERFORMANCE
 % Performance on each skill for
each nation was recorded, which
gave indicative numbers for
overall levels.
 Since the tools were not identical,
these scores are not comparable
between nations.
0
10
20
30
40
50
60
70
80
S1 S2 S3 S4
0
10
20
30
40
50
60
70
S1 S2 S3 S4
0
10
20
30
40
50
60
70
80
S1 S2 S3 S4
S1 Letters, Sounds & Spellings
S2 Words: Meanings & contexts
S3 Sentence: Structure & understanding
S4 Reading and Comprehension
EthiopiaIndiaVietnam

10. 0
.002.004.006.008
Density
250 500 750
English Score
YL India W1 YL India W2
YL Vietnam W1 YL Vietnam W2
YL Ethiopia W1 YL Ethiopia W2
English Performance by YL Country and Wave A common scale for
cross-country and
cross-wave anchor
items was created
using IRT
 The scale was
transformed to have
a mean of 500 (in
Wave 1) and a
standard deviation
of 100
 In one school year
students in our
sample made an
average of 13 points
of progress on this
common scale
 But we also saw variation in scores and
progress across and within the three
countries
CROSS-COUNTRY ANALYSIS – A COMMON SCALE

11. 0
.002.004.006.008
250500750
EnglishScore
YLEthiopiaW1YLIndiaW1YLVietnamW1
0
250 500 750
English Score
YL Ethiopia W1 YL India W1 YL Vietnam W1
Level 1: Basic User
Level 2: Competent User
Level 3: Advanced User
SCALE ANCHORING

12. 0
250500750
EnglishScore
YLEthiopiaW1YLIndiaW1YLVietnamW1
0
.002.004.006.008
250 500 750
English Score
YL Ethiopia W1 YL India W1 YL Vietnam W1
Level 1 (basic user):
At this level, students can:
• Identify everyday vocabulary
(e.g. bag, banana);
• Recognise familiar antonyms
(e.g. start/stop);
• Comprehend short, simple
sentences and complete simple
dialogues with appropriate
questions or answers.
SCALE ANCHORING: BASIC USER

13. SCALE ANCHORING: COMPETENT USER
0
.002.004.006.008
250500750
EnglishScore
YLEthiopiaW1YLIndiaW1YLVietnamW1
0
.002.004.006.008
250 500 750
English Score
YL Ethiopia W1 YL India W1 YL Vietnam W1
Level 2 (competent user):
At this level, students can:
• Use correct question words
(e.g. where, when);
• Complete simple sentences
using appropriate verbs and
adjectives (e.g. lazy, slow,
eat);
• Read short, simple texts and
identify specific information
which can be retrieved directly
from the text.

14. SCALE ANCHORING: ADVANCED USER
0
250500750
EnglishScore
YLEthiopiaW1YLIndiaW1YLVietnamW1
0
.002.004.006.008
250 500 750
English Score
YL Ethiopia W1 YL India W1 YL Vietnam W1
Level 3 (advanced user):
At this level, students can:
• Recognise less commonly used
vocabulary (e.g. whisper)
• Identify the meaning of
unfamiliar words from the
context in which they are used
(e.g. scarce, precise);
• Read more complex texts on
abstract themes and make
inferences about purpose and
intent based on the entire text.

15. WHO ARE THE STUDENTS AT EACH LEVEL?
0
20406080
100
Percent
Level 1 Level 2 Level 3
Bottom tercile (poorest)
Middle tercile
Top tercile (least poor)
0
20406080
100
Percent
Level 1 Level 2 Level 3
Neither parent can read Only mother can read
Only father can read Both parents can read
0
20406080
100
Percent
Level 1 Level 2 Level 3
Rural
Urban
 Clear differences in the
background characteristics of
students at each level
 Students at level 1 (basic users of
English) are more likely to be in
rural areas, from poorer
households and have illiterate
parents than those at other levels
(across all three countries)

16.  Findings suggest that functional English
skills are unevenly distributed
 Both where children are (which country)
and who they are (background, wealth,
locality) affects their functional English
skills
 This has clear implications for equality of
future opportunities, given the increasing
importance of English– suggests English
skills may act as a ‘divider’ rather than an
‘equaliser’?
 Next steps:
 Exploring potential for linking items to the CEFR
scale to further the knowledge on ‘can-do’
indicators at different levels
 ‘Value-added’ analysis of the English test data to
understand more about the schools and classes
where children learn more
 Exploration of the student, school and teacher
level factors relating to progress in English
across three very different country contexts
DISCUSSION AND NEXT STEPS

17. Thank you!
Any questions or comments?