Jindani, F., & Shallwani, S. (May, 2011). Experiences in assessing early learning achievement in international contexts: adapting an ‘international’ tool or ‘locally’ developing a new one? Paper presented at the Annual Conference of the Comparative and International Education Society, Montreal. Abstract: In recent decades, countries around the world have aimed to increase primary school enrolment rates (Filmer, Hasan, & Pritchett, 2006). However, school quality varies widely and many children in school are not learning effectively. For example, a learning achievement study in India found that almost half of fifth grade children were unable to read at second grade proficiency levels (Pratham, 2007). If children do not establish basic literacy skills in the first few years of education, they are at a severe disadvantage for the remainder of their schooling and lives. In order to understand and improve the quality of education and students’ learning, it is necessary to have indicators and standards for early learning achievement, literacy in particular, at both local and global levels. Some researchers and evaluators have aimed to develop and adapt standard international indicators for use in various contexts around the world. However, others, coming from a social-constructivist perspective, have argued that such tools are products of the Western world and inappropriate for diverse contexts of the world. In this paper, we compare and contrast the two perspectives with literature and research evidence, as well as our own experiences in developing indicators of learning achievement in international contexts. We share the experience of adapting an ‘international’ tool (the Early Grades Reading Assessment) for the Cambodian context and the experience of developing a ‘local’ Learning Achievement Tool in Pakistan. We also deconstruct notions of ‘international’ and ‘local’, and critically reflect on our role precisely as the international researchers in these contexts. We argue that standards must be established both at local and international levels through reflection and explicit discussion about contexts, social positions, values, and purpose (Myers, 2004), and that these standards must then be continuously reflected upon and subject to change over time and place.

1. Adapting an international tool or locally
developing a new one?
Farah Jindani
Sadaf Shallwani
University of Toronto (OISE)
Jindani, F., & Shallwani, S. (May, 2011). Experiences in assessing early learning
achievement in international contexts: adapting an ‘international’ tool or ‘locally’
developing a new one? Paper presented at the Annual Conference of the
Comparative and International Education Society, Montreal.

2. Outline of Presentation
Background
Experiences adapting an international tool
Experiences developing a local tool
Reflections
A suggested framework for moving forward

3. Background
Higher enrollment rates globally
Concerns:
Completion rates
Levels of learning
Uganda: 46% of third graders and 50% of sixth
graders are not reaching expected levels of
competency in literacy
India: 47% of Grade 5 children could not read a story
text at a Grade 2 level of difficulty

4. Background
Where is the system failing?
Need indicators and standards to assess the
situation, identify areas needing intervention, and
monitoring progress
Need for indicators that are reliable, contextually
meaningful, and socially relevant
Two approaches: adapting an international tool or
developing a new tool

5. Cambodian Context
Ratanakiri is home to a number of ethnic minority
groups; Khmer 33% of population
Ethnic minority children have trouble accessing
relevant and meaningful education
CARE long history of education programs in
Cambodia
HCEP (2002): Bilingual model of education

6. Adapting an International Tool
Most assessments do not assess reading skills
before fourth grade
Lack knowledge tested by assessments or lack basic
reading and comprehension skills?
Remedial instruction
Most assessments based on word recognition tasks

7. International Tool
RDI International (2007) with support from World
Bank and USAID developed Early Grades Reading
Assessment (EGRA)
EGRA oral assessment designed to assess skills
known to predict literacy:
Recognizing letters of alphabet
Reading simple words
Understanding sentences and paragraphs

8. EGRA
Quick to conduct
Has been adapted in numerous international
contexts
Salient for testing Khmer literacy in HCEP in RTK
because of bilingual education

9. Adapting EGRA
EGRA toolkit developed by writer; shared with team
Participatory approach
Cross-referencing between English and Khmer during
adaptation by research team
Content validity of national curriculum and community school
texts
Sound recognition section on EGRA
Independent work by team and would then come back
together

10. Experiences in Field
Variability amongst data
EGRA based literacy assessment tool completed
within time frame (20 minutes)
Time frame per section deemed too short (60
seconds) for children
Curriculum validity; tested grade 4 students

11. Lessons Learned
Organizational issues in relation to context
Content issues (straight translation between
languages)
Benefits and supports

12. Pakistan
RCC programme – early childhood development
programme
Project of the Aga Khan Foundation, Pakistan;
funded by the Dutch government
Research study
Programme and comparison schools in four provinces
Assessing outcomes on school-level, classroom-level,
and child-level indicators
Child-level: learning achievement in early primary

13. Learning Achievement Tool
Decision to develop a tool locally
Concerns over appropriateness of using tools developed in
other contexts
Desire to have a tool that assesses children’s learning
according to local benchmarks / standards
Process of development
Identified National Curriculum benchmarks – English
literacy, Urdu literacy, and numeracy
Reviewed textbooks from different regions
Drafted questions/tasks
Collaborative process of reviewing, piloting, and revising
each task

14. Experiences in the field
Administration
Training, field practice, field supervision
Two RAs at a school, 1 RA with up to 5 children
Consent
Counterbalanced order
Explain each question with example to child
(individually) – give time, prompt at least twice
If child does not to 3 questions in a row, move on to
next portion
Took about 20 minutes to administer

15. Experiences in the field
Challenges
Difficult at first to administer
Fine line between explaining and leading
RA bias – desire for child to perform well
Strategies to overcome
Developed standard guidelines for RAs to use in
explaining the task and examples
Field supervision and support
Issues of research ethics with children

16. Lessons learned
Test development process
Difficulties developing questions relevant to varying
contexts of Pakistan
Need to assess reliability and validity
Importance of standard implementation of tool
Children from programme schools outperformed
children in comparison schools

17. Reflections
International-local: not binary
Tools
Different contexts within country
Role of international researcher
Power/presumed expertise
Internal struggles on decisions (e.g., presumed credibility
of ‘international’ tools)
How do we create/adapt tools to assess learning
achievement that are contextually meaningful and
credible?

18. A framework for moving forward
Quality in early childhood (Myers, 2004)
Ongoing process of discussion and dialogue in the centre
– all stakeholders
Discuss ‘what kind of society do we want?’ ‘what kind of
citizens are needed for that society?’
Derive areas agreement, construct major categories and
indicators
Use research and experience
Make value positions and assumptions explicitly
Distinguish purpose of using instruments
Ongoing – society is not static