Evaluating Taxpayer Training Programmes Designed to Improve Compliance: Evalu...
NBQuettaCIES
1. Every day. In times of crisis. For our future.
Shirin Lutfeali
CIES
March 10, 2015
Washington, DC
Numeracy Boost-a deeper look at
language minorities in Pakistan
2. Overview
1. What is Numeracy
Boost
2. Numeracy Boost
in Quetta
3. Results
4. Language groups
5. Assumptions and
Moving Forward
2
4. Numeracy Boost is inside and
outside the classroom
4
• 3 pronged approach-inside and outside the classroom
• Focus on the early grades
– Number and Operations
– Geometry
– Measurement
• Content and process skills
• Math is relevant and useful
• Building math knowledge through activities and
discussion
5. Let’s play a game!
5
The Rubbish Pit
• The objective is to make the largest 3-digit number that comes up on the
die
• 4 rolls
• No moving numbers around after you’ve written them down!
Rubbish Pit
6. Student Assessment
6
Tool Component Information Captured Test
1. Student
Background
Characteristics, Education, SES, HNE,
Time, & Health
30-40 questions
2. Counting Numbers & Operations Skills Count aloud to 100, one-to-
one correspondence, skip
count by 10s, skip count by
5s, count backward 10 -1
3. Number
Awareness
Numbers & Operations Skills 12 number IDs
6 number discrimination
problems
12 missing number problems
4. Operations Numbers & Operations Skills 10 timed addition and
subtraction problems
3 authentic addition and
subtraction problems
5 word problems
5. Measurement Measurement skills 4 stick measurement
questions
3 time of day questions
days of week and months of
year
7. Teacher Training
7
• Five modules
• Delivered monthly
• Content and process skills
• Games, discussions, group work
• In-class support and follow-up
• Government engagement
8. Community Action
Math Camps for Children
Weekly sessions led by volunteers
focusing on bolstering math
content
being taught in the classroom
through games, group work, and
discussions
Math at Home for Parents
Weekly sessions focusing on
simple activities they can do
at home with their children
8
Buying Food in the Market
Concepts: money, operations,
estimation
“Children need both explicit and direct
instruction as well as time to explore, play
and connect math to the real world.”
(Ginsberg, Lee and Boyd, 2008)
Leaves, Leaves Everywhere!
Concepts: geometry, composing
shapes
10. Numeracy Boost in Quetta
• Quetta is in Balochistan Province
• Linguistically and ethnically
diverse region
• Balochistan Education Program
(BEP) launched in 233 schools in
2010
• BEP focused on access,
retention, increasing the quality of
schools, and building capacity at
government level
• Literacy Boost implemented in
2012 and Numeracy Boost in
2014
10
12. Evaluation Sample and Student
Characteristics
Schools Children
Baselin Endline
Numeracy
Boost
18 354 306
Control 18 348 293
Total 36 702 599
• Grade 2
• 49.1% of sample is female
• 9.1 years old on average
• 88% attended katchi
• 69% do chores; 85% of girls and 56% of
boys
• 64% report studying between 1-2 hours a
day, while 18% don’t study at all and 18%
do so more than two hours a day
• 95% report having no books at home
12
No differences between Numeracy Boost and control groups , n=599
13. Average scores by treatment group
13
48% 48%
25%
5%
0%
50%
100%
Average baseline and gain for
total math score by treatment
group
control gain
Numeracy Boost gain***
baseline
element total score possible
one to one 22
zero 1
place value 6
number id 12
skip count 7
number sense 10
missing
numbers 5
timed addition 20
timed
subtraction 20
shapes 5
word problems 3
patterns 2
measurement 3
time 3 *p<.05; **p<.01; ***p<.001
14. Average baseline scores and gains by treatment
group: Counting and identifying numbers
14
99% 99%
34% 34%
67% 67%
47% 47%
1%
46%
18%
42%
0%
4%
8%
0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Numeracy
Boost
Control Numeracy
Boost
Control Numeracy
Boost
Control Numeracy
Boost
Control
one to one
correspondence
(of 22)*
place value
(of 6)***
number identification
(of 12)**
zero***
Control gain
15. Results-Average baseline scores and gains by
treatment group: Number sense
58% 58% 59% 59%
50% 50%
38% 32%
36%
5% 9%
12%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Numeracy
Boost
Control Numeracy
Boost
Control Numeracy
Boost
Control
skip count by 5's
(of 7)***
quantity discrimination
(of 10)***
missing numbers
(of 5)***
Control gain
*significant at the 5% level; **significant a the 1% level; ***significant at the 0.1% level
16. Average baseline scores and gains by
treatment group: Operations
16
15% 15% 8% 8%
37% 37%
25%
22%
49%
7%
5%
8%
0%
20%
40%
60%
80%
100%
Numeracy
Boost
Control Numeracy
Boost
Control Numeracy
Boost
Control
timed addition
(of 20)***
timed subtraction
(of 20)***
word problems
(of 3)***
Control gain
*significant at the 5% level; **significant a the 1% level; ***significant at the 0.1% level
17. Average baseline scores and gains by treatment
group: Geometry and measurement
17
43% 43%
60% 60%
47% 47%
32% 32%
46%
31%
41%
45%
15%
6%
0%
8%
0%
20%
40%
60%
80%
100%
Numeracy
Boost
Control Numeracy
Boost
Control Numeracy
Boost
Control Numeracy
Boost
Control
patterns
(of 2)***
shapes
(of 5)***
measurement
(of 3)***
time
(of 3)***
Control gain
*significant at the 5% level; **significant a the 1% level; ***significant at the 0.1% level
18. Math Gain in Numeracy Boost Schools
by Language Group
18
48% 53% 56% 56% 58% 56%
31%
29% 25% 25%
36%
23%
0%
20%
40%
60%
80%
100%
Urdu
(n=11)
Pashto
(n=113)
Brahvi
(n=98)
Balochi
(n=21)
Farsi*
(n=86)
Others
(n=18)
baseline gain
*p<.05; **p<.01; ***p<.001
20. Assumptions and Moving Forward
• Farsi is most similar to
Urdu, the LOI
• Farsi speaking
families…
o engaged more regularly in
community events
o encouraged children to
participate in Math Camps
o maintain a child friendly /
print rich home
environment
o place a high value on
education for their children
• Attendance in community
activities
• Home environment,
family support and
encouragement
• Teachers matter-teaching
learning circles, demo
sessions, model teaching
• Qualitative research
20
Assessment, teacher training, community action
Early grades-based on needs in country and research in the field of early math
Wanted to focus on both the conceptual skills-understanding concepts like one to one correspondence, addition and subtraction and measurement but also important process skills-like reasoning and proof, problem solving, discussion and communication, making connections
We did this by using a lot of simple games to help build children’s content and process skills. Let’s play one now! Rubbish pit example.
Wanted to focus on both the conceptual skills-understanding concepts like one to one correspondence, addition and subtraction and measurement but also important process skills-like reasoning and proof, problem solving, discussion and communication, making connections
We did this by using a lot of simple games to help build children’s content and process skills. Let’s play one now! Rubbish pit example.
Data collected electronically
The numeracy boost philosophy is very similar to the LB philosophy: because we are implementing an actual program, we want the assessment to act as a diagnostic as well as for impact evaluation. That means we want to know the full extent of children’s skills – so we don’t stop them based on time, only on struggling. We want to test children on as wide a variety of skills as possible, corresponding to the three conceptual areas endorsed by the National Council for the Teachers of Mathematics (NCTM) and the National Association for the Education of Young Children (NAEYC). They also closely align with the curriculums of the countries in which we implement NB; Malawi and Bangladesh (and Pakistan).
NB is not a curriculum-it’s a methodology and approach that can be mapped onto any country’s math curriculum
Toolkit includes 5 modules for teachers, focusing on early primary grades, covering concepts related to three broad domains-Number and Operations, Geometry and Measurement
Contextualized prior to training, incorporating country competencies and benchmarks, adapting the formative assessment tool, linking with textbook
The assessment tool is for 2nd graders and includes a home background component, as well as a range of questions both timed and untimed on 3 domains
Community action-math camps for kids and math at home sessions for parents with activities they can do in the home to support math learning
Balochistan is largest province by land mass-44% of the country’s land but smallest in terms of population. Less than 5% of country’s pop lives here
Balochi, Pashtun, Brahvi, Farsi-languages. Ethnicities include Balochis, Hazara, Pashtuns, Sindhis, Punjabis, Turkmen
BEP provided trainings for teachers on child friendly environment and SHN, had trainings for principals and SMC members, and also provided school rebuilding
NB implemented in same schools which had received LB in the previous year
Balochistan province-BEP 233 schools total since 2010, just ended in 2014
NB in 2013 in 18 schools-previously had LB and other inputs
Control schools also had other inputs
14.6 percent of the sampled children were not found at the second time of data collection.
Attrition doesn’t seem to be linked to any observed characteristic-migrations due to movement of family
Numeracy Boost students significantly outperform their control school counterparts on total math score, ending the year at 75% correct on average.
Same pattern: significantly greater gains for NB students, little change for controls. NB students headed for mastery across these three skills
Started of the same at baseline
NB made gains in these somewhat more advanced skills
Geometry and measurement show great gains, little movement in controls. Implies that if these are taught, they can be mastered by most.
While all language groups started off at similar levels of average math scores, Farsi-peaking students made significantly greater progress compared to children who speak all other languages with the exception of Urdu-speakers. These 86 children are concentrated in 4 schools, with others scattered across additional schools, so it could be informative for the program team to consider activities and strengths that might be generalized as NB moves forward.
Community activities participation-missing number, subtraction, word problems, some geometry-in Bangladesh
LB+NB gains in Count to 100 (p<0.10) Skip count by 5s (p<0.05)Missing Number (p<0.01) Authentic subtraction (p<0.05) and timed subtraction (p<0.01) Subtraction fluency gain is higher (p<0.10)Word problems (p<0.05)
In Balochistan children from the Persian speaking community outperformed peers from other language groups and we have anecdotal information about the level of involvement of their fmailies and their home environment
Boys outperformed girls in both Bang and Pak-but girls did better on lit. Girls read to more, read more to others. Boys sent to market and help with farming, buying selling, etc.
Children in farsi speaking schools concentrated in 4 schools…what is it about these schools and teachers?
