Ilm Ideas Evidence and Advocacy Accelerator for Grant Partners

1. What to research and how?
Research questions, sampling and all
that
Faisal Bari
Associate Prof. of Economics, LUMS
Associate Fellow, IDEAS
(With contribution from Dr. Farooq
Naseer, IDEAS)

2. Outline
• What does the TNA tell us
• Framing of research issues, questions and
tools. This appears to be simpler than it
is….demands reflexivity
• Sampling and related issues…all about power

3. Ilm-Ideas TNA: Research Tools
0
2
4
6
8
10
12
14
16
18
Focus Groups Interviews HH Surveys School level
survey
Case studies Secondary
analysis of data
Statistical models Other
Research tools used most often
1 2 3 4 5

4. Ilm-Ideas: Tools Required
11
13
12
15
13
11
14
14
10
15
14
17
14
13
6
4
5
2
4
6
3
3
7
2
3
0
3
4
Conducting a research needs assessment and/or defining research
objectives
Identifying priority research questions
Selecting research sites and developing criteria for the selection
Selecting and justifying the sampling strategy and target numbers
Sampling – selecting the research target group
Conducting desk research to identify good practice examples within and
outside the country of similar researches undertaken
Developing research indicators
Developing research tools and instruments
Piloting the research instruments
Conducting qualitative and quantitative research tools and instruments
Management of data collection/fieldwork, including the control, supervision
and debriefings of field workers/interviewers
Use of data analysis software and systems
Conduct data interpretation and analysis
Report writing
Priority Non-Priority

5. Research Capacity
8
7
7
6
7
10
7
10
12
10
10
3
9
11
8
10
8
8
8
5
8
7
3
5
5
7
5
4
1
0
2
3
2
2
2
0
2
2
2
7
3
1
0 2 4 6 8 10 12 14 16 18
Conducting research needs assessment and/or defining research objectives
Identifying priority research questions
Selecting research sites and developing criteria for the selection
Selecting and justifying the sampling strategy and target numbers
Sampling – selecting the research target group
Conducting desk research to identify good practice examples
Developing research indicators
Developing qualitative and quantitative research tools and instruments
Piloting the research instruments
Conducting qualitative and quantitative research tools and instruments
Management of data collection/fieldwork, including the control,
supervision and debriefings of field workers/interviewers
Use of data analysis software and systems
Conduct data interpretation and analysis
Report writing
Organizational Capacity - Research
High Medium Low

6. Main Challenges in Policy Research
• Getting concerned institutions engaged and motivated
• Data management
• Interpreting data
• Report writing
• Availability of updated data
• Accessing policy documents
• Low experience/expertise in conducting policy research
• Access and availability of public expenditure documents
• Discrepancy in government data/inaccurate govt. data
• Dearth of qualitative research experts in the country
• Lack of interest within policy circles
• Shortage of sector experts
• Community based research
• Sometimes funding agency and govt. interests don’t match

7. Framing Research
• Can you tell whether you are drinking Coca Cola?
• For a single person: coke or not
• For a single person: coke or other colas
• For many people: coke or not
• For many people: coke or other colas
• Trivial? Think of cure for cancer 10% total cure
versus 50 percent improvement for 50% (but not
cure)

8. Framing Research: Examples
• Private Public Partnerships in Education:
Adopt a school programme
• Importance and need: 25 A and quality issues
• Variation in legal frameworks: Punjab and
Sindh
• Variation in models: PEN, CARE, SEF
• Variations across time: do models mature.
What is the exit strategy

9. Framing Research: Examples
• Remedial education for teachers (will come back
to this at the end too)
• DSD reports, PEC results….content knowledge of
teachers is a significant issue
• How to remedy that? CPD already in place
• Something that is scale-able also
• Using DTEs to reach teachers (Maths and Science)
• Use technology to reduce cost

10. Framing Research: Examples MFN
• Post fact impact evaluation…one way MFN
paper
• Introductory paras set the context and
question
• Issue of composite effect…rather than
isolating contributions. Child friendly (teacher
training, materials, parental
involvement)…better learning

11. Framing Research: Examples MFN
• Propensity score matching (not gold
standard…but best available here)
• Two stage matching: Schools and then
children (need both school and
children/family characteristics)
• School level matching: geography, medium,
level of school

12. Framing Research: Examples MFN
• Within school blocks….child matching
• Robustness
• Children joining…dropped…selection bias
• Treatment and control children…good match
on average

13. Framing Research: Examples MFN
• Mining….Item Response Theory (IRT)
• Possibility of leakage (teacher transfers,
student transfers)
• No non-cognitive testing….where gains might
be large too
• Could we check if the effect was different on
the weakest/strongest students

14. Framing Research: Examples
• Tahir Andrabi and the recent education
recovery paper.
• Distance from Fault Line as the independent
variable
• How is that established? And What is its
importance
• The results are insightful….the ‘hey, wait a
minute’ moment

15. Sampling Issues:
• Statistics Refresher: Summarizing data
• Sampling:
– Minimizing error
– Representativeness
• Hypotheses testing
• Power

16. Data: Summarizing
• Variation is what we study: variation is King
• Statistics helps us summarize data by using
two important features of a dataset:
–the average (mean, center)
• what is the average age of participants in this room?
• Is it important?....not a technical issue only (The deer
hunter)
–the variance (variability, spread)
• by how much does age vary across participants?
• Again….is it important…and when (50 or 0/100)

17. Distribution

18. Population and Sample
• Measuring the population gives us the truth!
(assuming there is no measurement error)
– But we usually cannot survey the entire
population
– Hence we must draw a sample
• How do we choose the sample?
• How large should be the sample?

19. Sample
• Sample must be representative of the population:
– Draw a random sample
– Jute example, skulls, Indian census
• But still, the sample is not some fixed subset of the
population so each sample will be different!
• This is called “sampling error.” How to reduce it?
– Draw a larger sample.
– But how large? (depends on the hypothesis of interest
and sampling error… want to maximize the “power” to
reject incorrect hypotheses)

20. Simple Random Sampling
• List every individual in the population of interest
(population size: N)
• Decide on a sample size based on ‘power’
calculations (sample size: n < N)… to be discussed
• Randomly pick n individuals from the population
such that each individual has a positive chance of
being picked
• Examples:
• Toss a coin
• Draw lots out of a basket
• Use a computer software

21. Stratified Random Sampling
• Mark separate sub-groups (or strata) in the
population list before drawing a random
sample from each
• Stratified Sampling
– For adequate representation
of different sub-groups (i.e. strata)
in the population
– For a given sample size, reduces the
sampling error as compared to the
un-stratified simple random sampling
• Trade-off between the cost of doing stratification and the
smaller sample size needed
• Fraction sampled could be different across strata; improves
across-group comparisons

22. Two Nice Results
• Before we turn to hypothesis testing and the concept
of statistical power, important to recognize that the
sample average behaves well in large samples
• Law of Large Numbers
– The sample average will approach the true
population average as the sample size increases
• Central Limit Theorem
– The sample average will tend to be normally
distributed, around the true population average
value, as the sample size increases

23. Normal Distribution
𝑀𝑒𝑎𝑛: 𝜇 =
𝑥𝑖
𝑛
𝑆𝑡. 𝑑𝑒𝑣: 𝜎 =
(𝑥𝑖 − 𝜇)
2
𝑛

24. Hypothesis Testing
• Suppose the average pre-training knowledge of M&E
in the population is 3/10 points on a standardized
test
• How can we empirically test whether this course
improves M&E knowledge?
• In statistical terms, this test can be stated as follows:
– H0 or the null hypothesis: This hypothesis states what you
would like to disprove i.e. “no effect”.
– H1 or the alternative hypothesis: The course improves
M&E knowledge i.e. “positive effect”.

25. Hypothesis Testing
• Ex-post, administer the test on multiple cohorts of course
participants –OR– use statistical theory to decide based on
just one cohort
• When is the average test score of course participants in a
cohort “significantly” (i.e. statistically) higher than 3?
• That is, allowing for
sampling error, when
can we be “confident”
that we are observing a
real improvement in M&E
scores?
• Depends on the sampling error
in average test score

26. Hypothesis Testing
• Suppose, you want to test a promising
intervention designed to improve
• (M&E) education. Question: Is the intervention
(“treatment”) effective?

27. Statistical Power
• The power of a test is the probability of
correctly rejecting the null hypothesis
• In other words, power is the probability of
correctly declaring the treatment as beneficial
• Hence, Statistical Power = 1 – Prob(Type-II
error)

28. Importance of getting power right
Testing a new ‘miracle’ cure for cancer
– Power too low; missed a large treatment effect
– Power too high; wasted resources in doing a large study to
declare a tiny, clinically irrelevant effect as statistically
significant
– Power just right; have a good chance of detecting
reasonably sized effects, but not tiny ones

29. Power: Main Ingredients
For a given significance level, power depends on
the following:
1. Sample Size
2. Assumed Effect Size under H1
3. Variance of outcome in the study population
4. Proportion of sample in T vs C
5. Clustering

30. Power Sample Size
• Increasing the sample size reduces the
sampling error (i.e. sample-to-sample
variation) in the sample average

31. Treatment Effect

32. Variance
• The “sampling error” in the sample average,
sigma^2/n, is directly proportional to the
(“natural”) variance in the outcome variable in
the population
• There is sometimes very little we can do to
reduce the noise
• The underlying variance is what it is
• We can try to “absorb” variance:
– controlling for other variables

33. Clustering:
• You want to know how close the upcoming
national elections will be
• Method 1: Randomly select 50 people from the
entire population
• Method 2: Randomly select 10 families, and ask
five members of each family their opinion
• Method 2 will yield relatively imprecise/noisy
estimates if the political opinion within families
does not tend to vary a lot (high “intra-cluster
correlation”)

34. Sampling Frames for Examples Used
• For PPP
• For remedial education for teachers
• Why did MFN go the way he did

35. And last but not least
• Happy hunting
• Thank you