Test your impact evlaution knowledge with our quiz from the Transfer Project Impact Evaluation Training with AERC in Nairobi in July 2019.

1. Who Wants To Be A
AERC
Millionaire?

2. Rules
1. There are 17 questions with 4 multiple choice
options, you need to guess the correct answer
2. The questions will be read twice, then you have 30
seconds to record the correct answer
3. You must work with your group – each group has
one judge/score keeper
4. No cheating, you cannot ask a trainer or ask another
group what the correct answer is
5. The team with the most points at the end wins a
prize (no, you cannot know what this is before the
end of the game)

3. Example
question

4. Where are the trainers from?
A Germany, USA, Tanzania, Ghana
B Ghana, Tanzania, Ecuador, Italy, USA
C India, USA, Ghana, Tanzania, Spain, Peru’
D Italy, Ghana, Peru’, Tanzania
Time’s Up!!!

5. Where are the trainers from?
A Germany, USA, Tanzania, Ghana
B Ghana, Tanzania, Ecuador, Italy, USA
C India, USA, Ghana, Tanzania, Spain, Peru’
D Italy, Ghana, Peru’, Tanzania

6. Question 1

7. USD100,000

8. In impact evaluation, the
counterfactual is…
A The outcome of the control or comparison group
B The outcome of the treatment group at baseline
C The outcome of the treatment group in the absence
of the program
D The outcome of the control group had they
received treatment
Time’s Up!!!

9. In impact evaluation, the
counterfactual is…
A The outcome of the control or comparison group
B The outcome of the treatment group at baseline
CThe outcome of the treatment group in the absence
of the program
D The outcome of the control group had they received
treatment

10. Question 2

11. USD100,000

12. Spillover is when…
A You spill your tea onto the table
B The controls obtain the treatment
C The intervention affects the control group
D A different program affects the treatment
group
Time’s Up!!!

13. Spillover is when…
A You spill your tea onto the table
B The controls obtain the treatment
C The intervention affects the control group
D A different program affects the treatment
group

14. Question 3

15. USD100,000

16. Contamination is when…
A The drinking water is dirty
B The control obtains the treatment
C When those assigned to treatment in
practice do not receive it
D You cough without covering your mouth
Time’s Up!!!

17. Contamination is when…
A The drinking water is dirty
B The control obtains the treatment
C When those assigned to treatment in
practice do not receive it
D You cough without covering your mouth

18. Question 4

19. USD100,000

20. External validity is…
A When someone external to our organization tells you
that your impact estimate is valid
B The clothes you are wearing are appropriate for the
occasion
C The result is applicable to other settings
D The outside temperature is good for field
work
Time’s Up!!!

21. External validity is…
A When someone external to our organization tells you
that your impact estimate is valid
B The clothes you are wearing are appropriate for the
occasion
C The result is applicable to other settings
D The outside temperature is good for field
work

22. Question 5

23. USD200,000

24. In oversubscription randomization, intervention
is given only to a subset of eligible participants
because…
A This approach ensures that a valid control group is
present
B It is a common knowledge that not everybody takes
the intervention even when it is offered
C Programs usually do not have enough resources to
provide intervention to all eligible participants immediately
D People who apply late do not deserve the program in
the first place
Time’s Up!!!

25. In oversubscription randomization, intervention
is given only to a subset of eligible participants
because…
A This approach ensures that a valid control group is
present
B It is a common knowledge that not everybody takes
the intervention even when it is offered
C Programs usually do not have enough resources to
provide intervention to all eligible participants immediately
D People who apply late do not deserve the program
in the first place

26. Question 6

27. USD200,000

28. In propensity score matching we assume that…
A Program participation is determined by observables
B Program participation is determined by unobservable
C The region of common support is increasing as the
propensity score increase
D The likelihood of participating in the program increases
as the score decreases
Time’s Up!!!

29. In propensity score matching we assume that…
A Program participation is determined by observables
B Program participation is determined by unobservable
C The region of common support is increasing as the
propensity score increase
D The likelihood of participating in the program increases
as the score decreases

30. Question 7

31. USD200,000

32. The balancing property in PSM ensures
that…
A Sample observations of participants and nonparticipants
are balanced in some predefined way
B Allocation of project resources is balanced in different
stages of the projects
C Means of control variables are the same for participants
and nonparticipants whose propensity scores are close
D The sample sizes in the treatment and comparison
groups are equal
Time’s Up!!!

33. The balancing property in PSM ensures
that…
A Sample observations of participants and nonparticipants
are balanced in some predefined way
B Allocation of project resources is balanced in different
stages of the projects
C Means of control variables are the same for participants
and nonparticipants whose propensity scores are close
D The sample sizes in the treatment and comparison
groups are equal

34. Question 8

35. USD200,000

36. When can we use Regression
Discontinuity Analysis (RDA)?
A When you get married
B When the slopes of the functional for are linear
C When there is a discontinuity between outcome and
score/rating variable
D When the cut-off point is exogenous
Time’s Up!!!

37. When can we use Regression
Discontinuity Analysis (RDA)?
A When you get married
B When the slopes of the functional for are linear
C When there is a discontinuity between outcome and
score/rating variable
D When the cut-off point is exogenous

38. Question 9

39. USD400,000

40. “Timing of effects” is a potential threat to validity
that refers to…
A The year when the program started
B The year when you conduct the evaluation
C The time needed to run an impact model in
Stata
D The time needed for the intervention to
affect the outcome
Time’s Up!!!

41. “Timing of effects” is a potential threat to validity
that refers to…
A The year when the program started
B The year when you conduct the evaluation
C The time needed to run an impact model in
Stata
D The time needed for the intervention to
affect the outcome

42. Question 10

43. USD400,000

44. The main reason for using a longitudinal data
model in impact evaluation is…
A To use data the program has been accumulating in its
routine information system over time
B Because it is more sexy
C To deal with potential endogeneity of the program
variable
D To increase the precision of your impact estimates
Time’s Up!!!

45. The main reason for using a longitudinal data
model in impact evaluation is…
A To use data the program has been accumulating in its
routine information system over time
B Because it is more sexy
C To deal with potential endogeneity of the program
variable
D To increase the precision of your impact estimates

46. Question 11

47. USD400,000

48. A difference-in-differences with fixed
effects model is useful to address…
A Endogeneity bias that comes from large standard
errors
B Endogeneity bias that comes from unobserved
factors that changed between baseline and endline
C Lack of precision that comes from small sample size
D Endogeneity bias that comes from unobserved
factors that do not change between baseline and
endline
Time’s Up!!!

49. A difference-in-differences with fixed
effects model is useful to address…
A Endogeneity bias that comes from large standard
errors
B Endogeneity bias that comes from unobserved
factors that changed between baseline and endline
C Lack of precision that comes from small sample size
D Endogeneity bias that comes from unobserved
factors that do not change between baseline and
endline

50. Question 12

51. USD400,000

52. The design effect due to clustering is
given by: DE= 1+ρ(m-1).
What does the m in the design effect
equation represent?
A The total number of cluster
B The average cluster size
C The Intra-cluster correlation coefficient
D The minimum detectable difference
Time’s Up!!!

53. The design effect due to clustering is
given by: DE= 1+ρ(m-1).
What does the m in the design effect
equation represent?
A The total number of cluster
B The average cluster size
C The Intra-cluster correlation coefficient
D The minimum detectable difference

54. Question 13

55. USD800,000

56. Which of these is NOT a criticism of RCTs?
A They are expensive to execute
B They often lack external validity
C They are unethical
D They cannot be easily understood
Time’s Up!!!

57. Which of these is NOT a criticism of RCTs?
A They are expensive to execute
B They often lack external validity
C They are unethical
D They cannot be easily understood

58. Question 14

59. USD800,000

60. A 50-50 split of treatment and control is
recommended because…
A It is the most ethical design
B It reduces the bias of the treatment effect
C It reduces the variance of the treatment
effect
D It mitigates the effects of cluster-based
sampling
Time’s Up!!!

61. A 50-50 split of treatment and control is
recommended because…
A It is the most ethical design
B It reduces the bias of the treatment effect
C It reduces the variance of the treatment
effect
D It mitigates the effects of cluster-based
sampling

62. Question 15

63. USD800,000

64. Adding covariates (additional X variables)
to your regression…
A Can improve the efficiency (variance) of your
estimate
B Can reduce the bias of your estimate if X is
correlated with T
C Can increase the bias of your estimate if T affects X
D All of the above
Time’s Up!!!

65. Adding covariates (additional X variables)
to your regression…
A Can improve the efficiency (variance) of your
estimate
B Can reduce the bias of your estimate if X is
uncorrelated with T
C Can increase the bias of your estimate if T affects X
D All of the above

66. Question 16

67. USD800,000

68. What is the meaning of this expression:
E(Y1/P=1) – E(Y0/P=1) ?
A The selection bias
B The counterfactual
C The ATT
D A measure of endogeneity
Time’s Up!!!

69. What is the meaning of this expression:
E(Y1/P=1) – E(Y0/P=1) ?
A The selection bias
B The counterfactual
C The ATT
D A measure of endogeneity

70. JACKPOT

71. USD1,000,000

72. What is the meaning of this expression:
E(Y0/P=0) – E(Y0/P=1) ?
A The selection bias
B The counterfactual
C The ATT
D A measure of endogeneity
Time’s Up!!!

73. What is the meaning of this expression:
E(Y0/P=0) – E(Y0/P=1) ?
A The selection bias
B The counterfactual
C The ATT
D A measure of endogeneity

74. And the winner is:
Drum roll. . .