1. Randomized Social Experiments
Marc Gurgand
Paris School of Economics and J-PAL
OECD - May 19, 2014

2. Randomized experiments in three sentences

3. Randomized experiments in three sentences
Evaluate the impact of social policies on measurable outcomes
Compare treatment and control goup
Randomize status to make the two groups comparable

4. Influential experiments
Class size : STAR (US 1985)
Welfare : Self-sufficiency project (Canada, 1994)
Local environment : Moving to Opportunity (US, 1994)
Conditional Cash Transfer : Progressa (Mexico, 1997)
Renewal : Development (Esther Duflo), Europe (Fonds
d’exp´eriementation jeunesse, Education Endowment Fundation,
Trygfonden/Aarhus, ...)

5. Methodology in two slides

6. Counterfactual and causal effects

7. Counterfactual and causal effects
Objective : measure the causal effect of some intervention (ex.
class size reduction) on some outcomes (ex. test scores)

8. Counterfactual and causal effects
Objective : measure the causal effect of some intervention (ex.
class size reduction) on some outcomes (ex. test scores)
Counterfactual
What would have happened (to a given individual)
had he/she not been subject to the intervention

9. Counterfactual and causal effects
Objective : measure the causal effect of some intervention (ex.
class size reduction) on some outcomes (ex. test scores)
Counterfactual
What would have happened (to a given individual)
had he/she not been subject to the intervention
Causal impact defined (and measured) by the difference between
outcomes in the obseved state and outcomes in the counterfactual
state

10. Randomized experiments

11. Randomized experiments
Objective : build-up counterfactuals (“same individual”)

12. Randomized experiments
Objective : build-up counterfactuals (“same individual”)
Treament and control group randomized
The outcome in one group is a counterfactual for the other group
Comparison of the two outcomes measures causal impact

13. Randomized experiments
Objective : build-up counterfactuals (“same individual”)
Treament and control group randomized
The outcome in one group is a counterfactual for the other group
Comparison of the two outcomes measures causal impact
Any systematic difference between the two groups must result form
the intervention

14. Randomized experiments
Objective : build-up counterfactuals (“same individual”)
Treament and control group randomized
The outcome in one group is a counterfactual for the other group
Comparison of the two outcomes measures causal impact
Any systematic difference between the two groups must result form
the intervention
Literature moves away from simple 0/1 experiments towards more
complex designs that inform policy more precisely

15. Two examples in labor policy

16. Example : Counseling the unemployed

17. Example : Counseling the unemployed
Important and costly active policy

18. Example : Counseling the unemployed
Important and costly active policy
Recent trend in subcontracting to private operators

19. Example : Counseling the unemployed
Important and costly active policy
Recent trend in subcontracting to private operators
Comparing counseled and not counseled ; or before and after
implementation of counseling : does not make sense.

20. Experiment 1

21. Experiment 1
200 PES offices in France, 44,000 unemployed in the target

22. Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :

23. Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counseling

24. Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counseling
Intensive private provider counseling

25. Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counseling
Intensive private provider counseling
standard PES track

26. Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counseling
Intensive private provider counseling
standard PES track
Mesure exit towards employment in the three groups

27. Intensive Counseling against standard track
After 6 months : entry rate increases from 20% to 30% with PES ;
from 20% to 24% with private providers
(source : Behaghel et al. 2014)

28. Displacement ?

29. Displacement ?
Two theories on overall impact of counseling :

30. Displacement ?
Two theories on overall impact of counseling :
1 Jobs are in given quantity and quickly filed : if one worker
accesses jobs faster, this is detrimental to another worker

31. Displacement ?
Two theories on overall impact of counseling :
1 Jobs are in given quantity and quickly filed : if one worker
accesses jobs faster, this is detrimental to another worker
2 Vacancies and unemployed workers coexist in large
proportion : counseling reduces frictions, moves the Beveridge
curve and can improve treated outcomes without decreasing
the untreated’s situation

32. Displacement ?
Two theories on overall impact of counseling :
1 Jobs are in given quantity and quickly filed : if one worker
accesses jobs faster, this is detrimental to another worker
2 Vacancies and unemployed workers coexist in large
proportion : counseling reduces frictions, moves the Beveridge
curve and can improve treated outcomes without decreasing
the untreated’s situation
The usual random experiment does not answer this question

33. Experiment 2

34. Experiment 2
235 PES offices interpreted as local labor markets

35. Experiment 2
235 PES offices interpreted as local labor markets
Randomize offices : in some no intensive counseling at all

36. Experiment 2
235 PES offices interpreted as local labor markets
Randomize offices : in some no intensive counseling at all
In others, randomize some unemployed that are offered
intensive counseling

37. Experiment 2
235 PES offices interpreted as local labor markets
Randomize offices : in some no intensive counseling at all
In others, randomize some unemployed that are offered
intensive counseling
11 000 workers followed during 20 months

38. Exit to jobs within 8 monthsn the treatment
ly to have found
s mostly driven
so driven by job
nd there was no
racts. However,
12 months and
no more likely
mparison group
The low overall impact of being offered inten
counseling is partly due to the relatively low take
figure 3: employment rates among treated a
untreated job seekers
40%
35%
30%
25%
Pure Com
parison Group
Com
parison Group in
Treated Areas
Treatm
ent Group
Pure Com
parison Group
Com
parison Group in
Treated Areas
Treatm
ent Grou
competitive job marketall job markets
Significantly different from the pure comparison group
oyment
(source : Crepon et al. 2013)

39. Lessons on job counseling

40. Lessons on job counseling
Efficient technology (but no long run effects)

41. Lessons on job counseling
Efficient technology (but no long run effects)
Private providers no more efficient (and less cost effective)
than PES (similar RCTs in Germany & Sweden)

42. Lessons on job counseling
Efficient technology (but no long run effects)
Private providers no more efficient (and less cost effective)
than PES (similar RCTs in Germany & Sweden)
Evidence in favor of displacement effects : benefit is around
zero

43. Lessons on job counseling
Efficient technology (but no long run effects)
Private providers no more efficient (and less cost effective)
than PES (similar RCTs in Germany & Sweden)
Evidence in favor of displacement effects : benefit is around
zero
Strong statements that require RCT to be credible
(but additional evidence is needed)