The document discusses the importance of evidence-informed policy making in the public sector through experimentation. It outlines the stages of experimentation, conditions for when to experiment, and the rationale behind it, emphasizing the need for careful framing and evaluation. Additionally, it addresses the challenges and risks associated with policy experiments, such as organizational culture and potential bias.

1. EXPERIMENTS IN THE
PUBLIC SECTOR:
GETTING THE BASICS
RIGHT
Piret Tõnurist
Innovation Specialist, Observatory
of Public Sector Innovation, OECD
June 27, 2017

2. ADAPTIVE POLICY MAKING: WAITING OR GOING?
27-June-2017 Governing better through evidence-informed policy making 2
 What is evidence under uncertainty?
 Golden decision rule in PA: careful attention to detail,
‘good’ professional or well-considered judgment
 Accurate versus complex decisions – ‘good’ decisions
vary greatly
 Don’t be paralyzed by uncertainty!

3. WHAT ARE EXPERIMENTS?
27-June-2017 Governing better through evidence-informed policy making 3
 Procedures to support, refute, or validate a hypothesis usually by creating
an intervention which is observed, measured and evaluated
Aerial view of the Hawthorne Works, ca. 1925.

4. PROCESS OF EXPERIMENTATION
27-June-2017 Governing better through evidence-informed policy making 4
 Frame the problem
 Identify the intended goals
 Identify possible solutions
 Set an hypothesis
 Select a method
 Define the levels of success
 Test
 Review and communicate
results
 Make a decision: pull out, re-
test or scale?
 Launch
 Monitor and evaluate

5. WHEN TO EXPERIMENT?
27-June-2017 Governing better through evidence-informed policy making 5
 When you don’t know the right answer
 If you know, do! Don’t get caught in the call
for action and ignore well-grounded prior
evidence!
 What is the source of uncertainty
(substantive, strategic)?
 When there is room for act
 When proof-of-concept can lead to broader
positive changeDon’t love the method, love the problem!

6. WHY EXPERIMENT?
27-June-2017 Governing better through evidence-informed policy making 6
 A ‘controlled small-scale’
 A ‘phase-wise policy design tools’
 ‘Variation promoters’
 ‘Alternative evaluation modes’
 ‘Looking glasses to the future’
 ‘Adaption devices’ or ‘learning
accelerators’

7. GO-TO THINGS WHEN PREPARING EXPERIMENTS
27-June-2017 Governing better through evidence-informed policy making 7
 Randomized assignment of subjects whenever possible
 Size of the sample
 Time-frame of the experiment
 Are control conditions needed and what they should be
 Examine limits to causality and indirect effects
 Examine external validity
 What are the success factors (when to stop, re-test or scale)
 Transparent publication of results
 What are the next steps after an experiment

8. MAIN CHALLENGES: DON’T HIT THE TARGET, BUT MISS
THE POINT!
27-June-2017 Governing better through evidence-informed policy making 8
 Underlining cultural, organizational shift
 Financial risks & future discounting
 How many experiments are allowed to be scaled up?
 Politics of policy experiments (donor influence, accepting uncertainty,
reputational stake, risk of manipulation, institutional barriers etc.)
 Evaluation: attribution and causality
 Bias towards prototyping
 Obtaining ‘evidential support’ for pre-determined policy solutions
 Diffusion: ‘craft rather than science’
 Ethics of experimentation

9. THANK YOU FOR
THE ATTENTION!
LETS
EXPERIMENT!
27-June-2017 Governing better through evidence-informed policy making 9