Presentation of the methodology for Explanatory Evaluation at EAPRIL2012

1. Explanatory Evaluation:
how to better understand effects
of interventions
Henk Sligte
Kohnstamm Institute for Educational Research
University of Amsterdam
Henk.Sligte@UvA.NL
http://kohnstamminstituut.uva.nl/htm/english.htm

2. Research for
Dutch Ministry of Education
• Report (in Dutch)
• Pater, C., Sligte, H., van Eck, E
(2012). Verklarende evaluatie,
een methodiek. Amsterdam:
Kohnstamm Instituut
• Explanatory Evaluation, a
methodology for policy
evaluation
http://www.kohnstamminstituut.uva.nl/rapporten/pdf/ki882.pdf

3. Evaluation
• Evidence based policy (and practice): effect
(impact) evaluation  what works?
• Dutch Ministry: we need explanations for
effects we need a methodology to
complement impact evaluation
• The other side of the medal: explanatory
evaluation what works for whom in what
circumstances? How and why does the
policy work (or not)? Finer granularity is
needed
• In future: both types to be applied

4. Tentative question
here and now:
If the research
method works in
explaining effects of
policy interventions,
can it then also work
in explaining
interventions and
innovations in
education, in schools?

5. Effect (Impact) evaluation
• Through (quasi)experiments demonstration of
causal relation between intervention and
found effects
– Experiment: Controlled Randomized Trials
– Quasi: natural experiment
– Difference-in-difference: compare with business as
usual (the whole population minus the
experimental)
Pretest-posttest model
Compare the experimental groups with similar
control or reference groups
Effects can exclusively be ascribed to intervention

6. BUT:
• The rationale (the why) for these effects
remains unknown: black box
• Explanatory evaluation focuses specifically
on this and can further validate the results
of an impact assessment (effect evaluation)
• Questions addressed:
– How and why does an intervention work?
– If there is no or a smaller effect than expected,
why is that the case?
– Which unwanted / unintended effects occur (also
unexpected positive)?

7. The development of
the methodology
• Literature study, especially Pawson, R. &
Tilley, N. (1997). Realistic Evaluation.
London: SAGE Publications
• Study of two Dutch policy interventions in
the area of early school leaving
– VM2: prevent drop-out
– De Wijkschool: stimulate drop-outs because of
multiple problems (drugs, criminality, debts,
broken families, etc) to go back to school

8. Effects of VM2
• VM2
– School drop-out in first year vocational
education enormous
– Intervention: combine systems of preparatory
and regular vocational education
– Effects: compare experimental group with
population as a whole on numbers of dropouts
– Conclusion: less dropout… “in xx percentpoint
of the cases a significant lower rate of dropouts
was the result”…
– How to find more differential effects and how to
understand them better?

9. Explanatory evaluation (EE)
• Realistic Evaluation: get under the surface of the
direct observable...
• What works for whom in what circumstances and
how and why?
• Ex ante (before), ex durante (during), ex post (after)
• Central to answering this question is the
reconstruction of the policy (or program) theory
• The policy theory is the sum of assumptions or
hypotheses that explain how the policy should work
These can be represented in statements like:
• If (intervention) ... Then ... (Outcome), and in some
cases also conditions But (take care that…)

10. EE: contexts differ
• In the reconstruction of the theory, it is important to
have an eye for the conditions and factors that play a
role in a variety of contexts.
• The context (or conditions) can be physical, but also
social, for example, certain groups or characteristics
of target groups (an intervention works only for
motivated students).
• Attention to circumstances helps explain why an
intervention works better in one case than in the other
case.
• The intervention works through certain mechanisms,
mechanisms that make "things work" and can be seen
as the driving force behind an intervention.

11. EE
• Analysis of policy documents
• Creation of concrete causal generative schemes
(=hypotheses) on the basis of this analysis
• Context-Mechanism-Outcome (CMO) schemes
– Problem mechanisms, leading to unwanted outcomes or
results in (parts of the) society
– Intervention that generates Change Mechanism(s)
– Change mechanisms overcome problem mechanisms
– Better outcomes
• Check with staff members (at Ministry) that were
responsible for policy intervention
• Adjust CMO-schemes, make new CMO-schemes

13. EE: field studies
• Test the theory, test the hypotheses (CMO)
• Enter the field studies. The researcher must
have the different groups of actors in the
picture.
• Consider what information to whom can be
achieved, what should be asked from whom:
who knows what?
• Formulate relevant questions for each (type
of) informant to answer and think about the
most appropriate method for each type
involved persons.

14. Field studies
• Reconstruction and comparison with the
policy theory takes place in a learning
dialogue at different levels.
• Types (levels) of respondents:
– Policy implementers (e.g. project leaders)
– Intermediaries (e.g. teachers)
– Target group (e.g. students, elderly, etc)
• Researchers: open attitude but focus on
understanding and consensual knowledge
(do we agree that this is what really
happened?)

15. EE: Field studies
• Each actor from his own role in the
operation of the policy can reflect on
assumed mechanisms and give an
explanation of (un) planned and (un)
desired effects.
• Question to the dialogue partners is
– whether the assumptions indeed work for them
– are assumptions about their behaviour correct,
– what side effects they face.
• Cyclical process: stop rules...

16. EE: Critical evaluation
• Here the explanations are found why the
policy intervention has or has not the desired
effects, the goal of an explanatory evaluation.
• If the effects of an intervention are not as
positive as expected, the researcher may ask
the following questions:
– Is the theory or policy measure(s) plausible?
– Was the theory sufficiently differentiated?
– Was implementation successful?
– Were necessary conditions met?
– What should be modified?
• Learning effects for policy makers…

17. Can EE work for educational
practice?

18. R&D in education
• Development  interventions/innovations
• Do you measure effects (outcomes) and
how? Pre-test/post-test?
• Compare with reference/control groups?
• What do you measure?
– Knowledge, skills, attitudes, higher order skills
(reflection, learn to learn, learn to see activities
as learning), motivation? Other, new things?
• Find out what works: outcomes, effects
• Find out what works for whom in what
contexts and why

19. Explanatory evaluation
• What is the theory behind the Development?
• Reconstruct the assumptions (ifthen/CMO)
• Distinguish assumptions at levels:
– The ideas, the theories
– The structures and systems developed and used
– The anticipated effects on various groups of actors
• Do field research
• Evaluate whether your assumptions are shared
with various actors at different levels: test the
hypotheses

20. Generic interview scheme
• THEN – NOW – LATER
• Contexts-Mechanisms-Outcomes
• Start with NOW
• What Outcomes realised? Differentiate (level
of ideas-theories, structures-systems,
behaviour)
• THEN: What Hypotheses? What Processes
(mechanisms) caused the Outcomes?
• What crucial success & failure factors?
• LATER: what new/adjusted outcomes?
• How to achieve the outcomes?

21. 2) Go back to start of project 4) Go to end of project
3) What Hypotheses? 6) What Interventions
What Mechanisms and Mechanisms
Caused the Outcomes? are needed to achieve
Success and failure the Outcomes?
factors?
PAST PRESENT FUTURE
1) What Outcomes are now realized? 5) What Outcomes
a) ideas/theories are expected?
b) structures/new roles What to be realized?
c) the workfloor What new ideas/theories?
Differences in Contexts? What success and failure?

22. ????
??