Presentation of Explanatory Evaluation at the Barcelona meeting of the EEE-project: Orchestrating

1. Some words on evaluating
(technological?) interventions
(EE of EEE?)
Henk Sligte
Kohnstamm Institute for Educational Research
University of Amsterdam
Henk.Sligte@UvA.NL
http://kohnstamminstituut.uva.nl/htm/english.htm

2. Evaluation
• Evidence based policy and practice: effect
(impact) evaluation  what works?
• The other side of the medal: explanatory
evaluation: what works for whom in what
circumstances and why? Finer granulanity
• Assignment of the Dutch Ministry of
Education to develop a method to
complement impact evaluation
• Based on policy interventions, can it work
for (technological) innovations, for EEE?

3. Effect (Impact) evaluation
• Through (quasi)experiments demonstration of
causal relation between intervention and
found effects
– Experiment: Controlled Randomized Trials
– Quasi: natural experiment
– Difference-in-difference: compare with business as
usual (the whole population minus the
experimental)
Pretest-posttest model
Compare with a similar control or reference groups
Effects can exclusively be ascribed to intervention

4. BUT:
• The rationale (the why) for these effects
remains unknown: black box
• Explanatory evaluation focuses specifically
on this and can further validate the results
of an impact assessment (effect evaluation)
• Questions addressed:
– How and why does an intervention work?
– If there is no or a smaller effect occurs than
expected, why is that the case?
– Which unwanted / unintended effects occur (also
unexpected positive)?

5. Intervention
• VM2
– School drop-out in first year vocational
education enormous
– Intervention: combine systems of preparatory
and normal vocational education
– Effects: compare experimental group with
population as a whole on numbers of dropouts
– Conclusion: less dropout… “in xx percentpoint
of the cases a significant lower chance of
dropout”…
– How do find more differential effects and how to
understand them better?

6. Explanatory evaluation (EE)
• Pawson, R. & Tilley, N. (1997). Realistic Evaluation.
London: SAGE Publications
• What works for whom in what circumstances and
why?
• Ex ante (before), ex durante (during), ex post (after)
• Central to answering this question is the
reconstruction of the policy (or program) theory
• The policy theory is the sum of assumptions or
hypotheses that explain how the policy should work
These can be represented in statements like:
• If (intervention) ... Then ... (Outcome), and in some
cases also conditions But (take care that…)

7. EE: contexts
• In the reconstruction of the theory, it is important to
have an eye for the conditions and factors that play a
role in a variety of contexts.
• The context (or conditions) can be physical, but also
social, for example, certain groups or characteristics
of target groups (an intervention works only for
motivated students).
• Attention to circumstances helps explain why an
intervention works better in one case than in the other
case.
• The intervention works through certain mechanisms,
mechanisms that make "things work" and can be seen
as the driving force behind an intervention.

8. EE
• Creation of concrete causal schemes:
Context-Mechanism-Outcome schemes
– Problem mechanism
– Intervention that generates Change Mechanism(s)

9. EE: field studies
• Test the theory, test the hypotheses
• Enter the field studies. The researcher must
the different groups of actors in the picture.
• Consider what information to whom can be
achieved, what should be asked from whom:
who knows what?
• Formulate relevant questions for each (type
of) informant to answer and think about the
most appropriate method for each type
involved persons.

10. Field studies
• Reconstruction and comparison with the
policy theory takes place in a learning
dialogue at different levels.
• Types (levels) of respondents:
– Policy implementers (e.g. project leaders)
– Intermediaries (e.g. teachers)
– Target group (e.g. students, elderly, etc)
• Researchers: open attitude but focus on
understanding and consensual knowledge
(do we agree that this is what really
happened?)

11. EE: Field studies
• Each actor from his own role in the
operation of the policy can reflect on
assumed mechanisms and give an
explanation of (un) planned and (un)
desired effects.
• The question to the audience itself is
– whether the assumptions indeed work for them
– are assumptions about their behavior correct,
– what side effects they face.
• Cyclical process: stop rules...

12. EE: Critical evaluation
• Here the explanations are found why the
policy intervention has or has not the desired
effects, the goal of an explanatory evaluation.
• If the effects of an intervention are not as
positive as expected, the researcher may ask
the following questions:
– Is the theory or policy measure(s) plausible?
– Was the theory sufficiently differentiated?
– Was implementation successful?
– Were necessary conditions met?
– What should be modified?
• Learning effects for policy makers…

13. Effects of EEE??
• What is the intervention? Is it similar to
policy interventions?
• How to measure effects? Pre-test/post-test?
• Compare with reference/control groups?
• What do you measure?
– Knowledge, skills, attitudes, higher order skills
(reflection, learn to learn, learn to see activities
as learning), motivation?
– Other, new things?

14. EE of EEE?
• What is the theory behind EEE?
• Reconstruct the assumptions
• Distinguish assumptions at levels:
– The ideas, the theories
– The structures and systems developed and used
– The anticipated effects on various groups of
actors
• Do field research
• Evaluate whether your assumptions are
shared with various actors: test the
hypotheses

15. Generic interview scheme
• THEN – NOW – LATER
• Contexts-Mechanisms-Outcomes
• Start with NOW
• What Outcomes realised? Differentiate (level
of ideas-theories, structures-systems,
behaviour)
• THEN: What Processes (mechanisms) caused
the Outcomes?
• What crucial success & failure factors?
• LATER: what new/adjusted outcomes?
• How to achieve the outcomes?