The document discusses challenges in evaluating the mechanisms and theories behind how interventions and programs work. It notes that while an intervention may predict an outcome like increased physical activity, understanding how and why it worked requires testing the underlying program theory against alternatives. Evaluating mechanisms of change is difficult due to sample size limitations and the complexity of interventions. Overfitting theories to limited data is also a problem. The document raises open questions about whether program theories can be treated like other scientific theories and subjected to replication, falsification, and cross-validation, and how best to study mechanisms of action in complex, real-world interventions.

1. www.mattiheino.com
How or why did my
intervention work?
Challenges in evaluating mechanisms and
program theories
@heinonmatti

2. www.mattiheino.com
What do you mean “it worked”?
• A predicted
change in a
pre-specified
outcome:
Physical activity

3. www.mattiheino.com
What do you mean “this is how it
worked”?
(Note: simplified logic model)

4. www.mattiheino.com
What do you mean “this is how it
worked”?
(Note: simplified logic model)
Program theory draws from…
• COM-B system
• Control Theory
• Theory of Planned Behaviour
• Self-Determination Theory
• (Goal Setting Theory
• Self-Efficacy Theory
• Self-Regulation Theory
• Social Cognitive Theory…)

5. www.mattiheino.com
Observations
Theory testing in science(s)
Statistical
hypothesis
Substantive theory
and assumptions
Substantive
hypothesis

6. www.mattiheino.com
Observations
Theory testing in science(s)
Statistical
hypothesis
Substantive theory
and assumptions
Substantive
hypothesis
The causal stream: each line
means “From this, it follows
that…” – deductive!

7. www.mattiheino.com
Observations
Theory testing in science(s)
Statistical
hypothesis
Substantive theory
and assumptions
Substantive
hypothesis
The causal stream: each line
means “From this, it follows
that…” – deductive!

8. www.mattiheino.com
Observations
Theory testing in science(s)
Statistical
hypothesis
Substantive theory
and assumptions
Substantive
hypothesis
The causal stream: each line
means “From this, it follows
that…” – deductive!
The inferential stream: each
line means “From this, what
can we conclude about…?”
– inductive!

9. www.mattiheino.com
Theory testing in science(s)
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions

10. www.mattiheino.com
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Program
theory holds

11. www.mattiheino.com
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Program
theory holds
Other theories
work in this
context

12. www.mattiheino.com
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Program
theory holds
Other theories
work in this
context
No systematic,
unmeasured
confounders

13. www.mattiheino.com
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Program
theory holds
Other theories
work in this
context
No systematic,
unmeasured
confounders
Accelerometers,
bioimpedance
devices etc.
measure what
they’re
supposed to

14. www.mattiheino.com
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Program
theory holds
Other theories
work in this
context
No systematic,
unmeasured
confounders
Accelerometers,
bioimpedance
devices etc.
measure what
they’re
supposed to
Intervention
delivered as
planned

15. www.mattiheino.com
Intervention  PA link tells us
something’s going on
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Intervention
Physical
activity

16. www.mattiheino.com
Intervention  PA link tells us
something’s going on
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Intervention
Physical
activity
… But only to the
extent, that increased
PA (in comparison to
controls) was a “damn
strange coincidence”
absent intervention.
(see Meehl 1990)

17. www.mattiheino.com
Intervention  PA link tells us
something’s going on
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Intervention
Physical
activity
“the only admissible positive
evidence for a theory
are the corpses of its rivals.”
‒ Lakatos (1978, on Popper’s solution)

18. www.mattiheino.com
Absence of link only tells us
something was wrong!
Theory
Auxiliary
theories
All else equal
(ceteris paribus)
Assumptions of
instruments
Experimental
conditions
Intervention
Physical
activity

19. www.mattiheino.com
Theory testing in science(s)
1. Replicate experiment to see that finding holds
• “we may say that a phenomenon is experimentally
demonstrable when we know how to conduct an
experiment which will rarely fail to give us a statistically
significant result.”
‒ Fisher (1947); see Mayo 2016
• Also, see Heino, Fried & LeBel (2017)

20. www.mattiheino.com
Theory testing in science(s)
1. Replicate experiment to see that finding holds
2. Investigate boundary conditions and mechanisms
of action

21. www.mattiheino.com
About those
mechanisms…

22. www.mattiheino.com
About those
mechanisms…
Intervention
Physical
activity
Autonomous
motivation
(See here for an additional
sample size problem)

23. www.mattiheino.com
About those
mechanisms…
Intervention
Physical
activity
Autonomous
motivation
- Spousal support?
- Media exposure?
- Habit?
- Past experiences?
- …

24. www.mattiheino.com
Self-efficacy
About those
mechanisms…
Intervention
Physical
activity
Autonomous
motivation
Intention
etc…

25. www.mattiheino.com
Overfitting

26. www.mattiheino.com
Overfitting
(2016) Link
Seemingly supported theories fail
to generalise to slightly different
situations, or other samples from
the same population!

27. www.mattiheino.com
Link“Attempts to corroborate
statistically significant
subgroup differences are
rare; when done, the
initially observed
subgroup differences are
not reproduced.”

28. www.mattiheino.com
Thanks! Summary and open
questions:
• Scientific knowledge grows by risky tests of theories;
replication and falsification. Currently, we do neither.
– Are program theories inherently different to other scientific
theories?

29. www.mattiheino.com
Thanks! Summary and open
questions:
• Scientific knowledge grows by risky tests of theories;
replication and falsification. Currently, we do neither.
– Are program theories inherently different to other scientific
theories?
• Mechanisms are extremely difficult to study. Currently, we
claim mediation but remain ignorant about causation.
– (How) should we study mechanisms of action in complex
interventions for complex systems?

30. www.mattiheino.com
Thanks! Summary and open
questions:
• Scientific knowledge grows by risky tests of theories;
replication and falsification. Currently, we do neither.
– Are program theories inherently different to other scientific
theories?
• Mechanisms are extremely difficult to study. Currently, we
claim mediation but remain ignorant about causation.
– (How) should we study mechanisms of action in complex
interventions for complex systems?
• Overfitting is a problem, though
cross-validation can help. Currently,
sample sizes remain inadequate.
– How to make models simple
enough, but not simpler?

31. www.mattiheino.com
Bonus
slide
I’d be happy to have
a conversation on
how all this applies
to real life decision
making!

32. www.mattiheino.com
Bonus
slide 2
Link
Link
How do we figure out,
which pieces of
information truly count,
so that we can ditch the
misleading ones?