The presentation was delivered at the Canadian Evaluation Society Conference in Ottawa Canada. Using rigorous evaluations in a multi-site framework, Pawson & Tilley's realist approach is tested using Canadian evalutions of funded crime prevention projects.

1. The Feasibility of the Realist Approach with
Canadian Evaluations
Donna Smith-Moncrieffe
Canadian Evaluation Society
Ottawa Convention Centre
June 17,2014

2. 2
Summary of the Presentation
1. Public Safety: Crime Prevention Evaluation Research
2. Realist Approach Using Multi-site evaluations
1. Using Multi-site theories to assess the feasibility of the Realist Approach
3. Case Study 1
● Assessing the Critical Elements in the Theory of Change
● Assessing between-site differences
● What Works for Whom and in What Circumstances?
4. Case Study 2
● Understanding the Context
● Outcome Assessment
● Utilizing New Theory to Strengthen the Intervention
4. Summary/Recommendations/Discussion

3. 3

4. Realist Approach using multi-site evaluations
4

5. Case Study 1: Stop Now and Plan®
Assessing the critical elements in the Theory of Change
5

6. Case Study 1 Cont’d:
Assessing Between-site Differences
Key elements Site 1 Toronto Site 2 Edmonton Site 2
Cree, Quebec
Effect sizes related
to emotional
regulation
(aggression)
Moderate to High
(0.54-1.17 effect
size)
Method: Quasi-
experiment with
comparison group
Moderate to High
(0.4-0.61 effect
size)
Method: Quasi-
experiment with
comparison group
Statistical
Assumptions not
met
Method: Single site
longitudinal analysis
Fidelity rating High High Low
Contextual
differences
• Project
implemented as
per the model
• Project
implemented as
per the model
• Literacy in
English
• Parental
engagement is
low
• Logistical
Challenges
• Cultural
differences 6

7. Case Study 1 cont’d :
What Works for Whom and it What Circumstances?
● In this case we identified Site 3 as not reaching the typical reductions in aggression;
● Questions about what worked for whom and in what circumstances were
answered via assessing the between-site differences
- Aboriginal youth who reside in rural communities will have more difficulty
completing the required number of cognitive therapy sessions due to distance
between the home and the program location;
- These Aboriginal youth will have more difficulty understanding the content if their
first language (Cree) and cultural norms are not adapted in their curriculum;
- The parents of Aboriginal youth at risk who reside in rural communities have a
higher rate of concern related to stigmas and therefore their involvement in
parental groups are significantly reduced;
These three key variables in the Cree site affected the achievement of typical
outcomes. These confounding variables were not present in the other two sites.
7

8. Case Study 2: Alternative Suspension Program
Step 1: Understanding the Context
Examining the Context Description
What is the aim of the program? To reduce crime by :
•Decreasing school disciplinary action
•Increasing school attachment
•Decreasing antisocial behavior
What is the nature of the target
population?
• Youth between 12-17 years who are at risk
of being suspended or have been
suspended
• Persistent behavior challenges
• Difficulties in academic and social life
In what context and settings does it
operate?
• Urban setting
• High deprivation, high dropout rates,
juvenile delinquency and criminality
• At-risk communities (low education rate,
high unemployment rate, high proportions of
single-parent families and of low-income
household
What are the prevailing theories
about how school suspensions
reduce crime?
• Drop out of school is a pathway to
delinquency and criminality
• Anti-social behavior in school has been
8

9. Case Study 2 cont’d:
Step 2: Outcome Assessment
● Interim school data
results (2012- 2013 )
indicate that
although the three
sites have delivered
the same program,
Edmonton has had
greater impact on
key variables;
● The original theory
works best for
Edmonton and not
the other sites;
● Examine what works
for whom in Step 3
Variable Site 1
Edmonton
Site 2
Moncton
Site 3
Chilliwack
Disciplinary
actions
71.4% decrease 45.3%
decrease
60.5%
decrease
Improvement
in School
Behaviour
(end of year)
73.5 %
improvement
45.6 %
improvement
68.3%
Improvement
9

10. Case Study 2 cont’d
Step 3: Utilizing New Theory to Strengthen the Intervention
● What are the elements that may be contributing to efficacy in one site and not
another?
● At the interim stage of the evaluation, the ANOVA results indicate that the
following variables may be contributing to between-site differences:
- Previous suspension history varies by site with Edmonton having
relatively lower rates of prior school suspensions than Moncton. Further
analysis is required to determine if this variable is contributing to outcome
differences
- School act legislation enforced in one site
● The Edmonton site was limited to YMCA activities only during the
suspension period. Participants in the two other sites had access to a
variety of other complementary activities outside of the YMCA.
● During the final two years of this longitudinal evaluation, these variables will
need to be addressed and then tested via multivariate analyses to determine
what elements enhance efficacy for the participants in their unique contexts
10

11. Summary
● Realist Theory is feasible and useful in a Canadian context:
- Prioritize evaluation questions that focus on the context;
- Examine the elements in the theory of change;
- Use a multi-site framework where possible to examine between-site
differences;
- Modify elements of the program or develop a new theory of change to
enhance favorable change for program participants operating in unique
contexts; and
● The approaches used in Case Study 1 and 2 can be used retrospectively or
can be used at the interim evaluation stage to apply modifications and
enhance outcomes
11

12. Recommendations
● Use program models with standardized content to facilitate the
identification of what works for whom and in what circumstances;
● Use mixed model techniques (multivariate analysis and qualitative
methods to isolate the between-site differences and the elements that
are contributing to variation in the original theory of change;
● Develop detailed theories of change and consider modifying theories
during long term evaluations to ensure that the appropriate context is
considered in the “evidence chain”;
● Ensure that stakeholders understand that the context, and theory of
change in each of the sites are as important as the quantitative
outcomes;
● Develop and share new theories of change for each of the sites to
clarify what works, for whom in each site. 12

13. Discussion
● How will we advance the Realist Approach in Canada?
● How will we ensure that evaluators use innovation and rigor to build
and rebuild theories of change that take into consideration the context
of each intervention?
● Who will facilitate and support the development and exchange of
information that will ensure this approach is integrated into evaluation
plans?
13

14. 14
Thank You
References
Astwood Strategy (2014) Impact Evaluation Final Report, Public Safety Canada
Laliberté, D. (2014). Alternative Suspension Impact Evaluation Summary, 2013-2014 Updates, Public Safety,
to be published.
Pawson R., & Tilley, N. (2006) Realist Evaluation. Development Policy Review Network Thematic Meeting, Re
Report on Evaluation , Amsterdam, Netherlands
R.A. Malatest & Associates Ltd (2014). Multi-site Evaluation of the Alternative Suspension Program. Second
Annual Report, Submitted by Glenda Shaw-Garlock, G. to Danièle Laliberté, Senior Evaluation Analyst.
Smith-Moncrieffe, D (2013) Stop Now and Plan: Impact Evaluation Summary 2013-2014, Public Safety
Canada
Treasury Board of Canada, Theory-Based Approaches to Evaluation: Concepts and Practices , Published by the
Government of Canada
Special thanks to evaluation team members: Donna Smith-Moncrieffe, Danièle Laliberté and Giselle Rosario for
compiling evaluation data for analysis and use for this presentation