Donna Smith-Moncrieffe presented on innovative synthesis methodologies at the Canadian Evaluation Society Conference. She discussed the National Crime Prevention Centre's need to synthesize data from 13 crime prevention programs with different target groups, interventions, and outcomes. She described using elements of realist synthesis and multiple case study synthesis to analyze data across the programs. Key steps included identifying questions, inclusion criteria, entering data, analysis, and reporting aggregate results. The presentation concluded that these synthesis methods were well-suited for combining non-standard data and that additional planning is needed to fully apply realist and case study approaches.

1. Innovative Synthesis Methodologies
Donna Smith-Moncrieffe
Canadian Evaluation Society Conference
Fairmont Royal York, Toronto
June 9-12, 2013

2. 2
Summary of the Presentation
 NCPC Mandate
 Government Context
 Evaluation considerations at the corporate level (Performance)
 Evaluation considerations at the project’s level (Knowledge products)
 Overview of 3 Types of synthesis methods
 Key points
 Challenges
 Case Study (Step by Step Process)
 Collating data from projects with different target groups, interventions and outcomes
 Synthesis tables
 Providing reporting options for summarizing various statistical results: percentages,
Effect Sizes, T tests and F coefficients
 Conclusions

3. 3

4. Government Context: Evaluation Approaches
4

5. Why did we need to Synthesize the Data?
● We needed to respond to the questions about, “what works in crime
prevention”
● We needed to provide findings and conclusive statements about 13
programs (total of 36 project sites) with different:
- Target groups;
- Interventions;
- Outcomes; and
- Evaluation designs
● Single site repeated design
● Quasi-experiment design (sometimes with a matched
comparison group)
5

6. Types of Synthesis Methods
6

7. Synthesis Method Requirements for quality
and Rigor
7

8. What approach did we use to Synthesize 13
Evaluations?
8

9. Step 1: Identify Key Questions
What behaviour related changes were made (i.e. changes in violent
offending, police contact and victimization?
­ What percentage of projects made favorable changes in relation to
the criminal justice system (i.e. reduced violent offending, police
contacts etc..)?
­ How much change was made?
­ What contextual and program related factors contributed to these
changes (i.e. geographical location, partnerships, resources etc..)?
9

10. Step 2: What are the inclusion criteria?
● In the Government context, we need to use as many evaluation studies
as possible in the synthesis; even when they do not meet rigid
evaluation standards.
● Evaluation studies that met the following minimum requirements were
included in the synthesis:
­ Study uses quantitative data (descriptive or inferential statistics);
­ Measures that correspond with key crime related outcomes (i.e.
association with anti­social peers, emotional regulation, substance
abuse, offending, education, employment etc..);
­ Study used at least the basic evaluation design (single group pre­
post test repeated measures design); and
­ At least pre and post test measures had been available
10

11. Step 3: Enter the Data (Partial Sample)
Name of
Study
Description
of Variable
Sample
Size
Result P Level Type of
Change
Level of
Study
Project
Intervention
Toronto
Arrests E=76
C=43
F=0.291 P=.823 No
Change
Exemplary
Multi-
Systemic
Therapy
Youth not
arrested for
an offence
E=28
C=12
E=89.3%
C=66.7%
N/A Favorable Basic
Towards No
Drugs
Weapon
Carrying
E=847
C=54
F=0.63 P<0.01 Favorable Notable
Velocity Police
contacts
E=87 69%
(reduction)
N/A Favorable Notable
Interpretation of Results
11

12. Step 4: Analysis and Interpretation
Synthesis for 9 evaluation studies measuring changes in criminal
behaviours
Overall interpretation
Background Information
•Number of evaluation studies: 9
•Number of sites: 11
•Common measures include: 1) Arrests; 2) Non-Violent Offending; 3) Violent
offending; 4) Criminal Victimization; and 5)Weapon Carrying
Quantitative results
•% of change by sub-measure: 50%
•% of projects demonstrating favorable change (showing at least one stat.
significant result at the project level: 75%
•% of projects demonstrating unfavorable change:0%
•% of projects demonstrating no change: 25%
Qualitative results
•Provide qualitative results by model, site or project level to qualify the quantitative
findings
12

13. Step 5 : Reporting Aggregate Data
13

14. Step 5: Reporting Aggregate Data
(Sample of Approach # 1)
14

15. Step 5: Reporting (Sample of Approach #2
reporting on magnitude of change )
15

16. Conclusions
● What synthesis method works best when the evaluation studies have a
variety of interventions, outcomes and statistical measures?
­ Elements of the Realist Synthesis review and Multiple­case Study
Synthesis are recommended; and
­ NCPC utilized elements of both of these synthesis methods but could
benefit from responding to the Why questions incorporated in the
Multiple­ Case Study Synthesis
● What reporting method works best to summarize non-standard data?
­ Approach #1 and # 2 are both useful; and
­ Assess your audience to determine the best reporting method
16

17. Conclusions cont’d
● What planning activities would be required to utilize both Realist and
Multiple Case study Methods?
­ Develop strategic direction (strategic plan to help focus the direction of
the evaluation);
­ Prioritize key areas to minimize complexities;
­ Ensure that the information management system is designed to:
● Clarify the unit of analysis (by model, by project, evaluation
questions, etc..);
● Respond to How and Why questions such as: How does the
program produce these effects or How does the intervention
contribute to change?; and
● Collect data about the context for each model, site or project.
17

18. References for Recommended Synthesis
Methods
18

19. 19
Thank You
Contact:
Donna Smith­Moncrieffe, BSc. Crim Dip, MSc.
Senior Evaluation Advisor
Public Safety Canada, National Crime Prevention Centre
Policy, Research and Evaluation
Tel: 416­952­0423
Fax: 416­952­0483
Email: donna.smith­moncrieffe@ps­sp.gc.ca
Web : www.PublicSafety.gc.ca | www.SecuritePublique.gc.ca