Thank you for the presentation. I appreciate you sharing the evaluation challenges and mitigation strategies.

1. Improving Accountability and Quality Assurance
An Analysis of Interim Evaluations
of Gang Prevention Projects across Canada
Presented by Donna Smith-Moncrieffe, BSc. Crim Dip, MSc.
Canadian Evaluation Society Conference
May 31-Jun 3, 2009, Ottawa

2. Presentation Outline
 National Crime Prevention Centre (NCPC) mandate
 Youth Gang Prevention fund (YGPF) and accountability framework
 Improving accountability
 Corporate level (assessment at the department level)
 Branch level (using rigorous evaluation methods within
NCPC to attribute outcomes of interest to the program)
 Key Challenges/Lessons Learned and mitigation strategies in
conducting impact evaluations of gang prevention projects
 Conclusion : Link between strong evaluations and accountability
2

3. NCPC Mandate/Core Activities
Mission statement:
To provide national leadership on effective and cost-efficient ways to
both prevent and reduce crime by addressing known risk factors in
high risk populations and places
Core activities:
 supporting targeted interventions in local communities
 building and sharing practical knowledge with policy makers and
practitioners
3

4. NCPC priorities:
Provide funding to the following target groups/crime issues:
 Children and youth at risk
 Crime prevention in Aboriginal communities
 Prevent recidivism among high-risk groups
 Priority crime issues (youth gang, drug-related crimes)
4

5. Youth Gang Prevention Fund (YGPF)
 Fund operates between Oct 2006 and March 2011 (5 years)
 Funding for anti-gang initiatives is provided in areas where
gangs exist or are emerging with a focus on youth aged 9-24
 Specialized fund developed to ensure that Canadian best practices
in gang crime prevention can be disseminated
5

6. Current NCPC Gang Projects
 17 funded across Canada
 Gang projects distributed across seven provinces (higher
concentration in B.C, Prairies and Ontario)
 Range of funds per project ($750,000 to $4,900,000.00 over a
four year period)
 Evidence-based practices for gang prevention include
“wraparound”, case management, activities addressing multiple
risk factors, and use of gang risk assessment tools. See NCPC
website publications: NCPC Model and Promising Programs
(2008)
 Projects have been operating between 6 and 12 months
6

7. Current NCPC Gang Impact Evaluations
 Evaluations are contracted out to 3rd party evaluators
 Typical evaluation allocation is approximately $180,000.00 over
four years
 15/17 evaluations are funded through the contribution
agreement with the project; two are funded through independent
contracts with NCPC
 Project evaluations need to provide information for NCPC and
departmental performance reporting (DPR), in terms of both
outputs (e.g. # of projects, # of activities implemented), and
outcomes (e.g. changes in risk factors, reduced offending)
7

8. Improving Accountability: Corporate
 The YGPF Accountability and Audit Framework (ARAF)
provides detail on how to monitor, evaluate and report on results
 Reporting to Treasury Board :
 3 year Interim Progress Report (March, 2009)
 Five Year Outcome-based Evaluation (April, 2011)
 YGPF data feed into crime prevention performance indicators in
annual Departmental Performance Reports (DPRs) and
Strategic Reviews
8

9. Improving Accountability: Branch Level
 6 Senior evaluation analysts work in regions
Strengthening internal where they review and provide advice on
evaluation capacity evaluation deliverables
Evaluation frameworks, annual reports and final
Clear Reporting report due dates are specified in the contribution
Requirements agreement and contract
YGPF National Logic model specifies expected
Tool Development outcomes and activities
YGPF Evaluation Guidelines provide key
indicators, literature and evaluation designs to 3rd
party evaluation contractors
Specific tasks and requirements stated (i.e. using
Detailed Request for police, health and education sources to verify
Proposals and contracts participant change)
Clauses allowing for discontinuation of the
evaluation if deliverables are not satisfactory 9

10. Methods of Analysis (Branch level only):
Interim results
 15 annual evaluation reports (submitted January, 2009)
completed by 3rd party contracted evaluators submitted for
review to NCPC evaluation analysts
 6 NCPC evaluation analysts summarized annual reports and
identified key challenges in : 1. Evaluation methods; 2. Program
delivery and 3. Working effectively with 3rd party evaluation
contractors
 Author prioritized top 5 challenges and mitigation strategies
proposed by NCPC evaluation analysts to increase knowledge
dissemination and improve the final stages of these gang
evaluations.
10

11. YGPF Evaluation Challenges and Mitigation
Strategies:
Target Group/Risk Assessment Issue
Difficulty in determining if target group is
Issue/Challenge really gang involved or at risk of gang
involvement
•Not always sure if the program is being offered
to the right target group
• The evaluation results for key gang related
Implications outcomes may not show changes if the target
group is not at the right risk levels
•NCPC evaluation analysts at the outset provide
Mitigation Strategy sources for risk assessment tools and encourage
adaptation of U.S tools
•Hold 3rd party evaluation contractors
accountable for explaining all aspects of the
tool (i.e. cut off scores , composite risk indices) 11

12. Challenges:
Collaborating in developing risk assessment
tools
No! I am an But, I am Project manager
evaluator, risk and risk assessments
assessments are not require evaluation skills!
my thing!!
Can we please just
collaborate on this
one!?!?
12

13. YGPF Evaluation Challenges and Mitigation Strategies:
Internal Threat to Validity Issues
Evaluations not using a comparison group (50% )
need to comprehensively identify how they are
Issue/Challenge addressing threats to internal validity
• Threats to internal validity could be compromised.
Example: Six months into the evaluation, the local police
changed gang definition. Without identification of this
Implications change, reduction in gangs could be attributed to program
and not to the definition change.
• 3rd party evaluation contractor to outline an
exhaustive/comprehensive list of ways to mitigate
Mitigation Strategy
internal threats to validity
• 3rd party evaluation contractor to monitor and
document all potential confounding variables during
project implementation 13

14. YGPF Evaluation Challenges and Mitigation
Strategies: Construct Validity Issues
Balancing the need to use standardized
instruments with the need to customize
Issue/Challenge instruments to appropriately reflect unique
program activities and crime related outcomes
of interest
•Key outcomes of interest unique to the program
Implications may not be tested for change
•Encourage customization of standardized surveys
•Encourage use of techniques such as factor analysis
Mitigation Strategy
once modified standardized tool is tested, to ensure
reliability and validity
14

15. YGPF Evaluation Challenges and Mitigation Strategies:
Strengthening Theory of Change
Balancing the need to identify a theory of
Issue/Challenge change with stringent project timelines and
with ‘spending’ performance standards
•A weak theory of change means that the intervention
Implications to at risk youth is not likely to contribute to changes in
key outcomes of interest
•Require development of comprehensive logic model
Mitigation Strategy during the development phase
•Encourage 3rd party evaluation contractors to
challenge program staff (e.g. identify required dosage
for key activities)
15

16. YGPF Evaluation Challenges and Mitigation Strategies:
Choose an Appropriate “Advice Strategy”
Identify and utilize an “advice strategy” that
Issue/Challenge increases quality assurance (QA) and accountability
while providing the project/evaluation with
appropriate levels of autonomy
Using a “soft advice approach” (advice without
Implications follow up) reduces quality assurance and accountability
A “stringent advice approach” increases QA and
accountability but can reduce evaluation autonomy
Mitigation Strategy At minimum , utilize a “moderate advice approach”.
Ensure all internal reviewers/analysts use a similar
approach and advise 3rd party contractors accordingly
16

17. Balancing Quality Assurance, Accountability
and Evaluation/Project Autonomy
Soft Advice Approach Moderate Advice Approach Stringent Advice Approach
•Negotiation and prioritization
•No negotiation re: •Some negotiation to determine of key changes
differences in opinion what is agreed upon •Follow up until all
•No follow up to ensure •At least 1 time follow up to suggested improvements
advice is incorporated ensure key improvements have have been incorporated
• No approvals/sign offs been incorporated • Approvals and sign-offs
Increasing levels of QA and Accountability
Increasing Levels of Project/Evaluation Autonomy

18. Conclusion:
Summary of key Lessons Learned
•Ensure 3rd party evaluators are involved with the
Target Group/Risk Assessment development and ensure project staff understand how to apply
the tool when screening potential participants
•Comprehensive documentation and explanation of how
Internal Threats to Validity
internal threats to validity are being addressed is required,
especially when comparison group has not materialized as
planned
Construct Validity •Customize standardized tools, and utilize multivariate
analysis (i.e. factor analysis) to ensure modified tools are still
reliable and valid
Theory of Change •Evaluators need to challenge the program theory and
provide active suggestions about intensity of program
activities and their relationship to the outcomes of interest
Advice Strategy
• Consider the ‘moderate advice approach’ to increase
quality of the final product 18

19. Summary:
Link Between Sound Evaluations and
Increased Accountability and QA
 Increase confidence in data reported to Treasury Board to justify
Departmental spending
 Strengthen projects and enable them to fulfil obligations of
contribution funding
 Increase confidence in knowledge about ‘what works’ in crime
prevention with youth gangs – knowledge that will influence
national practice
19

20. Contact Information
Donna Smith-Moncrieffe
Senior Evaluation Analyst
Public Safety Canada, National Crime Prevention Center
Telephone: 416-952-0423
E-mail: donna.smith-moncrieffe@ps.gc.ca
20