1. Long Term Complex Evaluations:
Challenges, Mitigation Strategies and Effective
Practice
Donna Smith-Moncrieffe
Victoria Conference Centre, B.C, May 2-5, 2010

2. Presentation Outline
 National Crime Prevention Centre (NCPC) mandate
 Youth Gang Prevention Fund (YGPF)
 YGPF Evaluations
 Challenges and Strategies used in Long Term Complex
Evaluations
 Development Phase (6 months)
 Challenges and Mitigation Strategies
 Implementation Phase (3 years)
 Challenges and Mitigation Strategies
 Summary

3. NCPC Mandate/Core Activities
Mission statement:
To provide national leadership on effective and cost-efficient ways to both
prevent and reduce crime by addressing known risk factors in high risk
populations and places
Core activities:
 Provide funding to support targeted interventions in local
communities
 building and sharing practical knowledge with policy makers and
practitioners

4. NCPC priorities
Provide funding to the following target groups/crime issues:
 Children and youth at risk
 Crime prevention in Aboriginal communities
 Prevent recidivism among high-risk groups
 Priority crime issues (youth gang, drug-related crimes)

5. Youth Gang Prevention Fund (YGPF)
 Fund operates between Oct 2006 and March 2011 (5 years)
 Funding for anti-gang initiatives is provided in areas where
gangs exist or are emerging with a focus on youth aged 9-24
 Specialized fund developed to ensure that Canadian best
practices in gang crime prevention can be disseminated

6. Current NCPC Gang Projects
 17 funded across Canada
 Gang projects distributed across seven provinces (higher
concentration in B.C, Prairies and Ontario)
 Range of funds per project ($750,000 to $4,900,000.00 over a
four year period)
 Evidence-based practices for gang prevention include
“wraparound”, case management, activities addressing multiple
risk factors, and use of gang risk assessment tools. See NCPC
website publications: NCPC Model and Promising Programs
(2008)
 Most projects have been operating 1-2 years

7. Current NCPC Gang Impact Evaluations
 Project recipients are required to hire external evaluation teams
 Typical evaluation allocation is approximately $180,000.00 over four
years
 15/17 evaluations are funded through the contribution agreement with
the project; two are funded through independent contracts with NCPC
 Project evaluations need to provide information for NCPC and
departmental performance reporting (DPR), in terms of both outputs (e.g.
# of projects, # of activities implemented), and outcomes (e.g. changes in
risk factors, reduced offending)

8. YGPF Evaluation Reporting
Requirements
The YGPF Accountability and Audit Framework (ARAF) provides detail on how to
monitor, evaluate and report on results . The evaluation reporting Requirements are
as follows:
NCPC/Public Safety reporting
 Annual Departmental Performance Reports
 Strategic Review (every five years)
 3 year Interim Progress Report (March, 2009)
 Five Year Outcome-based Evaluation to Treasury Board (April, 2011)
External Evaluator
 Evaluation Framework (6 months after project start date)
 3 Annual reports (1 due each year January 31)
 Final Evaluation Report (March 2011)

9. YGPF Timelines: Evaluation Deliverables
SOLICITATION DEVELOPMENT ANNUAL F TREASURY
F OF REPORTS BOARD REPORTING
PROJECTS PHASE U
U
N
N JANUARY
D
D 17 17
2009 YGPF
Projects EVALUATION SUMMATIVE
Funded FRAMEWORKS 2010 EVALUATION
S DEVELOPED
E
T 2011 N
A
D
R
S
T
S
October 2006- 6 2009 March 2011
2006 2007 Months 2011 2011

10. Development Phase:
Tools and Resources Needed
6 Senior evaluation analysts work in regions where they
Strengthening internal review and provide advice on evaluation deliverables
evaluation capacity
Evaluation frameworks, annual reports and final report
Clear reporting due dates are specified in the contribution agreement and
requirements contract
YGPF National Logic model specifies expected outcomes
Tool development and activities
YGPF Evaluation Guidelines provide key indicators,
literature and evaluation designs to 3rd party evaluation
teams
Specific tasks and requirements stated (i.e. using police,
Detailed proposals and health and education records to validate self-reported
contracts changes)
Clauses allowing for discontinuation of the evaluation if
deliverables are not satisfactory

11. Development Phase (First 6 months) :
Challenges
Challenge Strategies
 Risk Assessment Tool  Ensure the tool incorporates
Development : relevant risk factors from the
literature.
 Difficulty in determining if
target group is really gang  Utilize statistical tests to ensure
involved, at risk of gang construct validity exists (i.e.
chronbach’s alpha, factor analysis)
involvement or just at risk of
offending
 Encourage evaluators to track what
risk levels/cut-off scores are high
 Evaluators viewed this as a enough to warrant program
program staff responsibility intervention
(i.e. demonstrating behavioural
change with low risk participants is
limited)

12. Development Phase (First 6 months) :
Challenges
Challenges Strategies
 Lack of a Comparison Group:
 Evaluations not using a  Request that the feasibility of
comparison group (50% ) conducting alternative
need to comprehensively evaluation designs be
identify how they are demonstrated
addressing threats to internal
validity  Ensure the analytical plan
identifies confounding variables
 Internal threats such as and controls for them.
maturation and history are
common threats in long term
evaluations

13. Development Phase (First 6 months) :
Challenges
Strategies
Challenge
 Evaluators need to be proactive and
 Balancing quality assurance and challenge the program theory (i.e.
stringent timelines even when the program is still
developing by taking the evaluability
assessment approach)
 Developing an evaluation
framework in six months while  Consider stringent timelines while
the project is still developing ensuring that the intensity of
program activities and their
 Balancing the need to identify a relationship to the outcomes of
‘theory of change’ while meeting interest is logical prior to evaluating
stringent project timelines and the program
meeting ‘spending’ performance
standards (i.e. prepare an evidence-based
narrative to accompany the logic
model

14. Development Phase Challenges:
Completing an Evaluation Framework
I understand but we
are still developing
I really need to the program and
complete this this will likely take
evaluation time!!!!!!!
framework
soon……

15. Implementation Phase:
How to Improve the evaluation and
program prior to the end of the project
Utilize
Evaluation Advisory Committees Information
and
apply
Lessons
Interim Evaluation of the Fund Learned prior
to the
end of the
Evaluation
Annual Evaluation Reports

16. Implementation Phase: Challenges and
Mitigation Strategies
Challenge Strategies
 Encourage evaluation team to use
 Four levels of implementation
multilevel statistical models.
can be challenging:
 Multilevel analysis allows variance in
 Community outcome variables to be analyzed at
 Peer hierarchical levels (i.e. separate
 Family regression equations for each level)
 Individual
 Imbalances between the levels can be
 Planned interventions at the peer taken into account (i.e. interactive
and family levels were limited effects between individuals and
families can still be considered with
small samples from parents).

17. Implementation Phase: Challenges and
Mitigation Strategies
Challenge Strategies
 Long term evaluations usually  Ensure quality assurance
protocol is developed
mean frequent staff turnover
 New staff should be trained to
 Administration of ensure data collection is
questionnaires and data conducted in a standard manner
collection activity is mostly
conducted by program staff  Evaluators should encourage the
development of fidelity tools
that clearly outline minimum
levels of implementation, quality
and treatment intensity

18. Implementation Phase: Challenges and
Mitigation Strategies
 Defining program completion  Clearly identify activities that
for these projects have been meet requirements for program
challenging completion
 Administrators may keep
eager participants in the
 Develop a fidelity tool/index that
program for longer periods at minimum identifies dosage for:
 Length of stay and treatment
intensity is often not  Implementation
commensurate (i.e.  Treatment Intensity
participant stays for a year  Quality
but only 30% of required
treatment is completed)

19. Sample Fidelity Tool:
Implementation and Treatment Intensity
Sample Fidelity Tool
Required Element Measure
2 Case management sessions per week 4 Hours per week
16 Hours per month
96 Hours for the total
program
1 Two hour group session per month 2 Hours per month
12 Hours for the total
program
1 Two hour family counselling session in the home 2 hours per month
over a 6 month period 12 Hours for the total
program
Trained clinician leading individual and group Yes : at least 80% of
sessions the time

20. Implementation Phase: Challenges and
Mitigation Strategies
Challenges Strategies
 Managing external evaluation  Funding agency evaluation staff
teams over a long period time frequently share expectations and
deliverable due dates.
 External evaluation teams have
 Staff overseeing the evaluations
competing priorities
adopt a moderate advice
 Balancing the production of
approach that ensures technical
quality deliverables within advice is incorporated in a timely
stringent time frames can be manner
challenging
 Project autonomy can conflict
with NCPC evaluation
 Evaluation teams need to seek
expectations clarification frequently

21. Implementation Strategy:
Balancing Quality Assurance (QA) and
Accountability
Soft Advice Approach Moderate Advice Approach Stringent Advice Approach
•Negotiation and prioritization
•No negotiation re: •Some negotiation to determine of key changes
differences in opinion what is agreed upon •Follow up until all
•No follow up to ensure •At least 1 follow up to suggested improvements
advice is incorporated ensure key improvements have have been incorporated
• No approvals/sign offs been incorporated • Approvals and sign-offs
Increasing levels of QA and Accountability
Increasing Levels of Project/Evaluation Autonomy

22. Summary:
 Long term evaluations require structures that will allow continuous
feedback prior to the final evaluation report
 Continuous feedback should be provided to improve the program and to allow
the youth to achieve optimal programming and results
 Proactivity on the part of the external evaluation team is required to
ensure the risk assessment tool, the ‘theory of change’, comparison group
and treatment intensity are all comprehensively assessed during the
planning stages

23. Summary Cont’d
 Complex evaluations evaluating different levels of data (i.e. community,
family and individual levels) should consider the use of multilevel data
modelling
 Complex evaluations require that fidelity tools at least measure adherence
of implementation activities and treatment intensity
 Encourage a correlation of fidelity levels with outcomes of interest to
complement pre and post test data and improve efforts to isolate levels of
program attribution

24. Contact Information
Donna Smith-Moncrieffe
Senior Evaluation Advisor
Public Safety Canada
Policy, Research and Evaluation Division
National Crime Prevention Center
E-mail: donna.smith-moncrieffe@ps.gc.ca