Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
Evaluating Long Term Complex Evaluations 2010
1. Long Term Complex Evaluations:
Challenges, Mitigation Strategies and Effective
Practice
Donna Smith-Moncrieffe
Victoria Conference Centre, B.C, May 2-5, 2010
2. Presentation Outline
National Crime Prevention Centre (NCPC) mandate
Youth Gang Prevention Fund (YGPF)
YGPF Evaluations
Challenges and Strategies used in Long Term Complex
Evaluations
Development Phase (6 months)
Challenges and Mitigation Strategies
Implementation Phase (3 years)
Challenges and Mitigation Strategies
Summary
3. NCPC Mandate/Core Activities
Mission statement:
To provide national leadership on effective and cost-efficient ways to both
prevent and reduce crime by addressing known risk factors in high risk
populations and places
Core activities:
Provide funding to support targeted interventions in local
communities
building and sharing practical knowledge with policy makers and
practitioners
4. NCPC priorities
Provide funding to the following target groups/crime issues:
Children and youth at risk
Crime prevention in Aboriginal communities
Prevent recidivism among high-risk groups
Priority crime issues (youth gang, drug-related crimes)
5. Youth Gang Prevention Fund (YGPF)
Fund operates between Oct 2006 and March 2011 (5 years)
Funding for anti-gang initiatives is provided in areas where
gangs exist or are emerging with a focus on youth aged 9-24
Specialized fund developed to ensure that Canadian best
practices in gang crime prevention can be disseminated
6. Current NCPC Gang Projects
17 funded across Canada
Gang projects distributed across seven provinces (higher
concentration in B.C, Prairies and Ontario)
Range of funds per project ($750,000 to $4,900,000.00 over a
four year period)
Evidence-based practices for gang prevention include
“wraparound”, case management, activities addressing multiple
risk factors, and use of gang risk assessment tools. See NCPC
website publications: NCPC Model and Promising Programs
(2008)
Most projects have been operating 1-2 years
7. Current NCPC Gang Impact Evaluations
Project recipients are required to hire external evaluation teams
Typical evaluation allocation is approximately $180,000.00 over four
years
15/17 evaluations are funded through the contribution agreement with
the project; two are funded through independent contracts with NCPC
Project evaluations need to provide information for NCPC and
departmental performance reporting (DPR), in terms of both outputs (e.g.
# of projects, # of activities implemented), and outcomes (e.g. changes in
risk factors, reduced offending)
8. YGPF Evaluation Reporting
Requirements
The YGPF Accountability and Audit Framework (ARAF) provides detail on how to
monitor, evaluate and report on results . The evaluation reporting Requirements are
as follows:
NCPC/Public Safety reporting
Annual Departmental Performance Reports
Strategic Review (every five years)
3 year Interim Progress Report (March, 2009)
Five Year Outcome-based Evaluation to Treasury Board (April, 2011)
External Evaluator
Evaluation Framework (6 months after project start date)
3 Annual reports (1 due each year January 31)
Final Evaluation Report (March 2011)
9. YGPF Timelines: Evaluation Deliverables
SOLICITATION DEVELOPMENT ANNUAL F TREASURY
F OF REPORTS BOARD REPORTING
PROJECTS PHASE U
U
N
N JANUARY
D
D 17 17
2009 YGPF
Projects EVALUATION SUMMATIVE
Funded FRAMEWORKS 2010 EVALUATION
S DEVELOPED
E
T 2011 N
A
D
R
S
T
S
October 2006- 6 2009 March 2011
2006 2007 Months 2011 2011
10. Development Phase:
Tools and Resources Needed
6 Senior evaluation analysts work in regions where they
Strengthening internal review and provide advice on evaluation deliverables
evaluation capacity
Evaluation frameworks, annual reports and final report
Clear reporting due dates are specified in the contribution agreement and
requirements contract
YGPF National Logic model specifies expected outcomes
Tool development and activities
YGPF Evaluation Guidelines provide key indicators,
literature and evaluation designs to 3rd party evaluation
teams
Specific tasks and requirements stated (i.e. using police,
Detailed proposals and health and education records to validate self-reported
contracts changes)
Clauses allowing for discontinuation of the evaluation if
deliverables are not satisfactory
11. Development Phase (First 6 months) :
Challenges
Challenge Strategies
Risk Assessment Tool Ensure the tool incorporates
Development : relevant risk factors from the
literature.
Difficulty in determining if
target group is really gang Utilize statistical tests to ensure
involved, at risk of gang construct validity exists (i.e.
chronbach’s alpha, factor analysis)
involvement or just at risk of
offending
Encourage evaluators to track what
risk levels/cut-off scores are high
Evaluators viewed this as a enough to warrant program
program staff responsibility intervention
(i.e. demonstrating behavioural
change with low risk participants is
limited)
12. Development Phase (First 6 months) :
Challenges
Challenges Strategies
Lack of a Comparison Group:
Evaluations not using a Request that the feasibility of
comparison group (50% ) conducting alternative
need to comprehensively evaluation designs be
identify how they are demonstrated
addressing threats to internal
validity Ensure the analytical plan
identifies confounding variables
Internal threats such as and controls for them.
maturation and history are
common threats in long term
evaluations
13. Development Phase (First 6 months) :
Challenges
Strategies
Challenge
Evaluators need to be proactive and
Balancing quality assurance and challenge the program theory (i.e.
stringent timelines even when the program is still
developing by taking the evaluability
assessment approach)
Developing an evaluation
framework in six months while Consider stringent timelines while
the project is still developing ensuring that the intensity of
program activities and their
Balancing the need to identify a relationship to the outcomes of
‘theory of change’ while meeting interest is logical prior to evaluating
stringent project timelines and the program
meeting ‘spending’ performance
standards (i.e. prepare an evidence-based
narrative to accompany the logic
model
14. Development Phase Challenges:
Completing an Evaluation Framework
I understand but we
are still developing
I really need to the program and
complete this this will likely take
evaluation time!!!!!!!
framework
soon……
15. Implementation Phase:
How to Improve the evaluation and
program prior to the end of the project
Utilize
Evaluation Advisory Committees Information
and
apply
Lessons
Interim Evaluation of the Fund Learned prior
to the
end of the
Evaluation
Annual Evaluation Reports
16. Implementation Phase: Challenges and
Mitigation Strategies
Challenge Strategies
Encourage evaluation team to use
Four levels of implementation
multilevel statistical models.
can be challenging:
Multilevel analysis allows variance in
Community outcome variables to be analyzed at
Peer hierarchical levels (i.e. separate
Family regression equations for each level)
Individual
Imbalances between the levels can be
Planned interventions at the peer taken into account (i.e. interactive
and family levels were limited effects between individuals and
families can still be considered with
small samples from parents).
17. Implementation Phase: Challenges and
Mitigation Strategies
Challenge Strategies
Long term evaluations usually Ensure quality assurance
protocol is developed
mean frequent staff turnover
New staff should be trained to
Administration of ensure data collection is
questionnaires and data conducted in a standard manner
collection activity is mostly
conducted by program staff Evaluators should encourage the
development of fidelity tools
that clearly outline minimum
levels of implementation, quality
and treatment intensity
18. Implementation Phase: Challenges and
Mitigation Strategies
Defining program completion Clearly identify activities that
for these projects have been meet requirements for program
challenging completion
Administrators may keep
eager participants in the
Develop a fidelity tool/index that
program for longer periods at minimum identifies dosage for:
Length of stay and treatment
intensity is often not Implementation
commensurate (i.e. Treatment Intensity
participant stays for a year Quality
but only 30% of required
treatment is completed)
19. Sample Fidelity Tool:
Implementation and Treatment Intensity
Sample Fidelity Tool
Required Element Measure
2 Case management sessions per week 4 Hours per week
16 Hours per month
96 Hours for the total
program
1 Two hour group session per month 2 Hours per month
12 Hours for the total
program
1 Two hour family counselling session in the home 2 hours per month
over a 6 month period 12 Hours for the total
program
Trained clinician leading individual and group Yes : at least 80% of
sessions the time
20. Implementation Phase: Challenges and
Mitigation Strategies
Challenges Strategies
Managing external evaluation Funding agency evaluation staff
teams over a long period time frequently share expectations and
deliverable due dates.
External evaluation teams have
Staff overseeing the evaluations
competing priorities
adopt a moderate advice
Balancing the production of
approach that ensures technical
quality deliverables within advice is incorporated in a timely
stringent time frames can be manner
challenging
Project autonomy can conflict
with NCPC evaluation
Evaluation teams need to seek
expectations clarification frequently
21. Implementation Strategy:
Balancing Quality Assurance (QA) and
Accountability
Soft Advice Approach Moderate Advice Approach Stringent Advice Approach
•Negotiation and prioritization
•No negotiation re: •Some negotiation to determine of key changes
differences in opinion what is agreed upon •Follow up until all
•No follow up to ensure •At least 1 follow up to suggested improvements
advice is incorporated ensure key improvements have have been incorporated
• No approvals/sign offs been incorporated • Approvals and sign-offs
Increasing levels of QA and Accountability
Increasing Levels of Project/Evaluation Autonomy
22. Summary:
Long term evaluations require structures that will allow continuous
feedback prior to the final evaluation report
Continuous feedback should be provided to improve the program and to allow
the youth to achieve optimal programming and results
Proactivity on the part of the external evaluation team is required to
ensure the risk assessment tool, the ‘theory of change’, comparison group
and treatment intensity are all comprehensively assessed during the
planning stages
23. Summary Cont’d
Complex evaluations evaluating different levels of data (i.e. community,
family and individual levels) should consider the use of multilevel data
modelling
Complex evaluations require that fidelity tools at least measure adherence
of implementation activities and treatment intensity
Encourage a correlation of fidelity levels with outcomes of interest to
complement pre and post test data and improve efforts to isolate levels of
program attribution
24. Contact Information
Donna Smith-Moncrieffe
Senior Evaluation Advisor
Public Safety Canada
Policy, Research and Evaluation Division
National Crime Prevention Center
E-mail: donna.smith-moncrieffe@ps.gc.ca