Evaluating Long Term Complex Evaluations 2010

Long Term Complex Evaluations:Challenges, Mitigation Strategies and Effective Practice Donna Smith-MoncrieffeVictoria Conference Centre, B.C, May 2-5, 2010

Presentation Outline National Crime Prevention Centre (NCPC) mandate  Youth Gang Prevention Fund (YGPF) YGPF Evaluations Challenges and Strategies used in Long Term Complex Evaluations  Development Phase (6 months)  Challenges and Mitigation Strategies  Implementation Phase (3 years)  Challenges and Mitigation Strategies Summary

NCPC Mandate/Core ActivitiesMission statement:To provide national leadership on effective and cost-efficient ways to bothprevent and reduce crime by addressing known risk factors in high riskpopulations and places Core activities:  Provide funding to support targeted interventions in local communities  building and sharing practical knowledge with policy makers and practitioners

NCPC prioritiesProvide funding to the following target groups/crime issues: Children and youth at risk Crime prevention in Aboriginal communities Prevent recidivism among high-risk groups Priority crime issues (youth gang, drug-related crimes)

Youth Gang Prevention Fund (YGPF) Fund operates between Oct 2006 and March 2011 (5 years) Funding for anti-gang initiatives is provided in areas where gangs exist or are emerging with a focus on youth aged 9-24 Specialized fund developed to ensure that Canadian best practices in gang crime prevention can be disseminated

Current NCPC Gang Projects 17 funded across Canada Gang projects distributed across seven provinces (higher concentration in B.C, Prairies and Ontario) Range of funds per project ($750,000 to $4,900,000.00 over a four year period) Evidence-based practices for gang prevention include “wraparound”, case management, activities addressing multiple risk factors, and use of gang risk assessment tools. See NCPC website publications: NCPC Model and Promising Programs (2008) Most projects have been operating 1-2 years

Current NCPC Gang Impact Evaluations Project recipients are required to hire external evaluation teams Typical evaluation allocation is approximately $180,000.00 over four years 15/17 evaluations are funded through the contribution agreement with the project; two are funded through independent contracts with NCPC Project evaluations need to provide information for NCPC and departmental performance reporting (DPR), in terms of both outputs (e.g. # of projects, # of activities implemented), and outcomes (e.g. changes in risk factors, reduced offending)

YGPF Evaluation Reporting RequirementsThe YGPF Accountability and Audit Framework (ARAF) provides detail on how tomonitor, evaluate and report on results . The evaluation reporting Requirements areas follows:NCPC/Public Safety reporting  Annual Departmental Performance Reports  Strategic Review (every five years)  3 year Interim Progress Report (March, 2009)  Five Year Outcome-based Evaluation to Treasury Board (April, 2011)External Evaluator  Evaluation Framework (6 months after project start date)  3 Annual reports (1 due each year January 31)  Final Evaluation Report (March 2011)

YGPF Timelines: Evaluation Deliverables SOLICITATION DEVELOPMENT ANNUAL F TREASURY F OF REPORTS BOARD REPORTING PROJECTS PHASE U U N N JANUARY D D 17 17 2009 YGPF Projects EVALUATION SUMMATIVE Funded FRAMEWORKS 2010 EVALUATION S DEVELOPED E T 2011 N A D R S T SOctober 2006- 6 2009 March 20112006 2007 Months 2011 2011

Development Phase: Tools and Resources Needed 6 Senior evaluation analysts work in regions where theyStrengthening internal review and provide advice on evaluation deliverablesevaluation capacity Evaluation frameworks, annual reports and final reportClear reporting due dates are specified in the contribution agreement andrequirements contract YGPF National Logic model specifies expected outcomesTool development and activities YGPF Evaluation Guidelines provide key indicators, literature and evaluation designs to 3rd party evaluation teams Specific tasks and requirements stated (i.e. using police,Detailed proposals and health and education records to validate self-reportedcontracts changes) Clauses allowing for discontinuation of the evaluation if deliverables are not satisfactory

Development Phase (First 6 months) : ChallengesChallenge Strategies Risk Assessment Tool  Ensure the tool incorporates Development : relevant risk factors from the literature.  Difficulty in determining if target group is really gang  Utilize statistical tests to ensure involved, at risk of gang construct validity exists (i.e. chronbach’s alpha, factor analysis) involvement or just at risk of offending  Encourage evaluators to track what risk levels/cut-off scores are high  Evaluators viewed this as a enough to warrant program program staff responsibility intervention (i.e. demonstrating behavioural change with low risk participants is limited)

Development Phase (First 6 months) : ChallengesChallenges Strategies Lack of a Comparison Group:  Evaluations not using a  Request that the feasibility of comparison group (50% ) conducting alternative need to comprehensively evaluation designs be identify how they are demonstrated addressing threats to internal validity  Ensure the analytical plan identifies confounding variables  Internal threats such as and controls for them. maturation and history are common threats in long term evaluations

Development Phase (First 6 months) : Challenges StrategiesChallenge  Evaluators need to be proactive and Balancing quality assurance and challenge the program theory (i.e. stringent timelines even when the program is still developing by taking the evaluability assessment approach)  Developing an evaluation framework in six months while  Consider stringent timelines while the project is still developing ensuring that the intensity of program activities and their  Balancing the need to identify a relationship to the outcomes of ‘theory of change’ while meeting interest is logical prior to evaluating stringent project timelines and the program meeting ‘spending’ performance standards (i.e. prepare an evidence-based narrative to accompany the logic model

Development Phase Challenges:Completing an Evaluation Framework I understand but we are still developing I really need to the program and complete this this will likely take evaluation time!!!!!!! framework soon……

Implementation Phase: How to Improve the evaluation andprogram prior to the end of the project UtilizeEvaluation Advisory Committees Information and apply LessonsInterim Evaluation of the Fund Learned prior to the end of the Evaluation Annual Evaluation Reports

Implementation Phase: Challenges and Mitigation StrategiesChallenge Strategies  Encourage evaluation team to use Four levels of implementation multilevel statistical models. can be challenging:  Multilevel analysis allows variance in  Community outcome variables to be analyzed at  Peer hierarchical levels (i.e. separate  Family regression equations for each level)  Individual  Imbalances between the levels can be Planned interventions at the peer taken into account (i.e. interactive and family levels were limited effects between individuals and families can still be considered with small samples from parents).

Implementation Phase: Challenges and Mitigation StrategiesChallenge Strategies Long term evaluations usually  Ensure quality assurance protocol is developed mean frequent staff turnover  New staff should be trained to  Administration of ensure data collection is questionnaires and data conducted in a standard manner collection activity is mostly conducted by program staff  Evaluators should encourage the development of fidelity tools that clearly outline minimum levels of implementation, quality and treatment intensity

Implementation Phase: Challenges and Mitigation Strategies Defining program completion  Clearly identify activities that for these projects have been meet requirements for program challenging completion  Administrators may keep eager participants in the  Develop a fidelity tool/index that program for longer periods at minimum identifies dosage for:  Length of stay and treatment intensity is often not  Implementation commensurate (i.e.  Treatment Intensity participant stays for a year  Quality but only 30% of required treatment is completed)

Sample Fidelity Tool: Implementation and Treatment IntensitySample Fidelity ToolRequired Element Measure2 Case management sessions per week 4 Hours per week 16 Hours per month 96 Hours for the total program1 Two hour group session per month 2 Hours per month 12 Hours for the total program1 Two hour family counselling session in the home 2 hours per month over a 6 month period 12 Hours for the total programTrained clinician leading individual and group Yes : at least 80% ofsessions the time

Implementation Phase: Challenges and Mitigation StrategiesChallenges Strategies Managing external evaluation  Funding agency evaluation staff teams over a long period time frequently share expectations and deliverable due dates.  External evaluation teams have  Staff overseeing the evaluations competing priorities adopt a moderate advice  Balancing the production of approach that ensures technical quality deliverables within advice is incorporated in a timely stringent time frames can be manner challenging  Project autonomy can conflict with NCPC evaluation  Evaluation teams need to seek expectations clarification frequently

Implementation Strategy:Balancing Quality Assurance (QA) and Accountability Soft Advice Approach Moderate Advice Approach Stringent Advice Approach •Negotiation and prioritization•No negotiation re: •Some negotiation to determine of key changes differences in opinion what is agreed upon •Follow up until all•No follow up to ensure •At least 1 follow up to suggested improvements advice is incorporated ensure key improvements have have been incorporated• No approvals/sign offs been incorporated • Approvals and sign-offs Increasing levels of QA and Accountability Increasing Levels of Project/Evaluation Autonomy

Summary: Long term evaluations require structures that will allow continuous feedback prior to the final evaluation report  Continuous feedback should be provided to improve the program and to allow the youth to achieve optimal programming and results Proactivity on the part of the external evaluation team is required to ensure the risk assessment tool, the ‘theory of change’, comparison group and treatment intensity are all comprehensively assessed during the planning stages

Summary Cont’d Complex evaluations evaluating different levels of data (i.e. community, family and individual levels) should consider the use of multilevel data modelling Complex evaluations require that fidelity tools at least measure adherence of implementation activities and treatment intensity  Encourage a correlation of fidelity levels with outcomes of interest to complement pre and post test data and improve efforts to isolate levels of program attribution

Contact Information Donna Smith-Moncrieffe Senior Evaluation Advisor Public Safety CanadaPolicy, Research and Evaluation Division National Crime Prevention CenterE-mail: donna.smith-moncrieffe@ps.gc.ca

Evaluating Long Term Complex Evaluations 2010

by Donna Smith-Moncrieffe

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Evaluating Long Term Complex Evaluations 2010 Presentation Transcript