Clarifying the nature of the addiction treatment systemAchieving change in a climate of fragmentationBranding DTFP as Quality Improvement InitiativeChanging the system from an output to outcomes culture Needing to better understand incentives and clarify capacitiesPaying attention to supply and demandNeed to clarify ownership of the projectUncertainty around key partner roles and expectationsReluctance to change roles over timePlanning for use - tools in themselves will not bring about change.Need to embed spread processes into Accountability agreementsUtilize or develop bottom-up models of spreadNeed to clarify the role of various actors in spread efforts Need to fund spread and sustain activities for long enough to see change
How can evaluators assess readiness for developmental evaluation in a system of actors?Comfort with nonlinearity, emergence, unpredictability, and atypical evaluationWhat is the role of evaluation in disrupting systems and encouraging change?How can evaluators foster co-evolution with partners who are not ready for developmental approaches?
Ces 2013 - Doing Developmental Evaluation at the System Level
+Doing DevelopmentalEvaluation at theSystem Level:Lessons learned froman addiction systemchange initiativeTuesday, June 11th, 2013Jenna van Draanen, Kaileah McKellar, Sanjeev Sridharan
+Agenda An overview of the DTFP Evaluation DTFP Evaluation Methods DTFP Evaluation Results Developmental Evaluation at the System Level Lessons learned Lingering questions for the field of evaluation
+Drug Treatment Funding Program Addiction treatment system change Health Canada-funded project, delivered through the OntarioMinistry of Health 11 system change projects (3 streams) Multiple project leads and organizations 5 year timeline compressed to 18 months
+DTFPFundingSelection ofProjects &OrganizationsProject ImplementationTesting &validationDevelopmentof Tools andresourcesDevelopment ofInter-organizationalNetworksSystem-level evaluation& Evaluation capacitybuildingUptake&SpreadSystem Changeand practiceimprovement• Increased accessand use ofevidence• EnhancedCollaborativeProblem-SolvingCapacity• EnhancedEvaluative CulturePlanning forsustainingnetworksCollaboration & Co-DevelopmentKnowledge ExchangeSphere of DirectControlCollaborativeActivitiesEvaluationActivities4Project-levelactivitiesSphere of DirectInfluenceSphere of IndirectInfluence
+Developmental Evaluation of DTFP Incomplete knowledge 1) Spread processes within the system need to be developed 2) Relationships between different stakeholder groups need to bestrengthened.
+Developmental Evaluation of DTFP Purpose To bring evaluative thinking to DTFP and to help DTFP to evolveand create change Multiple and flexible methods Key-informant interviews, case studies, surveys, capacity-buildingworkshops, demand mapping, observation, videography,document review, realist synthesis, and timeline of impactexercises Heavy engagement What needs to be developed?
+Results Enhancements seen in evaluation and performancemeasurement, evidence-informed practice, knowledgeexchange projects At the project level Case studies giving insight into the system performance
+Results Themes Define the system Pay attention to dynamics Lack of ownership Tools themselves will not create change Planning for spread8
+Developmental Evaluation Challenges Multiple simultaneous impacts Projects having limited sphere of direct control Timeline of impact Providing feedback in an environment of uncertainty
+Lessons Learned Focus on learning Developingrelationships Rapid feedback Redundancy in agood thing