Implementation of an Evaluation Model for Evaluating Complex Health Research OutcomesPresentation Transcript
Implementation of an Evaluation Model for Evaluating Complex Health Research Outcomes American Evaluation Association November 1-5 2011 Kathryn Graham (PhD), Heidi Chorzempa (MSc)
Overview of Presentation•CAHS Implementation and Results•Lessons Learnt•Next Steps
Our Organization and Mandate •Alberta Innovates Health Solutions has a three decades long history of supporting medical/health research •“Support, for the economic and social well-being of Albertans, health research and innovation activities aligned to meet Government of Alberta priorities, including, without limitation, activities directed at the development and growth of the health sectors, the discovery of new knowledge and the application of that knowledge”.
Why We Evaluate Health Research? “…..economic and social well-being…. Source: Canadian Academy of Health Sciences 2009
Implementation Approach and Considerations Inputs Results TIMING – SHORT TO LONG TERM TIMING – SHORT TO LONG TERM 1. Evidence-Based Practice • Environmental scans Organizational • Theoretical & Conceptual Models (CAHS) and Health Socio- • Retrospective Data Mine Studies Research • Prospective reporting Economic Inputs, • Psychometric Testing Processes, Impact Reach, Outputs Categories 2. Practice-Based Evidence • Integated into the business model & Outcomes Missi • Stakeholder/Actors need, values, useOrganiz • Evaluation tools e.g. collaborative logic models 3. Top Down and or Bottom Up Indicators & Indicators & • Policy/Priority Driven and or Discovery Driven Metrics Metrics 4. Context • Strategy, External Factors, Clusters, Domains 5. Levels of Aggregation Macro, Meso, Micro Mixed Methodology Approach 6. Target Audience • Tailor results to stakeholder needs & values
Drivers for Retrospective Study • Verification: Using CAHS - does it make sense? Were the impact categories, sub categories and measures meaningful? • Applying Theory to Practice “in Context” - Alberta’s story • Evidence Informed Decision Making: Lessons learned informing design of new AIHS programs
Retrospective Data Mine Study 242 health researchers supported Pillar I: Pillar II: Pillar III: Pillar IV: Biomedical Clinical Health Population Research Research Services Health • Scholar • Clinical • Scholar • Population • Scientist Investigator • Scientist Health • Sr. Scholar • Sr. Scholar InvestigatorProgram Duration 5 3+3 5 3+3 years years years years
How we did it…. • A retrospective review of finalized grants from 2004-2008 • Population of a database • Review 855 annual reports across 8 types of personnel grants • Recognizing project limitations: -Attribution and contribution -Self-report -Data variability -Outcome underestimation -Timing of measurement -Retrofitting
Retrofitting the data to the model Annual Report Template Advancing Capacity Informing Health Socioeconomic Knowledge Building Decision Impact Impact Making # trainees # of collaborations # of diagnostic/ # publications # patents - Grad students prognosis # of influences in advancements # book - Post Docs # spin-off - Summer students guidelines # of treatment companies chapters - Other trainees # of influences in advancements policies # products in # abstracts * Infrastructure # of access development # of media citations improvements # of invited * External PI funding # of influences in # of safety amount public education talks improvements
Impact Summary Research Diffusion 100% Research That produces That influences That affects That contribute activity results decision making healthcare and to changing in…. health risk factors health, social 90% 21% National/ Global Research 11 collaborations and economic with Health wellbeing 15% Authorities Health Care Knowledge Research Results 100% Alberta •42 improvements 1196 Collaborations with other Researchers •5 guidelines Pool to health care Innovates effectiveness Health Improvements •3833 25 collaborations •8 improvements to Solutions health care in health and publications with Industries efficiency & wellbeing •Across 4 •4398 •17 products accessibility •Qualitative Research Pillars abstracts developed ResultsAdvancing Improvements in: Health •386 book •30 therapeuticsKnowledge Status and chapters •12 diagnosis/ 17 collaborations •prognosisCapacity function, •2789 with Government techniquesBuilding well-being, invited talks economic •4 policies Economic & conditions •3 data servicesInforming Research Capacity Social ImpactDecision •$217,381 in attracted Determinants of •107 patentsMaking funding Health •10 products in 18 collaborations •Personal development •990 trainees on with Non-profits behaviour •5 spin-off companiesHealth average per award year •Social/culturalImpact •8 guidelines determinants •14 new laboratories •EnvironmentalSocioeconomic The PublicImpact •16 media events Impacts feed back into inputs for future research
Timeline of Innovation 2001-2008 “We were the first lab in the world to describe the functional “In addition, these studies showed for consequences of mutations in T-type the first time, that chimerism channels linked to childhood absence “The second paper in J Neurophysiol was cited by the induces a dominant form of humoral epilepsy in humans” Faculty of 1000 Biology as highly innovative. This tolerance, allowing B cell tolerance to Sr. Scholar, year 3 paper explained why thalamic stimulation is not multiple donors.” Bio. Scholar, year 5“I have designed a new approach where a target gene can broadcast widely to the motor cortex and why DBS doesbe cloned into a large number of expression vectors by Significance: II found differences in the not disrupt motor control. It also helped explain the Significance: II showed that a combinedcombinatorial assembly of the target gene with a limited calcium channels in epileptic patients mechanism of action of subcortical DBS for severe protein can improve tolerance of donatednumber of pre-made DNA fragments.” Scholar, year 2 compared to non-epileptics. This could refractory depression. “ pancreatic tissue, which could improve elucidate the pathophysiology of epilepsy CI, year 4 long-term success rates of isletSignificance: II created a procedure that can disperse as well as potential therapeutic venues.many copies of one gene with limited resources. This transplantation in diabetics. Significance: Helps explain how deep braincould be used to increase research productivity (for stimulation works, which could lead toexample, placing the gene responsible for insulin inmany different cell types to try to determine the cell that improvements on this treatment modality.produces insulin the most efficiently. 2001 2002 2003 2004 2005 2006 2007 2008 “This lab contains state-of-the-art computing “The data in this manuscript were the first to show that infrastructure… and likely the first in a medical isoprostanes have the ability to modulate inflammatory imaging lab in Canada)” Bio.Scholar, year 2. processes with relevance to airway respiratory diseases. “ Sr. Scholar, year 3 Significance: II has cutting edge technology which could be used to improve diagnostic/prognostic Significance: II found evidence for the use of isoprostanes modalities or be used for research (higher (a form of anti-inflammatories) in the treatment of resolution images may display more information conditions such as asthma and COPD, where inflammation that is needed for progression). is the main source of disease progression. “In 2007, we published the first major report “The former was developed in collaboration with _____& _____and represents the first imaging tool to assess disease activity in both from the Alberta Diabetes Surveillance System spine and sacroiliac joints of patients with AS.” Sr. Scholar, year 2 (ADSS), in collaboration with the Health Significance: II developed a tool that can be used to assess Strategies and Surveillance Branches of Alberta progression of anklosying spondylitis (a chronic and painful Health and Wellness (AHW).” H.Scholar, year 4 condition). This can help determine guidelines for treatment (ex. When to use a specific therapy) Significance: The increased surveillance abilities resulting from ADSS will help inform health authorities’ efforts to reduce the *Please note that this timeline only represents a small proportion of the 320 innovation statements captured during the review. burden of diabetes on Albertans.
Example of the Balanced Scorecard Scorecard Information – Strategy Map Indicators Scorecard Data – Dash board Courtesy PM2 Linking and Rating Processes Linking and Ranking Projects
Lessons Learnt • Verification – “YES WE COULD DO IT” • Theory to Practice Lessons – Implementation - Complemented CAHS model with other models - Imbed knowledge translation – decision making rubrics – progress markers - Link collaborative activity to outcomes - Tell the story through qualitative and quantitative indicators • Context Lessons - Alberta Innovates System – innovation - Link organizational and program goals and objectives to CAHS categories - Tease out Innovation • Challenges • Change Management – theory of change • Utilization – is this a useful model for evaluating health research funding
Values and Valuing of Measurement “We are noticing a shift around the measurement of scientific value versus social/public value as well as the concept of collective action and shared value. Using a staged implementation approach, the CAHS model provided us with a best practice framework to measure benefits across the spectrum.
Contact Information Kathryn Graham (PhD) Director, Performance Management Alberta Innovates Health Solutions 1500 – 10104 103 Avenue N.W. Edmonton, Alberta, Canada T5J 4A7 Email: email@example.com Heidi Chorzempa (MSc) Manager, Performance Management Email: firstname.lastname@example.org THANK YOU