Acknowledge, the Performance Management Team: Heidi Chorzempa, May Kharaghani, Liza Chan, Daniel Zhang and Andrew LeJeune I am going to present on implementating the CAHS model, results of a retrospective study, lessons learnt from implementing overall and next steps.
The Edmonton based Alberta Innovates Health Solutions is a not for profit provincial health research funding agency. In early 2010 AHFMR was incorporated under the new realigned research and innovation system designed to strengthen the provinces role as a world leader in using science to seek solutions. Alberta Innovates Health Solutions was established in 1980 to support medical/health research AIHS has provided unprecedented opportunities in health research and innovation Contributed > $1 billion to the scientific community Supported > 8,500 researchers approximately Our mandate is to support the economic and social well-being of Albertans,
Definition CAHS identified three main objectives for conducting evaluation in the context of health research: Evaluation helps account for money that is spent Evaluation helps advocate for more funding Evaluation helps optimize research outcomes Attribution and Contribution Time lags Counterfactual Hard to Measure
When measuring outcomes and impacts, a number of considerations need to be taken into account and choice made, all of which influence the measures Organization’s mission Evidence Base – conceptual models e.g. CAHS, other frameworks in existence Stakeholder needs and values Evaluation tools such as logic models and or Balanced Score cards Whether the approach is top down i.e. priority and or policy driven and or bottom up i.e. curiosity driven research Context, Alberta Health Research and Innovation Strategy (AHRIS looked at 12 strategic focus areas, and 3 foundational areas, Highly Skilled People, Knowledge Translation, Innovation Platforms) Multi-faceted Approach: Mixed Methodology – Qualitative and Quantitative Indicators, narrative Multi-Method – case studies to bibliometrics Multi-Dimensional – spectrum of outputs, outcomes and impact Level of aggregation, need to determine the unit of analysis and appropriate level of measurement, e.g. individual versus system level Target audience – again to need.
Verification: Retrofit to Personnel Support Grant - Ability to retrofit and operationalize the CAHS model to an existing grant program based on an open discovery model Theory to Practice: Investigate contextual issues in operationalizing the CAHS model to an Alberta context, cross the 4 CIHR pillars, Alberta context, feasibility and usability Evidence Informed Decision Making: Test the model retrospectively for feasibility, usability, issues etc. prior to piloting it prospectively with two other grant programs.
The study was based on personnel support to fund the career and livelihoods of hundreds of remarkable researchers each year. Resulting in an approximate 40 million dollar investment each year. Across CIHR research pillars as per Gretchen’s presentation
A retrospective review of annual progress reports finalized from 2004-2008 Population of a database containing 584 variables Recognizing project limitations: Attribution and contribution Self-report Data variability Outcome underestimation Retrofitting
Documents Vision, mission, goals, values, business planning documents, corporate and operational, individual performance management plans 08/01/12
Dangling Indicators/Measures must be linked to something e.g. mission, objective, question etc. Metrics only approaches are behind the times, “Best Practice combines narratives with relevant qualitative and quantitative indicators to gauge broader social, environmental, cultural and economic public value. Limited consultation between policy-makers and the research evaluation community has led to a lack of policy-learning from international developments” (Claire Donovan 2011 Research Evaluation) Verification – “YES WE COULD DO IT” Theory to Practice Lessons – Implementation Complimented model with process measures for the linkage and exchange e.g. collaborations, networks, etc. Need to both qualititative and quantitative indicators and narratives to tell the story Context Lessons Change in Research Landscape and Strategy Organizational Excellence – Balanced Score Cards KT model to enhance Redesign – Goals and objectives Innovation Collective action Evidence Informed Decision Making Utilization – user perspective – data capture rules Change Management - awareness, education, systems
Implementation of an Evaluation Model for Evaluating Complex Health Research Outcomes
Implementation of an Evaluation Model for Evaluating Complex Health Research Outcomes American Evaluation Association November 1-5 2011 Kathryn Graham (PhD), Heidi Chorzempa (MSc)
Overview of Presentation•CAHS Implementation and Results•Lessons Learnt•Next Steps
Our Organization and Mandate •Alberta Innovates Health Solutions has a three decades long history of supporting medical/health research •“Support, for the economic and social well-being of Albertans, health research and innovation activities aligned to meet Government of Alberta priorities, including, without limitation, activities directed at the development and growth of the health sectors, the discovery of new knowledge and the application of that knowledge”.
Why We Evaluate Health Research? “…..economic and social well-being…. Source: Canadian Academy of Health Sciences 2009
Implementation Approach and Considerations Inputs Results TIMING – SHORT TO LONG TERM TIMING – SHORT TO LONG TERM 1. Evidence-Based Practice • Environmental scans Organizational • Theoretical & Conceptual Models (CAHS) and Health Socio- • Retrospective Data Mine Studies Research • Prospective reporting Economic Inputs, • Psychometric Testing Processes, Impact Reach, Outputs Categories 2. Practice-Based Evidence • Integated into the business model & Outcomes Missi • Stakeholder/Actors need, values, useOrganiz • Evaluation tools e.g. collaborative logic models 3. Top Down and or Bottom Up Indicators & Indicators & • Policy/Priority Driven and or Discovery Driven Metrics Metrics 4. Context • Strategy, External Factors, Clusters, Domains 5. Levels of Aggregation Macro, Meso, Micro Mixed Methodology Approach 6. Target Audience • Tailor results to stakeholder needs & values
Drivers for Retrospective Study • Verification: Using CAHS - does it make sense? Were the impact categories, sub categories and measures meaningful? • Applying Theory to Practice “in Context” - Alberta’s story • Evidence Informed Decision Making: Lessons learned informing design of new AIHS programs
Retrospective Data Mine Study 242 health researchers supported Pillar I: Pillar II: Pillar III: Pillar IV: Biomedical Clinical Health Population Research Research Services Health • Scholar • Clinical • Scholar • Population • Scientist Investigator • Scientist Health • Sr. Scholar • Sr. Scholar InvestigatorProgram Duration 5 3+3 5 3+3 years years years years
How we did it…. • A retrospective review of finalized grants from 2004-2008 • Population of a database • Review 855 annual reports across 8 types of personnel grants • Recognizing project limitations: -Attribution and contribution -Self-report -Data variability -Outcome underestimation -Timing of measurement -Retrofitting
Retrofitting the data to the model Annual Report Template Advancing Capacity Informing Health Socioeconomic Knowledge Building Decision Impact Impact Making # trainees # of collaborations # of diagnostic/ # publications # patents - Grad students prognosis # of influences in advancements # book - Post Docs # spin-off - Summer students guidelines # of treatment companies chapters - Other trainees # of influences in advancements policies # products in # abstracts * Infrastructure # of access development # of media citations improvements # of invited * External PI funding # of influences in # of safety amount public education talks improvements
Impact Summary Research Diffusion 100% Research That produces That influences That affects That contribute activity results decision making healthcare and to changing in…. health risk factors health, social 90% 21% National/ Global Research 11 collaborations and economic with Health wellbeing 15% Authorities Health Care Knowledge Research Results 100% Alberta •42 improvements 1196 Collaborations with other Researchers •5 guidelines Pool to health care Innovates effectiveness Health Improvements •3833 25 collaborations •8 improvements to Solutions health care in health and publications with Industries efficiency & wellbeing •Across 4 •4398 •17 products accessibility •Qualitative Research Pillars abstracts developed ResultsAdvancing Improvements in: Health •386 book •30 therapeuticsKnowledge Status and chapters •12 diagnosis/ 17 collaborations •prognosisCapacity function, •2789 with Government techniquesBuilding well-being, invited talks economic •4 policies Economic & conditions •3 data servicesInforming Research Capacity Social ImpactDecision •$217,381 in attracted Determinants of •107 patentsMaking funding Health •10 products in 18 collaborations •Personal development •990 trainees on with Non-profits behaviour •5 spin-off companiesHealth average per award year •Social/culturalImpact •8 guidelines determinants •14 new laboratories •EnvironmentalSocioeconomic The PublicImpact •16 media events Impacts feed back into inputs for future research
Timeline of Innovation 2001-2008 “We were the first lab in the world to describe the functional “In addition, these studies showed for consequences of mutations in T-type the first time, that chimerism channels linked to childhood absence “The second paper in J Neurophysiol was cited by the induces a dominant form of humoral epilepsy in humans” Faculty of 1000 Biology as highly innovative. This tolerance, allowing B cell tolerance to Sr. Scholar, year 3 paper explained why thalamic stimulation is not multiple donors.” Bio. Scholar, year 5“I have designed a new approach where a target gene can broadcast widely to the motor cortex and why DBS doesbe cloned into a large number of expression vectors by Significance: II found differences in the not disrupt motor control. It also helped explain the Significance: II showed that a combinedcombinatorial assembly of the target gene with a limited calcium channels in epileptic patients mechanism of action of subcortical DBS for severe protein can improve tolerance of donatednumber of pre-made DNA fragments.” Scholar, year 2 compared to non-epileptics. This could refractory depression. “ pancreatic tissue, which could improve elucidate the pathophysiology of epilepsy CI, year 4 long-term success rates of isletSignificance: II created a procedure that can disperse as well as potential therapeutic venues.many copies of one gene with limited resources. This transplantation in diabetics. Significance: Helps explain how deep braincould be used to increase research productivity (for stimulation works, which could lead toexample, placing the gene responsible for insulin inmany different cell types to try to determine the cell that improvements on this treatment modality.produces insulin the most efficiently. 2001 2002 2003 2004 2005 2006 2007 2008 “This lab contains state-of-the-art computing “The data in this manuscript were the first to show that infrastructure… and likely the first in a medical isoprostanes have the ability to modulate inflammatory imaging lab in Canada)” Bio.Scholar, year 2. processes with relevance to airway respiratory diseases. “ Sr. Scholar, year 3 Significance: II has cutting edge technology which could be used to improve diagnostic/prognostic Significance: II found evidence for the use of isoprostanes modalities or be used for research (higher (a form of anti-inflammatories) in the treatment of resolution images may display more information conditions such as asthma and COPD, where inflammation that is needed for progression). is the main source of disease progression. “In 2007, we published the first major report “The former was developed in collaboration with _____& _____and represents the first imaging tool to assess disease activity in both from the Alberta Diabetes Surveillance System spine and sacroiliac joints of patients with AS.” Sr. Scholar, year 2 (ADSS), in collaboration with the Health Significance: II developed a tool that can be used to assess Strategies and Surveillance Branches of Alberta progression of anklosying spondylitis (a chronic and painful Health and Wellness (AHW).” H.Scholar, year 4 condition). This can help determine guidelines for treatment (ex. When to use a specific therapy) Significance: The increased surveillance abilities resulting from ADSS will help inform health authorities’ efforts to reduce the *Please note that this timeline only represents a small proportion of the 320 innovation statements captured during the review. burden of diabetes on Albertans.
Example of the Balanced Scorecard Scorecard Information – Strategy Map Indicators Scorecard Data – Dash board Courtesy PM2 Linking and Rating Processes Linking and Ranking Projects
Lessons Learnt • Verification – “YES WE COULD DO IT” • Theory to Practice Lessons – Implementation - Complemented CAHS model with other models - Imbed knowledge translation – decision making rubrics – progress markers - Link collaborative activity to outcomes - Tell the story through qualitative and quantitative indicators • Context Lessons - Alberta Innovates System – innovation - Link organizational and program goals and objectives to CAHS categories - Tease out Innovation • Challenges • Change Management – theory of change • Utilization – is this a useful model for evaluating health research funding
Values and Valuing of Measurement “We are noticing a shift around the measurement of scientific value versus social/public value as well as the concept of collective action and shared value. Using a staged implementation approach, the CAHS model provided us with a best practice framework to measure benefits across the spectrum.
Contact Information Kathryn Graham (PhD) Director, Performance Management Alberta Innovates Health Solutions 1500 – 10104 103 Avenue N.W. Edmonton, Alberta, Canada T5J 4A7 Email: firstname.lastname@example.org Heidi Chorzempa (MSc) Manager, Performance Management Email: email@example.com THANK YOU