Acknowledge, the Performance Management Team: Amy Wong, Heidi Chorzempa, May Kharaghani, Daniel Zhang and Andrew LeJeune (What about me? Liza)
Evaluating complex systems requires using novel approaches as well as multimethod approach to provide an evidence base for policy making as well as informing program and organizational decision making and demonstrating results.
The Edmonton based Alberta Innovates Health Solutions is a not for profit provincial health research funding agency. In early 2010 AHFMR was incorporated under the new realigned research and innovation system designed to strengthen the provinces role as a world leader in using science to seek solutions.
Belief that supporting health research will lead to health, health system benefit.
One of the main stays are surveys and asking researchers for information hence administrative burden
Comprehensive and Integrated:Mixed Methodology – Qualitative and Quantitative collection, analysis and reportingMulti method – case studies to bibliometricsMulti dimensionalIntegrated into the business modelPhased Approach
Measures the “output” of individuals/research teams, institutions, and countries, to identify national and international networks, and to map the development of new (multi-disciplinary) fields of science and technology. CAHS : QUESTIONS INDICATORS/MEASURESACTIVITY/PPRODUCITVITY WHAT HAS BEEN THE LEVEL OF ACTIVITY IN TERMS OF KNOWLEDGE GENERATION BY CANADA-BASED MEDICAL RESEARCHERS NUMBER OF PUBLICATIONSCONTEXTUAL AND STRUCTURALQUALITYOUTREACH
Capacity building, advancing knowledge and informing decision making are the enabling outcomes to achieve health and socio-economic impact
Changes in the external environment drive change in AIHS which also impacts how we assess success and what is important to our different key stakeholder groups
MethodsExpert Opinion – Peer ReviewSurveysInterviewsCase StudiesMeasures the “output” of individuals/research teams, institutions, and countries, to identify national and international networks, and to map the development of new (multi-disciplinary) fields of science and technology. CAHS : QUESTIONS INDICATORS/MEASURESACTIVITY/PPRODUCITVITY WHAT HAS BEEN THE LEVEL OF ACTIVITY IN TERMS OF KNOWLEDGE GENERATION BY CANADA-BASED MEDICAL RESEARCHERS NUMBER OF PUBLICATIONSCONTEXTUAL AND STRUCTURALQUALITYOUTREACH
Measures the “output” of individuals/research teams, institutions, and countries, to identify national and international networks, and to map the development of new (multi-disciplinary) fields of science and technology.Difficult to account for organizational S&T mandate, priorities and levels of resources ($, infrastructure and FTEs)
ARC – a direct measure of scientific impact based on paper citation counts (2-year citation window). Normalized by subfields of Science and yearARIF – a proxy for the ‘quality’ of the journal in which papers are published. It is based on symmetric impact factors of journal. Normalized by subfields of science and year.Specialization Index – an indicator of the intensity of research of a given geographic or organizational entity (e.g. a country) in a given research area (e.g. domain, field) relative to the intensity of the reference entity (e.g. the world) in the same research area. CAHS :QUESTIONS INDICATORS/MEASURESACTIVITY/PPRODUCITVITY WHAT HAS BEEN THE LEVEL OF ACTIVITY IN TERMS OF KNOWLEDGE GENERATION BY CANADA-BASED MEDICAL RESEARCHERS NUMBER OF PUBLICATIONSCONTEXTUAL AND STRUCTURALQUALITYOUTREACH
Average Relative Citation and Specialized IndexAll questions addressed1 represents0 is the world levelThe results suggest that within a 20 year window, most countries had made some significant advances in terms of improving the degree of research specialization (SI) and the quality of knowledge ARC produced in the Medical Sciences. The size of the circles is proportional to the number of papers produced by the entity. The position of an entity 1 top right – high impact scientific production, entities specialize in the given domain and their activities have a high impact – their papers are more frequently cites that the world average in this field.
Measures the “output” of individuals/research teams, institutions, and countries, to identify national and international networks, and to map the development of new (multi-disciplinary) fields of science and technology.
Figure 8-5 shows a patent tree for one of the first 50 completed ATP projects, A project to develop wafer ion-implantation, carried out by Diamond Semiconductor Group (DSG). Patent trees represent one aspect of knowledge dissemination, the patent trees extend our understanding of the influence of the new knowledge created on others. – relevance, extent of dissemination, and impact.
Advantage - Able to illustrate the density of connectivity, which is important for identifying key papers (from the researcher’s own lens) and the researchers’ most influential periodDisdvantabe3D objects can be difficult to understandDifferent arrangements of the research events could provide quite different impressionWe simplify the highly complex diagram shown previously in Figure 4 by showing only the linkages that are directly or indirectly linked to Patent 312 and move the relevant events inside the circle (Figure 6). Patent 312, granted in 1999, has observable direct linkages to three other key researcher markers: Grant 151 (1999), Paper 16 (1993) and Paper 20 (1993). The latter two have direct linkages to six and four papers respectively. These papers are further connected to four older papers. The oldest paper goes back to 1977 (Paper 0). Therefore, we would conclude that there was a time lag of 22 years, between Paper 0 and Patent 312.
Historical TracingEmphasis on tracing chronologically a series of interrelated developments leading from research to ultimate outcomes or from outcomes back to the factors that spawn them.Forward TracingWhen the objective is to evaluate a given project, forward tracing, where the analyst starts with the research of interest and traces the evolution of related events from the point forward, is generally more manageable and cost-effective than backward tracing….
Mental Health Retrosight investigates the translation of basic, clinical and interventional research into clinical application and community practice.Uses bibliometrics and literature review for hot topics and going forwardUses delphi surveys and clinical guidelines for interventions going backwards
This graph is blurry [Liza C.]
Transcript of "Effective Evaluation: Innovative Methods, Tools and Promising Practices"
Effective Evaluation: Innovative Methods, Tools and Promising Practices Canadian Evaluation Society May 2011 Kathryn Graham (PhD)
Alberta Innovates Health Solutions • Research and Innovations System (2010) - Formerly Alberta Heritage Foundation for Medical Research • Established in 1980 to support medical/health research • AIHS has provided unprecedented opportunities in research • Contributed > $1 billion to the scientific community • Supported > 8,500 researchers approximately
AIHS Mission “to support research and innovation activities to improve the health and well-being of Albertans and create, through innovation, health related social and economic benefits for Albertans”.
Methodological Issues• Attribution and Contribution• Time lags• Counterfactual• Administrative Burden
AIHS Performance Management and Evaluation Framework Multi-faceted Approach: • Mixed Methodology – Qualitative and Quantitative • Multi-Method – case studies to bibliometrics • Multi-Dimensional – five outcome categories
Hybrid Model Performance Evaluation Impact Management
Pathway to Impact • # quality of life • # modifiable risk factors • # accessibility Health Impact Enabling Outcomes Lead to Impact • # reports of inventions • # spin-off companies and jobs • Social benefits Socioeconomi c Impact
Emerging Methodologies – BibliometricsBibliometrics: are a set of methods and proceduresused in the measurement of bibliographic records”including scientific and technological outputs(e.g., journal publications and patents; ScienceMetrix, 2009)Uses:• Impact assessments• Productivity trends• Collaborative relationships• Program innovation• Patterns and intensity of knowledge dissemination
Considerations• Time lags – time must pass for citations to occur – self citations• Need to account for differences in organizations, technical fields, disciplines• Bias of databases towards English language• Contribution - uncertainty whether outputs are result of one grant or multi-grant• Multiple Lines of Evidence – complimentary – emerging indicators
An Example: : Bibliometric Indicators • Publications – in top journals, top 20 researchers, comparative publication rates – Productivity/Activity • Specialization Index – Contextual and Structural • Average Relative Impact Factor - Quality • Average Relative Citation (ARC) – Quality and Uptake • Collaboration rates: interprovincial, international, top 10, field comparisons, academic user collaborations - OutreachApril 2011 12
Positional Analysis for Canada and Selected Countries, 1985-1989 2 US UK France Canada CanadaARC 1 Sw eden Australia Sw itzerland Denmark Finland Norw ay 0 New -Zealand 0 1 2 SI The ARC and the SI for the period is estimated using the average ARC and SI over the period Source – Science Metrix and Graph CIHR
Positional Analysis, for Canada and Selected Countries, 1990-1999 2 US UK France Canada Canada ARC Australia 1 Sw eden Sw itzerland Denmark Finland Norw ay 0 New -Zealand 0 1 2 SI The ARC and the SI for the period is estimated using the average ARC and SI over the period Source – Science Metrix and Graph CIHR
Positional Analysis for Canada and Selected Countries, 2000-2008 2.0 US UK France Canada Canada AustraliaARC 1.0 Sw eden Sw itzerland Denmark Finland Norw ay 0.0 New -Zealand 0.0 1.0 2.0 SI The ARC and the SI for the period is estimated using the average ARC and SI over the period Source – Science Metrix and Graph CIHR
Emerging Methodologies – Patent CitationTreesPatent citation trees show forward citing of thepatents generated by funded completed projects.Uses:• Citations can be used to track the dissemination of technical knowledge to subsequent publications and patents.• Help understand the influence of new knowledge created on others.• Relevance, extent of dissemination and impact.
Current Methodologies – Patent Trees Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 121.
An Example: Network Diagrams Source Time Lags a Feasibility study Rand Europe – Draft 2011
Case Studies Using Pay BackCase studies are in-depth investigations into aprogram, project, facility or phenomenon, usually to examine whathappened, to describe the context in which it happened, to explorehow and why, and to consider what would have happenedotherwise.• Historical Tracing – start with outcomes back• Forward Tracing – start with research of interest and trace related events going forward
Impact Assessments – Pay Back CaseStudiesUse:• Identify exemplary or best-practice experiences• Identify the long term payback from health research• Explore factors associated with successful research translation• Provide insights to inform future funding policyConsiderations:• Results may not be generalizable
Outcome Mapping – Developmental Evaluation Outcome mapping is a participatory method for planning, monitoring and evaluation; focuses on changes in behaviour of those with whom the project or programs works and oriented towards social change and organizational learning.Uses:• Focus on outcomes as changes in behaviour• Sphere of influence• Logical path
Next Steps - Performance Composite Index • Using Factor Analysis and Regression Analysis reduce the number of metrics and construct index in relation to each program goals. • Develop weighting system for goals – evidence from other studies • Ability to get a clear view of program performance. Across a portfolio of programs. • Indicate performance at different stages of the innovation process •
Promising Practices• Science of Science Network• Alliances• Innovation Platforms – Collective measurement and Shared Value• Forum on Research Evaluation Methodologies
ResourcesBuxton MJ & Hanney S (1996) How can payback from health services research be assessed? Journal of HealthServices Research & Policy, vol 1, no 1, pp 35-43.Panel on Return on Investment in Health Research. (2009). Making an Impact: A Preferred Framework and Indicators toMeasure Returns on Investment in Health Research. Canadian Academy of Health Sciences: Ottawa, CanadaEarl S. et al. (2001), Outcome Mapping: Building Learning and Reflection into Development Programs. InternationalDevelopment Research Centre, CanadaGamble, J. (2008). A developmental evaluation primer. The J.W. McConnell Family Foundation: Montréal.Michael Quinn Patton (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use.Guilford Press: New York.Kaplan, RS & Norton, DP (1992). The Balanced Scorecard: Measures that Drive Performance. Harvard BusinessReview, (January-February), pp. 71-79.Kaplan, RS & Norton, DP (1996). Using the Balanced Scorecard as a Strategic Management System. Harvard BusinessReview, (January/February), pp. 75-85Kaplan, RS & Norton, DP (1996). Balanced Scorecard: Translating Strategy into Action. Harvard Business School Press:Cambridge, MassachusettsKaplan RS & Norton DP (2000). The Strategy Focused Organization. Harvard Business School Press: USAKaplan RS & Norton DP (2004). Strategy Maps: Converting intangible assets into tangible outcomes. Harvard BusinessSchool Press: BostonMcLaughlin, JA, & Jordan, JB (2004). Logic Models. In, J. S. Wholey, H. P. Hatry & K. E. Newcomer (Eds). Handbook ofPractical Program Evaluation, 2nd Edition .San Francisco, CA: Jossey-Bass.Mercure, S, Cote, G, Archambault, E. (2007). Scientific Positioning of Alberta in Health Research Specialties. Science-Metrix. Montreal, Canada. Retrieved from http://www.ahfmr.ab.ca/download.php/b833a02029ebd9e62fd286168dfc2615
Contact information Kathryn Graham (PhD) Director, Performance Management Alberta Innovates Health Solutions 1500 – 10104 103 Avenue N.W. Edmonton, Alberta, Canada T5J 4A7 Email: email@example.com THANK YOU