Evaluating Problem Gambling KTE

Anne Bergen
Anne BergenPrincipal at Knowledge to Action Consulting
Evaluating
Problem Gambling KTE
Part of the OPGRC’s Moving Research Forward workshop series
February 2015
get in touch
Anne Bergen, PhD
knowledgetoaction.ca
@anne_bergen
Reuse & Attribution
This work is licensed under a
Creative Commons Attribution 4.0 International License.
Attribute this work as:
Bergen, A. (2015). Evaluating Problem Gambling KTE.
Moving Research Forward Workshop Series, Ontario
Problem Gambling Research Centre, Guelph ON.
Introductions
Why KTE
evaluation?
Evaluation can help you understand:
• Need for KTE activity
• Quality of KTE activity
• Reach of KTE activity
• Impact of KTE activity
KTEà Impact?
What kind of impact?
For whom?
Under what conditions?
When?
Based on what evidence?
What counts as
evidence?
Evaluation research
vs.
evaluation Research
how used?
to be published?
funded?
What are your
evaluation goals?
Evaluation Continuum
How will you use the evaluation results?
– What?
– So What?
– Now What?
Outputs Engagement Uptake Use Impact
Causal
Attributions
key questions for KTE
1.  What research knowledge should be transferred?
2.  To whom should research knowledge be
transferred?
3.  By whom should research knowledge be
transferred?
4.  How should research knowledge be transferred?
5.  With what effect should research knowledge be
transferred?
Adapted from: Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., &
Abelson, J. (2003). How can research organizations more effectively transfer
research knowledge to decision makers? Milbank Quarterly, 81(2), 221-248.
key questions for KTE evaluation
1.  What research knowledge was transferred?
2. To whom was research knowledge
transferred?
3.  By whom was research knowledge
transferred?
4. How was research knowledge transferred?
5.  With what effect was research knowledge
transferred?
Adapted from: Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., & Abelson, J. (2003).
How can research organizations more effectively transfer research knowledge to decision makers?
Milbank Quarterly, 81(2), 221-248.
Evaluation Over KTE Project Lifespan
•  Before: Needs assessment
•  Creation: Developmental evaluation
– user experience/ usability testing
•  During: Process (formative) evaluation
– quality & quantity of outputs
•  After: Summative (outcome) evaluation
– outcomes & impacts
Evaluation Goals
• What aspects of KTE do you
want to evaluate?
• What are you going to do with
the resulting information?
Evaluation Framework
•  Goals
•  Values
•  Approach
•  Logic model/ theory of change
•  Targets of change (stakeholders)
•  Methods, metrics, & measures
What are your
evaluation values?
More impact.
Inform Consult Involve Collaborate Empower
Less control. Slower process.
Adapted from Arnstein’s (1969) Ladder of Public Participation and the
IAP2 Spectrum of Public Participation
How engaged are your stakeholders in your
research & KTE activities?
What is your
evaluation context?
Research Funding Organization
Program of Research A
Project
A1
Product
A1a
Product
A1b
Project
A2
Product
A2a
Product
A2b
Product
A2c
Project
A3
Product
A3a
Product
A3b
Program of
Research B
Project
B1
Product
B1a
Product
B1b
Project
B2
Product
B2a
Product
B2b
Research Funding Organization
Program of Research A
Project
A1
Product
A1a
Product
A1b
Project
A2
Product
A2a
Product
A2b
Product
A2c
Project
A3
Product
A3a
Product
A3b
Program of
Research B
Project
B1
Product
B1a
Product
B1b
Project
B2
Product
B2a
Product
B2b
Context
Research Funding Organization
Program of Research A
Project
A1
Product
A1a
Product
A1b
Project
A2
Product
A2a
Product
A2b
Product
A2c
Project
A3
Product
A3a
Product
A3b
Program of
Research B
Project
B1
Product
B1a
Product
B1b
Project
B2
Product
B2a
Product
B2b
Evaluation Context (macro & micro)
•  Micro - (time, money, resources, intended
use, intended users, reporting requirements,
etc…)
•  Macro – (culture, leadership, evaluation,
external systems, political climate, etc….)
What approach to
evaluation will you
take?
http://betterevaluation.org
What Kind of Evaluation?
Evaluation research
vs.
evaluation Research
how used?
to be published?
funded?
practical & theoretical considerations
Audience/ Targets of Change
q  gamblers and problem gamblers
q  gambling industry
q  gambling researchers
q  general public
q  policymakers
q  subsets of the general public (e.g., youth)
q 
Methods
q  Qualitative
q  Quantitative
q  Mixed Methods
q  Prospective
q  Retrospective
q  .
q  Participatory?
q  Descriptive?
q  RCT?
q  Pre-Post?
q  Counterfactual?
q  Contribution analysis?
q  ..
What is the
theory of change?
(logical links between activities &
desired outcomes)
Theory of Change
Problem or
Issue
1
Needs &
Assets
2 Desired
Results
(outputs, outcomes)
3
Influential
Factors
4
KTE
Strategies
5
Assumptions
6
Adapted from Kellogg Foundation Theory of Change template
http://www.wkkf.org/resource-directory/resource/2007/07/spark-theory-of-change
Theory of Change
Problem or Issue
1
Needs & Assets
2
Desired Results
(outputs, outcomes)
3
Influential Factors
4
KTE Strategies
5
Assumptions
6
Adapted from Kellogg Foundation Theory of Change template
http://www.wkkf.org/resource-directory/resource/2007/07/spark-theory-of-change
Another approach: logic models
Assumptions
Think about individuals, groups, systems, &
assumptions
Activities Short-Term
Outcomes
Intermediate
Outcomes
Long-Term
Outcomes
Systems &
Policy Change
Initiatives
• Share
resources
• Create
partnerships
Awareness
Systems
Changes
Reduced harm
from gambling
Partnerships
Individual
Change
Initiatives
• Provide
education and
resources on…
• Provide
environmental
cues to…
Knowledge &
skills
Intention &
Behaviour
Changes
Psychosocial
Gambling
context
An example:
process-focused
KTE evaluation
Final	
  research	
  summary	
  created	
  
Increased	
  	
  
awareness	
  of	
  
research	
  
scope	
  among	
  
project	
  team	
  
members	
  
and	
  partners.	
  
	
  
Increase	
  in	
  	
  
perceived	
  
ownership	
  
over	
  project.	
  
	
  
Increased	
  
clarity	
  and	
  
relevance	
  of	
  
summary	
  
	
  
Increased	
  
(clear	
  
language)	
  
communica=
on	
  and	
  
transla=on	
  
skills.	
  
Document	
  
summary	
  
candidates	
  and	
  
full	
  ar=cles	
  in	
  
database.	
  
	
  
Create	
  
agreement	
  on	
  
process	
  for	
  
summaries	
  
within	
  project	
  
team.	
  
Hire	
  students	
  
from	
  across	
  
University.	
  
Format	
  
summary	
  using	
  
standard	
  
template	
  
Student	
  
writers	
  work	
  
with	
  faculty	
  to	
  
revise	
  
summary	
  
draDs	
  
Provide	
  
training	
  for	
  
writers	
  and	
  
stakeholders	
  
(staff).	
  
Increased	
  
accessibility	
  
of	
  summary	
  
	
  
Increased	
  
trust	
  of	
  
research	
  
product	
  
(successful	
  
branding)	
  
	
  
Ac=vi=es	
  Short	
  Term	
  Goals	
  Long	
  Term	
  Goals	
  
Increase	
  in	
  	
  
inter-­‐silo	
  
communica=
on.	
  
Increased	
  uptake	
  and	
  use	
  of	
  summary	
  and	
  original	
  
research	
  
Increased	
  
knowledge	
  
sharing	
  within	
  
and	
  outside	
  of	
  
academia	
  
Increased	
  
quality	
  of	
  
summary	
  
	
  
Increase	
  in	
  project	
  team	
  and	
  partner	
  
collabora=on:	
  
	
  
Industry	
  Communi=es	
  Policy-­‐Makers	
   Prac==oners	
  
Staff	
  Students	
   Faculty	
  
Preparation
Make	
  research	
  summaries	
  available	
  through:	
  
Social	
  Media	
  
Tradi=onal	
  
Media	
  
Government	
  
Partners	
  
Google	
  Search	
  University	
  
Business	
  
Support	
  Office	
  
DisseminationProduction
Internal	
  	
   Collabora=on	
  
KMb	
  
ins=tu=onal	
  
partners	
  
Collabora=on	
  
with	
  
Researchers	
  
and	
  Research	
  
Users	
  	
  
Logic model for a university-
based plain language summary
project
A	
  mid-­‐term	
  evalua=on	
  included	
  a	
  review	
  of	
  products	
  produced,	
  
processes	
  used	
  (including	
  collabora=on	
  across	
  the	
  Project	
  Team,	
  
the	
  system	
  for	
  deciding	
  on	
  summary	
  topics)	
  and	
  success	
  of	
  the	
  
social	
  network	
  strategy.	
  
Final	
  research	
  summary	
  created	
  
Increased	
  	
  
awareness	
  of	
  
research	
  
scope	
  among	
  
project	
  team	
  
members	
  
and	
  partners.	
  
	
  
Increase	
  in	
  	
  
perceived	
  
ownership	
  
over	
  project.	
  
	
  
Increased	
  
clarity	
  and	
  
relevance	
  of	
  
summary	
  
	
  
Increased	
  
(clear	
  
language)	
  
communica=
on	
  and	
  
transla=on	
  
skills.	
  
Document	
  
summary	
  
candidates	
  and	
  
full	
  ar=cles	
  in	
  
database.	
  
	
  
Create	
  
agreement	
  on	
  
process	
  for	
  
summaries	
  
within	
  project	
  
team.	
  
Hire	
  students	
  
from	
  across	
  
University.	
  
Format	
  
summary	
  using	
  
standard	
  
template	
  
Student	
  
writers	
  work	
  
with	
  faculty	
  to	
  
revise	
  
summary	
  
draDs	
  
Provide	
  
training	
  for	
  
writers	
  and	
  
stakeholders	
  
(staff).	
  
Increased	
  
accessibility	
  
of	
  summary	
  
	
  
Increased	
  
trust	
  of	
  
research	
  
product	
  
(successful	
  
branding)	
  
	
  
Ac=vi=es	
  Short	
  Term	
  Goals	
  Long	
  Term	
  Goals	
  
Increase	
  in	
  	
  
inter-­‐silo	
  
communica=
on.	
  
Increased	
  uptake	
  and	
  use	
  of	
  summary	
  and	
  original	
  
research	
  
Increased	
  
knowledge	
  
sharing	
  within	
  
and	
  outside	
  of	
  
academia	
  
Increased	
  
quality	
  of	
  
summary	
  
	
  
Increase	
  in	
  project	
  team	
  and	
  partner	
  
collabora=on:	
  
	
  
Industry	
  Communi=es	
  Policy-­‐Makers	
   Prac==oners	
  
Staff	
  Students	
   Faculty	
  
Preparation
Make	
  research	
  summaries	
  available	
  through:	
  
Social	
  Media	
  
Tradi=onal	
  
Media	
  
Government	
  
Partners	
  
Google	
  Search	
  University	
  
Business	
  
Support	
  Office	
  
DisseminationProduction
Internal	
  	
   Collabora=on	
  
KMb	
  
ins=tu=onal	
  
partners	
  
Collabora=on	
  
with	
  
Researchers	
  
and	
  Research	
  
Users	
  	
  
1.  Count & describe
2.  Ask respondents
(stakeholders) to
describe collab
process/why various
pieces were
important
1.  Post-training survey
(quality of training;
knowledge of plain
language)
2.  Writer & faculty
survey (quality of
process & outputs)
3.  Useful? Easy to
use? Clear? (1)
focus groups of
policy makers,
researchers, and
community partners
and (2) post focus-
group survey
4. Focus group/
survey for
stakeholders.
1. # tweets, RT, etc…
2.  # followers
3. # hits & downloads
An example:
descriptive
KTE evaluation
Cooper, A. (2011).
Knowledge Mobilization
in Education
A cross-case analysis of
44 research brokering
organizations across
Canada.
Unpublished doctoral
thesis, University of
Toronto, Toronto,
Ontario, Canada.
Cooper’s (2011) Brokering Functions of KM Strategies diagram
shows possible activities & outcomes
An example:
outcome-focused
KTE evaluation
Contribution analysis
1.  Map a “journey” (theory of change) that links activities
to increased research uptake to outcomes
2.  Identify assumptions and risks for each stage of the
journey
3.  Identify indicators for research uptake, use and impact
4.  Collect evidence.
5.  Review the map against your evidence. Identify gaps in
evidence, then try to fill the gaps.
6.  Write your contribution story. Explain the journey from
inputs to impacts.
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18, 270-280. doi:10.1177/1356389012451663
Morton. (2013). Assessing research impact: A case study of participatory research. Centre for Research on Families and
Relationships. https://www.era.lib.ed.ac.uk/handle/1842/6562
Morton, S. (2013).
Assessing research
impact: A case study
of participatory
research. Centre for
Research on Families
and Relationships, 66.
www.crfr.ac.uk
Example of “impact journey” diagram from Sarah Morton’s work
What measures and
indicators will you
use?
(metrics to stories)
Outputs vs. Outcomes
Outputs Engagement Uptake Use Impact
Causal
Attributions
measurement time & complexity
KTE indicators for:
•  Number/type of KTE products
•  Ease of use/ user experience
•  Timing/ relevance (meets audience needs)
•  Awareness/ attitudes/ beliefs/ knowledge
•  Self-reported intentions/ behaviour
•  Networks/ relationships/ collaborations
•  Systems/ policies/ organizational culture
Social Process, Social Impact
q  Quality & quantity of relationships
q  1:1 relationships; organizational relationships
q  Meetings
q  Requests & referrals
q  Co-produced products
q  Social network analysis
q  .
Group Feedback
What did you create?
+/- method: user experience testing
•  10 minutes). Review the workbook on your own. Mark the margins
with a plus (+) or minus (-) to record your positive and negative
reading experiences.
•  (10 + 10 minutes). With a partner, review the motives for each of
the +/- in the document. Ask probing questions where needed.
Record your responses and note the top 3 most important positive
and negative experiences.
•  (15 minutes). As a small group, give feedback on top 3 +/- for each
partner. What was effective/ not effective and why?
•  (15 minutes). As a large group, review trends in +/- experiences for
each group.
Create your own KTE
evaluation framework
Evaluation Framework
•  Your research
•  Your KTE
•  Evaluation goals
•  Values
•  Approach
•  Logic model/ theory of change
•  Targets of change (stakeholders)
•  Methods, metrics, & measures
W3
What?
So What?
Now What?
What?
What happened?
What did I observe?
What stood out for me?
So What?
Why is that important? How did
my understanding change?
Now What?
How will my practice change:
…over the next 3 months?
…over the next year?
1 of 53

Recommended

Skills & ideas for #ProblemGamblingKTE by
Skills & ideas for #ProblemGamblingKTE Skills & ideas for #ProblemGamblingKTE
Skills & ideas for #ProblemGamblingKTE Anne Bergen
2.3K views52 slides
Evaluating research impact: From a specific case to general guidelines. by
Evaluating research impact: From a specific case to general guidelines. Evaluating research impact: From a specific case to general guidelines.
Evaluating research impact: From a specific case to general guidelines. Anne Bergen
380 views41 slides
Bioinformatic core facilities discussion by
Bioinformatic core facilities discussionBioinformatic core facilities discussion
Bioinformatic core facilities discussionJennifer Shelton
754 views11 slides
SCC2011 - Evaluation: Facing the tricky questions by
SCC2011 - Evaluation: Facing the tricky questionsSCC2011 - Evaluation: Facing the tricky questions
SCC2011 - Evaluation: Facing the tricky questionsBritish Science Association
697 views37 slides
Research-To-Impact Canvas - one page planning tool for research, KT and Comme... by
Research-To-Impact Canvas - one page planning tool for research, KT and Comme...Research-To-Impact Canvas - one page planning tool for research, KT and Comme...
Research-To-Impact Canvas - one page planning tool for research, KT and Comme...KBHN KT
2K views1 slide
"Hybrid" KT & Project Planning Template by
"Hybrid" KT & Project Planning Template "Hybrid" KT & Project Planning Template
"Hybrid" KT & Project Planning Template KBHN KT
4K views55 slides

More Related Content

What's hot

"Hybrid" KT & Project Planning GUIDE by
"Hybrid" KT & Project Planning GUIDE"Hybrid" KT & Project Planning GUIDE
"Hybrid" KT & Project Planning GUIDEKBHN KT
1.2K views57 slides
Infographic Guide of Guides ACCESSIBLE2 by
Infographic Guide of Guides ACCESSIBLE2Infographic Guide of Guides ACCESSIBLE2
Infographic Guide of Guides ACCESSIBLE2KBHN KT
946 views27 slides
Strengthening an Organization’s Capacity to Demand and Use Data by
Strengthening an Organization’s Capacity to Demand and Use DataStrengthening an Organization’s Capacity to Demand and Use Data
Strengthening an Organization’s Capacity to Demand and Use DataMEASURE Evaluation
1.8K views29 slides
Emerging Participatory Research Methods for Digital Health Communications by
Emerging Participatory Research Methods for Digital Health CommunicationsEmerging Participatory Research Methods for Digital Health Communications
Emerging Participatory Research Methods for Digital Health CommunicationsAlex Bornkessel
2.2K views21 slides
Module 6.D Implementation: project management by
Module 6.D Implementation: project management Module 6.D Implementation: project management
Module 6.D Implementation: project management Togar Simatupang
74 views56 slides
Information Use Map by
Information Use MapInformation Use Map
Information Use MapMEASURE Evaluation
1.3K views26 slides

What's hot(20)

"Hybrid" KT & Project Planning GUIDE by KBHN KT
"Hybrid" KT & Project Planning GUIDE"Hybrid" KT & Project Planning GUIDE
"Hybrid" KT & Project Planning GUIDE
KBHN KT1.2K views
Infographic Guide of Guides ACCESSIBLE2 by KBHN KT
Infographic Guide of Guides ACCESSIBLE2Infographic Guide of Guides ACCESSIBLE2
Infographic Guide of Guides ACCESSIBLE2
KBHN KT946 views
Strengthening an Organization’s Capacity to Demand and Use Data by MEASURE Evaluation
Strengthening an Organization’s Capacity to Demand and Use DataStrengthening an Organization’s Capacity to Demand and Use Data
Strengthening an Organization’s Capacity to Demand and Use Data
MEASURE Evaluation1.8K views
Emerging Participatory Research Methods for Digital Health Communications by Alex Bornkessel
Emerging Participatory Research Methods for Digital Health CommunicationsEmerging Participatory Research Methods for Digital Health Communications
Emerging Participatory Research Methods for Digital Health Communications
Alex Bornkessel2.2K views
Module 6.D Implementation: project management by Togar Simatupang
Module 6.D Implementation: project management Module 6.D Implementation: project management
Module 6.D Implementation: project management
Togar Simatupang74 views
KM Impact Challenge - Sharing findings of synthesis report by kmimpactchallenge
KM Impact Challenge - Sharing findings of synthesis reportKM Impact Challenge - Sharing findings of synthesis report
KM Impact Challenge - Sharing findings of synthesis report
kmimpactchallenge354 views
Shared Rationales in Group Activities by Lu Xiao
Shared Rationales in Group ActivitiesShared Rationales in Group Activities
Shared Rationales in Group Activities
Lu Xiao882 views
DIN: Danish Design-driven Innovation by Marcin Monko
DIN: Danish Design-driven InnovationDIN: Danish Design-driven Innovation
DIN: Danish Design-driven Innovation
Marcin Monko799 views
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most... by InsightInnovation
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
InsightInnovation494 views
The Use of Digital Media by Small and Medium-Sized Enterprises by Mark Anthony Camilleri
The Use of Digital Media by Small and Medium-Sized EnterprisesThe Use of Digital Media by Small and Medium-Sized Enterprises
The Use of Digital Media by Small and Medium-Sized Enterprises
Using a Developmental Approach to Evaluating Innovative Extension Programs by Sarah Baughman
Using a Developmental Approach to Evaluating Innovative Extension ProgramsUsing a Developmental Approach to Evaluating Innovative Extension Programs
Using a Developmental Approach to Evaluating Innovative Extension Programs
Sarah Baughman389 views
Stakeholder Engagement Guide of Guides by KBHN KT
Stakeholder Engagement Guide of GuidesStakeholder Engagement Guide of Guides
Stakeholder Engagement Guide of Guides
KBHN KT13K views
103-Basco Identifying metrics to fully assess the impact of federal R&D inves... by innovationoecd
103-Basco Identifying metrics to fully assess the impact of federal R&D inves...103-Basco Identifying metrics to fully assess the impact of federal R&D inves...
103-Basco Identifying metrics to fully assess the impact of federal R&D inves...
innovationoecd388 views
NHS England with Agile Elephant: Collaboration at scale across health & ... by Janet Parkinson
NHS England with Agile Elephant:  Collaboration at scale across health & ...NHS England with Agile Elephant:  Collaboration at scale across health & ...
NHS England with Agile Elephant: Collaboration at scale across health & ...
Janet Parkinson83 views

Similar to Evaluating Problem Gambling KTE

Evaluability Assessments and Choice of Evaluation Methods by
Evaluability Assessments and Choice of Evaluation MethodsEvaluability Assessments and Choice of Evaluation Methods
Evaluability Assessments and Choice of Evaluation MethodsDebbie_at_IDS
2.8K views25 slides
U koersythesis&evaluation by
U koersythesis&evaluationU koersythesis&evaluation
U koersythesis&evaluationloumcgill
585 views18 slides
Evaluation research-resty-samosa by
Evaluation research-resty-samosaEvaluation research-resty-samosa
Evaluation research-resty-samosaResty Samosa
9.8K views79 slides
Visualising Learner Behaviours in MOOCs - Ascilite 2018 presentation by
Visualising Learner Behaviours in MOOCs - Ascilite 2018 presentationVisualising Learner Behaviours in MOOCs - Ascilite 2018 presentation
Visualising Learner Behaviours in MOOCs - Ascilite 2018 presentationUniversity of Newcastle, NSW.
370 views35 slides
How to Write a Strong Fellowship Application Webinar slides.pptx by
How to Write a Strong Fellowship Application Webinar slides.pptxHow to Write a Strong Fellowship Application Webinar slides.pptx
How to Write a Strong Fellowship Application Webinar slides.pptxEleanorCollard
80 views26 slides
Designing useful evaluations - An online workshop for the Jisc AF programme_I... by
Designing useful evaluations - An online workshop for the Jisc AF programme_I...Designing useful evaluations - An online workshop for the Jisc AF programme_I...
Designing useful evaluations - An online workshop for the Jisc AF programme_I...Rachel Harris
7.6K views21 slides

Similar to Evaluating Problem Gambling KTE(20)

Evaluability Assessments and Choice of Evaluation Methods by Debbie_at_IDS
Evaluability Assessments and Choice of Evaluation MethodsEvaluability Assessments and Choice of Evaluation Methods
Evaluability Assessments and Choice of Evaluation Methods
Debbie_at_IDS2.8K views
U koersythesis&evaluation by loumcgill
U koersythesis&evaluationU koersythesis&evaluation
U koersythesis&evaluation
loumcgill585 views
Evaluation research-resty-samosa by Resty Samosa
Evaluation research-resty-samosaEvaluation research-resty-samosa
Evaluation research-resty-samosa
Resty Samosa9.8K views
How to Write a Strong Fellowship Application Webinar slides.pptx by EleanorCollard
How to Write a Strong Fellowship Application Webinar slides.pptxHow to Write a Strong Fellowship Application Webinar slides.pptx
How to Write a Strong Fellowship Application Webinar slides.pptx
EleanorCollard80 views
Designing useful evaluations - An online workshop for the Jisc AF programme_I... by Rachel Harris
Designing useful evaluations - An online workshop for the Jisc AF programme_I...Designing useful evaluations - An online workshop for the Jisc AF programme_I...
Designing useful evaluations - An online workshop for the Jisc AF programme_I...
Rachel Harris7.6K views
Community engagement - what constitutes success by contentli
Community engagement - what constitutes successCommunity engagement - what constitutes success
Community engagement - what constitutes success
contentli586 views
Evaluation Workshop by Noel Hatch
Evaluation WorkshopEvaluation Workshop
Evaluation Workshop
Noel Hatch 719 views
Monitoring R&D functions by Nandita Das
Monitoring R&D functionsMonitoring R&D functions
Monitoring R&D functions
Nandita Das182 views
Evaluation training for wellcome trust 15th may by Bruce Etherington
Evaluation training for wellcome trust 15th mayEvaluation training for wellcome trust 15th may
Evaluation training for wellcome trust 15th may
Bruce Etherington656 views
SOEDS, 11th April 2022 How to Evaluate CSR Projects and Programmes.pptx by RAKESHNANDAN7
SOEDS, 11th April 2022 How to Evaluate CSR Projects and Programmes.pptxSOEDS, 11th April 2022 How to Evaluate CSR Projects and Programmes.pptx
SOEDS, 11th April 2022 How to Evaluate CSR Projects and Programmes.pptx
RAKESHNANDAN75 views
Botor_project_research_methodology_2016 by Shayne Botor
Botor_project_research_methodology_2016Botor_project_research_methodology_2016
Botor_project_research_methodology_2016
Shayne Botor234 views

Recently uploaded

Java Simplified: Understanding Programming Basics by
Java Simplified: Understanding Programming BasicsJava Simplified: Understanding Programming Basics
Java Simplified: Understanding Programming BasicsAkshaj Vadakkath Joshy
322 views155 slides
Gopal Chakraborty Memorial Quiz 2.0 Prelims.pptx by
Gopal Chakraborty Memorial Quiz 2.0 Prelims.pptxGopal Chakraborty Memorial Quiz 2.0 Prelims.pptx
Gopal Chakraborty Memorial Quiz 2.0 Prelims.pptxDebapriya Chakraborty
695 views81 slides
Recap of our Class by
Recap of our ClassRecap of our Class
Recap of our ClassCorinne Weisgerber
88 views15 slides
MercerJesse2.1Doc.pdf by
MercerJesse2.1Doc.pdfMercerJesse2.1Doc.pdf
MercerJesse2.1Doc.pdfjessemercerail
273 views5 slides
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB... by
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection
88 views113 slides
Thanksgiving!.pdf by
Thanksgiving!.pdfThanksgiving!.pdf
Thanksgiving!.pdfEnglishCEIPdeSigeiro
264 views17 slides

Recently uploaded(20)

BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB... by Nguyen Thanh Tu Collection
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
Dance KS5 Breakdown by WestHatch
Dance KS5 BreakdownDance KS5 Breakdown
Dance KS5 Breakdown
WestHatch99 views
Solar System and Galaxies.pptx by DrHafizKosar
Solar System and Galaxies.pptxSolar System and Galaxies.pptx
Solar System and Galaxies.pptx
DrHafizKosar106 views
Narration lesson plan by TARIQ KHAN
Narration lesson planNarration lesson plan
Narration lesson plan
TARIQ KHAN61 views
Monthly Information Session for MV Asterix (November) by Esquimalt MFRC
Monthly Information Session for MV Asterix (November)Monthly Information Session for MV Asterix (November)
Monthly Information Session for MV Asterix (November)
Esquimalt MFRC72 views
Ch. 7 Political Participation and Elections.pptx by Rommel Regala
Ch. 7 Political Participation and Elections.pptxCh. 7 Political Participation and Elections.pptx
Ch. 7 Political Participation and Elections.pptx
Rommel Regala111 views
Ch. 8 Political Party and Party System.pptx by Rommel Regala
Ch. 8 Political Party and Party System.pptxCh. 8 Political Party and Party System.pptx
Ch. 8 Political Party and Party System.pptx
Rommel Regala54 views
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx by Ms. Pooja Bhandare
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptxPharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx
Pharmaceutical Inorganic chemistry UNIT-V Radiopharmaceutical.pptx
Ms. Pooja Bhandare113 views
When Sex Gets Complicated: Porn, Affairs, & Cybersex by Marlene Maheu
When Sex Gets Complicated: Porn, Affairs, & CybersexWhen Sex Gets Complicated: Porn, Affairs, & Cybersex
When Sex Gets Complicated: Porn, Affairs, & Cybersex
Marlene Maheu85 views
Drama KS5 Breakdown by WestHatch
Drama KS5 BreakdownDrama KS5 Breakdown
Drama KS5 Breakdown
WestHatch98 views
Relationship of psychology with other subjects. by palswagata2003
Relationship of psychology with other subjects.Relationship of psychology with other subjects.
Relationship of psychology with other subjects.
palswagata200352 views

Evaluating Problem Gambling KTE

  • 1. Evaluating Problem Gambling KTE Part of the OPGRC’s Moving Research Forward workshop series February 2015
  • 2. get in touch Anne Bergen, PhD knowledgetoaction.ca @anne_bergen
  • 3. Reuse & Attribution This work is licensed under a Creative Commons Attribution 4.0 International License. Attribute this work as: Bergen, A. (2015). Evaluating Problem Gambling KTE. Moving Research Forward Workshop Series, Ontario Problem Gambling Research Centre, Guelph ON.
  • 6. Evaluation can help you understand: • Need for KTE activity • Quality of KTE activity • Reach of KTE activity • Impact of KTE activity
  • 7. KTEà Impact? What kind of impact? For whom? Under what conditions? When? Based on what evidence?
  • 9. Evaluation research vs. evaluation Research how used? to be published? funded?
  • 11. Evaluation Continuum How will you use the evaluation results? – What? – So What? – Now What? Outputs Engagement Uptake Use Impact Causal Attributions
  • 12. key questions for KTE 1.  What research knowledge should be transferred? 2.  To whom should research knowledge be transferred? 3.  By whom should research knowledge be transferred? 4.  How should research knowledge be transferred? 5.  With what effect should research knowledge be transferred? Adapted from: Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision makers? Milbank Quarterly, 81(2), 221-248.
  • 13. key questions for KTE evaluation 1.  What research knowledge was transferred? 2. To whom was research knowledge transferred? 3.  By whom was research knowledge transferred? 4. How was research knowledge transferred? 5.  With what effect was research knowledge transferred? Adapted from: Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision makers? Milbank Quarterly, 81(2), 221-248.
  • 14. Evaluation Over KTE Project Lifespan •  Before: Needs assessment •  Creation: Developmental evaluation – user experience/ usability testing •  During: Process (formative) evaluation – quality & quantity of outputs •  After: Summative (outcome) evaluation – outcomes & impacts
  • 15. Evaluation Goals • What aspects of KTE do you want to evaluate? • What are you going to do with the resulting information?
  • 16. Evaluation Framework •  Goals •  Values •  Approach •  Logic model/ theory of change •  Targets of change (stakeholders) •  Methods, metrics, & measures
  • 18. More impact. Inform Consult Involve Collaborate Empower Less control. Slower process. Adapted from Arnstein’s (1969) Ladder of Public Participation and the IAP2 Spectrum of Public Participation How engaged are your stakeholders in your research & KTE activities?
  • 20. Research Funding Organization Program of Research A Project A1 Product A1a Product A1b Project A2 Product A2a Product A2b Product A2c Project A3 Product A3a Product A3b Program of Research B Project B1 Product B1a Product B1b Project B2 Product B2a Product B2b
  • 21. Research Funding Organization Program of Research A Project A1 Product A1a Product A1b Project A2 Product A2a Product A2b Product A2c Project A3 Product A3a Product A3b Program of Research B Project B1 Product B1a Product B1b Project B2 Product B2a Product B2b
  • 22. Context Research Funding Organization Program of Research A Project A1 Product A1a Product A1b Project A2 Product A2a Product A2b Product A2c Project A3 Product A3a Product A3b Program of Research B Project B1 Product B1a Product B1b Project B2 Product B2a Product B2b
  • 23. Evaluation Context (macro & micro) •  Micro - (time, money, resources, intended use, intended users, reporting requirements, etc…) •  Macro – (culture, leadership, evaluation, external systems, political climate, etc….)
  • 24. What approach to evaluation will you take?
  • 26. What Kind of Evaluation? Evaluation research vs. evaluation Research how used? to be published? funded? practical & theoretical considerations
  • 27. Audience/ Targets of Change q  gamblers and problem gamblers q  gambling industry q  gambling researchers q  general public q  policymakers q  subsets of the general public (e.g., youth) q 
  • 28. Methods q  Qualitative q  Quantitative q  Mixed Methods q  Prospective q  Retrospective q  . q  Participatory? q  Descriptive? q  RCT? q  Pre-Post? q  Counterfactual? q  Contribution analysis? q  ..
  • 29. What is the theory of change? (logical links between activities & desired outcomes)
  • 30. Theory of Change Problem or Issue 1 Needs & Assets 2 Desired Results (outputs, outcomes) 3 Influential Factors 4 KTE Strategies 5 Assumptions 6 Adapted from Kellogg Foundation Theory of Change template http://www.wkkf.org/resource-directory/resource/2007/07/spark-theory-of-change
  • 31. Theory of Change Problem or Issue 1 Needs & Assets 2 Desired Results (outputs, outcomes) 3 Influential Factors 4 KTE Strategies 5 Assumptions 6 Adapted from Kellogg Foundation Theory of Change template http://www.wkkf.org/resource-directory/resource/2007/07/spark-theory-of-change
  • 33. Assumptions Think about individuals, groups, systems, & assumptions Activities Short-Term Outcomes Intermediate Outcomes Long-Term Outcomes Systems & Policy Change Initiatives • Share resources • Create partnerships Awareness Systems Changes Reduced harm from gambling Partnerships Individual Change Initiatives • Provide education and resources on… • Provide environmental cues to… Knowledge & skills Intention & Behaviour Changes Psychosocial Gambling context
  • 35. Final  research  summary  created   Increased     awareness  of   research   scope  among   project  team   members   and  partners.     Increase  in     perceived   ownership   over  project.     Increased   clarity  and   relevance  of   summary     Increased   (clear   language)   communica= on  and   transla=on   skills.   Document   summary   candidates  and   full  ar=cles  in   database.     Create   agreement  on   process  for   summaries   within  project   team.   Hire  students   from  across   University.   Format   summary  using   standard   template   Student   writers  work   with  faculty  to   revise   summary   draDs   Provide   training  for   writers  and   stakeholders   (staff).   Increased   accessibility   of  summary     Increased   trust  of   research   product   (successful   branding)     Ac=vi=es  Short  Term  Goals  Long  Term  Goals   Increase  in     inter-­‐silo   communica= on.   Increased  uptake  and  use  of  summary  and  original   research   Increased   knowledge   sharing  within   and  outside  of   academia   Increased   quality  of   summary     Increase  in  project  team  and  partner   collabora=on:     Industry  Communi=es  Policy-­‐Makers   Prac==oners   Staff  Students   Faculty   Preparation Make  research  summaries  available  through:   Social  Media   Tradi=onal   Media   Government   Partners   Google  Search  University   Business   Support  Office   DisseminationProduction Internal     Collabora=on   KMb   ins=tu=onal   partners   Collabora=on   with   Researchers   and  Research   Users     Logic model for a university- based plain language summary project
  • 36. A  mid-­‐term  evalua=on  included  a  review  of  products  produced,   processes  used  (including  collabora=on  across  the  Project  Team,   the  system  for  deciding  on  summary  topics)  and  success  of  the   social  network  strategy.   Final  research  summary  created   Increased     awareness  of   research   scope  among   project  team   members   and  partners.     Increase  in     perceived   ownership   over  project.     Increased   clarity  and   relevance  of   summary     Increased   (clear   language)   communica= on  and   transla=on   skills.   Document   summary   candidates  and   full  ar=cles  in   database.     Create   agreement  on   process  for   summaries   within  project   team.   Hire  students   from  across   University.   Format   summary  using   standard   template   Student   writers  work   with  faculty  to   revise   summary   draDs   Provide   training  for   writers  and   stakeholders   (staff).   Increased   accessibility   of  summary     Increased   trust  of   research   product   (successful   branding)     Ac=vi=es  Short  Term  Goals  Long  Term  Goals   Increase  in     inter-­‐silo   communica= on.   Increased  uptake  and  use  of  summary  and  original   research   Increased   knowledge   sharing  within   and  outside  of   academia   Increased   quality  of   summary     Increase  in  project  team  and  partner   collabora=on:     Industry  Communi=es  Policy-­‐Makers   Prac==oners   Staff  Students   Faculty   Preparation Make  research  summaries  available  through:   Social  Media   Tradi=onal   Media   Government   Partners   Google  Search  University   Business   Support  Office   DisseminationProduction Internal     Collabora=on   KMb   ins=tu=onal   partners   Collabora=on   with   Researchers   and  Research   Users     1.  Count & describe 2.  Ask respondents (stakeholders) to describe collab process/why various pieces were important 1.  Post-training survey (quality of training; knowledge of plain language) 2.  Writer & faculty survey (quality of process & outputs) 3.  Useful? Easy to use? Clear? (1) focus groups of policy makers, researchers, and community partners and (2) post focus- group survey 4. Focus group/ survey for stakeholders. 1. # tweets, RT, etc… 2.  # followers 3. # hits & downloads
  • 38. Cooper, A. (2011). Knowledge Mobilization in Education A cross-case analysis of 44 research brokering organizations across Canada. Unpublished doctoral thesis, University of Toronto, Toronto, Ontario, Canada. Cooper’s (2011) Brokering Functions of KM Strategies diagram shows possible activities & outcomes
  • 40. Contribution analysis 1.  Map a “journey” (theory of change) that links activities to increased research uptake to outcomes 2.  Identify assumptions and risks for each stage of the journey 3.  Identify indicators for research uptake, use and impact 4.  Collect evidence. 5.  Review the map against your evidence. Identify gaps in evidence, then try to fill the gaps. 6.  Write your contribution story. Explain the journey from inputs to impacts. Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18, 270-280. doi:10.1177/1356389012451663 Morton. (2013). Assessing research impact: A case study of participatory research. Centre for Research on Families and Relationships. https://www.era.lib.ed.ac.uk/handle/1842/6562
  • 41. Morton, S. (2013). Assessing research impact: A case study of participatory research. Centre for Research on Families and Relationships, 66. www.crfr.ac.uk Example of “impact journey” diagram from Sarah Morton’s work
  • 42. What measures and indicators will you use? (metrics to stories)
  • 43. Outputs vs. Outcomes Outputs Engagement Uptake Use Impact Causal Attributions measurement time & complexity
  • 44. KTE indicators for: •  Number/type of KTE products •  Ease of use/ user experience •  Timing/ relevance (meets audience needs) •  Awareness/ attitudes/ beliefs/ knowledge •  Self-reported intentions/ behaviour •  Networks/ relationships/ collaborations •  Systems/ policies/ organizational culture
  • 45. Social Process, Social Impact q  Quality & quantity of relationships q  1:1 relationships; organizational relationships q  Meetings q  Requests & referrals q  Co-produced products q  Social network analysis q  .
  • 46. Group Feedback What did you create?
  • 47. +/- method: user experience testing •  10 minutes). Review the workbook on your own. Mark the margins with a plus (+) or minus (-) to record your positive and negative reading experiences. •  (10 + 10 minutes). With a partner, review the motives for each of the +/- in the document. Ask probing questions where needed. Record your responses and note the top 3 most important positive and negative experiences. •  (15 minutes). As a small group, give feedback on top 3 +/- for each partner. What was effective/ not effective and why? •  (15 minutes). As a large group, review trends in +/- experiences for each group.
  • 48. Create your own KTE evaluation framework
  • 49. Evaluation Framework •  Your research •  Your KTE •  Evaluation goals •  Values •  Approach •  Logic model/ theory of change •  Targets of change (stakeholders) •  Methods, metrics, & measures
  • 51. What? What happened? What did I observe? What stood out for me?
  • 52. So What? Why is that important? How did my understanding change?
  • 53. Now What? How will my practice change: …over the next 3 months? …over the next year?