Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluating Problem Gambling KTE

980 views

Published on

Evaluating Problem Gambling KTE. (2015). Part of the "Moving Research Forward" Workshop Series for the Ontario Problem Gambling Research Centre.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Evaluating Problem Gambling KTE

  1. 1. Evaluating Problem Gambling KTE Part of the OPGRC’s Moving Research Forward workshop series February 2015
  2. 2. get in touch Anne Bergen, PhD knowledgetoaction.ca @anne_bergen
  3. 3. Reuse & Attribution This work is licensed under a Creative Commons Attribution 4.0 International License. Attribute this work as: Bergen, A. (2015). Evaluating Problem Gambling KTE. Moving Research Forward Workshop Series, Ontario Problem Gambling Research Centre, Guelph ON.
  4. 4. Introductions
  5. 5. Why KTE evaluation?
  6. 6. Evaluation can help you understand: • Need for KTE activity • Quality of KTE activity • Reach of KTE activity • Impact of KTE activity
  7. 7. KTEà Impact? What kind of impact? For whom? Under what conditions? When? Based on what evidence?
  8. 8. What counts as evidence?
  9. 9. Evaluation research vs. evaluation Research how used? to be published? funded?
  10. 10. What are your evaluation goals?
  11. 11. Evaluation Continuum How will you use the evaluation results? – What? – So What? – Now What? Outputs Engagement Uptake Use Impact Causal Attributions
  12. 12. key questions for KTE 1.  What research knowledge should be transferred? 2.  To whom should research knowledge be transferred? 3.  By whom should research knowledge be transferred? 4.  How should research knowledge be transferred? 5.  With what effect should research knowledge be transferred? Adapted from: Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision makers? Milbank Quarterly, 81(2), 221-248.
  13. 13. key questions for KTE evaluation 1.  What research knowledge was transferred? 2. To whom was research knowledge transferred? 3.  By whom was research knowledge transferred? 4. How was research knowledge transferred? 5.  With what effect was research knowledge transferred? Adapted from: Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision makers? Milbank Quarterly, 81(2), 221-248.
  14. 14. Evaluation Over KTE Project Lifespan •  Before: Needs assessment •  Creation: Developmental evaluation – user experience/ usability testing •  During: Process (formative) evaluation – quality & quantity of outputs •  After: Summative (outcome) evaluation – outcomes & impacts
  15. 15. Evaluation Goals • What aspects of KTE do you want to evaluate? • What are you going to do with the resulting information?
  16. 16. Evaluation Framework •  Goals •  Values •  Approach •  Logic model/ theory of change •  Targets of change (stakeholders) •  Methods, metrics, & measures
  17. 17. What are your evaluation values?
  18. 18. More impact. Inform Consult Involve Collaborate Empower Less control. Slower process. Adapted from Arnstein’s (1969) Ladder of Public Participation and the IAP2 Spectrum of Public Participation How engaged are your stakeholders in your research & KTE activities?
  19. 19. What is your evaluation context?
  20. 20. Research Funding Organization Program of Research A Project A1 Product A1a Product A1b Project A2 Product A2a Product A2b Product A2c Project A3 Product A3a Product A3b Program of Research B Project B1 Product B1a Product B1b Project B2 Product B2a Product B2b
  21. 21. Research Funding Organization Program of Research A Project A1 Product A1a Product A1b Project A2 Product A2a Product A2b Product A2c Project A3 Product A3a Product A3b Program of Research B Project B1 Product B1a Product B1b Project B2 Product B2a Product B2b
  22. 22. Context Research Funding Organization Program of Research A Project A1 Product A1a Product A1b Project A2 Product A2a Product A2b Product A2c Project A3 Product A3a Product A3b Program of Research B Project B1 Product B1a Product B1b Project B2 Product B2a Product B2b
  23. 23. Evaluation Context (macro & micro) •  Micro - (time, money, resources, intended use, intended users, reporting requirements, etc…) •  Macro – (culture, leadership, evaluation, external systems, political climate, etc….)
  24. 24. What approach to evaluation will you take?
  25. 25. http://betterevaluation.org
  26. 26. What Kind of Evaluation? Evaluation research vs. evaluation Research how used? to be published? funded? practical & theoretical considerations
  27. 27. Audience/ Targets of Change q  gamblers and problem gamblers q  gambling industry q  gambling researchers q  general public q  policymakers q  subsets of the general public (e.g., youth) q 
  28. 28. Methods q  Qualitative q  Quantitative q  Mixed Methods q  Prospective q  Retrospective q  . q  Participatory? q  Descriptive? q  RCT? q  Pre-Post? q  Counterfactual? q  Contribution analysis? q  ..
  29. 29. What is the theory of change? (logical links between activities & desired outcomes)
  30. 30. Theory of Change Problem or Issue 1 Needs & Assets 2 Desired Results (outputs, outcomes) 3 Influential Factors 4 KTE Strategies 5 Assumptions 6 Adapted from Kellogg Foundation Theory of Change template http://www.wkkf.org/resource-directory/resource/2007/07/spark-theory-of-change
  31. 31. Theory of Change Problem or Issue 1 Needs & Assets 2 Desired Results (outputs, outcomes) 3 Influential Factors 4 KTE Strategies 5 Assumptions 6 Adapted from Kellogg Foundation Theory of Change template http://www.wkkf.org/resource-directory/resource/2007/07/spark-theory-of-change
  32. 32. Another approach: logic models
  33. 33. Assumptions Think about individuals, groups, systems, & assumptions Activities Short-Term Outcomes Intermediate Outcomes Long-Term Outcomes Systems & Policy Change Initiatives • Share resources • Create partnerships Awareness Systems Changes Reduced harm from gambling Partnerships Individual Change Initiatives • Provide education and resources on… • Provide environmental cues to… Knowledge & skills Intention & Behaviour Changes Psychosocial Gambling context
  34. 34. An example: process-focused KTE evaluation
  35. 35. Final  research  summary  created   Increased     awareness  of   research   scope  among   project  team   members   and  partners.     Increase  in     perceived   ownership   over  project.     Increased   clarity  and   relevance  of   summary     Increased   (clear   language)   communica= on  and   transla=on   skills.   Document   summary   candidates  and   full  ar=cles  in   database.     Create   agreement  on   process  for   summaries   within  project   team.   Hire  students   from  across   University.   Format   summary  using   standard   template   Student   writers  work   with  faculty  to   revise   summary   draDs   Provide   training  for   writers  and   stakeholders   (staff).   Increased   accessibility   of  summary     Increased   trust  of   research   product   (successful   branding)     Ac=vi=es  Short  Term  Goals  Long  Term  Goals   Increase  in     inter-­‐silo   communica= on.   Increased  uptake  and  use  of  summary  and  original   research   Increased   knowledge   sharing  within   and  outside  of   academia   Increased   quality  of   summary     Increase  in  project  team  and  partner   collabora=on:     Industry  Communi=es  Policy-­‐Makers   Prac==oners   Staff  Students   Faculty   Preparation Make  research  summaries  available  through:   Social  Media   Tradi=onal   Media   Government   Partners   Google  Search  University   Business   Support  Office   DisseminationProduction Internal     Collabora=on   KMb   ins=tu=onal   partners   Collabora=on   with   Researchers   and  Research   Users     Logic model for a university- based plain language summary project
  36. 36. A  mid-­‐term  evalua=on  included  a  review  of  products  produced,   processes  used  (including  collabora=on  across  the  Project  Team,   the  system  for  deciding  on  summary  topics)  and  success  of  the   social  network  strategy.   Final  research  summary  created   Increased     awareness  of   research   scope  among   project  team   members   and  partners.     Increase  in     perceived   ownership   over  project.     Increased   clarity  and   relevance  of   summary     Increased   (clear   language)   communica= on  and   transla=on   skills.   Document   summary   candidates  and   full  ar=cles  in   database.     Create   agreement  on   process  for   summaries   within  project   team.   Hire  students   from  across   University.   Format   summary  using   standard   template   Student   writers  work   with  faculty  to   revise   summary   draDs   Provide   training  for   writers  and   stakeholders   (staff).   Increased   accessibility   of  summary     Increased   trust  of   research   product   (successful   branding)     Ac=vi=es  Short  Term  Goals  Long  Term  Goals   Increase  in     inter-­‐silo   communica= on.   Increased  uptake  and  use  of  summary  and  original   research   Increased   knowledge   sharing  within   and  outside  of   academia   Increased   quality  of   summary     Increase  in  project  team  and  partner   collabora=on:     Industry  Communi=es  Policy-­‐Makers   Prac==oners   Staff  Students   Faculty   Preparation Make  research  summaries  available  through:   Social  Media   Tradi=onal   Media   Government   Partners   Google  Search  University   Business   Support  Office   DisseminationProduction Internal     Collabora=on   KMb   ins=tu=onal   partners   Collabora=on   with   Researchers   and  Research   Users     1.  Count & describe 2.  Ask respondents (stakeholders) to describe collab process/why various pieces were important 1.  Post-training survey (quality of training; knowledge of plain language) 2.  Writer & faculty survey (quality of process & outputs) 3.  Useful? Easy to use? Clear? (1) focus groups of policy makers, researchers, and community partners and (2) post focus- group survey 4. Focus group/ survey for stakeholders. 1. # tweets, RT, etc… 2.  # followers 3. # hits & downloads
  37. 37. An example: descriptive KTE evaluation
  38. 38. Cooper, A. (2011). Knowledge Mobilization in Education A cross-case analysis of 44 research brokering organizations across Canada. Unpublished doctoral thesis, University of Toronto, Toronto, Ontario, Canada. Cooper’s (2011) Brokering Functions of KM Strategies diagram shows possible activities & outcomes
  39. 39. An example: outcome-focused KTE evaluation
  40. 40. Contribution analysis 1.  Map a “journey” (theory of change) that links activities to increased research uptake to outcomes 2.  Identify assumptions and risks for each stage of the journey 3.  Identify indicators for research uptake, use and impact 4.  Collect evidence. 5.  Review the map against your evidence. Identify gaps in evidence, then try to fill the gaps. 6.  Write your contribution story. Explain the journey from inputs to impacts. Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18, 270-280. doi:10.1177/1356389012451663 Morton. (2013). Assessing research impact: A case study of participatory research. Centre for Research on Families and Relationships. https://www.era.lib.ed.ac.uk/handle/1842/6562
  41. 41. Morton, S. (2013). Assessing research impact: A case study of participatory research. Centre for Research on Families and Relationships, 66. www.crfr.ac.uk Example of “impact journey” diagram from Sarah Morton’s work
  42. 42. What measures and indicators will you use? (metrics to stories)
  43. 43. Outputs vs. Outcomes Outputs Engagement Uptake Use Impact Causal Attributions measurement time & complexity
  44. 44. KTE indicators for: •  Number/type of KTE products •  Ease of use/ user experience •  Timing/ relevance (meets audience needs) •  Awareness/ attitudes/ beliefs/ knowledge •  Self-reported intentions/ behaviour •  Networks/ relationships/ collaborations •  Systems/ policies/ organizational culture
  45. 45. Social Process, Social Impact q  Quality & quantity of relationships q  1:1 relationships; organizational relationships q  Meetings q  Requests & referrals q  Co-produced products q  Social network analysis q  .
  46. 46. Group Feedback What did you create?
  47. 47. +/- method: user experience testing •  10 minutes). Review the workbook on your own. Mark the margins with a plus (+) or minus (-) to record your positive and negative reading experiences. •  (10 + 10 minutes). With a partner, review the motives for each of the +/- in the document. Ask probing questions where needed. Record your responses and note the top 3 most important positive and negative experiences. •  (15 minutes). As a small group, give feedback on top 3 +/- for each partner. What was effective/ not effective and why? •  (15 minutes). As a large group, review trends in +/- experiences for each group.
  48. 48. Create your own KTE evaluation framework
  49. 49. Evaluation Framework •  Your research •  Your KTE •  Evaluation goals •  Values •  Approach •  Logic model/ theory of change •  Targets of change (stakeholders) •  Methods, metrics, & measures
  50. 50. W3 What? So What? Now What?
  51. 51. What? What happened? What did I observe? What stood out for me?
  52. 52. So What? Why is that important? How did my understanding change?
  53. 53. Now What? How will my practice change: …over the next 3 months? …over the next year?

×