Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NCCMT Spotlight Webinar: Guide to Policy-Influence Evaluation


Published on

Interested in learning how to evaluate your policy influence?

Do you promote the uptake and dissemination of population health interventions? Are you interested in exploring public health–related case studies of policy influence? The Guide to Policy-Influence Evaluation can help!

This guide was developed by the Public Health Agency of Canada’s Innovation Strategy and produced by Cathexis Consulting.

How can the Guide to Policy-Influence Evaluation help you?

The Guide to Policy-Influence Evaluation was developed to help organizations use policy influence to improve the uptake and evaluation of evidence-based population health interventions. This process is divided into the four steps of evaluation planning. Each step includes two or more resources to support it. The resources are then summarized and important highlights are presented as they related to each step.

This webinar includes an overview of the Guide by its developers, followed by a presentation from a community based organization who evaluated the impact on policies within their work to promote healthier weights.

The Guide to Policy-Influence Evaluation includes three public health–related case studies:
•Healthy weights among Aboriginal children and youth
•Anti-bullying for primary schools
•Food security and healthy weights

To see the summary statement of this method developed by NCCMT, click here:

The National Collaborating Centre for Methods and Tools is funded by the Public Health Agency of Canada and affiliated with McMaster University. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada.

NCCMT is one of six National Collaborating Centres (NCCs) for Public Health. The Centres promote and improve the use of scientific research and other knowledge to strengthen public health practices and policies in Canada.

Published in: Healthcare
  • Be the first to comment

  • Be the first to like this

NCCMT Spotlight Webinar: Guide to Policy-Influence Evaluation

  1. 1. Follow us @nccmt Suivez-nous @ccnmo Funded by the Public Health Agency of Canada | Affiliated with McMaster University Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The views expressed here do not necessarily reflect the views of the Public Health Agency of Canada.. Guide to Policy-Influence Evaluation Presenters: Marla Steinberg, PhD CE Anima Anand, PhD Michele Hopkins, RSW, MSW Shannon Bradley Dexter, MSc January 25, 2017 1:00 – 2:30 PM ET
  2. 2. Follow us @nccmt Suivez-nous @ccnmo 2 Housekeeping Use Chat to post comments and/or questions during the webinar • ‘Send’ questions to All (not privately to ‘Host’) Connection issues • Recommend using a wired Internet connection (vs. wireless), • WebEx 24/7 help line • 1-866-229-3239 Participant Side Panel in WebEx Chat
  3. 3. Follow us @nccmt Suivez-nous @ccnmo 3 After Today The PowerPoint presentation (in English and French) and English audio recording will be made available. These resources are available at: PowerPoint: Audio Recording:
  4. 4. Follow us @nccmt Suivez-nous @ccnmo 4 How many people are watching today’s session with you? Poll Question #1 A. Just me B. 1-3 C. 4-5 D. 6-10 E. >10
  5. 5. Follow us @nccmt Suivez-nous @ccnmo Your profession? Put a √ on your answer (or RSVP via email) / Epidemiologist Management (director, supervisor, etc.) Allied health professionals (nurse, dietician, dental hygenist, etc.) Librarian Physician / Dentist Other 5
  6. 6. Follow us @nccmt Suivez-nous @ccnmo Guide to Policy-Influence Evaluation Episode 30 6
  7. 7. NCC Infectious Diseases Winnipeg, MB NCC Methods and Tools Hamilton, ON NCC Healthy Public Policy Montreal, QC NCC Determinants of Health Antigonish, NS NCC Aboriginal Health Prince George, BC NCC Environmental Health Vancouver, BC 7
  8. 8. Registry of Methods and Tools Online Learning Opportunities WorkshopsMultimedia Public Health+ Networking and Outreach NCCMT Products and Services 8
  9. 9. Follow us @nccmt Suivez-nous @ccnmo 9 Poll Question #2 How familiar are you with the method or tool we are discussing today? A. I am not familiar with the method or tool B. I have heard of the method or tool C. I have used the method or tool
  10. 10. Follow us @nccmt Suivez-nous @ccnmo 10 Presenter Shannon Bradley Dexter, MSc Senior Policy Analyst, Innovation Strategy, Public Health Agency of Canada
  11. 11. Follow us @nccmt Suivez-nous @ccnmo 11 Presenter Marla Steinberg, PhD CE Evaluation Consultant
  12. 12. Follow us @nccmt Suivez-nous @ccnmo 12 Presenter Anima Anand, PhD Project Lead, Healthy Weights for Children, The Bridge Youth & Family Services
  13. 13. Follow us @nccmt Suivez-nous @ccnmo 13 Presenter Michele Hopkins, RSW, MSW Project Coordinator, Healthy Weights for Children, The Bridge Youth & Family Services
  14. 14. Evaluating Policy-Influence NCCMT Webinar January 25, 2017 Marla Steinberg, Ph.D. CE Evaluation Consultant Anima Anand, Ph.D. and Michele Hopkins M.S.W. Healthy Weights for Children Project Shannon Bradley Dexter Innovation Strategy Public Health Agency of Canada
  15. 15. Welcome By the end of this webinar, you should be able to: 1. List the complexities of evaluating policy-influence. 2. Assess how the Guide to Policy-Influence Evaluation can support your work.
  16. 16. Establishing our Learning Community Poll: What is your interest in policy-influence? A. I am a practitioner who does policy-influence work B. I am a public health manager or decision-maker with a mandate for policy-influence work C. I am a funder who funds policy-influence work D. I am an evaluator who evaluates policy-influence work E. I am a student interested in policy-influence work F. Other G. I don’t really know what policy-influence work is…………….. Poll Question #3
  17. 17. Have you ever….. • Wondered how you get senior decision makers and politicians to take an interest in your program or issue? • Wondered how best to support the scale-up of an intervention? • Wondered how to evaluate your influence on policy? • Wondered what should be considered a realistic achievement for policy- influence work?
  18. 18. The Guide to Policy-Influence Evaluation may be just what you need….
  19. 19. Needs, definitions, and the complexity of policy-influence work and evaluation Setting the context for the guide
  20. 20. Origins of the Guide: The Public Health Agency of Canada’s Innovation Strategy (IS) • Funds population health intervention research in diverse communities across Canada to build evidence about "what works" "for who" and "in what context" to reduce health inequities. • Promotes and shares intervention results to inform future policy, program design, and other actions that will improve population health. • Funding provided in three phases. Current focus areas: Mental Health Promotion and Achieving Healthier Weights
  21. 21. IS Evaluation Requirements • IS projects are required to undertake a comprehensive evaluation including assessing the uptake of evidence or knowledge generated through the project. • Uptake is often achieved through the “influence on policy.” THE ISSUE: Projects needed guidance, tools, and resources to effectively report on how IS projects were influencing policy. They wanted to be able to tell their policy-influence stories!
  22. 22. Exploring the world of policy-influence work
  23. 23. Definitions
  24. 24. What is policy? • Formal and wordy definition: Policy is a means of governing action with the aim of attenuating or promoting particular phenomena occurring in the population. Policies can outline rules, provide principles that guide action, set roles and responsibilities, reflect values and principles, as well as state intentions. Policies can be enacted by all levels of government (federal, provincial, regional and municipal), community organizations, businesses, and schools. Polices can guide programs, practice or education. • Adapted from the National Collaborating Centre for Healthy Public Policy (2010) Accessed at: • Endorsed by 95% of IS project respondents in the policy evaluation needs assessment conducted in the fall of 2013.
  25. 25. What is policy-influence work? Policy-influence work supports the uptake or spread of evidence-based interventions.
  26. 26. An example “With partner organizations, we developed a food security questionnaire that went to all party leaders during the recent provincial election to see where they stand on food security issues. The next step will be holding them to some promises. We have also been working with the municipality to develop a food charter and food strategy. Prior to this we prepared a fact sheet on food security as a municipal issue, which was distributed to supporters and encouraged people to write in support of including food security in the Regional Plan” Policy maker education Policy development Public awareness campaign Advocacy
  27. 27. Other terms……….
  28. 28. What is unique about policy-influence work and its evaluation?
  29. 29. Policy-Influence is Complex
  30. 30. “Policy-influence is a highly complex process shaped by a multitude of interacting forces and actors. Outright success, in terms of achieving specific, hoped-for changes in policy, is rare, and the work that does influence policy is often unique and rarely repeated or replicated, with many incentives working against the sharing of ‘good practice’” (Jones, 2011, p. 1)
  31. 31. A lot of policy-influence is about being there when the window opens and being nimble to jump on board. A lot of policy work is stand and wait and then jump when you need to act” Bonnie Leadbeater – WITS (Mental Health IS project)
  32. 32. Policy-making and policy- influence are processes
  33. 33. All roads lead to Rome….. There are a wide variety of activities that can influence policy
  34. 34. There are a variety of policy impacts: (1) Conceptual – changing the thinking of key stakeholders (2) Instrumental – changing actions Carol Weiss
  35. 35. Jones (2011) offers further elaborations on the range of policy impacts: 1. Framing debates and getting issues on to the political agenda; drawing attention to new issues and affecting the awareness, attitudes or perceptions of key stakeholders 2. Encouraging discursive commitments; affecting language and rhetoric to promote the recognition of specific groups or endorsements of policy recommendations 3. Securing procedural change; changes in the process whereby policy decisions are made, such as opening new spaces for policy dialogue 4. Affecting policy content 5. Influencing behaviour change in key actors: policy change requires changes in behavior and implementation at various levels in order to be meaningful and sustainable
  36. 36. Attributing Change is Difficult
  37. 37. Theories of change are useful (required!)
  38. 38. There are a variety of methods for evaluating policy-influence
  39. 39. Working Group of IS Projects Created Needs Assessment Completed Guide Developed Case Studies Added Final Production and Translation The development of the guide
  40. 40. Recommends a limited number of useful (and free) resources that are organized around a four- step evaluation planning process
  41. 41. Four steps are aligned with other evaluation planning frameworks CDC Framework for Evaluation in Public Health
  42. 42. Includes three case studies: WITS - anti-bullying program for primary schools from the mental health stream Healthy Weights Connections - System change intervention to improve public health services for aboriginal children and families Our Food - Creating a food strategy for Halifax
  43. 43. Includes completed evaluation plans for the cases developed through using the resources
  44. 44. User Experience Insights
  45. 45. Set of “raw” ingredients To be selected, prepared, adapted, and consumed as needed to suit your context.
  46. 46. What it is not…… A step-by-step recipe to be rigidly followed
  47. 47. What it is not…… A step-by-step recipe to be rigidly followed A resource on how to evaluate the implementation, effectiveness or impact
  48. 48. Who is the audience Funders Projects or organizations involved in policy-influence work Evaluators Assumes a basic understanding of evaluation
  49. 49. Uses for the guide Read it to understand the complexities of and options for policy-influence work and its evaluation
  50. 50. Uses for the guide Read it to understand the complexities of policy- influence evaluation Use some of the frameworks to develop evaluation requirements for funded projects
  51. 51. Uses for the guide Read it to understand the complexities of policy- influence evaluation Use some of the frameworks to develop project- specific evaluation requirements Use the tools to work with projects to plan their policy-influence work and evaluation
  52. 52. Healthy Weights for Children Project created an innovative family education model ‘Healthy Together’© (HT) • Integrated healthy eating and physical activity within group learning to bring families together to learn to make healthier choices and build healthy relationships. • Established a collaborative national network to guide program integration within core practice and policy.
  53. 53. • 21 sites in 8 Provinces and Territories (BC, AB, SK, MB, ON, NL, NB, NWT) • 150+ trained professionals across Canada • 1,000+ children, youth and caregivers • Community context: • Northern, rural, urban, Aboriginal, multi-cultural • Low income, immigrant/refugee, foster families Snapshot of HT
  54. 54. Program effectiveness • HT demonstrated that inclusion of healthy cooking/eating and physical activity within group learning was key to participant engagement, leading to positive behaviour change. • Reduced consumption of sugar sweetened beverages • Increased physical activity • Reduced screen time • Relationships that developed during HT sessions led to building social support for families, particularly in underserved communities.
  55. 55. HT ‘policy influence’ needs • An effective policy development framework • Concerted effort by multi-sectorial partners to promote healthier weight practices and policy • Tools to measure the impact of policy influence work
  56. 56. Complexity of HT policy influence • In order to be effective, HT needed to – Be responsive – Be adaptable – Build capacity – Enhance knowledge uptake
  57. 57. HT policy influence - process Policy Development Placement on Policy Agenda Policy Adoption Policy Implementation Policy Maintenance
  58. 58. Tool for HT Policy Influence Work –Goals & Activities Practice/ Policy Goal Activities/Tactics Timeline  What is an example of a practice/ policy related to achieving healthier weights?  How do you envision Healthy Together influencing practice/policy discussions?  What practice/policy goal(s) do you think is possible within your sphere of influence?  What activities/tactics will you engage in, to achieve the above goals, and the estimated timeline?  Comments Thank you for your participation! Office/Agency Organization/Network Community Province Other
  59. 59. HT policy influence activities • Program uptake/practice change by sites • Establishing relationships with key decision makers • Policy makers’ education/ capacity building • Policy development/maintenance • Policy implementation/evaluation • Demonstration projects/pilots • Electronic outreach through social media • Briefings/presentations
  60. 60. • Adoption of HT approach into core practice • Communities/systems engaged in ‘buy-in’ • Stakeholders leverage partnerships to support uptake Ultimate outcome: ‘Vulnerable children will experience greater equality of health outcomes and achieve healthier weights. ‘ HT policy outcomes
  61. 61. HT theory of change • If families at-risk of developing unhealthy weights come together through programming that enables healthy eating, increases enjoyment of physical activity and strengthens the family bond, then members will improve their overall health and overall quality of life with skills and habits that will last.
  62. 62. How do I get the guide? NCCMT: e1340385f1b593d3bb9f50f6a38.pdf the guide?
  63. 63. Wrap Up – Key Messages • Policy-influence in order to support scale-up of evidence-based programs is a concern to many funders and organizations • There is a burgeoning literature on how to do and evaluate policy- influence work • The Guide to Policy-Influence Evaluation will help you navigate this literature with a select number of high quality useful resources • The Guide can help to: • Develop evaluation requirements for policy-influence work • Support projects and organization to do and evaluate policy-influence work
  64. 64. Thank You! Questions/Comments/Contact The Guide:
  65. 65. Follow us @nccmt Suivez-nous @ccnmo 66 Your Comments/Questions • Use Chat to post comments and/or questions • ‘Send’ questions to All (not privately to ‘Host’) Chat Participant Side Panel in WebEx
  66. 66. Follow us @nccmt Suivez-nous @ccnmo 67 Poll Question #4 Could this method or tool be useful in practice? A. Very useful B. Somewhat useful C. Not at all useful D. Don’t know
  67. 67. Follow us @nccmt Suivez-nous @ccnmo 68 Your Feedback is Important Please take a few minutes to share your thoughts on today’s webinar. Your comments and suggestions help to improve the resources we offer and plan future webinars. The short survey is available at: GLBz
  68. 68. Follow us @nccmt Suivez-nous @ccnmo 69 Poll Question #5 Would you be willing to receive an email from the NCCMT with a follow-up survey in 2-3 months? A. Yes B. No
  69. 69. Follow us @nccmt Suivez-nous @ccnmo 70 Join us for our next webinar NCCMT Spotlight on Methods & Tools: CDC Clear Communication Index Date: Wednesday, February 1, 2017 Time: 1:00 – 2:30pm EST Interested in evidence-based criteria for clear communication of public health material? Do you develop and review public health communication materials? Are you interested in enhancing clarity and ease of understanding of these materials? Join us for a webinar on how the Clear Communication Index can help you. Register at: evidence/onstage/g.php?MTID=eb180a59b9994f1d4da0729d c3d31345d
  70. 70. Follow us @nccmt Suivez-nous @ccnmo 71 Poll Question #6 What are your next steps? (Check all that apply) A. Access the method/tool referenced in the presentation B. Read the NCCMT summary about the method/tool described today C. Consider using the method/tool in practice D. Tell a colleague about the method/tool
  71. 71. Follow us @nccmt Suivez-nous @ccnmo Funded by the Public Health Agency of Canada | Affiliated with McMaster University Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The views expressed here do not necessarily reflect the views of the Public Health Agency of Canada.. For more information about the National Collaborating Centre for Methods and Tools: NCCMT website Contact: