NCCMT Spotlight Webinar: Guide to Policy-Influence Evaluation

Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
Guide to Policy-Influence Evaluation
Presenters:
Marla Steinberg, PhD CE
Anima Anand, PhD
Michele Hopkins, RSW, MSW
Shannon Bradley Dexter, MSc
January 25, 2017 1:00 – 2:30 PM ET

2
Housekeeping
Use Chat to post comments and/or
questions during the webinar
• ‘Send’ questions to All (not
privately to ‘Host’)
Connection issues
• Recommend using a wired
Internet connection (vs.
wireless),
• WebEx 24/7 help line
• 1-866-229-3239
Participant Side
Panel in WebEx
Chat

3
After Today
The PowerPoint presentation (in English and French)
and English audio recording will be made available.
These resources are available at:
PowerPoint: http://www.slideshare.net/NCCMT/
Audio Recording:
https://www.youtube.com/user/nccmt/videos

4
How many people are watching
today’s session with you?
Poll Question #1
A. Just me
B. 1-3
C. 4-5
D. 6-10
E. >10

Your profession?
Put a √ on your answer (or RSVP via email)
/
Epidemiologist Management (director,
supervisor, etc.)
Allied health
professionals (nurse,
dietician, dental
hygenist, etc.)
Librarian Physician / Dentist Other
5

Guide to Policy-Influence
Evaluation
http://www.nccmt.ca/resources/search/241
Episode 30
6

NCC
Infectious
Diseases
Winnipeg, MB
NCC
Methods
and Tools
Hamilton, ON
NCC Healthy
Public Policy
Montreal, QC
NCC
Determinants
of Health
Antigonish, NS
NCC
Aboriginal
Health
Prince George, BC
NCC
Environmental
Health
Vancouver, BC
7

Registry of Methods and Tools
Online Learning
Opportunities
WorkshopsMultimedia
Public Health+
Networking and
Outreach
NCCMT Products and Services
8

9
Poll Question #2
How familiar are you with the
method or tool we are discussing
today?
A. I am not familiar with the method or tool
B. I have heard of the method or tool
C. I have used the method or tool

10
Presenter
Shannon Bradley Dexter, MSc
Senior Policy Analyst,
Innovation Strategy, Public
Health Agency of Canada

11
Presenter
Marla Steinberg, PhD CE
Evaluation Consultant

12
Presenter
Anima Anand, PhD
Project Lead, Healthy
Weights for Children, The
Bridge Youth & Family
Services

13
Presenter
Michele Hopkins, RSW, MSW
Project Coordinator, Healthy
Weights for Children, The
Bridge Youth & Family
Services

Evaluating Policy-Influence
NCCMT Webinar
January 25, 2017
Marla Steinberg, Ph.D. CE
Evaluation Consultant
Anima Anand, Ph.D. and Michele Hopkins M.S.W.
Healthy Weights for Children Project
Shannon Bradley Dexter
Innovation Strategy
Public Health Agency of Canada

Welcome
By the end of this webinar, you should be able to:
1. List the complexities of evaluating policy-influence.
2. Assess how the Guide to Policy-Influence Evaluation can support your work.

Establishing our Learning Community
Poll: What is your interest in policy-influence?
A. I am a practitioner who does policy-influence work
B. I am a public health manager or decision-maker with a mandate
for policy-influence work
C. I am a funder who funds policy-influence work
D. I am an evaluator who evaluates policy-influence work
E. I am a student interested in policy-influence work
F. Other
G. I don’t really know what policy-influence work is……………..
Poll Question #3

Have you ever…..
• Wondered how you get senior decision
makers and politicians to take an
interest in your program or issue?
• Wondered how best to support the
scale-up of an intervention?
• Wondered how to evaluate your
influence on policy?
• Wondered what should be considered
a realistic achievement for policy-
influence work?

The Guide to Policy-Influence Evaluation may
be just what you need….

Needs, definitions, and the complexity of policy-influence work and evaluation
Setting the context for the guide

Origins of the Guide: The Public Health
Agency of Canada’s Innovation Strategy (IS)
• Funds population health intervention research in
diverse communities across Canada to build evidence
about "what works" "for who" and "in what context"
to reduce health inequities.
• Promotes and shares intervention results to inform
future policy, program design, and other actions that
will improve population health.
• Funding provided in three phases.
Current focus areas: Mental Health
Promotion and Achieving Healthier Weights

IS Evaluation Requirements
• IS projects are required to undertake a
comprehensive evaluation including assessing the
uptake of evidence or knowledge generated
through the project.
• Uptake is often achieved through the “influence
on policy.”
THE ISSUE: Projects needed guidance, tools, and resources to
effectively report on how IS projects were influencing policy. They
wanted to be able to tell their policy-influence stories!

Exploring the world of policy-influence work

What is policy?
• Formal and wordy definition:
Policy is a means of governing action with the aim of attenuating or promoting
particular phenomena occurring in the population.
Policies can outline rules, provide principles that guide action, set roles and
responsibilities, reflect values and principles, as well as state intentions.
Policies can be enacted by all levels of government (federal, provincial, regional
and municipal), community organizations, businesses, and schools.
Polices can guide programs, practice or education.
• Adapted from the National Collaborating Centre for Healthy Public Policy
(2010) Accessed at: http://www.ncchpp.ca/docs/MethodPP_EN.pdf
• Endorsed by 95% of IS project respondents in the policy evaluation needs
assessment conducted in the fall of 2013.

What is policy-influence work?
Policy-influence work supports the
uptake or spread of evidence-based
interventions.

An example
“With partner organizations, we developed a food
security questionnaire that went to all party leaders
during the recent provincial election to see where
they stand on food security issues. The next step will
be holding them to some promises. We have also
been working with the municipality to develop a food
charter and food strategy. Prior to this we prepared
a fact sheet on food security as a municipal issue,
which was distributed to supporters and encouraged
people to write in support of including food security
in the Regional Plan”
Policy maker education
Policy development
Public awareness
campaign
Advocacy

What is unique about policy-influence work
and its evaluation?

“Policy-influence is a highly complex
process shaped by a multitude of
interacting forces and actors.
Outright success, in terms of
achieving specific, hoped-for changes
in policy, is rare, and the work that
does influence policy is often unique
and rarely repeated or replicated,
with many incentives working against
the sharing of ‘good practice’”
(Jones, 2011, p. 1)

A lot of policy-influence is
about being there when the
window opens and being
nimble to jump on board. A
lot of policy work is stand
and wait and then jump
when you need to act”
Bonnie Leadbeater – WITS
(Mental Health IS project)

Policy-making
and policy-
influence are
processes

All roads lead
to Rome…..
There are a
wide variety
of activities
that can
influence
policy

There are a variety of
policy impacts:
(1) Conceptual – changing the
thinking of key
stakeholders
(2) Instrumental – changing
actions
Carol Weiss

Jones (2011) offers further elaborations on the
range of policy impacts:
1. Framing debates and getting issues on to the political
agenda; drawing attention to new issues and affecting
the awareness, attitudes or perceptions of key
stakeholders
2. Encouraging discursive commitments; affecting
language and rhetoric to promote the recognition of
specific groups or endorsements of policy
recommendations
3. Securing procedural change; changes in the process
whereby policy decisions are made, such as opening
new spaces for policy dialogue
4. Affecting policy content
5. Influencing behaviour change in key actors: policy
change requires changes in behavior and
implementation at various levels in order to be
meaningful and sustainable

http://tclocal.org/food/
Attributing Change is Difficult

Theories of change are
useful (required!)

There are a variety of methods for evaluating
policy-influence

Working
Group of IS
Projects
Created
Needs
Assessment
Completed
Guide
Developed
Case Studies
Added
Final
Production
and
Translation
The development of the guide

Recommends a
limited number
of useful (and
free) resources
that are
organized
around a four-
step evaluation
planning
process

Four steps are
aligned with other
evaluation planning
frameworks
CDC Framework for
Evaluation in Public
Health

Includes three case studies:
WITS - anti-bullying program
for primary schools from the
mental health stream
Healthy Weights Connections
-
System change intervention
to improve public health
services for aboriginal
children and families
Our Food - Creating a food
strategy for Halifax

Includes
completed
evaluation
plans for the
cases
developed
through using
the resources

Set of “raw” ingredients
http://www.otisfundraisingideas.com/product/our-ingredients
To be selected,
prepared, adapted,
and consumed as
needed to suit your
context.

What it is not……
A step-by-step recipe to be rigidly followed

What it is not……
A step-by-step recipe to be rigidly followed
A resource on how to evaluate the implementation, effectiveness or
impact

Who is the audience
Funders
Projects or organizations involved in policy-influence work
Evaluators
Assumes a basic understanding of evaluation

Uses for the guide
Read it to understand the complexities of and
options for policy-influence work and its evaluation

Uses for the guide
Read it to understand the complexities of policy-
influence evaluation
Use some of the frameworks to develop evaluation
requirements for funded projects

Uses for the guide
Read it to understand the complexities of policy-
influence evaluation
Use some of the frameworks to develop project-
specific evaluation requirements
Use the tools to work with projects to plan their
policy-influence work and evaluation

Healthy Weights for Children Project
created an innovative family education model
‘Healthy Together’© (HT)
• Integrated healthy eating and physical activity
within group learning to bring families together
to learn to make healthier choices and build
healthy relationships.
• Established a collaborative national network to
guide program integration within core practice
and policy.

• 21 sites in 8 Provinces and Territories
(BC, AB, SK, MB, ON, NL, NB, NWT)
• 150+ trained professionals across Canada
• 1,000+ children, youth and caregivers
• Community context:
• Northern, rural, urban, Aboriginal, multi-cultural
• Low income, immigrant/refugee, foster families
Snapshot of HT

Program effectiveness
• HT demonstrated that inclusion of healthy
cooking/eating and physical activity within group
learning was key to participant engagement, leading to
positive behaviour change.
• Reduced consumption of sugar sweetened beverages
• Increased physical activity
• Reduced screen time
• Relationships that developed during HT sessions led to
building social support for families, particularly in
underserved communities.

HT ‘policy influence’ needs
• An effective policy development framework
• Concerted effort by multi-sectorial partners to promote
healthier weight practices and policy
• Tools to measure the impact of policy influence work

Complexity of HT policy influence
• In order to be effective, HT needed to
– Be responsive
– Be adaptable
– Build capacity
– Enhance knowledge uptake

HT policy influence - process
Policy
Development
Placement on
Policy Agenda
Policy
Adoption
Policy
Implementation
Policy
Maintenance

Tool for HT Policy Influence Work –Goals & Activities
Practice/ Policy Goal Activities/Tactics Timeline
 What is an example of a practice/ policy related to achieving healthier weights?
 How do you envision Healthy Together influencing practice/policy discussions?
 What practice/policy goal(s) do you think is possible within your sphere of influence?
 What activities/tactics will you engage in, to achieve the above goals, and the estimated timeline?
 Comments
Thank you for your participation!
Office/Agency
Organization/Network
Community
Province
Other

HT policy influence activities
• Program uptake/practice change by sites
• Establishing relationships with key decision makers
• Policy makers’ education/ capacity building
• Policy development/maintenance
• Policy implementation/evaluation
• Demonstration projects/pilots
• Electronic outreach through social media
• Briefings/presentations

• Adoption of HT approach into core practice
• Communities/systems engaged in ‘buy-in’
• Stakeholders leverage partnerships to support uptake
Ultimate outcome:
‘Vulnerable children will experience greater equality of health
outcomes and achieve healthier weights. ‘
HT policy outcomes

HT theory of change
• If families at-risk of developing unhealthy
weights come together through programming
that enables healthy eating, increases enjoyment
of physical activity and strengthens the family
bond, then members will improve their overall
health and overall quality of life with skills and
habits that will last.

How do I get the guide?
NCCMT:
http://www.nccmt.ca/uploads/media/media/0001/01/c3374b8be4b35
e1340385f1b593d3bb9f50f6a38.pdf
the guide?

Wrap Up – Key Messages
• Policy-influence in order to support scale-up of evidence-based
programs is a concern to many funders and organizations
• There is a burgeoning literature on how to do and evaluate policy-
influence work
• The Guide to Policy-Influence Evaluation will help you navigate this
literature with a select number of high quality useful resources
• The Guide can help to:
• Develop evaluation requirements for policy-influence work
• Support projects and organization to do and evaluate policy-influence work

Thank You!
Questions/Comments/Contact
Marlasteinberg@telus.net
Anima.anand@thebridgeservices.ca
Michele.hopkins@thebridgeservices.ca
shannon.bradleydexter@canada.ca
www.healthy-together.ca
The Guide: http://www.nccmt.ca/resources/search/241

66
Your Comments/Questions
• Use Chat to post comments
and/or questions
• ‘Send’ questions to All (not
privately to ‘Host’)
Chat
Participant Side
Panel in WebEx

67
Poll Question #4
Could this method or tool be useful
in practice?
A. Very useful
B. Somewhat useful
C. Not at all useful
D. Don’t know

68
Your Feedback is Important
Please take a few minutes to share your thoughts
on today’s webinar.
Your comments and suggestions help to improve
the resources we offer and plan future webinars.
The short survey is available at:
https://nccmt.co1.qualtrics.com/SE/?SID=SV_8ib8Yz9pRao
GLBz

69
Poll Question #5
Would you be willing to receive an
email from the NCCMT with a
follow-up survey in 2-3 months?
A. Yes
B. No

70
Join us for our next webinar
NCCMT Spotlight on Methods & Tools: CDC Clear
Communication Index
Date: Wednesday, February 1, 2017
Time: 1:00 – 2:30pm EST
Interested in evidence-based criteria for clear communication
of public health material? Do you develop and review public
health communication materials? Are you interested in
enhancing clarity and ease of understanding of these
materials? Join us for a webinar on how the Clear
Communication Index can help you.
Register at: https://health-evidence.webex.com/health-
evidence/onstage/g.php?MTID=eb180a59b9994f1d4da0729d
c3d31345d

71
Poll Question #6
What are your next steps? (Check all
that apply)
A. Access the method/tool referenced in the
presentation
B. Read the NCCMT summary about the
method/tool described today
C. Consider using the method/tool in practice
D. Tell a colleague about the method/tool

Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
For more information about the
National Collaborating Centre
for Methods and Tools:
NCCMT website www.nccmt.ca
Contact: nccmt@mcmaster.ca

NCCMT Spotlight Webinar: Guide to Policy-Influence Evaluation

More Related Content

What's hot

Viewers also liked

Similar to NCCMT Spotlight Webinar: Guide to Policy-Influence Evaluation

More from The National Collaborating Centre for Methods and Tools

Recently uploaded

NCCMT Spotlight Webinar: Guide to Policy-Influence Evaluation