ALL the evidence webinar: Appraising and using evidence about community context and organizational resources (2023)

Welcome!
• This webinar will be recorded.
• Your microphone and camera will be turned off for the duration of the
webinar.
• To ensure accessibility, live captions can be enabled from the control
panel.

November 15, 2023
1:00-2:30 EST
Susan Snelling, PhD
Senior Knowledge Translation Specialist
ALL the Evidence:
Appraising and using evidence about community
context and organizational resources

Housekeeping
• Connection issues
• We recommend using a wired Internet
Connection
• If you are experiencing technical issues please
send a private message to Alyssa Kostopoulos
• Live transcripts are enabled
• Use the Q&A and chat to post questions
and/or comments throughout the webinar
• Post your questions in the Q&A
• Send questions about technical difficulties in a
private chat to Alyssa Kostopoulos
• Polling

After Today
After the webinar, access the recording (in English) at
www.youtube.com/nccmt and slides in English and French at
www.slideshare.net/NCCMT/presentations.

Disclaimer
5
• Funded by the Public Health Agency of Canada | Affiliated with McMaster University.
• Production of this presentation has been made possible through a financial contribution
from the Public Health Agency of Canada. The views expressed here do not necessarily
reflect the views of the Public Health Agency of Canada.

Land Acknowledgement
McMaster University recognizes and acknowledges that it is
located on the traditional territories of the Mississauga and
Haudenosaunee nations, and within the lands protected by
the Dish With One Spoon wampum agreement.

The National Collaborating Centres for Public Health

The National Collaborating Centre for Methods and Tools
• Facilitates and scales evidence-
informed decision making in public
health organizations
• Provides high-quality resources, real-
world training and practical mentorship
that respond to the evolving needs of
public health
• Advances stronger public health,
driven by the best-available evidence,
to improve the health and well-being of
every person living in Canada

Pre-webinar Polling Questions
1.How many people are watching today’s session with you?
A) Just Me
B) 2-3
C) 4-5
D) 6-10
E) >10
2. Have you visited the National Collaborating Centre for Methods and Tools’ website or used its resources before?
A) Yes
B) No
3. If you stated YES on the previous question, how many times have you used the NCCMT’s resources?
A) Once
B) 2-3 times
C) 4-10 times
D) 10+ times
E) I’ve never used the NCCMT’s resources.

Presenter
10
Susan Snelling, PhD
Senior Knowledge
Translation Specialist

Learning Objectives
11
1. Understand the process of appraising and applying evidence from
research and context.
2. Be familiar with the NCCMT QACE and RPA Tools to support you in
using the best available evidence.
3. Understand the application of the tools through an example scenario
featuring critical appraisal and multiple sources of evidence.

1 2 3 4
EIDM Tools Scenario Decision
Flow
Core Concepts in Evidence
Informed Decision Making

Evidence-informed Decision Making (EIDM)
13
The process of distilling and
disseminating the best
available evidence from
research, context and
experience, and using that
evidence to inform and
improve public health practice
and policy.

Four Domains of Evidence in Public Health
14

ALL the Evidence
Evidence-informed decision making
involves the use of multiple sources
of evidence.
Sounds good!
And…
• Not all evidence contributes in the
same way – different sources inform
us about different aspects of the
decision.
• Not all evidence should be weighted
equally – higher quality evidence (of
any kind) should have a greater
influence on the decision
What interventions
are effective?
Is there a need
locally?
Context for
intervention?
Do we have the
necessary
resources?

1 2 3 4
Tools to Support Use of Evidence
Flow

Tool Development
17
Academic and grey literature review
for existing tools and measures
Suggestions and feedback from
librarians, public staff in diverse roles,
and public health professionals
across Canada

Quality Assessment of Community Evidence (QACE)
18
The Quality Assessment of Community Evidence (QACE) Tools were developed
to assess the quality of evidence for:
“Is the quality of this evidence about community needs and preferences
good enough to influence decision making?”
www.nccmt.ca/qace

Core Dimensions of Quality for Local Health Status and
Community Evidence
19

Resource Planning and Assessment (RPA) Tool
20
Focuses on resource planning, capturing what is known about available resources
and the level of confidence in estimates of resource needs.
Resources
e.g., financial or
human resources
“Based on evidence about financial and human resources, is a given
intervention feasible?”
https://www.nccmt.ca/rpa

Core Dimensions of Resource Planning
21
Human Resources
• Personnel
• Skills, training,
certifications
• Collaboration with key
partners
Material Resources
• Equipment and
supplies
• Honoraria/incentives
• Space
Organizational Support
Resources
• Equity considerations
• Leadership
• Information technology
• Promotion/media/social
media
• Administrative support;
Enthusiasm; Buy-in

Quality Assessment of Research Evidence
22
Lots of tools available!
Try CASP as one option: https://casp-uk.net/casp-tools-checklists/
Research
e.g., qualitative or
quantitative research
“Is the quality of this research evidence about effectiveness
good enough to influence decision making?”
https://www.nccmt.ca/learning-modules

1 2 3 4
Scenario Using Multiple Sources
of Evidence
Flow

Scenario
24
Decision to Promote Active School Transportation
Critical Appraisal and Application of Multiple Sources of Evidence

Scenario: Active School Transportation
25
Decision: Should our community implement an Active School
Transport intervention?
Research
e.g., qualitative or
quantitative research
• Find research evidence about
what works
• Assess quality, retaining the
best-available evidence

What did the Jones et al (2019) systematic review include?
• Effectiveness of active transport interventions on physical activity and fitness in primary
school children (aged 4-11 years), compared to an inactive control.
• Seventeen eligible studies were included.
What does Jones (2019) tell us?
1. Active travel shows promise in increasing physical activity in primary school children.
a. Intervention group saw small, statistically significant difference in outcomes.
2. Walking school buses & educational strategies were most effective.
3. More complex interventions did not necessarily produce larger effect sizes.
4. Overall quality of included studies = weak.
29

30
• Find research evidence
• Assess quality, retaining
the best-available
evidence
• Active travel interventions
increase physical activity
in primary school children
– small effect.

Community Evidence
31
• What is the local need?
• Magnitude of the health issue in
the local setting
• Significance and importance of
the issue in comparison to other
community health concerns
• Context for Intervention
• Who else is working on this issue?
• Needs, interests and intervention
preferences of community members,
stakeholders, and consumer groups
• Support or opposition from the
public/government officials; Current
political climate

32
• Find evidence of local
community need and context
• Assess quality, retaining the best-
available evidence
• E.g., current physical activity data
• E.g., current distances traveled to
school
Community health
issues, local context:
QACE Tool A

33
QACE Tool A Is the quality of this evidence about local health issues and
local context good enough to influence decision making?
Relevant
Trustworthy
Equity-informed
Possible sources of evidence:
• School census
-Proportion of kids within walking distance
• Caregiver survey
-Current rates of active transport
• Mapping current routes to school
Quality Assessment:

34
Is the quality of this evidence about local health issues
and local context good enough to influence decision
making?”
QACE Tool A
Relevant
• Meaningful  Does this source address my topic of interest?
 Is this indicator relevant to my topic?
 Does this source allow me to determine the significance of this
issue compared to other issues?
• Applicable
• Transferable
 Does this source include data available at the regional/local level?
How important is regional/local level data for my topic of interest?
Summary of your assessment:
(e.g., quality, gaps and limitations)

35
making?
QACE Tool A
Trustworthy
• Methodologically
sound
 What methods were used? Were those appropriate methods for the topic?
 To what extent did the methods reduce the risk of bias?
 Are there conflicts of interest that could introduce bias into the evidence?
• Transparent
• Cognizant of
research evidence
 Does this source draw a conclusion? Is the conclusion based on evidence?
To what extent is the basis for that conclusion transparent?
 To what extent does the conclusion align with other available evidence
(“triangulation”)? What might account for any differences?
• Richness/
Saturation/
Adequacy of data
 Are there gaps in this data source? How significant are those gaps to a
complete understanding of the issue?

36
making?
QACE Tool A
Equity-Informed
• Representative of
community
 What is the level of analysis and reporting (e.g., census area or smaller)?
 Does this source provide data on the health status of specific groups in the
community?
 Where are the population health data gaps?
• Engaging
stakeholders
• Intersectional
• Inclusive
 Does this source include all groups, including disadvantaged groups?
 Were any population groups excluded from data collection (e.g., people
without telephones, no permanent housing, etc.)?
 Where are the population health data gaps?
• Culturally safe; ethical
data collection
 To what extent was the evidence collected in an ethical and culturally safe
way?
 Did the source abide by Tri-Council policies on ethical data collection?
 Were OCAP™ principles observed, if applicable?
 Were communities consulted about whether and how they wanted to
provide data?
 Were communities involved in the interpretation and sharing of the findings?

37
QACE Tool A Is the quality of this evidence about local health issues and
local context good enough to influence decision making?
Relevant
Trustworthy
Equity-informed
• School census
-Proportion of kids within walking distance
• Caregiver survey
-Current rates of active transport
• Mapping current routes to school
Local school data:
• 65% of students live within walking distance
of school
• 31% usually walk/wheel to school
Quality Assessment:

38
• Find evidence of local
community preferences
• Assess quality, retaining the
best-available evidence
• E.g., Other organizations
promoting active school
transport?
• E.g., focus group data on support
for active school transport among
parents? among school admin?
Community and
political preferences
and actions:
QACE Tool B

39
Is the quality of this evidence about community and political
preferences and actions good enough to influence decision
making?
QACE Tool B
Relevant
Trustworthy
Equity-informed
Quality Assessment:
Local survey data:
• Top reasons to not walk/wheel
• Top desired changes

making?
40
QACE Tool B
Relevant
• Meaningful  Does this source outline a clear question that is being answered?
 Does this source address my topic of interest?
• Applicable
• Transferable
 How applicable and transferable is this evidence to my community and local
context?
 How similar is the context of this source to my setting?

making?
41
QACE Tool B
Trustworthy
• Methodologically
sound
 What methods were used? Were those appropriate methods for the topic?
 To what extent did the methods reduce the risk of bias? (A separate quality
assessment tool such as the GRADE-CERQual can help answer this
question.)
 Are there conflicts of interest that could introduce bias into the evidence?
• Transparent
• Cognizant of research
evidence
 Does this source draw a conclusion? Is the conclusion based on evidence?
To what extent is the basis for that conclusion transparent?
 To what extent does the conclusion align with other available evidence
(“triangulation”)? What might account for any differences?
• Richness/ Saturation/
Adequacy of data
 To what extent is this evidence adequate for understanding all perspectives
on the issue?
 How significant are any gaps to a complete understanding of the issue?

making?
42
QACE Tool B
Equity-Informed
• Representative
of community
 How representative is this source of my community, including disadvantaged
groups?
 Have all affected community groups been considered? Whose perspectives are not
heard?
• Engaging
stakeholders
• Intersectional
• Inclusive
 How were affected communities engaged in the evidence gathering process?
Whose perspectives are not being considered?
 How participatory was the engagement of members of affected community groups?
 Where does the community engagement fall on the IAP2 spectrum: inform, consult,
involve, collaborate, empower?
• Culturally safe;
ethical data
collection
 To what extent was the evidence collected in an ethical and culturally safe way?
 Did the source abide by Tri-Council policies on ethical data collection?
 Were OCAP™ principles observed, if applicable?
 Were communities consulted about whether and how they wanted to provide data?
 Were communities involved in the interpretation and sharing of the findings?

43
making?
QACE Tool B
Relevant
Trustworthy
Equity-informed
Quality Assessment:
Local survey data:
• Top reasons to not walk/wheel:
• Family schedule 42%
• Safety/traffic 20%
• Too far 11%
• Top desired changes
• Improved street safety 52%
• Designated routes 19%
• Different bell times 4%

44

45
• What resources will be needed to
implement the proposed
intervention/program/service?
• Are the needed resources
available?
• Is the intervention/program/
service feasible from a resource
standpoint?
Human, material,
organizational support

46
Based on evidence about financial and human resources,
is this intervention feasible?”
RPA Tool
46
Human resources, e.g., coordinator time, volunteer time
Material resources, e.g., vests for walking school bus leaders
Organizational support resources, e.g., GPS research on safe
routes; other organizations that want to help
What resource gaps do we have?
How accurate and trustworthy is this information?

Consider the answers for all resource types to determine:
47
“What resources will be needed to
implement the proposed
intervention/program/service?”
“Are the needed resources
available?”
Resources
Needed:
Type,
Timeframe
and Cost
Resources
Available:
Type,
Timeframe
and Cost
Gap
(Available-
Needed)
Data
Sources
“Is the intervention/program/service
feasible from a resource standpoint?”

1 2 3 4
Decision Flow Using Multiple
Sources of Evidence
Flow

Decision Flow
51
Research:
Yes
Local need: Yes
Community support:
Yes
Resources: Yes
Resources: No
Community
Support: No
Local need: No
Research:
No

Decision Flow: Scenario
52
Research:
Yes
Local need: Yes
Community support:
Yes
Resources: Yes
Resources: No
Community
Support: No
Local need: No
Research:
No
Weak evidence
Small effect on PA
of education and
walking school bus
QA?
31% walk now
65% could walk
QA?
Address barriers through
education; then walking
school bus (pilot?)
Low cost
Feasible

Tools to Support Using ALL the Evidence
53
• The new QACE and RPA tools fill the gap in assessing quality of evidence for
public health decision-makers.
• Community evidence, including local health status and needs and community and
political preferences and actions, should be assessed for relevance,
trustworthiness, and being equity-informed.
• Evidence related to resources, including human, material and organizational,
should be assessed for intervention feasibility, resource gaps, and the accuracy
of the information
QACE tools: nccmt.ca/qace RPA tool: nccmt.ca/rpa

Need More? Book a KB Consult!
54
• NCCMT is here to support you!
• Knowledge Broker support is
available to help you apply these
tools.
• We’re holding ‘office hours’ on
Wednesdays from 1:00 pm to
1:30 pm EST.
• To book, email
nccmt@mcmaster.ca

Webinar Feedback
Your responses will be kept anonymous.
Please indicate your level of agreement with the following:
1. Participating in the webinar increased my knowledge and understanding of evidence-informed decision
making.
2. I will use the information from today’s webinar in my own practice.
3. Which of the following statements apply to your experience with the webinar today (check all that
apply):
□ The webinar was relevant to me and my public health practice
□ The webinar was effectively facilitated
□ The webinar had opportunities to participate
□ The webinar was easy to follow along
□ The webinar met my expectations
Strongly agree Agree Undecided Disagree Strongly Disagree
Strongly agree Agree Undecided Disagree Strongly Disagree

Contact Us
nccmt.ca
nccmt@mcmaster.ca
Scan for access to the
NCCMT website!

New Launch! User Dashboard
• Personalize your EIDM journey with
NCCMT’s NEW user dashboard!
• Track your progress as you work
through available training
opportunities.
• Save evidence syntheses and
publications.
• Build your own toolkit to match
your EIDM needs.
• Log in or sign up for a free NCCMT
account to take advantage of this
dashboard and enhance your
learning: nccmt.ca/login.
ACCESS MY DASHBOARD
Modules in Progress

Funded by the Public Health Agency of Canada | Affiliated with McMaster University. Production
of this presentation has been made possible through a financial contribution from the Public
Health Agency of Canada. The views expressed here do not necessarily reflect the views of the
Public Health Agency of Canada.
Thank You!

References
60
• Jones RA, Blackburn NE, Woods C, Byrne M, van Nassau F, & Tully MA. (2019). Interventions promoting
active transport to school in children: A systematic review and meta-analysis. Preventive Medicine, 123,
232-241.
• National Collaborating Centre for Methods and Tools. (2023). Resource Planning and Assessment (RPA)
Tool. https://www.nccmt.ca/rpa
• National Collaborating Centre for Methods and Tools. (2023). Quality Assessment of Community
Evidence (QACE) Tools. https://www.nccmt.ca/qace
• National Collaborating Centre for Methods and Tools. (2023). User’s Guide: Quality Assessment of
Community Evidence (QACE) Tools.
https://www.nccmt.ca/uploads/media/media/0001/02/21da73e270f256c3c008ed34627fb54b6d0f454c.pdf

ALL the evidence webinar: Appraising and using evidence about community context and organizational resources (2023)

Recommended

Recommended

More Related Content

Similar to ALL the evidence webinar: Appraising and using evidence about community context and organizational resources (2023)

Similar to ALL the evidence webinar: Appraising and using evidence about community context and organizational resources (2023) (20)

More from The National Collaborating Centre for Methods and Tools

More from The National Collaborating Centre for Methods and Tools (20)

Recently uploaded

Recently uploaded (20)

ALL the evidence webinar: Appraising and using evidence about community context and organizational resources (2023)