Health Program and Evaluation

1. Nova Scotia 811Telecare
Service Evaluation Plan
Nursing 5893 Dr. Lori Weeks
Steve Iduye, Xiaoping Zhuang, John Amoah-Gyekye.

2. Agenda
➢Background and Context
➢Goal and Objectives
➢Target Population
➢Evaluation Questions
➢Program Theory and Logic Model
➢Stakeholders Communication
➢Evaluation Design and Data Collection
➢Data Analysis Method
➢Resource
➢Evaluation Results and Recommendations

3. Background and Context
➢Inequitable access to health services
Disparities in health status
Outcomes between rural and remote residents.
➢Recent years, tele services have been widely used
Telemedicine： Provide health education
➢Healthlink 811：Telehealth programs
Nova Scotia Department of Health (DoH)
Registered Nurses
Over 125 different languages
24 hours a day, 7 days a week

4. Goal
➢To reduce hospital services utilization through 811 telecare
program
➢To evaluate if the program is a catalyst for patients self-
directed change in health behaviour

5. Objectives
➢To improve patients symptoms and Increase patients
confidence in self care by
➢To reduce hospital services utilization through 811 telecare
program by
95%
10%

6. Target Population
Residents of Nova Scotia with the following non-emergency
symptoms:
➢Abdominal Pain
➢Medication Questions
➢Chest Pain
➢Cough
➢Fever in 3 months old or older
➢or residents that need to clarify information regarding
medications

7. Evaluation Questions
➢Does 811 Telecare program reduce hospital utilization
services?
➢Is the information provided by Registered Nurses helpful in
improving patients symptoms?

8. 811 Telecare Service Program Logic Model
Inputs
Activities
Final Outcomes
Immediate Or
Intermediate Outcomes
Outputs
✓Budge
✓Experienced Registered Nurse
✓IT Specialists and Informaticians
✓Decision Support Software
✓Patients Database
✓Assessment of the urgency of a caller's
symptom(s)
✓Provision of health advice and information by
experienced registered nurses
✓24 hours per day, 7 days per week operation
✓TTY service for the hard of hearing
✓Service in over 120 different languages (third-
party language interpretation service)
✓Delivery of culturally competent care
✓All residents of Nova Scotia with non emergency
symptoms will start to call
✓Improved Callers Symptoms
✓Improved Callers Confidence in self care
✓Eliminate health knowledge deficit
✓It will improve callers Symptoms and Increased
patient's confidence in self-care by 95%
✓It will reduce hospital service utilization by 10%

9. Stakeholders Communication
• Phase One: The team conveyed our letter of intent to the
Telecare service coordinator. We also followed this letter
up with several calls in order to establish a relationship
with the main program office in Halifax.
• Phase Two: A video conference was arranged by the
program office with the team to answer all our questions
and to guide us in a direction they think is necessary.
• Phase Three: Our team will deliver the evaluation plan in
writing to the program office after we are done.

10. Evaluation Design
➢In line with the goals and objectives of this evaluation plan,
we use:
➢A group pretest and post test quasi-experimental design to
conduct the evaluation effect and impact.

11. Data sources to answer each question posed in the
above
• Feedback for caller Satisfaction
• Stakeholder Satisfaction from the System
• Reactions from the Community
• Phone Service Tele triaging impact
• 811 Demographic Information
• District Health Authorities.
• 811 Registered Nurses
• Funders

12. Data Collection Matrix - Matching data collection to key evaluation questions
Key evaluation
questions
Interviews Focus Group Questionnaire Case Study
Does 811 Telecare
program reduce
hospital utilization
services?
✔ ✔ ✔ ✔
Is the information
provided by
Registered Nurses
helpful in improving
patients symptoms?
✔ ✔ ✔ ✔

13. Data Collection Matrix - Data collection (secondary data) options
Option What might it include Example
Retrieving existing documents
and data
▪ Formal policy documents,
implementation plans and
reports
▪ Official statistics Programme
monitoring data
▪ Programme records
Review of programme planning
documents,
minutes from meetings, progress
reports
Collecting data from individuals
or groups
▪ Interviews – key informant,
individual, group, focus group
discussions, projective
techniques
▪ Questionnaires or surveys –
email, web, face to face,
mobile data
Key informant interviews with
representatives from relevant
departments, organizations
and/or the wider development
community
Interviews with programme
managers, programme

14. Limitations of using methods
• Interviews
➢Identifying individuals who are willing to be open to respond to the interview
• Focus Group
➢Scheduling the meeting with the number of people of the focus group.
➢challenge the difficulty in capturing the information being discussed by the group.
• Questionnaire
➢Some respond statement not clear and needed further clarification.
➢Extreme brief responses may need extra follow up questions making it a bit
difficult to interpret the data
• Case Study
➢An overwhelming amount of data to deal with.
➢Which evaluation questions needs to be refined for a couple of times

15. Data Collection
➢ This evaluation uses a mixed-method (quantitative and qualitative) data collection
approach.
➢ Each evaluation question uses multiple sources of information (triangulation of
evidence) to answer the evaluation question. For example, administrative data,
information about patient pain and temperature assessment are quantitative data
while patient that clarifies medication using 811 teleservice agent generates a
qualitative data.
➢ Statistical analysis is necessary to draw conclusions about quantitative data. E.g
Regression analysis is used to predict (including forecasting), inference, and
understanding causal relationships.
➢ Qualitative analysis software like NUDIST, ATLAS-ti could also be used in managing and
facilitating qualitative data analysis (Issel, 2014)

16. Data Collection- To Determine Sample Size
➢A power analysis will be conducted in order to determine
the minimum sample size needed for this study.
➢By using Gpower 3.1, at expected margin of error of 5%,
confidence level of 95%, with initial population size of
137,000 that participated in 811 telecare program last year,
and response distribution of 50% with power size of 80%,
the minimum sample size for the evaluation is 384.

17. Data Analysis Method
➢Data is collected and computerized for analysis using a
group pretest and post test approach mentioned above.
➢All the data will be coded and statistical calculations will be
done using R studio software at the end of data collection.

18. Data Analysis Method- Change Differences
Pretest (Baseline Measurement): Non emergency symptoms reported by
Nova Scotian residents include:
➢Abdominal Pain
➢Medication Questions
➢Chest Pain
➢Cough
➢Fever in 3 months old baby or older as reported by parents or
caregivers
➢No prior hospital or doctor's clinic visit for the same symptoms stated
above

19. Data Analysis Method- Change Differences
Intervention or Treatment
➢Provision of health information and assistance that is
clinically sound, Nova Scotia-appropriate and easily
accessible from anywhere in the province.

20. Data Analysis Method- Change Differences
Post test Measurement
➢Measurement of changes in clinical symptoms stated
above (positive or negative changes)
➢Measure the number of referral to hospital or doctor's clinic
for the same symptoms stated as baseline

21. Data Analysis Method- Change Difference
Calculation
➢ To measure the change difference, the initial pretest baseline is subtracted before the intervention
from the subsequent post test score (Issel, 2014). In other words, the amount of change = sum(each
post test- each pretest score) divided by Number of paired scores.
➢ According to Issel, (2014) this formula is good to calculate change difference in only one group,
because each participant difference score is calculated and then averaged across all participants.
➢ For example pain assessment pre and post test could be done by using numeric rating scale(0-10) to
assess level of pain in all the callers with pain symptoms. Pre and post comparison are made.
➢ Cough assessment tool with number awarded to frequency, sound, and duration. Pre and post
differences are drawn.
➢ Fever is measured by thermometer with pre and post temperature is compared
➢ Hospital or doctor's clinic visit for the same symptoms stated above is evidenced by the numbers of
referral made by the 811 telecare agents.

22. Resource Plan
Task Name Task Description Staffing Expected Costs Timeframe
Goal: Evaluating
reduction of hospital
utilization services
through 811 telecare
program.
Evaluating patients’
health behavior self-
directed change
All May 19 -June 30
1. Current State
Analysis
Online resource
research and
contacting clients for
more information to
Understand the
program.
Xiaoqing, 811 staff Internal Human
resource,
Time,
Internet connection,
Facilities such as
computer and place
May19-May26
2.Developing
Evaluation Questions
Create several
questions based on
program’s objectives
for evaluation.
Steve Internal Human
resource,
Time,
Internet connection,
May19-May26

23. Resource Plan
Task Name Task Description Staffing Expected Costs Timeframe
3.Collecting Data Identify what data
should be collected
and from who. Data
collection methods
will be selected.
John, 811 staff,
external data source
firm staff
Internal Human
resource,
Time,
Internet connection,
Facilities such as
computer, places,
phone call
May19-June2
4.Analyzing Data Analyze collected
data with analyzing
tools, such as
statistical software.
Steve, John,
Xiaoqing,
professional data
analysts
Internal Human
resource,
Time,
Internet connection,
Facilities such as
computer and place,
hire professional
data analysts
June3-June12

24. Resource Plan
Task Name Task Description Staffing Expected Costs Timeframe
5.Establishing Evaluation results Based on the data
analysis, evaluation
results can be
created.
Steve, John,
Xiaoqing,
professional
evaluators
Internal Human
resource,
Time,
Internet connection,
hire evaluators
June13-June16
6.Developing Recommendations Based on the
evaluation results,
recommendations
can be made.
Steve, John,
Xiaoqing
Internal Human
resource,
Time, Facilities such
as computer and
place
June16-June22
7.Presenting Evaluation Results
and Recommendations
Presenting the
evaluation results
and evaluation
recommendations to
clients.
Steve, John,
Xiaoqing
Internal Human
resource,
Time,
Internet connection,
Place and facilities
for presentation
June23

25. Evaluation Results and Recommendations
Care Continuum
➢Finding a means to communicate the patients health needs to the
family physician is critical in continuing care. This is made possible
through the following measures:
➢Conversation regarding patients health complaints should be
forwarded the next day to the patients family physician in text format
in the absence of an EHR.
➢811 tele-care database should continuously interfaces with electronic
health record for easy information accessibility by other care
providers.

26. Evaluation Results and Recommendations
Upgrade Care
➢Develop application which can be used for both voice contact and
video contact such as Facetime, Skype.
➢Patients with their own health ID which connected to their health
records to sign in this 811 application.
➢Patient with such app can show their symptoms and give more
information to the registered nurse and that can improve
healthcare delivery.

27. References
➢ Issel, L. (2014). Health program planning and evaluation : A practical, systematic
approach for community health (3rd ed.). Burlington, MA: Jones & Bartlett Learning
➢ Hill, Kimberly Anne. (2014, March). Determining the Impact of Telehealth on Rural and
Remote Health Care Service Delivery in Canada. 2014 University of Regina. Retrieved
from: http://ourspace.uregina.ca:8080/handle/10294/54897
➢ S. Mahadevan, M.B.A., M.Phil., K. Muralidhar, M.D., M.B.A., and D. Shetty, M.S.,
F.R.C.S. (2012, November). Tele-Education Service Using Telemedicine Network in
Healthcare Industry. TELEMEDICINE and e-HEALTH . MARY ANN LIEBERT, INC. VOL. 18
NO. 9.
➢ McKesson Canada. (2010, July/August). HEALTHLINK 811. Nova Scotia partners to
provide better access to care. Retrieved from: http://www.summitconnects.com
➢ For trusted, everyday health information and advice, call 811. Retrieved on 2016, May
29 from: https://811.novascotia.ca/

28. Questions?