1. Monitoring & Evaluation
Using
Evidence Based Participatory
Quality Improvement Techniques in
Public Health Programs
Presented by,
Mr. Rakesh. N. Pillai
Socio Research and Reform Foundation (NGO)
- SRRF
1

2. STEPS TAKEN TO IMPROVE THE QUALITY
OF A PUBLIC HEALTH PROGRAM
- AN EXAMPLE
India Diabetes Educator Program
2007-2011
Background
• India has 2nd leading global case load of Diabetes
• National prevalence is estimated at 8.3%*
• Diabetes Educators are not recognized as a health
professional at the start of this program
2
*Source: International Diabetes Federation, Diabetes Atlas 5, 2011

3. Program Objective
• To develop a cadre of Diabetes Educators in
India, creating a favorable environment to
increase their acceptance in patient care
• To increase positive self efficacy of Diabetes
Patients by promoting self care behaviour and
achieve clinical metabolic control
Partnerships
- Society for Action in Community Health – SACH India
- Project HOPE, USA & Other implementing partners
- Eli-Lilly, BD, Bayers & Many Others
3

4. Program Overview
4

5. M&E Team - Objective
• To improve the Quality of the Program
• Set up an Evidence based Continuous
Monitoring and Evaluation system
• Create a Health Management Information
System (HMIS)
• Establish a mechanism for regular / quarterly
evaluations and take corrective steps
• Overall to foster an environment for the
success of the program
5

6. What is M&E ???
Note: M&E should be done at the planning phase
Document
Program
Activities
Reduce
Biases
Generalize
Results
Summarize
Figures
M&E
Disseminate
Results
Establish
Uniformity
Create
Evidences
6

7. QC – Seven Step Formula
Significance :
‘You can’t get good without knowing the basics’
QC- 7 step formula is the basic procedure for
solving problems scientifically, rationally,
efficiently and effectively.
This is a problem solving strategy which allows
any individual to solve difficult problems
rationally and scientifically
7

8. What is QC – Seven Step Formula
Step Purpose
1
2
Action Items
Understand Topic
-Identify Problem
-Decide topics to address
Understand Situation -Mechanisms for collecting information
& Set targets
-Decide on targets (value and deadline)
3
Plan Activities
4
Analyse Causes
5
6
Consider &
implement counter
measures
Check results
7
Standardize &
-Decide ‘what to do’
-Decide schedule, Divide responsibilities
-Check present values of characteristics
-List possible causes, Items to tackle
-Propose ideas for counter measures
-Ways to put counter measures in place
-Check details of counter measures
-Compare results with targets
-Identify tangible and intangible benefits
Establish new standards & revise old, 8

9. Step 1 : Understanding the Topics –
Diabetes Educator Program
» Check the roles of each departments & roles
-
Central Co-ordinating Office
Finance & Admin
Program Implementing Team
Each Partner centres- MT’s & DE’s
» Check the policies & objectives of departments
» List various problems by each departments
» Select most appropriate problems to be tackled
> Pareto Principle – 80% of problems could be
attributed to 20% of the causes
Tools Used
a. Cause & Effect Diagram
b. Pareto Diagram
9

10. Components of the M&E System ?
M&E system should establish a system to;
1. map Performances of the Program
- through performance/outcome indicators
- during baseline, interim and final evaluation
2. map the Progress – through Process Indicators
- Quarterly reports
- Bi-annual reports etc
3. Detailed Implementation Plan (DIP)
4. Monitoring Cost & Time Schedules
10

11. Step 2 : Understand Situation & set
targets – Diabetes Educator Program
» Decide on control characteristics
» Survey the past trends & existing levels
» Investigate the flow of work & Processes
» What sorts of changes occurred over time
» Stratify data as per problems
» Statistical dispersion / deviation is measured
» Prioritise
Tools Used:
Graphs, Checklists, Histograms, Scatter Diagram
11

12. We Strategized M & E as;
PDCA Wheel
Act
Plan
Check
Do
Source: TQM Approach / Kaizen Principle
12

13. Step 3 : Planning M&E – Diabetes
Educator Program
We decided on;
a. Action Items
b. Schedule of activities
c. Activity Plan
Tools Used : Control Charts, Check lists etc.
13

14. Options considered for M&E
1. ‘Health Management Information Systems
(HMIS)’
- Many software programs are available
2. Finally decided to make a simple, Excel sheet
- ‘Program Design Tool (PD Tool)’
14

15. Step 4: Analysis
Questions to design ‘PD Tool’
According to
a. 5W1H [What, When, Where, Who, Why and
How]
b. 4M Technique [Manpower, Machines,
Materials and Methods]
• How many readings do we need?
• What period should we collect data?
• How to stratify data?
• What sampling or measurement methods?
15

16. Step 5: Design Counter Measures
Consider Problem from all sides
Collect ideas from experts
Collect ideas from various stakes in the
processes
Using lateral thinking
Select countermeasure proposals
- Effect
- Feasibility
- Economy
16

17. ‘PD Tool’ Definitions for Diabetes
Educator Program
1. Performance Indicators
- High level indicators that lead directly to
your Goal and Objectives
- Usually in Percentages
- Common for Baseline, Interim and Final
Objective
Foster an enabling
environment for diabetes
education and training in
India
Performance/Outcome
Indicator
% improvement in wellbeing
of Diabetes Patients
17

18. Definition’s & Examples ...Contd
2. Process Indicators
- Indicators that measure progress at the
activity level.
- Usually simple counts
Program Specific Activities
Process Indicators
Develop national curriculum
for diabetes education
Conduct Master Trainer
training
# of recognized curriculum
developed
# of Master Trainers trained
18

19. Definition’s & Examples ...Contd
3. Detailed Implementation Plan (DIP)
- Based on Process Indicators / Activity based
- Map the progress every quarter through
quarterly reports
Process
Target
Indicators
# Students
3000
certificate in
Diabetes
Education
Year 1
Year 2
Year 3
Year 4
740
810
860
920
19

20. DIP System
Hide blank rows
Hide blank rows
Year 2
Target LOP
Hide blank
rows
Hide blank
rows
Hide blank
rows
Hide blank
rows
Hide blank
rows
Hide blank
rows
Hide blank
rows
Hide blank
rows
1
# of Master Trainers trained
35
Actual achieved
# of MT's actively engaged
in conducting Diabetes
2 Education and training
Q1 Q 2 Q 3 Q 4 Q1 Q 2 Q 3 Q 4 Q1 Q 2
35
66
30
Actual achieved
# Students enrolled in PG
Certificate in Diabetes
3 Education
Year 4
Q3
Q4
Sep- Dec- Mar- Jun- Sep- Dec- Mar- Jun- Sep- DecJunMar-11
08 08 09 09 09 09 10 10 10 10
11
Process Indicators
Hide blank
rows
Year 3
35
22
22 12
30
44
4,325
325 500 500 500 500 500 500
500
500
4,542
304 464 482 480 470 500 411
448
450
# Students of PG Certificate
in Diabetes Education
3,000
4 presenting the Final Exam 3,000
197 80 90 100 100 200 200
547
581
197 80 90 100 100 200 200
547
Actual achieved
Actual achieved
4,325
2,855
20
581

21. Quarterly Report
Reporting
Period
Year
2011
Current Quarter
green
Scoring
green
Version 6.0,
Expected to Date
green
September 2010
Approvals
Beneficiaries
Project Name
Diabetes Educator Project
India
30-Dec-11
05058
Country
% Elapsed: 100% Project Code
Goal
Decrease morbidity and mortality related to diabetes in India
Outcome Indicators
Year 4
Indicator
Baseline
Final
Targe +/+/- baseline
Data of data collection (planned or
t
target
Date Result
30-Jun-11
actual)
30-Dec8
7.5
20%
80%
average a1c for patients
10
7.3
30-Dec61%
71% -5%
5%
% improvement in well being index (WHO 5)
10
66%
30-Dec11
16 -200%
300%
improvement in patients Self Efficacy score
10
14
Process Indicators
Current
Expected to Date
Quarter
%
Description
Target Actual
# of Master Trainers trained
0
0
189%
# of MT's actively engaged in training
0
0
147%
# Batches planned/initiated
0
0
99%
# Students enrolled i
0
0
105%
# Students presenting the Final Exam
0
0
103%
# Students graduating
0
0
103%
21
# Annual National Meeting
0
0
100%
1
2
3
4
5
6
7
8
9
10

22. Timeline Set for the Program Activities
Timeline
Version 3.0, September 2010
Project
Name:
Country:
India Diabetes Educator Project
Start
Date:
India
End
1-Jul-07 Dat
e:
Optional page
Activities
Program Specific Activities
1
2
3
4
5
Develop national curriculum for
diabetes education
Develop partnerships with leading
centers for diabetes education,
care and management
Develop regional network/TAG of
Des
Conduct Master Trainer training
Master trainers to train Diabetes
Educators
Year 1
Year 2
Year 3
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
Sep- Dec- Mar Jun- Sep- Dec- Mar Jun- Sep- Dec- Mar Jun07 07 -08 08 08 08 -09 09 09 09 -10 10
22

23. Program Design
Hi de bl ank rows
HMIS Developed
Versi
Hi de bl ank rows
Hi de bl ank rows
Hi de bl ank rows
Hi de bl ank rows
Project Name
Country
Donor
India Diabetes Educator Project
Location
India
Lilly, BD and Bayer
Start Date
End Date
India
1-Jul-07
30-Dec-11
$ award
Duration (# y
Problem Statement
Diabetes is a global pandemic and India has the highest case load of any country. While patient education is ke
self-management, no formal Diabetes Educator program exists at the national level. Poor self-management lead
and mortality and diabetes is currently the second leading cause fo death nationwide.
Program Snapshot
Launch a sustainable model for forming allied health professionals as diabetes educators nationwide, in partn
centers of excellence.
Hide blank rows
Hide blank rows
Hide blank rows
Hide blank rows
Goal
Objectives
Hide blank rows
Foster an enabling environment for diabetes education
1
and training in India
Hide blank rows
Decrease morbidity and mortality related to diabetes in India
Outcome Indicators
Data Source
Frequency
1
average a1c for patients
Evidences from
operations research
One time
Design and develop Diabetes Education distance learning
course for allied healthcare professionals benchmarked
on the best standards and incorporating Indian medical
practices
Hide blank rows
2
% improvement in well being index (WHO 5)
Evidences from
operations research
One time
Increase knowledge, enhance education skills and
improve practice of allied healthcare professionals in
diabetes education, care and management
Hide blank rows
3
improvement in patients Self Efficacy score
Evidences from
operations research
One time
2
3
Improve training outcomes among people living with
4 diabetes managed by allied healthcare professionals
Hide blank rows diabetes education by Project HOPE
trained in
Hide blank rows
Hide blank rows
Program Specific Activities
Hide blank rows
Process Indicators
Data Source
Frequency
1 Develop national curriculum for diabetes education
Hide blank rows
1
# of IDF recognized curriculum developed
Guidelines and peers/
experts
One time
Develop partnerships with leading centers for diabetes
2
education,
Hide blank rows care and management
2
# of centres adopting HOPE Diabetes Educator Program
Networks/ MOU
Annually
3 Develop regional network/TAG of Des
Hide blank rows
3
# of regional network/TAG of Diabetes Educators formed Networks/ MOU
4 Conduct Master Trainer training
Hide blank rows
4
# of Master Trainers trained
IDEP Database
Baseline & final
5
Hide blank rows
Master trainers to train Diabetes Educators
5
# of MT's actively engaged in conducting Diabetes
Education and training
IDEP Database
Semi-annually
6
Hide blank rows
Supervise/mentor Master Trainers
6
# Batches planned/initiated for imparting HOPE PG
Certificate in Diabetes Education
IDEP Database
Semi-annually
7
Hide blank rowsannual MT meetings
Conduct
7
# Students enrolled in PG Certificate in Diabetes
Education
IDEP Database
Semi-annually
8
Hide blank rowspartner/donor meetings
Conduct
8
# Students of PG Certificate in Diabetes Education
presenting the Final Exam
IDEP Database
Semi-annually
9 Disseminate results
Hide blank rows
9
# Students graduating with HOPE's PG Certificate in
Diabetes Education with > = 70% grade in each milestone IDEP Database
and overall
Annually
23
Annually

24. Step 6: Check Results
• Operational Research was Initiated
To evaluate the impact of HOPE-trained
DEs on patient outcomes
a. Clinical parameters
b. Psychosocial parameters
c. Behavior change parameters
24

25. Results – Baseline to 6 Months
Baseline
Mean (S.D)
8.13 (1.54)
148.7 (39.9)
6-Month
Mean (S.D)
7.35 (1.15)
130.4 (34)
27.1 (4.5)
26.0 (4.1)
< .0001
Total Cholesterol
174.7 (37.9)
163.4 (32.9)
< .0001
Triglycerides
143.4 (74.3 )
139 (55.8)
.06
HDL Cholesterol
42.2 (11.5)
59.3 (28.2)
< .0001
LDL Cholesterol
101.9 (31.2)
79.9 (36.8)
< .0001
Blood
Pressure
132.7 (13.8 )
80.9 (6.5)
129.1 (12.4)
79.7 (6.5)
< .0001
Clinical Measures
HbA1C
Fasting blood glucose (mg/dl)
Body mass index (BMI)
Systolic BP
Diastolic BP
Significance
< .0001
< .0001
< .0001
* All analyses conducted using paired t-tests

26. Results – Baseline to 6 mos
Psychosocial /Behavioral
Variables
Well-being
Baseline
6 mos
Mean (S.D) Mean (S.D)
Significance
61 (16.8)
11.1 (3.5)
65.7 (23.5)
13.7 (3.6)
< .0001*
< .0001*
% people eating 4+ fruits or
vegetable per day
14.8 %
43.9 %
< .0001**
% people drinking 2+ litres of
water per day
58.1 %
70.2 %
< .0001**
% people doing 30+ minutes of
exercise per day
22.8 %
71.9 %
< .0001**
% people doing self monitoring
15.2 %
54.6 %
< .0001**
Self Efficacy
* Analyses conducted using paired t-tests
** Analyses conducted using Chi Square

27. Conclusion
QC Seven Step Quality Improvement Technique is
an useful tool to strategise Monitoring and
Evaluation Plans, especially in large, multicentre,
Public Health Programs
Mastering this tools helps the Program Managers
to Plan, Do, Check and Act upon the Objectives
finally achieving the Program Goals
27

28. Thank You
28