Methods for Monitoring Quality of OVC Programs Florence Nyangara, PhD. Futures Group/MEASURE Evaluation Global Health Conference May 29 th , 2009

The Questions? How can we measure and monitor the Quality of OVC programs? Which indicators will provide the most useful information to monitor and improve quality?

How can we measure and monitor the Quality of OVC programs?

Which indicators will provide the most useful information to monitor and improve quality?

Presentation objectives Understand why monitor quality of services Tools and approaches to systematically assess and monitor the quality of OVC services standards i.e. CSI Indicators necessary to make informed decisions to improve, manage, and report on the quality of services provided to OVC?

Understand why monitor quality of services

Tools and approaches to systematically assess and monitor the quality of OVC services standards i.e. CSI

Indicators necessary to make informed decisions to improve, manage, and report on the quality of services provided to OVC?

Quality Monitoring Process Well Defined Quality Standards - established for most Dimensions of Quality, such as technical competence, effectiveness, efficiency, safety, and coverage - for each service domain, i.e. Education Plan a monitoring system and a set of tools to collect data (existing M&E tools - multipurpose)

Well Defined Quality Standards - established for most Dimensions of Quality, such as technical competence, effectiveness, efficiency, safety, and coverage - for each service domain, i.e. Education

Plan a monitoring system and a set of tools to collect data (existing M&E tools - multipurpose)

Ultimate Goal of Improving Quality Comply with the set Standards Set a Quality Monitoring System Standards Set Improved Outcomes Improving Quality of Services

Improved

Outcomes

Quality Monitoring System/Plan Define objectives and expected results Define key activities or processes Collect the data as usual (use existing tools) Decide what information your program needs to make decisions, i.e. select key indicators Analyze data and use the information to make changes that improve service quality & program Inputs Activities Outputs Outcomes

Define objectives and expected results

Define key activities or processes

Collect the data as usual (use existing tools)

Decide what information your program needs to make decisions, i.e. select key indicators

Analyze data and use the information to make changes that improve service quality & program

Monitoring QL Improvement M onitoring processes and the measurement of specific outcomes ensures that we are meeting acceptable standards Example in Education: Essential Action (STD) : Engage schools/communities to ensure OVC attend school Process Indicator: # of schools/communities engaged Example Outcome: % OVC attending school or advancing in class/grade level (performance)

M onitoring processes and the measurement of specific outcomes ensures that we are meeting acceptable standards

Example in Education:

Essential Action (STD) : Engage schools/communities to ensure OVC attend school

Process Indicator: # of schools/communities engaged

Example Outcome: % OVC attending school or advancing in class/grade level (performance)

Methods of collecting Data….. Observations Client satisfaction surveys Focus group interviews Checklist (supervision or home visits) Spot-checks Periodic assessments of outcomes for sample of OVC Conduct program evaluation (baseline, mid-term and end-line surveys)

Observations

Client satisfaction surveys

Focus group interviews

Checklist (supervision or home visits)

Spot-checks

Periodic assessments of outcomes for sample of OVC

Conduct program evaluation (baseline, mid-term and end-line surveys)

Tools….. Home visit forms Supervision checklist Survey questionnaires Child Status Index (CSI) Assessment – OUTCOMES which can also gives data on outputs: What services child received Frequency and source of support OVC needs and outcomes in each domain

Home visit forms

Supervision checklist

Survey questionnaires

Child Status Index (CSI) Assessment – OUTCOMES

which can also gives data on outputs:

What services child received

Frequency and source of support

OVC needs and outcomes in each domain

Example – PC3 Program Ethiopia, used the CSI to monitor the effects of QL Improvement 575 Tier III partners & 5 international partners (TI) Developed service quality standards to achieve set Goals (see Child Status Index) Conducted CSI assessments 3 times 6-months apart. A sample of 3,000 OVC were assessed in the 3 waves Results show improvement in all domains except Food & Nutrition which declined. We can say – QL improvement is associated with better outcomes

575 Tier III partners & 5 international partners (TI)

Developed service quality standards to achieve set Goals (see Child Status Index)

Conducted CSI assessments 3 times 6-months apart.

A sample of 3,000 OVC were assessed in the 3 waves

Results show improvement in all domains except Food & Nutrition which declined.

We can say – QL improvement is associated with better outcomes

Results – PC3 - CSI Assessment Mean scores 12 indicators during the three assessments Domains Jan-08 Jun-08 Mar-09 Domain Average 1 Food security 2.73 2.7 2.71 2.71 2 Nutrition& growth 2.88 2.89 2.89 2.89 3 Shelter 2.8 2.78 2.83 2.80 4 Care 2.89 2.95 3.02 2.95 5 Abuse and exploitation 3.18 3.25 3.28 3.24 6 Legal protection 3.16 3.23 3.28 3.22 7 Wellness 3.09 3.1 3.2 3.13 8 Health care services 2.92 3 3.1 3.01 9 Social behavior 3.04 3.08 3.16 3.09 10 Emotional health 3.16 3.23 3.27 3.22 11 Performance 2.92 2.94 3.03 2.96 12 Education & work 2.96 3.05 3.1 3.04

Mean scores 12 indicators during the three assessments Domains

Jan-08 Jun-08 Mar-09 Domain Average

1 Food security 2.73 2.7 2.71 2.71

2 Nutrition& growth 2.88 2.89 2.89 2.89

3 Shelter 2.8 2.78 2.83 2.80

4 Care 2.89 2.95 3.02 2.95

5 Abuse and exploitation 3.18 3.25 3.28 3.24

6 Legal protection 3.16 3.23 3.28 3.22

7 Wellness 3.09 3.1 3.2 3.13

8 Health care services 2.92 3 3.1 3.01

9 Social behavior 3.04 3.08 3.16 3.09

10 Emotional health 3.16 3.23 3.27 3.22

11 Performance 2.92 2.94 3.03 2.96

12 Education & work 2.96 3.05 3.1 3.04

Conclusion Define service quality standards Monitoring Steps: Ensure all essential actions for each domain are taken (use checklist) Progress in implementing the essential actions (# done) Outputs: Number of children reached with improved services (# OVC) Monitor outcome progress – periodic CSI assessments

Define service quality standards

Monitoring Steps:

Ensure all essential actions for each domain are taken (use checklist)

Progress in implementing the essential actions (# done)

Outputs: Number of children reached with improved services (# OVC)

Monitor outcome progress – periodic CSI assessments

Thank You!!