Improving measurement through Operations Research

Improving measurement through Operations Research Peter Winch [email_address] Department of International Health, Johns Hopkins Bloomberg School of Public Health and Technical Advisor on Operations Research, MCHIP Project

Brief history of measurement in NGO child survival projects (1)
Early projects
Wide variation in quality of evaluation
Lack of standardized outcome measures, sampling techniques and questionnaires
Hard to compare results between baseline and final, or between projects

Brief history of measurement in NGO child survival projects (2)
The response: Focus on indicators
Definition of standard indicators, measurement tools and sampling techniques (30-cluster sampling, LQAS)
Training/capacity building on evaluation methodology
NGOs have applied this expertise in measurement of indicators as they have become engaged in projects funded by: Global Fund, PEPFAR, PMI etc.

Measurement in NGO Operations Research projects
Measurement itself has not been the major focus of innovation in NGO operations research projects
The measurement tradition has been “indicatorcentric”:
Main form of measurement is cross-sectional household surveys
Main purpose of surveys is to measure indicators
Indicators are pre-defined, and calculated in standard ways
Not much experimentation with measurement, beyond measurement of standard indicators

Strengths of focus on indicators
Results of projects easily communicated to various audiences, including Congress
Can summarize results across projects
Can standardize:
Definition of indicators
Training and procedures for data collection
Training and procedures for data analysis

Weaknesses of focus on indicators
Less effort invested in developing innovative approaches to measurement
Many fundamental questions have not been addressed:
Why do some interventions work well in one site, but not in another?
How can we select the most appropriate intervention for a certain type of community?
Through what mechanism do community-based interventions achieve their impact?

Example: Odek et al. 2009
Odek et al. Effects of Micro-Enterprise Services on HIV Risk Behaviour Among Female Sex Workers in Kenya’s Urban Slums
Promotion of micro-enterprise as alternative to commercial sex work

Figure 2 from Odek et al. 2009

We invest great effort in measurement of indicators on the far right of this model

We invest relatively little effort in measurement of the variables in this box

NGO scorecard on measurement
Doing better
Practices in the home
Coverage, access to services, use of services
Quality of care
Room for improvement
Characteristics of communities
Steps on the way to improved practices, coverage and quality

Steps on the way to improved practices, coverage and quality
Community mobilization
Empowerment
Social capital
Strengthened household to hospital continuum of care

Usual measurement approach to these steps
Assume intervention area has it, and comparison area doesn’t have it
Intervention area is mobilized
Comparison area is not mobilized
This prevents us from answering important questions:
Why do some communities respond better than others to interventions?
What interventions are most appropriate for a community with characteristics X and Y?

Community mobilization, empowerment and social capital are constructs

What is a construct?
Also called “latent variable”
Abstract theoretical variable invented to represent a phenomenon of interest to scientists
Not directly measurable

What is a construct?

Measurement of constructs in psychology
Most developed in psychology
Psychology as a science depends on valid and reliable measurement of constructs
Some psychologists spend their entire careers working on measurement of one construct
Example: Depression

How do you measure a construct?
Different from an indicator
Generally have one single question
Think carefully about the numerator and the denominator
Usually put together a set of questions (items) related to the construct

Ways to measure constructs
Single item on a questionnaire
Frequently misleading e.g. “what is your monthly salary?” to measure wealth
Additive index: One point for each question where respondent answers YES
Complex scales

Example of additive indices: Measuring community mobilization

Community Action Cycle in community mobilization

Measuring community mobilization and the action cycle 1. Exposure to the process, engagement in the Community Action Cycle 2. Changes in individual awareness of problems, self-efficacy, commitment to change 3. Actions taken after being mobilized

1. Exposure to the process, engagement in the Community Action Cycle
Have you attended a meeting with your neighbors organized by X project? Y/N
Did you discuss/list the different health problems here in this village? Y/N
Did you discuss which problems are most important? Y/N
Did you discuss the causes of these problems? Y/N
Did you discuss what can be done about these problems? Y/N

2. Changes in individual awareness of problems, self-efficacy, commitment to change
“ I feel confident that I could provide feedback to the committee on the quality of services in the clinic”
Strongly disagree, disagree, agree, strongly agree
“ I intend to work with the committee to improve the quality of services in the clinic”
Strongly disagree, disagree, agree, strongly agree

3. Actions taken after being mobilized
I have provided feedback to the committee on the quality of services in the clinic Y/N
I have attended committee meetings to work on improving health services here Y/N
I am working with the committee to improve the quality of services in the clinic Y/N

What I am recommending
Keep measuring indicators, but don’t just measure indicators
Branch out into measurement of constructs
Be aware that time and effort is needed to develop good measures of constructs

Steps in developing a scale to measure a construct
Qualitative research to understand what is happening
Draft set of questions to measure the construct
Administer questions to a small sample
Statistical analysis to select the questions (items) that belong in the scale
Revision of the questions to measure the construct
Include questions in questionnaire administered to larger sample

Systems
“ Systems” is a core theme throughout many maternal and child health projects
Manifests itself in different ways:
District health systems
Household-to-hospital continuum of care
Referral networks
Supervision systems

Continuum of care for health services (Kerber et al. Lancet 2007)

Sociometric questionnaire
Questionnaire to elicit the structure of a social network
The questions are generally about relations with other people, rather than questions about individual knowledge, behavior etc.
Valuable tool for examining systems

Sociometric questionnaire applied to complete network: Steps
Define the population, and its boundaries
List all actors in the population/network
All health workers
All CHWs and TBAs
Select one or more questions to be asked regarding each of the other actors in the network
Create sociometric questionnaire with list of actors and questions
Often in table format

Example of sociometric questionnaire ID Name How many times a week to you talk to ____? How many times per month do you seek advice from ____? 001 Person #1 002 Person #2 003 Person #3 004 Person #4

Questions for a sociometric questionnaire
You can’t have very many!
Examples:
Whom do you go to for advice?
Who provides you with support for X?
Who have you talked to about how to treat sick children?
Who has referred a patient to you?
Who have you talked to on the mobile phone?

Sociometric questionnaire: Full matrix P1 P2 P3 P4 P1 5 4 10 P2 5 0 1 P3 2 0 2 P4 10 1 3

Sociometric questionnaire: Symmetrized lower-half matrix P1 P2 P3 P4 P1 P2 5 P3 3 0 P4 10 1 2.5

Analysis of the symmetrized lower-half matrix
Calculation of measures for actors in network: Degree centrality, betweenness, closeness centrality, etc.
Calculation of measures of network as a whole: Density, cohesion etc.
Multi-dimensional scaling
Sociogram

Examples of what could be done in maternal and child health projects
Referrals between facilities
Communication between health workers

Referrals between facilities
Take existing data, then enter into a lower-half matrix
From the lower-half matrix:
Manual analysis: Take map, draw lines between facilities, width indicates number of completed referrals
Multidimensional scaling on matrix
Calculate network density before and after the intervention

Health facility referral matrix Facility 1 Facility 2 Facility 3 Facility 4 Facility 1 Facility 2 20 Facility 3 5 3 Facility 4 0 0 5

Density: Method 1
“ Linkage density”
Number of facility pairs with any links
Total number of facility pairs
=4/6

Density: Method 2
“ Referral density”:
Total number of referrals
Total number of facility pairs
=33/6
Might also calculate referrals in relation to target population, e.g. maternal referrals per women of reproductive age

Comparisons of density
Comparisons can be made for trends in same referral system over time
Comparisons with other systems likely won’t be meaningful, due to differences in:
Number of facilities
Total population size
Population per facility

Multidimensional scaling 1 3 2 4 20 5 3 5

Improving measurement through Operations Research

by jehill3 on May 03, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 6

Statistics

Improving measurement through Operations Research — Presentation Transcript