Health system-evaluation-and-monitoring

Health System Evaluation and Monitoring
Dr.Ahmed-Refat AG Refat WWW.SlideShare.net/AhmedRefat

Evaluation & Monitoring
of

Health Systems
Dr. Ahmed-Refat AG Refat

12/12/2012
1


Evaluation Vs Monitoring
Evaluation is:

“The systematic gathering, analysis and reporting

of data about a system*/program** to assist in
decision making.”
*A “system” a set of inter-connected parts that have to function together to be effective
A health system consists of all organizations, people and actions whose primary intent is to
promote, restore or maintain health (WHO 2007)
** Program is a group of related activities intended to achieve specific outcomes.

2


When Is Program Evaluation Desirable?

Retrospective Evaluation - is often used when
programs have been functioning for some time.

3


Prospective Evaluation.
is conducted when a new program is being introduced
or when a major changes introduced
A prospective evaluation identifies ways to increase
the impact of a program on clients; it examines and
describes a program’s attributes; and, it identifies how
to improve delivery mechanisms to be more efficient
and less costly.

4


Formative Vs Summative Evaluation
The broadest and most common classification of
evaluation identifies two kinds of evaluation:

1. formative evaluation. =
evaluation of components of a program other than
their outcomes.

5


A formative evaluation may evaluate the degree of need
for the program, or the activities used by the program to
achieve its desirable outcomes, but without evaluating
the degree of outcome.
Formative evaluation is analogous to Total Quality Management
(TQM) and Continuous Quality Improvement (CQI) since all these
approaches are a commitment to constantly improve operations,
processes and activities to meet client requirements efficiently,
consistently and cost-effectively.
Formative evaluation then evaluates whether the program will use, or
does use, the right mix and volumes of human resources, materials and
activities to carry out the program.

6


2. summative evaluation=
evaluation of the degree to which a program has
achieved its desired outcomes, and the degree to which
any other outcomes (positive or negative) have resulted
from the program.

Summative evaluations examine the changes that should or
did occur as a result of the program(outcomes).

7


Evaluation
Prospevtive – Retrospective
• prospective, meaning it determines what ought to
happen (and why); or
• retrospective, meaning it determines what actually
happened (and why).

8


Evaluation
Internal And External
1. Internal evaluation (sometimes called self
evaluation), in which people within a program
sponsor, conduct and control the evaluation. Internal
evaluation can more fully engage the insights of
program personnel but runs the risk of overly
subjective evaluation results.

9


Evaluation
Internal And External
2. External evaluation, in which someone from
Outside the program. External evaluation has the
advantage of objectivity if done
Well.

11


Evaluation
Descriptive Analytical
Descriptive elements of the evaluation are meant to
answer four of the questions that are the hallmark of
good journalism, just as they are the hallmark of good
descriptive evaluation: WHO WHAT WHEN WHERE
Description alone does not answer WHY & HOW.
11


The why and how questions that are answered by
analytical evaluation.

12


13


14


15


Monitoring
Monitoring is the constant or
recurring collection and examination
of selected information on program
activity over the life of the program.

16


Monitoring Information can be used for two purposes:
1. to alert the program to changes in program operation
that might be signals of possible program failure; and

2. to provide a body of information that will be used
when each kind of evaluation is carried out.
Monitoring can emerge from prospective evaluations,
and can provide raw material for retrospective
evaluations.

17


The Difference between Outputs and Outcomes
An output is a measurable result of activities within
a program, reflecting the immediate result of the
activities but not directly reflecting the effect on
clients of the program.

An outcome is a measurable positive or negative
change to clients of a program or to other stakeholders.

18


19


21


22


Performance Indicators
in
Evaluation and Monitoring
of Health System

23


Indicators of access
Definition of Access
“access” refers to the presence or absence of physical
or economic barriers that people might face in using
health services.
Physical barriers are usually interpreted to mean

24


those related to the general supply and availability of
health services and distance from health facilities.
Economic barriers are usually interpreted to mean
those related to the cost of seeking and obtaining
health care, in relation to a patient’s or household’s
income. Many of the features of “access” are also
included in definitions of the structural aspects of quality
of care assessments.

25


Physical Access
 ! Percent of (rural, poor) population residing within X
kilometers of a health facility
kilometers of a health facility
 providing a package of basic health services
kilometers of a health facility staffed by a doctor
kilometers of a pharmacy
 Percent of the population residing within X kilometers of
a hospital

26


 ! Percent of population residing within X kilometers of a
hospital providing 24-hour emergency (obstetric) care
 ! Percent of population served by 24-hour ambulance
services
 ! Percent of health facilities equipped with telephones or
radios
 ! Population per doctor
 ! Population per nurse
 ! Population per hospital bed




27


 Economic Access
 ! Average total private cost of professionally treated
outpatient illness episode as a percent of monthly per
capita household income for consumers in the lowest per
capita income quintile
 ! Average total private cost of medicines for a typical
outpatient illness episode as a percent of monthly per
capita household income for consumers in the lowest per
capita quintile
 ! Average total private cost of an average hospital stay as
a percent of annual per capita household income for
consumers in the lowest per capita income quintile

28


29


Indicators of equity
“The concept of equity as it relates to
health systems may refer variously to
differences in health status, utilisation,
or access among different income,
socio-economic, demographic, ethnic,
and/or gender groups”

31


Equity can be 1. horizontal and 2. vertical equity.

Horizontal equity refers to the equal treatment of
equals regardless of gender, marital status and so on.

Vertical equity is based on the principle that individuals
who are unequal in society should be treated differently

31


Definition of Equity
The concept of equity as it relates to health systems may
refer variously to differences in health status, utilization, or
access among different income, socio-economic,
demographic, ethnic, and/or gender groups.
Most health sector reform efforts directed toward system
performance in low- and middle-income countries
concentrate on definitions of equity related to access and
utilization.

32


If equity is defined mainly in terms of ensuring access in
general—or universal access to a package of basic or cost-
effective health services in particular—
Access can be modified and calculated across the
population, or, using coverage rates that compare access
for different income, to socio-economic, demographic,
ethnic, gender, geographic, or other groups considered
especially undeserved.

33


34


Indicators of Quality
Definition of Quality
“Quality of care” is clearly a multi-dimensional concept
and one on which there is as yet no consensus
definition (even in the United States).

35


Quality of care
“that kind of care which is expected to maximize
an inclusive measure of patient welfare, after one
has taken account of the balance of expected
gains and losses that attend the process of care
in all its parts.

36


Quality of care is a multi-dimensional concept on which there is
little current consensus.
Three aspects of quality of care are commonly distinguished in
the literature, between ‘structural’, ‘process’ and ‘outcome’
dimensions .
‘Structural’ quality refers to whether appropriate resources
are in place to provide health care of a minimum standard
(personnel trained for their tasks, well maintained equipment and
buildings, a regular drug supply).

37


‘Process’ quality generally refers to activities occurring
during the interaction between the health system and the client
(i.e. whether good quality
care is actually delivered). Also, both the presence of
accreditation and the quality of professional training and
continuing education may affect quality.
‘Outcome’ quality, in addition to health status, can include
patient satisfaction and perceived quality.
Outcome flows from process and refers to “changes in a patient’s
current and future health status that can be attributed to the
antecedent health care.”

38


Structural Indicators
! Existence of national standards for professional qualifications of
health manpower, including
enforcement mechanisms
! Proportion of health workers possessing basic professional
qualifications, including skills for
specific primary health care services
! Existence of national facility standards, including enforcement
mechanisms
! Proportion of health facilities meeting basic structural standards,
based on the services to be
provided
! Presence of clear national standards for high priority health
services

39


! Proportion of facilities in which current diagnostic and treatment
guidelines are available in
writing
! Presence of a national quality assurance program, including
trained staff and established
procedures for quality design, monitoring and improvement. Sub-
indicators of the presence of
effective quality assurance include:
> Routine review and updating of technical standards
> Effective methods for communicating standards to the field level
> Routine application of methods for comparing performance with
standards
> Routine application of data-based problem-solving
methodologies

41


> Routine application of methods for incorporating community
input into system design and
management
! Proportion of health facilities that did not experience drug
stockouts during the preceding three
months
! Proportion of cases in which all recommended drugs are
available

41


Process Indicators
! Proportion of clinics in which services are fully integrated,
per national standards
! Proportion of health workers receiving appropriately timed
and effectively conducted supervision,
per national policy
! Proportion of patient contacts in which treatment received
is consistent with national diagnostic
and treatment protocols, including guidelines for client-
provider interaction
! Proportion of referrals made and consummated in
accordance with national guidelines and
standards
42


! Proportion of clients who know and understand essential
actions needed to complete treatments
and avoid future preventable conditions
! Proportion of clients who follow through to completion of
recommended treatments (drop out
rate)
! Client satisfaction
Indicators for Health System Performance 23

43


Indicators of efficiency
The important distinction between effectiveness and efficiency is
that the latter takes into account costs.

Efficiency is defined by health economists in several different
ways, and can be applied to health services as well as health
outcomes. Efficiency essentially concerns how and which
health services are produced, and has three dimensions:
technical, economic8 and allocative .

44


45


Indicators of Efficiency
! Outpatient visits per hour of physician labor (or per
physician)
! Outpatient visits per hour of nurse labor (or per nurse)
! Ratio of outpatient visits to personnel costs (i.e., unit
personnel costs)
! Cost per outpatient visit (or Operating cost per outpatient
visit)
! Cost per hospital bed-day (or per hospital admission or per
hospital discharge)
! Percent of outpatient visits obtained from the private sector
! Private hospital beds as percent of total

46


! Personnel expenditure as a percent of total recurrent health
expenditure
! Expenditure on drugs and supplies as a percent of total
recurrent health expenditure
! Number of nurses per doctor
! Number of nurses per hospital bed
! Number of doctors per hospital bed
! Ratio of average salary of government health worker (e.g.,
doctor, nurse, technician) with a given
level of experience to the income of a comparable private
sector health worker
! Salaries of government health workers are paid on time
(yes/no)

47


! Adequate performance incentives exist for government
health personnel (yes/no)
! Generic drug expenditure as a percent of total drug
expenditure
! Government health system uses basic drug list for
procurement (yes/no)
! Percent of government recurrent health budget spent on
public health services
! Primary health care expenditure as a percent of recurrent
costs
! Percent of total government drug expenditures allocated to
primary care facilities
! Fees are charged in all facilities (yes/no)

48


! Fee levels promote efficiency (yes/no)
! Referral system functions effectively (yes/no)
! Average length of hospital inpatient stay
! Hospital bed occupancy rate
! Percent of insured enrolled in plans which use copayments
and deductibles, managed care plans,
or plans subject to global budgeting

49


Sustainability
the capacity of the system to continue
its normal activities successfully in the
future
Most definitions of sustainability also include the
additional requirement that the system be able to
expand its activities as needed to keep up with
increases in demand due to economic and
population growth.

51


Sustainability includes both financial and
institutional dimensions.
Financial sustainability refers to the capacity of the
health system to replace withdrawn donor funds with
funds from other, usually domestic, sources.
Institutional sustainability refers to the capacity of the
system, if suitably financed, to assemble and manage
the necessary non-financial resources to successfully
carry on its normal activities.

51


Indicators of Sustainability

52


Financial Sustainability
! Percent of total health system financed by tax revenue
! Percent of government health system financed by tax
revenue
! Government health expenditure as percent of total
government budget
! Government health expenditure as percent of GDP
! Percent of total health spending financed by donors
! Percent of government health spending financed by donors

53


! Percent of government recurrent health spending financed
by donors
! Percent of government health expenditure directed to
primary care
! Percent of government health expenditure directed to
preventive care
! Percent of government health expenditure directed to MCH
services

54


Institutional Sustainability
! Foreign doctors as a percent of all doctors
! Number of months of foreign technical assistance funded
by donors
! Donor expenditures on technical assistance as a percent of
all donor health expenditures

55


Key characteristics
of
good service delivery
56


Key characteristics of good service delivery

.Comprehensiveness: A comprehensive
range of health services is provided,
appropriate to the needs of the target
population, including preventative, curative,
palliative and rehabilitative services and
health promotion activities.

57



.Accessibility: Services are directly and
permanently accessible with no undue barriers of cost,
language, culture, or geography. Health services are
close to the people, with a routine point of entry to the
service network at primary care level (not at the
specialist or hospital level). Services may be provided in
the home, the community, the workplace, or health
facilities as appropriate .

58



.Coverage:
Service delivery is designed so that all
people in a defined target population are
covered, i.e. the sick and the healthy, all
income groups and all social groups .

59



.Continuity

Service delivery is organized to provide an
individual with continuity of care across the
network of services, health conditions,
levels of care, and over the life-cycle.

61



.Quality:

Health services are of high quality, i.e. they
are effective, safe, centered on the
patient’s needs and given in a timely
fashion.

61


.Person-centredness:
Services are organized around the person, not the
disease or the financing. Users perceive health services
to be responsive and acceptable to them. There is
participation from the target population in service
delivery design and assessment. People are partners in
their own health care.

62


Coordination:
Local area health service networks are actively
coordinated, across types of provider, types of care,
levels of service delivery, and for both routine and
emergency preparedness. The patient’s primary care
provider facilitates the route through the needed
services, and works in collaboration with other levels
and types of provider. Coordination also takes place
with other sectors (e.g. social services) and partners
(e.g. community organizations).
63


Criteria for
Evaluating Individual
Indicators
64


The indicators should be selected according to the
following criteria:
 ! Validity. Does it measure what it is supposed to
measure
 Precision. Is the indicator clearly and unambiguously
defined?
 ! Reliability. Will two measurements of the indicator for
the same health system produce
 the same result?
 ! Timeliness. Is the indicator available on an annual
basis and without undue delay?
 ! Comparability. Can the indicator be used to compare
health systems meaningfully across countries?

65


 ! Additivity. Can the indicator be readily and
meaningfully applied to sub-regions and to population
sub-groups (e.g., gender, income)?
 ! Interpretability. Does a higher (or lower) value of the
indicator consistently imply that a health system performs
better?
 ! Cost. Is the cost manageable? There is often an
unavoidable tradeoff between cost, on the one hand, and
validity, reliability, and timeliness, on the other hand.

66


Cited References
Knowles, James C.,Charlotte Leighton, and Wayne Stinson, 1997. Measuring Results of Health
Sector Reform for System Performance: A Handbook of Indicators. Special Initiatives Report
No. 1. Bethesda, MD: Partnerships for Health Reform, Abt Associates Inc

Monitoring and evaluation of health systems strengthening An operational framework WHO, Geneva. October
2010
Margaret El. Kruk and Lynn P. Freedman.Assessing health system performance in developing countries: A
review of the literature. Health Policy 85 (2008) 263–276
http://www.ppmrn.net/storage/ppmrn/Margaret%20Elizabeth%20Kruk%20Lynn%20P.%20Freedman.pdf
/http://ocw.jhsph .edu/index.cfm/go/viewCourse/course/HSRE/coursePage/lectureNotes

/http://www.healthsystems2020.org/content/resource/detail/528

http://www.euro.who.int/en/who -we-are/partners/observatory/studies/performance -measurement-for -
health-system-improvement-experiences,-challenges-and-prospects

http://www.euro.who.int/en/who -we-are/partners/observatory/studies

67

Health system-evaluation-and-monitoring

More Related Content

What's hot

Similar to Health system-evaluation-and-monitoring

More from Ahmed-Refat Refat

Health system-evaluation-and-monitoring