Performance of Routine Information System Management Framework (PRISM) led by Natasha Kanagat
The PRISM framework consists of four tools to assess Routine Health Information System (RHIS) performance, identify technical, behavioral and organizational factors that affect RHIS, aid in designing priority interventions to improve performance and improve quality and use of routine health data.
Recording: http://universityofnc.adobeconnect.com/p1edhgz9zs7/
PRISM Tool: https://www.cpc.unc.edu/measure/publications/ms-11-46-d
1. Data Demand & Use:
PRISM Tool
Webinar Series - # 7
Tuesday, February 7, 2012
Presenters: Natasha Kanagat & Molly Cannon
2. Agenda
• Welcome - Webinar tips
• Brief overview of Data Demand and Use
• Presentation of Tool – PRISM
• Questions and Answers
• Wrap up
3. Troubleshooting
If you lose connectivity, re-enter the meeting
room by clicking on the link provided
If you are located in the US, you can rejoin by
clicking on the link or use the conference call
number provided
For other troubleshooting questions right click on
the host and choose “private chat”
Send an email to leah.gordon@unc.edu
4. Tips for Participating in the
Discussion
To comment, raise your hand by clicking on the
icon with person raising hand. You can then:
Speak into your microphone. Be sure its enabled.
Click on the microphone icon at the top of the
screen
Type into the Q&A function. You man enter
comments into the Q&A pod at any time
A recording of the webinar will be made available
at www.measureevaluation.org/ddu
6. “… without information, things are done
arbitrarily and one becomes unsure of
whether a policy or program will fail or
succeed. If we allow our policies to be guided
by empirical facts and data, there will be a
noticeable change in the impact of what we
do.”
National-level Policymaker, Nigeria
7. Definitions
Data use – Using data in the decision making
process
create or revise a program or strategic plan
develop or revise a policy
advocate for a policy or program
allocate resources
monitor a program
review must be linked to a specific decision making process
Data Demand - decision makers specify what kind of
information they want & seek it out
9. Improving Data-informed Decision
Making
Data Users & Data Producers
Tool Application Capacity Building Organizational Support
Monitoring & Evaluation System Improvements
11. Rationale for strong Health
Information Systems
Improving health
Essential foundation of public health
action and health systems strengthening
“Are we doing things right?”
12. HIS Data Sources
Reference: HMN Framework and Standards for Country Health
Information Systems, Second edition
13. Routine Health Information
System
A system that provides information at
regular intervals of a year or less through
mechanisms designed to meet
predictable information needs.
Includes paper-based or electronic health
records, and facility- and district-level
management information systems.
15. Challenges of implementing good
HIS
Parallel, uncoordinated systems
Vertical programs
Reporting requirements
Resource constraints
Poor data quality
Lack of data use for decision making
16. The data collection forms are too
complicated.” …
“What is the use of collecting data
when nobody uses it?” …
“Upper management is not
committed to RHIS activities.” …
23. Examples of technical interventions
Defining a set of essential indicators
Standardize a data generation architecture
based on best practices
Development of computerized data
analysis/presentation application: DSS,
Morocco
24. Examples of organizational and behavioral
interventions (let us go a step further...)
Create incentives for use of information
(Pakistan, Uganda)
Promote HMIS self-assessment (Uganda)
Introduce Performance Improvement Tools
focused on problem solving approach
(Thailand)
26. Mexico: Understanding
organizational and behavioral
factors affecting DQ and Data Use
27. How Were OBAT Findings Used?
• A website was created, all 33 state health departments
accessed the OBAT questionnaire. The information
processed by OBAT was relayed to the national level;
• National authorities able to prioritize interventions and
produce a plan for the incoming government to improve
information use and decision making at all levels;
• Measuring HMIS performance, based on funding
availability, was recognized as important.
28. Pakistan: Tracking quantitative HIS
performance measurements over time
Improvements in Data Accuracy and Use of
Information Before and After Pilot test in Pakistan,
2006
80
70
60
Percent
50 Accuracy
40
30 Use of info
20
10
0
Interventions
Before After
29. How were pilot test findings used?
A separate training manual developed to
assess data quality and use of information
Facilities and the districts implemented
monthly performance reviews using DHIS
information and recorded decisions on a new
DHIS register
A National Action Plan for DHIS, approved by
the Government of Pakistan in February 2007
31. The application of the PRISM
framework and its tools in various
countries has led to data quality
improvement and increased use of
information for decision making
Adaptable & Flexible
Can be used by a variety of
stakeholders
32. PRISM Resources
The PRISM framework and tools are available as free
downloads from the MEASURE Evaluation website:
https://www.cpc.unc.edu/measure/tools/monitoring-
evaluation-systems/prism
34. MEASURE Evaluation DDU Resources
www.measureevaluation.org/ddu
Data Demand and Use Tool Kit
Data Demand and Use Training Resources
Next webinar will be on February 14, 2012 at
9:00 am EST…Framework for Linking Data with
Action
To sign up for Data Use Net, send an email
to listserv@unc.edu. Leave the subject field
blank and in the body of the message type
„subscribe DataUseNet‟
36. MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for Health,
and Tulane University. Views expressed in this presentation do not
necessarily reflect the views of USAID or the U.S. Government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.
Editor's Notes
Global health context- The need for quality health care services is intimately known by all of us. Global HIV epidemic. There were an estimated 33 million people living with HIV at the close of 2008, the majority of whom either need or will soon need treatment.Approximately, one-third of the world‘s population is infected with TB..Each year, malaria causes nearly 1 million deaths, mostly among children under 5 years of age the health system is burdened by millions of clinical cases as well.In much of sub-Saharan Africa, the transition from high to low fertility has stalled. Also, young people—those below the age of 20—account for the largest proportion of the population. In the next few years, we will see larger numbers of people needing health services as this cohort ages. In the face of this demand we are experiencing Inadequate numbers and poor distribution of qualified health workers and an inadequate human resources system to support them.It is within this context of a high disease burden, a growing population, and insufficient health services, that it becomes extremely important for governments to make the best use of their limited resources. The need to develop strategies, policies, and interventions that are based on quality data and information is urgent.
The importance of data-informed decision making is expressed on this slide by a national-level policymaker in Nigeria who participated in a data use assessment conducted by MEASURE Evaluation. The assessment involved interviews with a range of professionals at the national, regional, and facility levels. The policymaker interviewed, stated… (READ SLIDE)“… without information, things are done arbitrarily and one becomes unsure of whether a policy or program will fail or succeed. If we allow our policies to be guided by empirical facts and data, there will be a noticeable change in the impact of what we do.” This statement nicely summarizes why we are here today to discuss the importance of improving data-informed decision making.
The framework presented here illustrates the entire cycle of evidence-based decision making, starting with basic M&E systems and the collection of information - through to the use of data and continued demand for data to repeat the cycle. This approach illustrates the ideal. You will note that in addition to the collection of quality data there are also the considerations of ensuring that the information is available and in a format that is easily understood by relevant stakeholders. This information is then interpreted and used to improve policies and programsThe cycle supports the assumption that the more positive experiences a decision maker has in using information to support a decision, the stronger the commitment will be to improving data collection systems and continuing to use the information they generate. This leads to repeat data use.You will note that this cycle is supported by coordination and collaboration. This coordination is among data users and data producers as well as between management systems and other organizational supports that facilitate and support data informed decision making. Lastly, the cycle is supported by CB to ensure that individuals are equipped with the skills to collect and use data.It is important to note that there are many opportunities for this process to break down. In the best designed M&E systems you often find lacklustre data use. Data is not being used as often as it should be.
How do we improve DDU?Firstly, build upon a commitment and ongoing efforts to improve M&E and information systems – this is the foundation of all data use improvement interventions.Identify and engaging data users and data producers is also critical. By data users we are referring to those whose primary function is to manage data systems and by data users we are referring to those whose primary function is to use data to monitor and improve health service delivery. These two groups don’t always work closely together. For data use to function as we saw on the previous slide, regular collaboration between these two groups is critical. It is also important to apply tools, build capacity and strengthen organizational systems to support data informed decision making. In this webinar series we will be discussing tool application (the pink box) and the types of tools MEASURE Evaluation has developed to facilitate DDU. The last webinar session of this series will address capacity building and at a later date we will offer a webinar on strengthening organizational supports to improve data demand and sue. The combination of tool application, capacity building and strengthening organizations are all complimentary and necessary elements of any strategy to improve the use of data in decision making.
Thanks Molly, Hi everyone, Good morning, afternoon or evening depending on where you are. Thank you for taking the time to attend this webinar. We will focus on PRISM applications for DDU, but I will quickly walk us through an overview of HIS, a couple of key definitions, introduce the PRISM tools and then move onto country specific PRISM assessments specific to DDU.
Reliable and timely health information is an essential foundation of public health action and health systems strengthening, at the national and international levels. Health Information Systems helps us understand what is going on in the health system. Identify and track areas of weaknesses and strength... ..
HIS data are usually generated either directly from populations or from the operations of health and other institutions. Population-based sources generate data on all individuals within defined populations and can include total population counts (such as the census and civil registration) and data on representative populations or subpopulations (such as household and other population surveys). What these data sources have in common is that they relate to the whole population, not only to groups using institutional services. Such data sources can either be continuous and generated from administrative records (such as civil registers) or periodic (such as cross-sectional household surveys).Institution-based sources generate data as a result of administrative and operational activities. These activities are not confined to the health sector and include police records (such as reports of accidents or violent deaths), occupational reports (such as work related injuries), and food and agricultural records (such as levels of food production and distribution). Within the health sector, the wide variety of health service data includes morbidity and mortality data among people using services; services delivered; drugs and commodities provided; information on the availability and quality of services; case reporting; and resource, human, financial and logistics information.Most data on the provision of clinical services or health status at the time of clinical encounters are generated “routinely” during the recording and reporting of services delivered.
RHIS attempt to produce timely and quality information about what is happening in health sector organizations. Ideally, RHIS information is used to guide day-to-day operations, track performance, learn from past results, and improve accountability.
RHIS Performance is characterized by: Data quality which is characterized by the relevance, accuracy, timeliness, and completeness of data. Information use is defined as decision makers explicitly considering information in policymaking, planning, management, and service delivery.
Health information systems have evolved in a haphazard and fragmented way as a result of administrative, economic, legal or donor pressures. For example, the responsibility for health data is often divided among different ministries or institutions, and coordination among these entities can be difficult due to financial and administrative constraints.Health information systems are further fragmented by disease-focused demands that often relate to donor requirements and international initiatives directed towards specific areas like malaria, HIV/AIDS or tuberculosis. Intense pressure for the rapid availability of data often contributes to the establishment of disease-specific information systems driven by performance-based funding.Within the health sector itself, health workers are overburdened by excessive reporting requirements from multiple and poorly coordinated subsystems that cannot deliver timely, accurate and complete data. Although a vast amount of data may be collected, only a small proportion is synthesized, analysed and used. Data UseData are often collected and presented in formats that are not user friendly, and without the synthesis or analysis required for proper day-to-day management or longer-term planning. There is little point in engaging in the time- and resource-consuming process of data collection if there is no commitment to analysing the data, disseminating the resulting information and using it to improve health system functioning. Poor data quality Decision-makers at all levels of the health system need information that is relevant, reliable and timely. But, even when high-quality information is available this does not guarantee its appropriate use in the decision making process. The ultimate objective of a routine health information system (RHIS) is to produce information for taking action in the health sector. “Are we doing things right?” “Are we doing the right things?” If things are being done correctly, the data should demonstrate that all activities were carried out as planned. Positive results should follow. Therefore there is a need to consistently assess RHIS system performance with the goal of understanding the factors that hinder RHIS performance .
Traditional assessments only partly address how to improve RHIS, because they look narrowly at technical issues like data collection forms,data collection methods or information technology.PRISM, which is the Performance of Routine Information System Management, acknowledges the broader context in which RHIS operate. PRISM places RHIS performance within the context of technical, behavioral and organizational determinants and the success of RHIS depends on the performance of these three inter-related areasBehavioral determinants include the knowledge, skills, attitudes, values, and motivation of the people who collect and use data;Technical determinants include data collection forms, processes, systems, and methods; andOrganizational determinants include information culture, structure, resources, and roles and responsibilities of key contributors at each level of the health system.Objective of the framework is to identify these determining factors with the aim of improving the performance of the RHIS and thus, increase data quality (precision, veracity, comprehensiveness, and timeliness) and promote the use of information in decision making.
There are four PRISM tools that together measure RHIS performance, processes and determinants:the RHIS performance diagnostic tool; Overall level of RHIS Performance:i.e. the level of data quality and use of information. Captures the technical determinants of RHIS performance, such as level of complexity of data collection forms and user-friendliness of information technology.(2) the RHIS overview tool; This tool examines technical determinants, like structure and design of existing information systems in the health sector,information flows, and interaction between different information systems. It allows users to understand the availability and status of RHIS resources necessary for RHIS implementation at the facility and district levels. (3) the RHIS management assessment tool; and This tool is designed to take rapid stock of the RHIS management practices and aids in developing recommendations for better management such as organizing, planning, performance improvement, training, supervision and finances.(4)the organizational and behavioural assessment tool (OBAT)This tool identifies behavioral and organizational factors affecting RHIS performance. Behavioral determinants include level of data demand, motivation, confidence, task competence, and problem-solving skills. Organizational factors include level of promotion of a culture of information, merit criteria, and use of RHIS information for performance appraisalFrom here you can see that the --RHIS Performance Diagnostic Tool. The primary component in the toolset, this determines the overall level of RHIS performance, looking separately at quality of data and use of information, to identify weak areas. This diagnostic tool identifies strengths and weaknesses; the other three tools identify the underlying technical, organizational, and behavioral reasons for those strengths and weaknesses.
Diagnostic Tool has 4 assessment forms: At the district and facility level each, there is a Quality of Data Assessment Form and a Use of Information Assessment Form This slide provides a snapshot of the type of questions on data use in the diagnostic tool in the Use of Info: District Assessment Form.
This slide provides a snapshot of the type of questions asked by the OBAT on the culture of information.
There are 3 types of interventions: technical, behavioral and organizational interventions.
In Mexico, the Ministry of Health was well-informed about the structure and management of the Mexican HMIS, but sought a deeper understanding of the organizational and behavioral factors affecting data quality and information use.The OBAT provides insight into: a) Promotion of a “culture of information”; b) HMIS tasks self-efficacy (confidence level); c) HMIS task competence; d) Knowledge of HMIS purpose and methods of checking data quality. The PRISM framework assumes that if organizations promote a culture of information, they will also improve their competence in conducting HMIS tasks, and thus improving their self confidence to carry out HMIS tasks. The promotion of a culture of information will be associated with knowing the purpose and methods for checking HMIS data quality. Using OBAT will help identify the HMIS’s organizational and behavioral factors.Findings: The survey revealed gaps between respondents’ perception of the promotion of a culture of information and their actual competence and knowledge of HMIS tasks. This indicates opportunities to bridge gaps:On average, 70% of respondents believed strongly that the MOH promotes checking data quality but only 57% of the respondents could describe at least two ways of checking data quality;On average, 71% of respondents believed strongly that the MOH promotes problem solving skills but only 23% of the respondents demonstrated skills in defining and solving problems;On average, 72% of respondents believed strongly that the MOH promotes use of HMIS information but only 52% of the respondents showed how to use HMIS information.
To strengthen HMIS performance with an emphasis on using information in the following ways:
On the basis of the findings, the following measures focusing on DDU were implemented: The development of the DHIS pilot test package included revised data collection forms for primary care facilities and new forms for hospitals as well as tools for continuous improvement of DHIS performance;An HMIS training manual was developed for the revised data collection registers and forms;A separate training manual was developed on assessing data quality and use of information for continuous improvement of health system performance;The HMIS team developed and pilot-tested a district software application for data entry, analysis, and report generation;During the pilot test, the facilities and the districts conducted monthly performance reviews using DHIS information and recorded decisions on a new DHIS register;A National Action Plan for scaling-up the DHIS, approved by the government of Pakistan in February 2007, was disseminated in a meeting attended by federal and provincial officials and international donors.
1. Costa Rica OBAT V. 2.0 20092. Ecuador PRISM V. 2.0 20103. Honduras OBAT V. 2.0 20064. Mexico OBAT V. 1.0 2005-20065. Paraguay PRISM V. 2.0 2006-20076. Peru PRISM V. 2.0 2008-20097. DominicanRepublic PRISM V. 2.0 2008-2009ChinaUgandaEthiopiaSouthAfrica
PRISM Tools can be adapted and applied at international, national or sub-national levels. The tools can be adapted to reflect variances in RHIS design, decision-making processes and stakeholders. The tools described in this document have been designed for a routine facility-based health information system. However, the tools can be adapted for other data sources, such as vital events registration systems, or non-routine health information systems, such as surveys. Flexible. The PRISM Tools were designed with the assumption that the organization has established a minimum set of RHIS processes, practices and infrastructure. Since they address elements that would be common to most any RHIS, the tools are broadly applicable to diverse organizations. The tools can be used to assess both categorical and integrated information systems, in public- and private-sector RHIS frameworks.Adaptable. Users can modify the tools to match the socio-demographic characteristics of respondents in a given organization. Similarly, the content of a tool can be adapted to meet the specifics of the given situation. The collected data can be analyzed manually or entered in any data analysis program such as Excel, EpiInfo, etc.
We can move to the Q & A segment. Our audience includes people with extensive country experience and this is a great opportunity to put forth your questions for insights into PRISM applications for data demand and use.