Assessment of Constraints to Data Use is a rapid assessment tool designed to identify barriers and constraints that inhibit effective practices in data use.
When welcoming folks, thank them for volunteering for the webinar. Introduce presenters and participants
Global health context- The need for quality health care services is intimately known by all of us. Global HIV epidemic. There were an estimated 33 million people living with HIV at the close of 2008, the majority of whom either need or will soon need treatment. Approximately, one-third of the world‘s population is infected with TB.. Each year, malaria causes nearly 1 million deaths, mostly among children under 5 years of age the health system is burdened by millions of clinical cases as well. In much of sub-Saharan Africa, the transition from high to low fertility has stalled. Also, young people—those below the age of 20—account for the largest proportion of the population. In the next few years, we will see larger numbers of people needing health services as this cohort ages. In the face of this demand we are experiencing Inadequate numbers and poor distribution of qualified health workers and an inadequate human resources system to support them. It is within this context of a high disease burden, a growing population, and insufficient health services, that it becomes extremely important for governments to make the best use of their limited resources. The need to develop strategies, policies, and interventions that are based on quality data and information is urgent.
The importance of data-informed decision making is expressed on this slide by a national-level policymaker in Nigeria who participated in a data use assessment conducted by MEASURE Evaluation. The assessment involved interviews with a range of professionals at the national, regional, and facility levels. The policymaker interviewed, stated… (READ SLIDE) “… without information, things are done arbitrarily and one becomes unsure of whether a policy or program will fail or succeed. If we allow our policies to be guided by empirical facts and data, there will be a noticeable change in the impact of what we do.” This statement nicely summarizes why we are here today to discuss the importance of improving data-informed decision making.
The framework presented here illustrates the entire cycle of evidence-based decision making, starting with basic M&E systems and the collection of information - through to the use of data and continued demand for data to repeat the cycle. This approach illustrates the ideal. You will note that in addition to the collection of quality data there are also the considerations of ensuring that the information is available and in a format that is easily understood by relevant stakeholders. This information is then interpreted and used to improve policies and programs The cycle supports the assumption that the more positive experiences a decision maker has in using information to support a decision, the stronger the commitment will be to improving data collection systems and continuing to use the information they generate. This leads to repeat data use. You will note that this cycle is supported by coordination and collaboration. This coordination is among data users and data producers as well as between management systems and other organizational supports that facilitate and support data informed decision making. Lastly, the cycle is supported by CB to ensure that individuals are equipped with the skills to collect and use data. It is important to note that there are many opportunities for this process to break down. In the best designed M&E systems you often find lacklustre data use. Data is not being used as often as it should be.
How do we improve DDU? Firstly, build upon a commitment and ongoing efforts to improve M&E and information systems – this is the foundation of all data use improvement interventions. Identify and engaging data users and data producers is also critical. By data users we are referring to those whose primary function is to manage data systems and by data users we are referring to those whose primary function is to use data to monitor and improve health service delivery. These two groups don’t always work closely together. For data use to function as we saw on the previous slide, regular collaboration between these two groups is critical. It is also important to apply tools, build capacity and strengthen organizational systems to support data informed decision making. In this webinar series we will be discussing tool application (the pink box) and the types of tools MEASURE Evaluation has developed to facilitate DDU. The last webinar session of this series will address capacity building and at a later date we will offer a webinar on strengthening organizational supports to improve data demand and sue. The combination of tool application, capacity building and strengthening organizations are all complimentary and necessary elements of any strategy to improve the use of data in decision making.
There are many factors that affect data use. Let’s consider why this happens. Here you see the three main determinants of a Routine Health Information System, including data use. We define ‘determinant’ as a determining or causal element or factor directly linked to data use. The three determinants highlighted are—Organizational, Technical, and Behavioral. Organizational determinants—these determinants relate to the organizational context that supports data collection, availability, and use, such as the identified procedures and the roles and responsibilities of those that collect, analyze, disseminate, and use data. Technical determinants—refer to the technical aspects of data collection processes and tools, such as the data collection processes, methods, and forms. Last, Behavioral determinants refer to the behavior of individuals who produce and use data. This would cover their skills, attitudes, values, and motivation.
In addition to organizational, technical, and behavioral determinants, we also need to consider that the political, cultural, and social contexts are important determinants of data demand and information use, because decision making, sharing of information, and data collection and reporting all occur within these contexts. It is important to address all of these areas when developing a strategy to improve data use. A full assessment of the routine health information system can be conducted to identify strengths and weaknesses in all of these areas, using the PRISM assessment developed by MEASURE Evaluation. However, we are going to present a rapid assessment that can be conducted. A future webinar will cover how to use PRISM.
The Assessment of Data Use Constraints is a tool developed by MEASURE Evaluation for rapid assessments; it assists users in improving understanding of the demand for data and the constraints on data use. These rapid assessment tools are based on the PRISM framework, there are specific tools that comprehensively assess a RHIS but this is designed to be a rapid, snapshot way of assessing the three potential groups of constraints. Specifically, it: Identifies existing barriers and constraints on data use. Identifies existing best practices in data use so these practices can be applied elsewhere. Formal planning should follow-up the mapping process. The information generated by this tool should be far more than a list of barriers and constraints. It should be forward-looking and prescriptive, showing ways that these obstacles and deficiencies can be overcome. These are areas that can usually be addressed with targeted interventions. The assessment is conducted by interviewing key informants at various levels of the health system. The assessment also can be used to examine processes within a facility or single organization and incorporated into health information and organizational capacity-building assessments at the national and subnational levels. The interview guide is organized by the three determinants of data use (as discussed in the previous slides).
As previously mentioned, the assessment of data use constraints is a series of questions asked of key informants. Depending on your needs, you can ask the questions to different types of key informants.
On this slide, you see an example of what the assessment tool looks like. As you can see, these questions are intended to identify technical constraints. The endemic shortage of computers is an obvious technical constraint, but there are other common technical issues that erode data quality. For instance, contributors could be defining health indicators differently, or using different sources for the same data element or indicator, or using different algorithms to report it.
Many information systems suffer from shortages of skilled people to manage, interpret, and use the data; and motivation and incentive to generate high quality data. For example, one health information unit, despite having an M&E system for HIV/AIDS, was still not getting the data it had requested from its service sites. Decision-maker attitudes Staff motivation Lack of “data culture”
Organizational processes might not support the use of data. For instance, officials might be reluctant to use data that has not been officially sanctioned. Perhaps the release of certain sensitive information—such as figures that reveal a measles outbreak—is tightly controlled. This information can be shared only by official protocol. More often, there are simply no channels or systematic processes to share data with people who could use it.
This is an excellent management tool – document so you are able to monitor progress, keep people on same page with common direction to create continued culture of data use. If you use this in your organization, you can also add another column for a measureable indicator so you can track progress over time.
This is an excellent management tool – document so you are able to monitor progress, keep people on same page with common direction to create continued culture of data use.
Assessment of Constraints to Data Use
Data Demand & Use: Assessment of Data Use Constraints Webinar Series - #1 Tuesday, January 17, 2012 Presenters: Molly Cannon and Tara Nutley
Agenda <ul><li>Welcome – webinar tips </li></ul><ul><li>Brief overview of Data Demand and Use </li></ul><ul><li>Presentation of tool - Assessment of Data Use Constraints </li></ul><ul><li>Questions and answers </li></ul><ul><li>Wrap up </li></ul>
Why improve data-informed decision making? Pressing need to develop health policies, strategies, and interventions
“… without information, things are done arbitrarily and one becomes unsure of whether a policy or program will fail or succeed. If we allow our policies to be guided by empirical facts and data, there will be a noticeable change in the impact of what we do.” National-level Policymaker, Nigeria
Definitions <ul><li>Data use – Using data in the decision making process </li></ul><ul><ul><li>create or revise a program or strategic plan </li></ul></ul><ul><ul><li>develop or revise a policy </li></ul></ul><ul><ul><li>advocate for a policy or program </li></ul></ul><ul><ul><li>allocate resources </li></ul></ul><ul><ul><li>monitor a program </li></ul></ul><ul><ul><li>review must be linked to a specific decision making process </li></ul></ul><ul><li>Data Demand - decision makers specify what kind of information they want & seek it out </li></ul>
MEASURE Materials/Resources Tools <ul><li>Quick Guide and Tool Kit </li></ul><ul><li>Assessment to Data Use Constraints </li></ul><ul><li>Information Use Map </li></ul><ul><li>Stakeholder Engagement </li></ul><ul><li>PRISM </li></ul><ul><li>Framework for Linking Data with Action </li></ul>Training <ul><li>Training Tool Kit (3 days) </li></ul><ul><li>Introduction to Basic Data Analysis and Interpretation for Health Programs </li></ul><ul><li>Integrating DDU into an M&E workshop </li></ul><ul><li>Pre-Service Nursing Education </li></ul><ul><li>High Impact Research </li></ul>Guidance Documents & Publications <ul><li>Seven Steps Guide </li></ul><ul><li>Case Study Series </li></ul><ul><li>Other papers </li></ul>
What determines Data Demand & Use? * Based on PRISM analytical framework (LaFond, Fields et al. (2005). The PRISM: An analytical framework for understanding performance of health information systems in developing countries. MEASURE Evaluation). ORGANIZATIONAL TECHNICAL BEHAVIORAL
What determines Data Demand & Use? POLITICS CULTURE SOCIETY * Based on PRISM analytical framework (LaFond, Fields et al. 2005 The PRISM: An Analytical Framework for Understanding Performance of Health Information Systems in Developing Countries. MEASURE Evaluation). ORGANIZATIONAL TECHNICAL BEHAVIORAL
Assessment of data use constraints tool <ul><li>Purpose </li></ul><ul><ul><li>To improve understanding of the demand for data and the constraints on data use. Designed for a rapid assessment of constraints to data use and overview of macro constraints </li></ul></ul><ul><li>Description </li></ul><ul><ul><li>Key informant interview guide designed to identify constraints in three categories: organizational, technical, and individual – two versions </li></ul></ul><ul><ul><li>Identifies effective practices in data use </li></ul></ul><ul><ul><li>Culminates with a planning matrix for addressing barriers to data use </li></ul></ul>
Assessment Versions <ul><li>National and sub-national assessment </li></ul><ul><ul><li>Provides a broad view of constraints at national and sub-national levels by collecting information from decision-makers on their current use of data and on their perceptions of the constraints to data use for evidence-based decision making. </li></ul></ul><ul><li>Facility level assessment </li></ul><ul><ul><li>Provides an understanding of constraints to both generating and using data at lower levels of the health system – and uses two different interview guides (one for data producers and other for data users). </li></ul></ul>
Both Versions - Interview and analysis process <ul><li>Conduct assessments </li></ul><ul><li>Analyze information </li></ul><ul><li>Identify priority barriers </li></ul><ul><li>Use the planning matrix template </li></ul><ul><li>Monitor progress on the planning matrix developed </li></ul>
NATIONAL AND SUB-NATIONAL ASSESSMENT <ul><li>Version 1 </li></ul>
Key Informants <ul><li>Interview between 20 and 25 individuals </li></ul><ul><ul><li>2/3 from national level </li></ul></ul><ul><ul><li>1/3 from provincial or district level </li></ul></ul><ul><ul><li>½ from the public sector </li></ul></ul><ul><ul><li>½ should include decision-makers from the NGO and private sector </li></ul></ul><ul><ul><li>Should include policymakers and program managers in the health sector or a related position in finance or planning. </li></ul></ul>
Six Sections of Key Informant Interview <ul><li>Logistical information and informed consent </li></ul><ul><li>Introductory questions – relative to last decision made in the organization </li></ul><ul><li>Technical constraints </li></ul><ul><li>Individual constraints </li></ul><ul><li>Organizational constraints </li></ul><ul><li>Closing questions – other contextual factors </li></ul>
Assessment of data use constraints tool Technical Constraints Technical constraints are related to the ability to generate high-quality data and analyses. RA8 Have you ever had an experience while making a policy or program-related decision when you were concerned about the quality of the information being used? RA9 Are there multiple sources of information or statistics for issues of importance to you, and have you experienced any problems caused by having different estimates? RA10 I am interested in knowing about technical capacity for collecting and using information. Does your agency have the technical capacity to produce reliable information without a lot of external technical assistance? RA11 Does your agency have the technical capacity to ensure access to and availability of reliable data? RA12 Has there been an occasion when data quality or local technical capacity made it difficult for you to use information in making a decision? RA13 How would you have gone about preventing this situation?
Assessment of data use constraints tool Individual Constraints Individual constraints are related to the skills, attitudes, values, and motivation of individuals . RA14 What specific challenges have you experienced among your staff when it comes to using data? Probe respondent for the following items following their response: awareness of data sources, technical skill, motivation, time and workload, lack of incentives or knowledge of the benefit to using data for policy change and program management.
Assessment of data use constraints tool Organizational Constraints Organizational constraints are related to challenges in using information that are due to how your organization functions . RA 15 How does your organization support having the necessary information to make decisions? RA 16 How does your organization support the prioritization and use of information in decision making? RA 17 How does your organization support training of staff in skills for using information in decision making? RA 20 What are the challenges your organization/agency experiences in sharing survey and research data? RA 22 Are there risks associated with sharing information? If so what are they? Record the response and the respondent’s openness or reluctance to answering this question.
Key Informants <ul><li>Has 2 interview guides – one for data producers and one for data users </li></ul><ul><li>Determine clinical area of interest (e.g., HIV/AIDS, malaria) </li></ul><ul><li>Interview at least 5 individuals from each facility </li></ul><ul><li>Include staff working in positions with a range of data production and use responsibilities, such as senior managers, clinicians, laboratory and pharmacy staff, counselors, and health information officers </li></ul>
Sections of Key Informant Interview Guide Data Users Data Producers Logistics/Background Logistics/Background Information Use for Decision Making Data Information and Flow Technical Barriers to Information Use Data Utilization Organizational Barriers to Information Use Barriers to Data Use Other Barriers to Information Use Other Barriers to Data Use
Planning Matrix for addressing barriers to data use # Barrier Proposed Intervention Steps Involved Person Responsible Other Stake-holders Gen. Time-line
Planning Matrix for addressing barriers to data use # Barrier Proposed Intervention Steps Involved Person Responsible Other Stake-holders Gen. Time-line 1 Lack of capacity at the facility level to produce quality and accurate data To build capacity of relevant key players in collecting, collating, and reporting data On the job training <ul><li>M&E specialist </li></ul><ul><li>Training coordinator </li></ul><ul><li>Chief of party </li></ul><ul><li>SACAs </li></ul>Oct 2012 2 Lack of computerized database at IP level to analyze and interpret data To develop a user-friendly and secure database at the IP level Training on the use of software <ul><li>Database consultants </li></ul><ul><li>Training coordinator </li></ul><ul><li>M&E specialist </li></ul><ul><li>Project manager </li></ul>Sept 2013
Adaptation to Key Informant Interviews <ul><li>Use Assessment materials in a workshop setting to identify the key constraints (1 to 3). </li></ul><ul><li>May also use it informally to stimulate understanding of usefulness of tool which can lead to a commitment to more formal use of tool </li></ul>
Assessment of Data Use Constraints Resources <ul><li>Tool is available on the MEASURE Evaluation website: www.measureevaluation.org/ddu-toolkit </li></ul><ul><ul><li>Instructions for use </li></ul></ul><ul><ul><li>Key informant interview guides </li></ul></ul><ul><ul><li>Planning matrix template </li></ul></ul><ul><ul><li>Implementation checklist </li></ul></ul><ul><li>Case examples from Kenya and Nigeria also on website: </li></ul><ul><ul><li>www.measureevaluation.org/ddu-publications </li></ul></ul>
MEASURE Evaluation DDU Resources <ul><li>www.measureevaluation.org/ddu </li></ul><ul><ul><li>Data Demand and Use Tool Kit </li></ul></ul><ul><ul><li>Data Demand and Use Training Resources </li></ul></ul><ul><li>Next webinar will be on January 24, 2012 at 9:00 am EST…Information Use Map </li></ul><ul><li>To sign up for Data Use Net, send an email to [email_address] . Leave the subject field blank and in the body of the message type ‘subscribe DataUseNet’ </li></ul>
<ul><li>MEASURE Evaluation is a MEASURE project funded by the </li></ul><ul><li>U.S. Agency for International Development and implemented by </li></ul><ul><li>the Carolina Population Center at the University of North Carolina </li></ul><ul><li>at Chapel Hill in partnership with Futures Group International, </li></ul><ul><li>ICF Macro, John Snow, Inc., Management Sciences for Health, </li></ul><ul><li>and Tulane University. Views expressed in this presentation do not </li></ul><ul><li>necessarily reflect the views of USAID or the U.S. Government. </li></ul><ul><li>MEASURE Evaluation is the USAID Global Health Bureau's </li></ul><ul><li>primary vehicle for supporting improvements in monitoring and </li></ul><ul><li>evaluation in population, health and nutrition worldwide. </li></ul>