Assessing M&E Systems for Data Quality USAID Mini University George Washington University Washington, DC, October 5th, 2007

Why is data quality important? Governments and donors collaborating to fight HIV/AIDS, TB, and malaria- “Three Ones” Accountability for funding and results reported increasingly important Quality data needed at program level for management decisions Unreliable data can impact on the appropriateness of management decisions Office of Inspector General, 2006

Governments and donors collaborating to fight HIV/AIDS, TB, and malaria- “Three Ones”

Accountability for funding and results reported increasingly important

Quality data needed at program level for management decisions

Data quality and the Program Cycle Data Quality Results Reporting Target Setting Improved Program & Resource Management

Data Quality Services: the REAL world In the real world, project activities are implemented in the field. These activities are designed to produce results that are quantifiable. Data: INFORMATION SYSTEM An information system represents these activities by collecting the results that were produced and mapping them to a recording system. Linking your services and your data … but how well does it all work?

Aggregated Data from PEPFAR Countries Source: 3 rd Annual PEPFAR report Where did these OVC numbers come from? Do you think they are good numbers? Why or why not? Make a flow chart of the data from sites to PEPFAR

Data Quality REAL WORLD In the real world, project activities are implemented in the field. These activities are designed to produce results that are quantifiable. INFORMATION SYSTEM An information system represents these activities by collecting the results that were produced and mapping them to a recording system. Data Quality: How well the information system represents the real world Data Quality Real World Information System 1. Accuracy 2. Reliability 3. Completeness 4. Precision 5. Timeliness 6. Integrity 7. Confidentiality

Dimensions of Data Quality What are some elements of data quality?

What are some elements of data quality?

Dimensions of Data Quality The data are protected from deliberate bias or manipulation for political or personal reasons. Integrity Clients are assured that their data will be maintained according to national and/or international standards for data. Confidentiality Data are up-to-date (current), and information is available on time. Timeliness The data have sufficient detail (e.g. collected by age, sex, etc.) Precision Completely inclusive: an information system represents the complete list of eligible names and not a fraction of the list. Completeness The data are measured and collected consistently (the same way with the same data collection instruments) over time. Reliability Valid data are considered accurate: They measure what they are intended to measure. Accuracy/ Validity

Information Flow Results Indicators Data sources Data collection/reporting systems

Support to PLWHA and/or families How are these data collected? Community health workers daily record of households visited Facility-level register Client intake forms How aggregated? How forwarded to next level? How forwarded to next level? How forwarded to national level? How forwarded to international level?

Community health workers daily record of households visited

Facility-level register

Client intake forms

How aggregated?

How forwarded to next level?

How forwarded to next level?

How forwarded to national level?

How forwarded to international level?

Program/project M&E Plan and List of Indicators What is the role of the central M&E Unit in the M&E system and data quality? What is the role of the intermediate aggregation level? What is the role of sites? Levels of the M&E System Data Requirements 1- Central/ national M&E Unit 2- Intermediate aggregation levels (e.g. district, region, state, sub-partner) 3- Service sites (health facility-based or community-based) Data Reporting Data Reporting

Brainstorming… … .what needs to be in place at each level of the M&E system to ensure good quality data?

… .what needs to be in place at each level of the M&E system to ensure good quality data?

DQ Systems Questions… How well does your information system function? Does your project/program link to the national information system? Are the definitions of indicators clear and understood at all levels? Do individuals and groups understand their roles and responsibilities? Does everyone understand the specific reporting timelines—and why they need to be followed?

How well does your information system function? Does your project/program link to the national information system?

Are the definitions of indicators clear and understood at all levels?

Do individuals and groups understand their roles and responsibilities?

Does everyone understand the specific reporting timelines—and why they need to be followed?

DQ Systems Questions, cont.… Are data collection instruments and reporting forms standardized and compatible? Do they have clear instructions? Do you have documented data review procedures for all levels…and use them? Are you aware of potential data quality challenges, such as missing data, double counting, lost to follow up? How do you address them? What are your policies and procedures for storing and filing data collection instruments?

Are data collection instruments and reporting forms standardized and compatible? Do they have clear instructions?

Do you have documented data review procedures for all levels…and use them?

Are you aware of potential data quality challenges, such as missing data, double counting, lost to follow up? How do you address them?

What are your policies and procedures for storing and filing data collection instruments?

What is the Link between M&E Systems and Data Quality? 1. Accuracy 2. Reliability 3. Completeness 4. Precision 5. Timeliness 6. Integrity 7. Confidentiality Dimensions of Data Quality 1- Central/ national M&E Unit 2- Intermediate aggregation levels (e.g. district, region, state, sub-partner) 3- Service sites (health facility-based or community-based) Data Reporting Data Reporting Levels of the M&E System

Strengthening M&E Systems through data Quality How can we promote good quality data?

How can we promote good quality data?

Multiagency Tools Under draft

SYSTEMS APPROACH INDICATOR APPROACH AUDITING APPROACH (also for capacity building) Three Multi-agency complementary DQ Tools 1- M&E Systems Strengthening Tool What data management systems should be in place to ensure data-quality? 3- Data Quality Assurance Tool for Program Level Indicators What are the Data Quality challenges in collecting specific indicator data (e.g., for ARV, for People Trained, etc.) 2- Data Quality Assessment (DQA) Tool Are appropriate data management systems in place? Is reported data accurate and valid?

1. Accuracy 2. Reliability 3. Completeness 4. Precision 5. Timeliness 6. Integrity 7. Confidentiality Functional Areas of an M&E System that Affect Data Quality Dimensions of Data Quality Data quality mechanisms and controls VII Data management processes VI Data collection and Reporting Forms and Tools V Links with the national reporting system VIII Indicator Definitions IV Data Reporting Requirements III Training II M&E Capabilities, Roles and Responsibilities I Functional Areas of an M&E System Needed to Ensure Quality

M&E Systems Strengthening Tools Systems approach Facilitator moderated Three levels: M&E Plan M&E Coordination Unit Data Reporting Systems

Systems approach

Facilitator moderated

Three levels:

M&E Plan

M&E Coordination Unit

Data Reporting Systems

Data Quality Assurance Tools Auditing approach Two components Systems Assessment Data Quality Verification Verification factors

Auditing approach

Two components

Systems Assessment

Data Quality Verification

Verification factors

DQA Protocol 1: Functional Areas of an M&E System that Affect Data Quality (from DQA Systems Assessment Protocol) Are there operational indicator definitions meeting relevant standards and are they systematically followed by all service points? 4 Indicator Definitions IV Has the Program/Project clearly documented (in writing) what is reported to who, and how and when reporting is required? 3 Data Reporting Requirements III Have the majority of key M&E and data-management staff received the required training? 2 Training II Are key M&E and data-management staff identified with clearly assigned responsibilities? 1 M&E Capabilities, Roles and Responsibilities I 11 Summary Questions 8 Functional Areas

DQA Protocol 1: Functional Areas of an M&E System that Affect Data Quality (from DQA Systems Assessment Protocol) Does the data collection and reporting system of the Program/Project link to the National Reporting System? 11 Links with National Reporting System VIII Are there clearly defined and followed procedures to periodically verify source data? 10 Are there clearly defined and followed procedures to identify and reconcile discrepancies in reports? 9 Are data quality challenges identified and are mechanisms in place for addressing them? 8 Data Quality Mechanisms and Controls VII Does clear documentation of collection, aggregation and manipulation steps exist? 7 Data Management Processes VI Are source documents kept and made available in accordance with a written policy? 6 Are there standard data-collection and reporting forms that are systematically used? 5 Data-collection and Reporting Forms and Tools V

Summary Question 1: Are key M&E and data-management staff identified with clearly assigned responsibilities?  The responsibility for recording the delivery of services on source documents is clearly assigned to the relevant staff (i.e., it is in their job description). 6   There are designated staff responsible for the review of aggregated numbers prior to submission to the next level (e.g. to the central M&E Unit). 5   There are designated staff responsible for reviewing the quality of data submitted by sub-reporting levels (e.g. regions, districts, service points). 4  The Program Manager(s) review(s) the aggregated numbers prior to the submission/release of reports from the M&E Unit. 3  There are staff dedicated to M&E and data management systems. 2  There is a documented organizational structure/chart that clearly identifies positions that have data management responsibilities. 1 I - M&E Capacities, Roles and Responsibilities Service Points Aggre-gation Levels M&E Unit Reporting System Level Component of the M&E System LIST OF ALL QUESTIONS List of Questions Related to the Data Management and Reporting System

DQA: Summary of M&E System Functional Areas Interpreting the findings: The larger the score, the stronger the component

Data verification: DQA Protocol 2 Perform “spot checks” to verify the actual delivery of services or commodities to the target populations. 5. Spot checks Perform “cross-checks” of the verified report totals with other data-sources (eg. inventory records, laboratory reports, etc.). 4. Cross-checks Trace and verify reported numbers: (1) Recount the reported numbers from available source documents; (2) Compare the verified numbers to the site reported number; (3) Identify reasons for any differences. 3. Trace and Verification Review availability and completeness of all indicator source documents for the selected reporting period. 2. Documentation Review Describe the connection between the delivery of services/commodities and the completion of the source document that records that service delivery. 1. Description Description Verification SERVICE DELIVERY POINT - 5 TYPES OF DATA VERIFICATIONS

DQ Dimensions, Levels of the M&E System and Functional Areas Dimensions of Data Quality Levels of the M&E System Functional Areas of an M&E System Needed to Ensure Quality Data quality mechanisms and controls VII Data management processes VI Data-collection and Reporting Forms and Tools V Links with the national reporting system VIII Indicator Definitions IV Data Reporting Requirements III Training II M&E Capabilities, Roles and Responsibilities I Functional Areas of an M&E System Needed to Ensure Quality 1. Accuracy 2. Reliability 3. Completeness 4. Precision 5. Timeliness 6. Integrity 7. Confidentiality Data Reporting 1- Central/ national M&E Unit 2- Intermediate aggregation levels 3- Service sites (health facility-based or community-based)

Data Quality Plan Template Country: Time Period: Program or Partner (e.g. National Program or Partner): TA needs Funding Timeline Responsibility Description of Strengthening Measure Functional Areas (8) DQ Plan Table 2. Functional Areas, Strengths and Weaknesses and Strengthening Measures Strengthening Measure(s) Needed* Weaknesses Strengths Summary Questions Functional Areas (8) DQ Plan Table 1. Functional Areas, Strengths and Weaknesses and Strengthening Measures Data quality mechanisms and controls VII Data management processes VI Data-collection and Reporting Forms and Tools V Links with the national reporting system VIII Indicator Definitions IV Data Reporting Requirements III Training II M&E Capabilities, Roles and Responsibilities I Functional Areas of an M&E System Needed to Ensure Quality Dimensions of Data Quality Levels of the M&E System

Country: Time Period: Program or Partner (e.g. National Program or Partner):

Data Quality Plan Template Country: Time Period: Program or Partner (e.g. National Program or Partner): TA needs Funding Timeline Responsibility Description of Strengthening Measure Functional Areas (8) DQ Plan Table 2 . Functional Areas, Strengths and Weaknesses and Strengthening Measures Strengthening Measure(s) Needed* Weaknesses Strengths Summary Questions Functional Areas (8) DQ Plan Table 1 . Functional Areas, Strengths and Weaknesses and Strengthening Measures Data quality mechanisms and controls VII Data management processes VI Data-collection and Reporting Forms and Tools V Links with the national reporting system VIII Indicator Definitions IV Data Reporting Requirements III Training II M&E Capabilities, Roles and Responsibilities I Functional Areas of an M&E System Needed to Ensure Quality Dimensions of Data Quality Levels of the M&E System

Country: Time Period: Program or Partner (e.g. National Program or Partner):

MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) through Cooperative Agreement GPO-A-00-03-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina in partnership with Constella Futures, John Snow, Inc., ORC Macro, and Tulane University. Visit us online at http://www.cpc.unc.edu/measure.