Building New Institutional Capacity in M&E: The Experience of National AIDS Coordinating Authority V F Kemerer Getting To Results IDEAS Global Assembly Johannesburg, South Africa 18 March 2009

Overview Background Use of Assessment Tools for Strategic Planning NACA Specific Capacity Building Interventions Measurement and Determination of Effectiveness and Outcomes

Background

Use of Assessment Tools for Strategic Planning

NACA Specific Capacity Building Interventions

Measurement and Determination of Effectiveness and Outcomes

Background

Background Coordinate the National HIV DMRS Lead National M&E System Development NAC M&E UNIT National Agencies USG & PEPFAR GFATM UN Family Misc Bi- & Multi-Laterals CB through TA using various tools & strategies to establish and/or improve performance

Strategic Vision: Build effective and sustainable M&E systems to provide comprehensive, high-quality information

Use of Assessment Tools for Strategic Planning SI Assessment MESST DQA/RDQA

SI Assessment

MESST

DQA/RDQA

SI Assessment Activities Map national HIV output indicators and identify data flow Outline challenges in the DMRS Assess M&E structures Stakeholder Analysis – National M&E Framework Stakeholder Analysis –HIV researchers, organisations, and research

Map national HIV output indicators and identify data flow

Outline challenges in the DMRS

Assess M&E structures

Stakeholder Analysis – National M&E Framework

Stakeholder Analysis –HIV researchers, organisations, and research

Linking SI Capacity and SI Planning with SI Products Timely, high quality strategic information products Unified SI Strategic Planning and Coordinated SI Support SI Systems Assessments Ongoing Monitoring & Feedback Building of National SI Systems

Timely, high quality strategic

information products

Use of Assessment Tools for Strategic Planning SI Assessment MESST DQA/RDQA

SI Assessment

MESST

DQA/RDQA

A new approach to M&E Assessments Donor-driven Country-driven Focused exclusively on capacities of specific silos Expanded to integration in the wider M&E Systems General approach More systematic approach (based on a checklist of items) Strictly Assessment Tool Broader Assessment, Management, Planning & Capacity Building Tool FROM … TO …

How does the M&E Strengthening Tool work? Step 1: Diagnosis Step 3: Action Plan Development of a Costed Action plan with: Identification of Strengths and Weaknesses Defining Strengthening Measures : Responsibility Timeline Funding Implication T/A Needs Diagnosis through completion of a Checklist: Answers to Statements/ Questions Provision of Comments/Explanations Step 2: Analysis A Summary Dashboard will be automatically generated : Displays the overall distribution of answers for each of the checklists’ sections Provides a more detailed analysis of the specific gaps that helps respondents complete the ACTION PLAN

Development of a Costed Action plan with:

Identification of Strengths and Weaknesses

Defining Strengthening Measures :

Responsibility

Timeline

Funding Implication

T/A Needs

Diagnosis through completion of a Checklist:

Answers to Statements/ Questions

Provision of Comments/Explanations

A Summary Dashboard will be automatically generated :

Displays the overall distribution of answers for each of the checklists’ sections

Provides a more detailed analysis of the specific gaps that helps respondents complete the ACTION PLAN

Expected Benefits Better identify M&E capacity gaps and corresponding strengthening measures , including through Technical Assistance (TA) Ensure that such investments contribute to the strengthening of the National Systems (avoiding parallel reporting systems) Guide investments in M&E (within the recommended range of 5%-10% of the overall Budget) Enhance the quality of programmatic data to improve program management

Better identify M&E capacity gaps and corresponding strengthening measures , including through Technical Assistance (TA)

Ensure that such investments contribute to the strengthening of the National Systems (avoiding parallel reporting systems)

Guide investments in M&E (within the recommended range of 5%-10% of the overall Budget)

Enhance the quality of programmatic data to improve program management

Use of Assessment Tools for Strategic Planning SI Assessment DDIU Tools MESST DQA/RDQA

SI Assessment

DDIU Tools

MESST

DQA/RDQA

DQA METHODOLOGY – 2 Protocols The methodology for the DQA includes two (2) protocols: Data Verifications (Protocol 1) Quantitative comparison of recounted to reported data and review of timeliness, completeness and availability of reports. 1 Assessment of Data Management Systems (Protocol 2) Qualitative assessment of the strengths and weaknesses of the data-collection and reporting system. 2

The methodology for the DQA includes two (2) protocols:

PURPOSE: Identify potential risks to data quality created by the data- management and reporting systems at: the M&E Management Unit; the Service Delivery Points; any Intermediary Aggregation Level (District or Region). The DQA assesses both (1) the design; and (2) the implementation of the data-management and reporting systems. The assessment covers 8 functional areas (HR, Training, Data Management Processes , etc.) PROTOCOL 2: Assessment of Data Management and Reporting Systems M&E Management Unit Service Delivery Sites / Organizations Intermediate Aggregation levels (eg. District, Region) Assess Data Management and Reporting Systems ( Part 1 )

PURPOSE: Identify potential risks to data quality created by the data- management and reporting systems at:

the M&E Management Unit;

the Service Delivery Points;

any Intermediary Aggregation Level (District or Region).

The DQA assesses both (1) the design; and (2) the implementation of the data-management and reporting systems.

The assessment covers 8 functional areas (HR, Training, Data Management Processes , etc.)

Assess Data Management and Reporting Systems ( Part 1 )

Objective of the Data Quality Assessment Tool The Data-Quality Assessment (DQA) Tool is designed to: assess the ability of data-management systems to collect manage, and report quality data . verify the quality of reported data for key indicators at selected sites ; and The DQA Tool is not intended to assess the entire M&E system; it relates and focuses more in depth into component 10 (i.e. Supportive supervision and data auditing ) of the “Organizing Framework for a Functional National HIV M&E System” . !

assess the ability of data-management systems to collect manage, and report quality data .

verify the quality of reported data for key indicators at selected sites ; and

NACA Specific Capacity Building Interventions MESST RDQA TA for DQ SI Assessment M&E Road Map 1/4ly Mentoring (SHAPMoS)

CB INTERVENTIONS

SI Assessment M&E Road Map Implementation Incorporation of Data Flow Findings into SHAPMoS Incorporation of the Institutional Inventory List into: National Programmatic Database based at the NACA Supporting primary efforts of WHO’s P-SAM Exercise Participation in the Launch of the National HIV M&E Strategy

Incorporation of Data Flow Findings into SHAPMoS

Incorporation of the Institutional Inventory List into:

National Programmatic Database based at the NACA

Supporting primary efforts of WHO’s P-SAM Exercise

Participation in the Launch of the National HIV M&E Strategy

SI Assessment M&E Road Map Implementation ToT M&E Basic Concepts 4 quarterly mentoring & coaching visits to Regional M&E Officers Support to the NACA to produce QSCR

ToT M&E Basic Concepts

4 quarterly mentoring & coaching visits to Regional M&E Officers

Support to the NACA to produce QSCR

Institutional Capacity Strengthened – Regional M&E Officers Internal monitoring systems developed and in place Trace and verification conducted in each Region at SDPs Regional QSCRs produced Information products developed for feedback to SDPs

Internal monitoring systems developed and in place

Trace and verification conducted in each Region at SDPs

Regional QSCRs produced

Information products developed for feedback to SDPs

CB INTERVENTIONS

Timeline: TA to Malawi 2007 2008 01 02 03 04 05 06 07 08 09 10 11 12 01 02 03 04 05 Data Quality Plan for MESST MESST: Malaria MESST: HIV Discussions with NAC: Next Steps Plan for DQ T/A M&E Plan Revision

RDQA Tool To Operationalize Statement of Work Support finalization of the Draft Data Quality Protocol Assist to roll-out the data quality protocol Conduct data quality training(s) for key implementing agencies Draft Services & Indicators Matrix Draft register(s) to be used by all CBO SDPs Draft clear instructions Draft DQ policies Draft data steps performed at each level of the DMRS Draft guidelines for quality controls Draft written back-up procedures Data Quality Training in 5 Zones

Support finalization of the Draft Data Quality Protocol

Assist to roll-out the data quality protocol

Conduct data quality training(s) for key implementing agencies

Institutional Capacity to Support Data Quality Developed Data quality plan written that includes guidance, reporting forms, and indicator matrix Key stakeholders at national and sub-national levels oriented to – and reviewed – new data quality documents NACA M&E Officers trained to train on data quality Key staff in all 5 Zones have been trained by NACA on data quality

Data quality plan written that includes guidance, reporting forms, and indicator matrix

Key stakeholders at national and sub-national levels oriented to – and reviewed – new data quality documents

NACA M&E Officers trained to train on data quality

Key staff in all 5 Zones have been trained by NACA on data quality

Conclusions NACA M&E Unit institutional capacity strengthened at National and Sub-National levels Data reporting more timely, with increasingly accurate, reliable and complete data Foundation for effective data use laid Documented systems establish institutional memory

NACA M&E Unit institutional capacity strengthened at National and Sub-National levels

Data reporting more timely, with increasingly accurate, reliable and complete data

Foundation for effective data use laid

Documented systems establish institutional memory

MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) through Cooperative Agreement GPO-A-00-03-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina in partnership with Futures Group, John Snow, Inc., Management Sciences for Health, ORC Macro International, and Tulane University. Visit us online at http://www.cpc.unc.edu/measure.