Reduce reporting burdens for community based organizationsTrack prevention, care, testing, and linkages at the community level
Harmonize PEPFAR and Global Fund community program indicatorsImprove the technical merit of community indicatorsHighlight a few of the emerging indicators that would fill gaps in knowledge about community programs
Ground work for the standardization of indicators to measure community HIV program performance began in 2009 with the international consultation on Community Based Information Systems hosted by MEASURE Evaluation Project. Among the recommendations was a request to develop a core set of indicators across donors to reduce reporting burdens. In the fall 2009, an inventory of all published community indicators (totaling 123) was developed. The indicator list included input, output and outcome indicators using both survey and program data. The list also included indicators that collected information on clinical services that relied on community support. In January 2010, an international consultation on community indicators hosted by MEASURE Evaluation Project in Washington D.C. recommended focusing on the standardization of indicators that track service delivery, referrals for services, quality of services, and health outcomes with data collected at the community level only. The previously mentioned criteria were developed during this meeting and later validated by our community of practice members. A follow up consultation in Geneva provided the Global Fund recommendation to include indicators that measure community-led activities to decrease stigma, decrease gender based violence, or support an enabling environment. Finally in the summer of 2010, PEPFAR completed an internal review, which included all headquarter TWGs. TWGs reviewed the indicator inventory,previous consultation recommendations, and gave support for the development of an on-line forum to further gather expert opinion around indicator standardization.
In 2011, a 12 week on-line forum series designed by MEASURE Evaluation Project was held from April to June 2011. Technical experts from the field and from multilateral headquarters joined a community of practice (CoP) on Knowledge Gateway in order to participate. The community of practice remains a dissemination and feedback tool, with 165 members representing 26 countries. Lead experts were recruited to facilitate a 2-week long forum on a special topic designed to forward the goals of The Community. There were 5 topics:outcomes for OVC, service provision for HIV prevention, monitoring community participation in HTC, tracking referrals and linkages, and standardizing existing care indicators. MEASURE staff facilitated a formal launch for one week and then a wrap-up of the forum series for another week. The wrap up aimed to gain consensus on the overall conclusions of the forum series and on a final set of indicators.
This criteria was proposed during the January 2010 meeting in D.C. and was validated by community of practice participants using an online poll.Indicators represent services that address the continuum of prevention, treatment, care, and supportIndicators are fully defined and currently used Indicators are collected (and used) at the community level and aggregated (and used) at the national levelIndicators require data collected from a community-based information system and not collected from a clinic-based information systemIndicators do not require data from surveys or special studies administered by the national authorities
10 indicators using a poll during the “wrap-up” forum.Each indicator is based on existing foundational indicatorsMembers were not successful in addressing the metrics for community-led activities to decrease stigma, decrease gender based violence, or support an enabling environment. This remains a gap.Prevention programs across the board struggle to monitor one to one and small group contacts and generally do not report on the number of unique individuals. The lack of evidence-based preventions programs implemented on a global level leaves managers unable to fully define the intervention to be counted. The only exceptions cited during an on-line forum were multi-session youth education programs and outreach worker facilitated referrals to HIV testing. Generally, the indicators currently required by Global Fund and PEPFAR are not feasible to collect and lack technical merit. Members suggested revising these indicators to better reflect information HIV prevention programs are currently able to collect, with the aim to improve the quality of the measures overtime. We think the prevention with positives (PwP) indicator introduced by PEPFAR requiring a minimum package of services is important to include in this set. As written, the indicator can be collected from clinical and community programs. The field test will take a closer look at how community programs report against this indicator.Monitoring condom and clean needle distribution and/or promotion is important, but the current practice to report the number of materials distributed is not sufficient. We propose tracking stock outs at the prevention site level. Generally HIV care, support, and nutrition programs can successfully track the total number of unique individuals receiving at least one care service (case loads) in a reporting period. During the field test, we are most interested in disaggregation such as type of services, community services versus clinical services at the national level, etc.Individual assessments for adults and children are necessary to ensure that appropriate care, support, and nutrition services are provided to each person. USAID advisor Amie Heap has spear headed a “Nutrition and HIV Indicator Set” that has been submitted to the UNAIDS Registry this year. We have borrowed from the Nutrition and HIV Indicator Set by adapting the nutrition assessment indicator to be a general care and support assessment indicator for children. We are also using the proportion of undernourished people living with HIV (PLHIV) is an essential community indicator from that set. Again, as written, the indicator can be collected from clinical and community programs. The field test will take a closer look at how community programs report against this indicator or if they report it at all.HIV testing programs are successfully tracking the total number of people receiving HIV test results in a reporting period. However, a theme across all areas, including HIV testing and counseling, was the need to understand who is receiving what service from whom. We would like to now what type of support Civil Society Organizations and National AIDS Program managers require to use the indicator data disaggregated by Risk/Need, Age, Sex, Type of Service, Location, and Type of Provider.Another way to improve comprehensive prevention, care and support is to prioritize improving linkages and referrals for clients. Community program participation in national referral guidelines and/or the development referral guidelines of their own is important. We know that referrals and linkages are weak and that an indicator measuring counter-referrals or completed referrals would be a good start in monitoring linkages.
Purpose: To engage key stakeholders in harmonizing a core set of community-based HIV program indicators that can be collected (and used) at the community level and be aggregated (and used) at the national level.The field test objectives are based on the 2010 UNAIDS Indicator Standards: Operational Guidelines for Selecting Indicators for the HIV ResponseThe field test was comprised of three phases: community HIV intervention mapping and site selection, CBO interviews, NAP interviews.Note that the field test is not a review or evaluation of current indicators or monitoring and evaluation systems. Rather, the field test conducted in Vietnam and Kenya was to generate lessons learned that informed the revision and finalization of the set of recommended indicators for community HIV programs. As a set, the CBOs should provide information for all nine indicators, but each CBO is not required to provide services and collect information for all nine indicators. The set of CBOs should include organizations with small staff and large staff. All CBOs should have well developed service delivery and strong to fair data collection systems. The set of CBOs does not have to be geographically representative of the country in which the field test occurs.
Consultations will be conducted with CBO directors, program managers, M&E directors, and database managers. Consultations will also be requested with National AIDS program directors and managers, M&E directors and database managers. Group consultations will be requested after individual interviews have been completed to help “connect the dots.”The consultations requested the following from the participant:An overview of their community program Their opinion on the usefulness of current indicatorsA description of their data collection processesThen we would introduce the draft indicator and ask forFeedback on the draft indicator’s usefulness and clarity of definitionAll consultations used a semi-structured interview guide that was founded on data quality assurance tools.
Kenya12 organizations in site selection phase6 local, 1 ING, and 2 natlgovt organizations in interview phase35interviews including the community and national levelsNairobi, nearby Central Province was included so that urban/rural variation,Coast Province for EBIsVietnam10 local organizations in site selection phase7 organizations in interview phase20 interviews including community and national levelHanoi area
Given the results from the field tests conducted in Vietnam and Kenya, a “package of services” outlined in any of the community program indicators should be illustrative not required at the international level. A focus on establishing national guidelines or a national package of services would improve the feasibility of every indicator, facilitating less reporting burden for the community level and improving validity and reliability across community sites. Of note, clear and specific guidance on care and support packages versus PwP packages is an important first step to better understanding the types of services PLHIV receive and from whom. It should also be noted that any information collected for community program indicators would require a paper-based system. This is relevant for both Vietnam and Kenya. Any data management systems adapted or developed for these indicators should reflect that need. While personnel have experience collecting and reporting on a wide array of indicators similar to those recommended, there are still issues regarding limited capacity of personnel to manage electronically-based systems.Strong evidence in both countries points to the feasibility and usefulness at the community and national level for the prevention indicators and the stock out indicators. Although, many experts have called for better tracking of individuals receiving care and support services, the field test points to the continuation of tracking the “number of adults and children infected and/or affected by HIV/AIDS who received a minimum of one care and support service during the reporting period.” As mentioned earlier, great caution is recommended in order to avoid duplication when tracking the care and support indicator and PwP indicator. A gap, not fully explored during this field test, may be the number households receiving care and support. The number of households may better reflect care and support programs in countries with high prevalence rates.The indicators for HIV testing services, nutrition assessments, needs assessments for children, and completed referrals were not feasible and are therefore not included in the final set of indicators. One system issue that arose was the lack of CBO capacity to assess and track progress of unique individuals over long periods of time. Tracking the number of people who received a service within a designated reporting period is possible in both countries. Managing individual, long-term records at the CBO, which would include assessment results, is not feasible in either country. Therefore, nutrition and OVC individual assessment indicators were not included in the final set. A general “completed referral” indicator was also eliminated from the indicator set for similar reasons. CBOs have limited capacity to document and track individual cases over an extended time. Another system issue that arose is how to best represent the supportive role community programs play in people’s access to clinical services. Although community programs are essential to linking community members to HIV testing, including HIV tests provided by CBO personnel or on CBO grounds, the management and reporting responsibilities fall on clinical systems in both countries. Therefore, the “number of people tested” and the “number test kit stock outs” were not included in the indicator set. We strongly recommend that clinic-based information systems track the number of people tested at community-based sites in order to improve HIV testing strategies. In the event that the PwP indicator is reported via the clinic-based information system only, such as in Kenya, we strongly recommend that the number of people reached with PwP services by CBOs is tracked within that system and used for program management. Conversely, if a country does not require PwP program data to be tracked by a clinic-based information system only, we recommend collecting a community-based PwP indicator, which is included in the recommended set.In order to document the work CBOs contribute to HIV testing programs and to invest in client referrals and program linkages, we recommend adding an indicator used in Vietnam. “Number and proportion of clients referred to HIV testing services that were tested and received their results,” can be found on page 16 of this report and is included in the final recommended set.Given the programming and epidemic situations found in Vietnam and Kenya, there are gaps in the indicator set presented outlined below. In Vietnam for example, one of the current gaps is the need for indicators that address programs for IDUs within the proposed indicator set (please refer to Annex 6 for a list of IDU related indicators currently used in Vietnam). This is particularly important for countries with concentrated epidemics such as that found in Vietnam. It is also relevant to some extent for the diverse epidemic found in Kenya where pockets of IDUs in certain areas, such as the coastal region, mimic concentrated epidemics. An additional gap highlighted by the Kenya field test is the need for indicators that address ART adherence and the tracing of ART defaulters. Kenyan respondents felt that the contributions that communities make toward tracing of ART defaulters should be documented and monitored. An indicator for the proportion of defaulters traced would be useful for the purpose of assessing quality of treatment services. In addition, the inclusion of such an indicator would sensitize community partners to the importance of adherence and could bring up issues around why people default. At least one INGO is piloting a model in which a list of defaulters is generated by CCC staff; CHWs trace those individuals in the community and attempt to persuade them to return to the clinic. Introducing a defaulter tracing indicator could help to formalize and expand this process.The final recommended indicators for community HIV programs are listed below. Reference sheets for each indicator follow the list.
Completed referral to a service like HIV testing may be closer to a an estimate in the beginning. Coupon system is working but new and imperfect. Also consider peer worker logs.
Peer educator to small group participant ratioNumber people reached with mass mediaPwP could be written as a community versus a clinical package. Or it could be written as a completed referral to a clinical service. Or vice a versa.
Also count numbers distributed
PLHIVOVCCare taker of OVCHead of householdAll members in the householdDependent on the nature of the service or the participation requirements
Completed referrals to a specific service would be usefulHTCPwPARTNutritionDrug RehabilitationThis completed referral to HTC represents all of the work CBOs put into outreach to vulnerable people. CBOs would count this if the test happens at their sponsored event, on their property, with their community worker. They would also count this indicator if the test happened at a clinic. A coupon system or community worker log would be used.The number of people tested would be a separate indicator reported by the clinical site using the official testing form.
Community HIV Program Indicators
Community HIV ProgramIndicatorsMEASURE EvaluationTulane UniversityMary Freyder
Purpose of Presentation Rationale for the community indicator project Donor organization consultations On-line technical forum series Field tests in Vietnam and Kenya Description of recommended indicators
Rationale Reduce reporting burdens for communitybased organizations Track prevention, care, testing, and linkagesat the community level
Aim Harmonize PEPFAR and Global Fundcommunity program indicators Improve the technical merit of communityindicators Highlight a few of the emerging indicators thatwould fill gaps in knowledge about communityprograms
Extensive International Consultation 2009 meeting Community Based Information SystemsMEASURE Evaluation in D.C. 2010 meeting Community Program IndicatorsMEASURE Evaluation in D.C. 2010 meeting Community Systems StrengtheningGlobal Fund in Geneva 2010 internal review of Community Program Indicatorsall PEPFAR TWGs in D.C.
Community of Practice“The overall goal of the Indicators for Community HIVPrograms CoP is to gather expert opinion on the bestmetrics for community HIV program performance and togain consensus on minimum standards for monitoringcommunity HIV services and health outcomes.”http://knowledge-gateway.org/registration/PreviewInvitation.ashx?c=24643e57-e93e-49df-ae1c-f798ea662a16&i=61fe624d-0aaa-49ab-abeb-83d0ef89e10a
Indicator Selection Criteria Represent services that address the continuum ofprevention, treatment, care, and support Fully defined and currently used Collected/used at the community level andaggregated/used at national level Collected by community-based information system(non-clinic based) Not include indicators from national level surveys orspecial studies
CoP Recommendations Oct 2011 Outreachencounters Completed DEBIs PwP Condom stock-outs Needle/Syringestock out Received HTC Completed referral One care service UndernourishedPLHIV Children assessedfor individual needs
Field Tests in Vietnam and Kenya To assess the need and utility of therecommended indicators for both CBOs andnational authorities. To determine the feasibility of collecting andanalyzing data for the recommended indicators. To assess whether or not the draftrecommended indicators are fully defined
Mixed Methods Approach Individual and small group consultations In the style of semi-structured interviews Review program records, documents and datamanagement systems In the style of Data Quality Assessments Request copies of relevant documents and forms
Field Test Participants Kenya 7 community organizations 35 interviews Vietnam 7 community organizations 20 interviews
Field Test Recommendations (I) “Minimum Package of Services” should beillustrative at the international level Focus on developing national guidelines forcommunity services is important Feasibility of data collection per local context Validity, i.e. more closely represents the service Reliability, i.e. less open to interpretation from onesite to another
Field Test Recommendations (II) Proposed prevention indicators are feasibleand useful encounters, standardized prevention interventions,stock outs, community PwP services One care service important and useful; thereis overlap with PwP Completed referrals to a specific service ispossible, but ambitious HIV testing, ART defaulters
Field Test Recommendations (III) Consider counting households receiving careand support Right now a mix of individuals and head ofhousehold under one care service The option to count individuals and/or households HTC and PWP indicators are from clinic-based information systems Disaggregate by type of site would very useful tolocal and national level
Field Test Recommendations (IV) Uncommon community services; indicatorsnot required at this time Nutrition assessments for PLHIV Individual needs assessments for children oradults Supporting or tracking all/various completedreferrals
Field Test Recommendations (V) Paper-based systems still relevant Possible to track # individuals in a reportingperiod Managing longitudinal records is not possible,therefore difficult to track: needs assessments completed referrals individual level outcomes
Field Test Recommendations (VI) Community services found during field testnot represented in indicator list HIV prevention with people who injected drugs Finding ART defaulters in the community ART adherence support
Final Indicators for Prevention Services Number of HIV prevention outreach encountersconducted during the reporting period Number of people who completed a standardizedHIV prevention intervention during the reportingperiod Number of people living with HIV/AIDS (PLHIV)reached with a minimum package of community-based or home-based interventions forPrevention with Positives (PwP) during thereporting period
Final Indicators for Prevention Materials Number and proportion of days during thereporting period that stock outs of condomsoccurred at prevention sites Number and proportion of days during thereporting period that stock outs of needles and/orsyringes occurred at prevention sites
Final Indicator for Care Services Number of adults and children infected and/oraffected by HIV/AIDS who received a minimumof one community-based or home-based careand support service during the reporting period PLHIV OVC Caretaker of OVC(s) Head of household All members of household
Final Indicator for Testing and Linkages Number and proportion of clients referred to HIVtesting and counseling services by a community-based program and received their test results inthe reporting period