Systems-level Monitoring and Evaluation of a Large-scale Combination HIV Prevention Initiative in 12 U.S. Cities
Systems-level Monitoring and Evaluation of a Large-scale Combination HIV Prevention Initiative in 12 U.S. Cities Holly H. Fisher, Tamika Hoyte, Dale Stratford, and Gary Uhl 2011 National HIV Prevention Conference Atlanta, GA National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention Division of HIV/AIDS Prevention
National HIV/AIDS Strategy July 2010- White House released National HIV/AIDS Strategy (NHAS) NHAS Goals: Reduce new HIV infections Increase linkage to care and overall health outcomes for persons living with HIV Reduce HIV-related health disparities Six federal agencies were asked to develop operational plans that describe how they will advance NHAS
National HIV/AIDS Strategy Dec. 2010- HHS operational plan1 is submitted Described: • ongoing efforts to align existing activities with NHAS • new initiatives that support NHAS goals HHS 12-Cities Project- accelerate comprehensive HIV/AIDS planning and cross-agency response in the 12 U.S. jurisdictions hardest hit by HIV/AIDS CDC ’s Enhanced Comprehensive HIV Prevention Plan (ECHPP) is one of the major 12-Cities Project initiatives1Department of Health and Human Services. HHS Operational Plan for National HIV/AIDS Strategy. Available at:http://aids.gov/federal-resources/policies/national-hiv-aids-strategy/nhas-operational-plan-hhs.pdf. Accessed April26, 2011.
ECHPP description Three-year project (Sept. 2010 to Sept. 2013) implemented by health depts in 12 MSAs with highest AIDS burden Project objectives Develop an enhanced plan that aligns the jurisdiction’s prevention and care activities with NHAS Identify/implement optimal combination of prevention, care, and treatment activities Implement activities at appropriate scale Increase targeting of highest risk groups
Priority populations African Americans/ People living with Blacks HIV/AIDS Hispanics/Latinos People at high risk Injection drug users with negative or High-risk unknown HIV status heterosexuals Men who have sex with men
Systems-level M & E ECHPP is a systems approach to HIV prevention Systems-level monitoring and evaluation… Multiple levels Broad assessment of entire entity, as well as some examination of individual components Components of interest can vary widely Breadth vs. depth trade-off New frontier for DHAP evaluators
Systems-level M & E Scaled, combination prevention programs require: Synthesis of multiple data and methods to arrive at overall picture of epidemic Innovative, systems-level evaluation approach An evaluation approach that’s flexible as programs may evolve, change over time Must assess optimal combinations of HIV prevention and continuum of care activities Triangulation of data, using mixed methods and data sources
Process Local process monitoring Measures and objectives are locally-defined Report to CDC site team for program improvement Core process monitoring Standardized measures across jurisdictions Measures were developed in collaboration with grantees Use of program data from other federal agency data sets
Outcome Analyze outcomes from clinical and behavioral surveillance data systems Medical Monitoring Project (MMP) National HIV Behavioral Surveillance (NHBS) (New) MSM Web surveillance project Collect additional outcome data via serial, cross-sectional surveys in six ECHPP cities Two surveys • Clinic survey for HIV-diagnosed ppl in care (modeled after MMP) • Community survey for IDUs and high-risk heterosexuals (modeled after NHBS)
Impact Analyze impact data from HIV surveillance systems: Case surveillance (eHARS) Behavioral and clinical surveillance (MMP, MSM web surveillance) Incidence surveillance Will also look at long-term outcomes from clinical and behavioral surveillance systems to assess impact
Monitoring & evaluation questionsProgram implementationWhich services and programs were provided?Which populations were reached?Were ECHPP (local) objectives met?What were barriers and facilitators to implementation? (atprogram-, agency-, and community-level)Allocation of resourcesHow was health dept funding allocated across ECHPPactivities?How was the funding allocated across ECHPP prioritypopulations?
Monitoring & evaluation questionsOutcomes among HIV-positive individuals:Was there a reduction in HIV risk behaviors?Was there an increase in service access and participation in HIVprevention activities?Was there an increase in positive health outcomes?Outcomes among high-risk, HIV-negative/HIV-unknownindividuals:Was there a reduction in HIV risk behaviors?Was there an increase in service access and participation in HIVprevention activities?
Monitoring & evaluation questionsImpact among priority populations:Was there a reduction in HIV incidence or indicators of HIVrisk?Was there an increase in linkage to, and impact of, preventionand care services for people living with HIV/AIDS?Was there a reduction in HIV-related health disparities?
Analysis and reporting Three-tiered approach that uses existing data systems, as well as new data collections Triangulation across data sources Health dept-reported program data Program data obtained from other federal agencies Population-based, self-report survey data Population-based, surveillance data
Analysis and reporting Descriptive reporting to demonstrate progress toward local objectives, NHAS goals Data synthesis and triangulation Trend analysis Statistical modeling to assess indicators over time Thematic analysis of qualitative and contextual info
ECHPP Evaluation Timeline Revised August 2011 Nov April Oct Oct Oct Oct Oct2007-08 2009 2010 2011 2011 2012 2013 2014 2015 Data types: ECHPP ECHPP Planning Implementation Process (10-10181) (10-10181) ECHPP Implementation Outcome (11-1117) Impact CDC-funded Program Data ECHPP Process Data Collection Other Federal Agency Data (e.g., HRSA, SAMHSA, CMS) Supplemental data collection (6 cities) Community and Community and clinic surveys clinic surveysNHBS- NHBS- NHBS- NHBS- NHBS- NHBS- NHBS- NHBS-MSM IDU HET MSM IDU HET MSM IDU MSM web MSM web MSM web MSM web surveillance surveillance surveillance surveillance MMP MMP MMP MMP MMP MMP Clinic survey Clinic survey HIV HIV HIV HIV HIV HIV HIV HIV case case case case case case case casesurveill. surveill. surveill. surveill. surveill. surveill. surveill. surveill. MSM web MSM web MSM web MSM web surveillance surveillance surveillance surveillance MMP MMP MMP MMP MMP MMP Data Triangulation, Synthesis, and Analysis
ECHPP evaluation “firsts” First time CDC’s DHAP will: Use data from non-CDC-funded HIV prevention/care activities Use national surveillance and program data together to assess impact Strategize and develop protocols related to data sharing across federal agencies (to enhance national monitoring)
ECHPP evaluation “firsts” First time CDC’s DHAP will: Integrate and synthesize information gathered from a variety of data sources to make a broad statement about how/whether public health strategies are working in highest prevalence areas Use a systems-level approach to monitor/evaluate any HIV prevention programs Attempt to link HIV prevention programs to community-level outcomes to community-level impact
Working with health dept staff Created M & E Workgroup To learn about: Reporting capacity across health depts Context within which ECHPP reporting will occur How relationships between health dept staff and their partners could affect data collection/reporting To develop process indicators To get buy-in on overall evaluation design
Working with CDC staff outside program evaluation ECHPP evaluation team comprised of staff with experience in: Program evaluation HIV testing Case and incidence Data management and surveillance analysis Behavioral and clinical ECHPP program surveillance implementation Regular input from senior managers
Working with federal partners Cross-agency Workgroup Data sharing Common metrics One ECHPP goal = Improve national-level coordination
Recent Milestones Awarded contract to estimate extent and availability of data on HIV testing in U.S. Held two-day consultation on systems-level evaluation methods
Consultation Purpose Inform systems-level M & E of national HIV prevention programs and special initiatives Participants Evaluation experts from various program areas at CDC External consultants with expertise in evaluating large and complex public health programs Topics Real-world successes and challenges Indicator measurement, analytic strategies Confounds, covariates, contextual factors
Key Recommendations Identify important/potential covariates that should also be monitored Identify interconnected systems that could contribute to findings Determine how you will distinguish real-world uncertainty from sampling uncertainty Identify intermediate outcomes for interventions like policy changes that may take a long time to be realized
Key Recommendations Be creative with data collection methods Conduct longitudinal and case studies to allow attribution and better link process/outcome /impact Modeling, modeling, and more modeling