Systems-level Monitoring and Evaluation   of a Large-scale Combination HIV  Prevention Initiative in 12 U.S. Cities   Holl...
OVERVIEW
Overview   ECHPP background and rationale   Monitoring and evaluation plan   Logic models   Analysis and reporting   ...
ECHPP BACKGROUND ANDRATIONALE
National HIV/AIDS Strategy   July 2010- White House released National    HIV/AIDS Strategy (NHAS)        NHAS Goals:     ...
National HIV/AIDS Strategy    Dec. 2010- HHS operational plan1 is submitted        Described:               • ongoing ef...
* Enhanced Comprehensive HIVPrevention Plans (ECHPP) project
ECHPP description   Three-year project (Sept. 2010 to Sept. 2013)    implemented by health depts in 12 MSAs with    highe...
Priority populations   African Americans/        People living with    Blacks                     HIV/AIDS   Hispanics/...
MONITORING ANDEVALUATION PLAN
Systems-level M & E   ECHPP is a systems approach to HIV    prevention   Systems-level monitoring and evaluation…     M...
Systems-level M & E   Scaled, combination prevention programs    require:     Synthesis of multiple data and methods to ...
Process   Local process monitoring     Measures and objectives are locally-defined     Report to CDC site team for prog...
Outcome   Analyze outcomes from clinical and behavioral    surveillance data systems     Medical Monitoring Project (MMP...
Impact   Analyze impact data from HIV surveillance    systems:     Case surveillance (eHARS)     Behavioral and clinica...
Monitoring & evaluation questionsProgram implementationWhich services and programs were provided?Which populations were re...
Monitoring & evaluation questionsOutcomes among HIV-positive individuals:Was there a reduction in HIV risk behaviors?Was t...
Monitoring & evaluation questionsImpact among priority populations:Was there a reduction in HIV incidence or indicators of...
LOGIC MODELS
ANALYSIS AND REPORTING
Analysis and reporting   Three-tiered approach that uses existing data    systems, as well as new data collections   Tri...
Analysis and reporting   Descriptive reporting to demonstrate progress    toward local objectives, NHAS goals   Data syn...
ECHPP Evaluation Timeline                                                                     Revised August 2011         ...
ECHPP evaluation “firsts”   First time CDC’s DHAP will:     Use data from non-CDC-funded HIV prevention/care      activi...
ECHPP evaluation “firsts”   First time CDC’s DHAP will:     Integrate and synthesize information gathered from a      va...
COLLABORATION WITHSTAKEHOLDERS
Working with health dept staff   Created M & E Workgroup   To learn about:     Reporting capacity across health depts  ...
Working with CDC staff outside program              evaluation   ECHPP evaluation team comprised of staff    with experie...
Working with federal partners   Cross-agency Workgroup   Data sharing   Common metrics   One ECHPP goal = Improve nati...
RECENT MILESTONES
Recent Milestones   Awarded contract to estimate extent and    availability of data on HIV testing in U.S.   Held two-da...
Consultation   Purpose     Inform systems-level M & E of national HIV prevention      programs and special initiatives ...
Key Recommendations   Identify important/potential covariates that    should also be monitored   Identify interconnected...
Key Recommendations   Be creative with data collection methods   Conduct longitudinal and case studies to allow    attri...
QUESTIONS AND DISCUSSION
For more information, please contact:         Dr. Holly Fisher         hfisher@cdc.gov         404-639-1940
Upcoming SlideShare
Loading in …5
×

Systems-level Monitoring and Evaluation of a Large-scale Combination HIV Prevention Initiative in 12 U.S. Cities

553 views

Published on

Published in: Health & Medicine
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
553
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Systems-level Monitoring and Evaluation of a Large-scale Combination HIV Prevention Initiative in 12 U.S. Cities

  1. 1. Systems-level Monitoring and Evaluation of a Large-scale Combination HIV Prevention Initiative in 12 U.S. Cities Holly H. Fisher, Tamika Hoyte, Dale Stratford, and Gary Uhl 2011 National HIV Prevention Conference Atlanta, GA National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention Division of HIV/AIDS Prevention
  2. 2. OVERVIEW
  3. 3. Overview ECHPP background and rationale Monitoring and evaluation plan Logic models Analysis and reporting Collaboration with stakeholders Recent milestones Questions and discussion
  4. 4. ECHPP BACKGROUND ANDRATIONALE
  5. 5. National HIV/AIDS Strategy July 2010- White House released National HIV/AIDS Strategy (NHAS) NHAS Goals:  Reduce new HIV infections  Increase linkage to care and overall health outcomes for persons living with HIV  Reduce HIV-related health disparities Six federal agencies were asked to develop operational plans that describe how they will advance NHAS
  6. 6. National HIV/AIDS Strategy Dec. 2010- HHS operational plan1 is submitted  Described: • ongoing efforts to align existing activities with NHAS • new initiatives that support NHAS goals  HHS 12-Cities Project- accelerate comprehensive HIV/AIDS planning and cross-agency response in the 12 U.S. jurisdictions hardest hit by HIV/AIDS CDC ’s Enhanced Comprehensive HIV Prevention Plan (ECHPP) is one of the major 12-Cities Project initiatives1Department of Health and Human Services. HHS Operational Plan for National HIV/AIDS Strategy. Available at:http://aids.gov/federal-resources/policies/national-hiv-aids-strategy/nhas-operational-plan-hhs.pdf. Accessed April26, 2011.
  7. 7. * Enhanced Comprehensive HIVPrevention Plans (ECHPP) project
  8. 8. ECHPP description Three-year project (Sept. 2010 to Sept. 2013) implemented by health depts in 12 MSAs with highest AIDS burden Project objectives  Develop an enhanced plan that aligns the jurisdiction’s prevention and care activities with NHAS  Identify/implement optimal combination of prevention, care, and treatment activities  Implement activities at appropriate scale  Increase targeting of highest risk groups
  9. 9. Priority populations African Americans/  People living with Blacks HIV/AIDS Hispanics/Latinos  People at high risk Injection drug users with negative or High-risk unknown HIV status heterosexuals Men who have sex with men
  10. 10. MONITORING ANDEVALUATION PLAN
  11. 11. Systems-level M & E ECHPP is a systems approach to HIV prevention Systems-level monitoring and evaluation…  Multiple levels  Broad assessment of entire entity, as well as some examination of individual components  Components of interest can vary widely  Breadth vs. depth trade-off New frontier for DHAP evaluators
  12. 12. Systems-level M & E Scaled, combination prevention programs require:  Synthesis of multiple data and methods to arrive at overall picture of epidemic  Innovative, systems-level evaluation approach  An evaluation approach that’s flexible as programs may evolve, change over time Must assess optimal combinations of HIV prevention and continuum of care activities Triangulation of data, using mixed methods and data sources
  13. 13. Process Local process monitoring  Measures and objectives are locally-defined  Report to CDC site team for program improvement Core process monitoring  Standardized measures across jurisdictions  Measures were developed in collaboration with grantees Use of program data from other federal agency data sets
  14. 14. Outcome Analyze outcomes from clinical and behavioral surveillance data systems  Medical Monitoring Project (MMP)  National HIV Behavioral Surveillance (NHBS)  (New) MSM Web surveillance project Collect additional outcome data via serial, cross-sectional surveys in six ECHPP cities  Two surveys • Clinic survey for HIV-diagnosed ppl in care (modeled after MMP) • Community survey for IDUs and high-risk heterosexuals (modeled after NHBS)
  15. 15. Impact Analyze impact data from HIV surveillance systems:  Case surveillance (eHARS)  Behavioral and clinical surveillance (MMP, MSM web surveillance)  Incidence surveillance Will also look at long-term outcomes from clinical and behavioral surveillance systems to assess impact
  16. 16. Monitoring & evaluation questionsProgram implementationWhich services and programs were provided?Which populations were reached?Were ECHPP (local) objectives met?What were barriers and facilitators to implementation? (atprogram-, agency-, and community-level)Allocation of resourcesHow was health dept funding allocated across ECHPPactivities?How was the funding allocated across ECHPP prioritypopulations?
  17. 17. Monitoring & evaluation questionsOutcomes among HIV-positive individuals:Was there a reduction in HIV risk behaviors?Was there an increase in service access and participation in HIVprevention activities?Was there an increase in positive health outcomes?Outcomes among high-risk, HIV-negative/HIV-unknownindividuals:Was there a reduction in HIV risk behaviors?Was there an increase in service access and participation in HIVprevention activities?
  18. 18. Monitoring & evaluation questionsImpact among priority populations:Was there a reduction in HIV incidence or indicators of HIVrisk?Was there an increase in linkage to, and impact of, preventionand care services for people living with HIV/AIDS?Was there a reduction in HIV-related health disparities?
  19. 19. LOGIC MODELS
  20. 20. ANALYSIS AND REPORTING
  21. 21. Analysis and reporting Three-tiered approach that uses existing data systems, as well as new data collections Triangulation across data sources  Health dept-reported program data  Program data obtained from other federal agencies  Population-based, self-report survey data  Population-based, surveillance data
  22. 22. Analysis and reporting Descriptive reporting to demonstrate progress toward local objectives, NHAS goals Data synthesis and triangulation Trend analysis Statistical modeling to assess indicators over time Thematic analysis of qualitative and contextual info
  23. 23. ECHPP Evaluation Timeline Revised August 2011 Nov April Oct Oct Oct Oct Oct2007-08 2009 2010 2011 2011 2012 2013 2014 2015 Data types: ECHPP ECHPP Planning Implementation Process (10-10181) (10-10181) ECHPP Implementation Outcome (11-1117) Impact CDC-funded Program Data ECHPP Process Data Collection Other Federal Agency Data (e.g., HRSA, SAMHSA, CMS) Supplemental data collection (6 cities) Community and Community and clinic surveys clinic surveysNHBS- NHBS- NHBS- NHBS- NHBS- NHBS- NHBS- NHBS-MSM IDU HET MSM IDU HET MSM IDU MSM web MSM web MSM web MSM web surveillance surveillance surveillance surveillance MMP MMP MMP MMP MMP MMP Clinic survey Clinic survey HIV HIV HIV HIV HIV HIV HIV HIV case case case case case case case casesurveill. surveill. surveill. surveill. surveill. surveill. surveill. surveill. MSM web MSM web MSM web MSM web surveillance surveillance surveillance surveillance MMP MMP MMP MMP MMP MMP Data Triangulation, Synthesis, and Analysis
  24. 24. ECHPP evaluation “firsts” First time CDC’s DHAP will:  Use data from non-CDC-funded HIV prevention/care activities  Use national surveillance and program data together to assess impact  Strategize and develop protocols related to data sharing across federal agencies (to enhance national monitoring)
  25. 25. ECHPP evaluation “firsts” First time CDC’s DHAP will:  Integrate and synthesize information gathered from a variety of data sources to make a broad statement about how/whether public health strategies are working in highest prevalence areas  Use a systems-level approach to monitor/evaluate any HIV prevention programs  Attempt to link HIV prevention programs to community-level outcomes to community-level impact
  26. 26. COLLABORATION WITHSTAKEHOLDERS
  27. 27. Working with health dept staff Created M & E Workgroup To learn about:  Reporting capacity across health depts  Context within which ECHPP reporting will occur  How relationships between health dept staff and their partners could affect data collection/reporting To develop process indicators To get buy-in on overall evaluation design
  28. 28. Working with CDC staff outside program evaluation ECHPP evaluation team comprised of staff with experience in: Program evaluation HIV testing Case and incidence Data management and surveillance analysis Behavioral and clinical ECHPP program surveillance implementation Regular input from senior managers
  29. 29. Working with federal partners Cross-agency Workgroup Data sharing Common metrics One ECHPP goal = Improve national-level coordination
  30. 30. RECENT MILESTONES
  31. 31. Recent Milestones Awarded contract to estimate extent and availability of data on HIV testing in U.S. Held two-day consultation on systems-level evaluation methods
  32. 32. Consultation Purpose  Inform systems-level M & E of national HIV prevention programs and special initiatives Participants  Evaluation experts from various program areas at CDC  External consultants with expertise in evaluating large and complex public health programs Topics  Real-world successes and challenges  Indicator measurement, analytic strategies  Confounds, covariates, contextual factors
  33. 33. Key Recommendations Identify important/potential covariates that should also be monitored Identify interconnected systems that could contribute to findings Determine how you will distinguish real-world uncertainty from sampling uncertainty Identify intermediate outcomes for interventions like policy changes that may take a long time to be realized
  34. 34. Key Recommendations Be creative with data collection methods Conduct longitudinal and case studies to allow attribution and better link process/outcome /impact Modeling, modeling, and more modeling
  35. 35. QUESTIONS AND DISCUSSION
  36. 36. For more information, please contact: Dr. Holly Fisher hfisher@cdc.gov 404-639-1940

×