1. Systems-level Monitoring and Evaluation
of a Large-scale Combination HIV
Prevention Initiative in 12 U.S. Cities
Holly H. Fisher, Tamika Hoyte, Dale Stratford, and
Gary Uhl
2011 National HIV Prevention Conference
Atlanta, GA
National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention
Division of HIV/AIDS Prevention

2. OVERVIEW

3. Overview
 ECHPP background and rationale
 Monitoring and evaluation plan
 Logic models
 Analysis and reporting
 Collaboration with stakeholders
 Recent milestones
 Questions and discussion

4. ECHPP BACKGROUND AND
RATIONALE

5. National HIV/AIDS Strategy
 July 2010- White House released National
HIV/AIDS Strategy (NHAS)
NHAS Goals:
 Reduce new HIV infections
 Increase linkage to care and overall health
outcomes for persons living with HIV
 Reduce HIV-related health disparities
 Six federal agencies were asked to develop
operational plans that describe how they will
advance NHAS

6. National HIV/AIDS Strategy
 Dec. 2010- HHS operational plan1 is submitted
 Described:
• ongoing efforts to align existing activities with NHAS
• new initiatives that support NHAS goals
 HHS 12-Cities Project- accelerate comprehensive
HIV/AIDS planning and cross-agency response in the
12 U.S. jurisdictions hardest hit by HIV/AIDS
 CDC ’s Enhanced Comprehensive HIV
Prevention Plan (ECHPP) is one of the major
12-Cities Project initiatives
1Department of Health and Human Services. HHS Operational Plan for National HIV/AIDS Strategy. Available at:
http://aids.gov/federal-resources/policies/national-hiv-aids-strategy/nhas-operational-plan-hhs.pdf. Accessed April
26, 2011.

7. *
Enhanced Comprehensive HIV
Prevention Plans (ECHPP) project

8. ECHPP description
 Three-year project (Sept. 2010 to Sept. 2013)
implemented by health depts in 12 MSAs with
highest AIDS burden
 Project objectives
 Develop an enhanced plan that aligns the jurisdiction’s
prevention and care activities with NHAS
 Identify/implement optimal combination of
prevention, care, and treatment activities
 Implement activities at appropriate scale
 Increase targeting of highest risk groups

9. Priority populations
 African Americans/  People living with
Blacks HIV/AIDS
 Hispanics/Latinos  People at high risk
 Injection drug users with negative or
 High-risk unknown HIV status
heterosexuals
 Men who have sex
with men

10. MONITORING AND
EVALUATION PLAN

11. Systems-level M & E
 ECHPP is a systems approach to HIV
prevention
 Systems-level monitoring and evaluation…
 Multiple levels
 Broad assessment of entire entity, as well as some
examination of individual components
 Components of interest can vary widely
 Breadth vs. depth trade-off
 New frontier for DHAP evaluators

12. Systems-level M & E
 Scaled, combination prevention programs
require:
 Synthesis of multiple data and methods to arrive at
overall picture of epidemic
 Innovative, systems-level evaluation approach
 An evaluation approach that’s flexible as programs may
evolve, change over time
 Must assess optimal combinations of HIV
prevention and continuum of care activities
 Triangulation of data, using mixed methods
and data sources

13. Process
 Local process monitoring
 Measures and objectives are locally-defined
 Report to CDC site team for program improvement
 Core process monitoring
 Standardized measures across jurisdictions
 Measures were developed in collaboration with
grantees
 Use of program data from other federal
agency data sets

14. Outcome
 Analyze outcomes from clinical and behavioral
surveillance data systems
 Medical Monitoring Project (MMP)
 National HIV Behavioral Surveillance (NHBS)
 (New) MSM Web surveillance project
 Collect additional outcome data via serial,
cross-sectional surveys in six ECHPP cities
 Two surveys
• Clinic survey for HIV-diagnosed ppl in care (modeled after
MMP)
• Community survey for IDUs and high-risk heterosexuals
(modeled after NHBS)

15. Impact
 Analyze impact data from HIV surveillance
systems:
 Case surveillance (eHARS)
 Behavioral and clinical surveillance (MMP, MSM web
surveillance)
 Incidence surveillance
 Will also look at long-term outcomes from
clinical and behavioral surveillance systems to
assess impact

16. Monitoring & evaluation questions
Program implementation
Which services and programs were provided?
Which populations were reached?
Were ECHPP (local) objectives met?
What were barriers and facilitators to implementation? (at
program-, agency-, and community-level)
Allocation of resources
How was health dept funding allocated across ECHPP
activities?
How was the funding allocated across ECHPP priority
populations?

17. Monitoring & evaluation questions
Outcomes among HIV-positive individuals:
Was there a reduction in HIV risk behaviors?
Was there an increase in service access and participation in HIV
prevention activities?
Was there an increase in positive health outcomes?
Outcomes among high-risk, HIV-negative/HIV-unknown
individuals:
Was there a reduction in HIV risk behaviors?
Was there an increase in service access and participation in HIV
prevention activities?

18. Monitoring & evaluation questions
Impact among priority populations:
Was there a reduction in HIV incidence or indicators of HIV
risk?
Was there an increase in linkage to, and impact of, prevention
and care services for people living with HIV/AIDS?
Was there a reduction in HIV-related health disparities?

19. LOGIC MODELS

20. ANALYSIS AND REPORTING

21. Analysis and reporting
 Three-tiered approach that uses existing data
systems, as well as new data collections
 Triangulation across data sources
 Health dept-reported program data
 Program data obtained from other federal agencies
 Population-based, self-report survey data
 Population-based, surveillance data

22. Analysis and reporting
 Descriptive reporting to demonstrate progress
toward local objectives, NHAS goals
 Data synthesis and triangulation
 Trend analysis
 Statistical modeling to assess indicators over
time
 Thematic analysis of qualitative and
contextual info

23. ECHPP Evaluation Timeline Revised August 2011
Nov April Oct Oct Oct Oct Oct
2007-08 2009 2010 2011 2011 2012 2013 2014 2015
Data types:
ECHPP ECHPP
Planning Implementation
Process
(10-10181) (10-10181)
ECHPP Implementation Outcome
(11-1117)
Impact
CDC-funded Program Data ECHPP Process Data Collection
Other Federal Agency Data (e.g., HRSA, SAMHSA, CMS) Supplemental data collection
(6 cities)
Community and Community and
clinic surveys clinic surveys
NHBS- NHBS- NHBS- NHBS- NHBS- NHBS- NHBS- NHBS-
MSM IDU HET MSM IDU HET MSM IDU
MSM web MSM web MSM web MSM web
surveillance surveillance surveillance surveillance
MMP MMP MMP MMP MMP MMP
Clinic survey Clinic survey
HIV HIV HIV HIV HIV HIV HIV HIV
case case case case case case case case
surveill. surveill. surveill. surveill. surveill. surveill. surveill. surveill.
MSM web MSM web MSM web MSM web
surveillance surveillance surveillance surveillance
MMP MMP MMP MMP MMP MMP
Data Triangulation, Synthesis, and Analysis

24. ECHPP evaluation “firsts”
 First time CDC’s DHAP will:
 Use data from non-CDC-funded HIV prevention/care
activities
 Use national surveillance and program data together
to assess impact
 Strategize and develop protocols related to data
sharing across federal agencies (to enhance national
monitoring)

25. ECHPP evaluation “firsts”
 First time CDC’s DHAP will:
 Integrate and synthesize information gathered from a
variety of data sources to make a broad statement
about how/whether public health strategies are
working in highest prevalence areas
 Use a systems-level approach to monitor/evaluate any
HIV prevention programs
 Attempt to link HIV prevention programs to
community-level outcomes to community-level impact

26. COLLABORATION WITH
STAKEHOLDERS

27. Working with health dept staff
 Created M & E Workgroup
 To learn about:
 Reporting capacity across health depts
 Context within which ECHPP reporting will occur
 How relationships between health dept staff and their
partners could affect data collection/reporting
 To develop process indicators
 To get buy-in on overall evaluation design

28. Working with CDC staff outside program
evaluation
 ECHPP evaluation team comprised of staff
with experience in:
Program evaluation HIV testing
Case and incidence Data management and
surveillance analysis
Behavioral and clinical ECHPP program
surveillance implementation
 Regular input from senior managers

29. Working with federal partners
 Cross-agency Workgroup
 Data sharing
 Common metrics
 One ECHPP goal = Improve national-level
coordination

30. RECENT MILESTONES

31. Recent Milestones
 Awarded contract to estimate extent and
availability of data on HIV testing in U.S.
 Held two-day consultation on systems-level
evaluation methods

32. Consultation
 Purpose
 Inform systems-level M & E of national HIV prevention
programs and special initiatives
 Participants
 Evaluation experts from various program areas at CDC
 External consultants with expertise in evaluating large
and complex public health programs
 Topics
 Real-world successes and challenges
 Indicator measurement, analytic strategies
 Confounds, covariates, contextual factors

33. Key Recommendations
 Identify important/potential covariates that
should also be monitored
 Identify interconnected systems that could
contribute to findings
 Determine how you will distinguish real-world
uncertainty from sampling uncertainty
 Identify intermediate outcomes for
interventions like policy changes that may
take a long time to be realized

34. Key Recommendations
 Be creative with data collection methods
 Conduct longitudinal and case studies to allow
attribution and better link process/outcome
/impact
 Modeling, modeling, and more modeling

35. QUESTIONS AND DISCUSSION

36. For more information, please contact:
Dr. Holly Fisher
hfisher@cdc.gov
404-639-1940