1. Funded by a grant from the Robert Wood Johnson Foundation
Thinking Ahead – Monitoring the Impact
of Health Reform
Elizabeth Lukanen, MPH
SHADAC, University of Minnesota
National Association of Medicaid Directors (NAMD)
Spring Meeting
Denver, Colorado
May 21, 2013
2. Presentation Overview
• Why should states develop a monitoring
framework (and why should Medicaid be
involved)?
• Steps to develop a monitoring or evaluation
framework
• State examples
2
4. 2015 and Beyond
4
Everyone will be clamoring for data
and analysis on the impact of health
reform
• States will be looking to report on “early
wins”
• Policymakers and operational staff will
need information to make ongoing
implementation decisions
• Heated debate is likely to continue and both sides will be
looking for information on the impact
• The media will be looking for ANY story
• The public and key stakeholders will want a progress
report
5. Objectives for Generating a
Monitoring/Evaluation Framework
• Encourages agreement on goals,
priorities, and how progress will be
measured
• Defines how each component of
reform (e.g., Medicaid, exchange)
contributes to those goals
• Establishes program/agency
collaboration to focus on the “big
picture”
• Avoids duplication of data collection
and provides consistency in
measurement
5
• Provides opportunity to select lead agency or individual accountable for
monitoring efforts
• Prepares state staff to respond to future questions from policymakers
6. 6
o National surveys and analyses are a great, especially
when cross-state comparisons are important, but…
• Each state will be unique in how it implements the ACA
• State-led efforts will track progress toward state
priorities
• States often have richer data to examine questions in-
depth
Why Should Monitoring Efforts be State-
Led?
Why not just rely on
national studies or 50-
state analyses from
other sources?
7. Why Should Medicaid Play a Role?
7
• Medicaid is “where it’s at”
– Even if you don’t plan for it, you will likely engage in
evaluation/monitoring work
• Many key evaluation measures will rely on
Medicaid data
– Assure consistency in reporting
– Avoid duplication of data collection and analysis
– Reduce analyst burden
• Define what it means to be successful
• Contribute to and be aware of the
messaging regarding impact of reform
8. Why Now?
• Define in advance what is important to measure –
helps identify successes and problem areas
• Establish a baseline prior to reform implementation
• Identify gaps in available data and ways to fill the
gaps
– Take advantage of opportunities to “build in” to new data
systems
• Stay ahead of “story”
8
Why can’t I focus on
implementation now and deal
with evaluation later?
9. Evaluation and Monitoring
Framework Development
Define scope
Choose and
operationalize measures
Select appropriate data
and identify data gaps
Setting benchmarks and
goals (or not)
Stakeholder engagement
9
10. Defining Scope
10
• Set focus
– Medicaid only, all health reform activities (state and federal?)
• Need to keep the number of topic areas manageable
– Access, cost, public health, impact on providers
• What are you trying to achieve?
– High Medicaid participation rates; good enrollee experience,
reduced uninsurance; low rate of coverage gaps
• What issues are policymakers most concerned about?
– Churn, continuity of coverage, provider capacity to care for newly
uninsured;
• Who is the audience?
11. Choosing Measures
• Keep the number of measures
manageable - prioritize
• Choose measures that are directly
related to policy goals and levers
• Think about near-/medium-/long-term
impacts and include some measures for
each
• Include some measures that might be
“early success signs” or “early warning
signs”
• Consider feasibility - existing data vs.
possibility of collecting new data
11
12. Operationalize the Measure
• Create a working definition or preferred method for
calculating the measure
– e.g., how do you calculate churn?
• Defining the “universe”
– e.g., population-wide? exchange
vs. total market?
• Specify the level of detail you want to capture
– e.g., disenrollment or disenrollment by reason
12
13. Select Appropriate Data
1. Conduct a data scan
2. Assess data against
a defined set of
criteria
3. Identify gaps
4. Prioritize ways of
filling gaps
13
14. Setting Benchmarks and Goals (or not)
• Possible benchmarks
– Change over time
– Defined ideal
– Other states
– National average
• The most useful goals are
– Realistic
– Specific
– Connected to specific actions/strategies and policy priorities
• Decisions will influence choices about data sources
• Consensus around goals and benchmarks can be
challenging
14
15. Stakeholder Engagement
• “Stakeholder” can be defined narrowly
or broadly
• Stakeholders can be engaged at any
point in the process
• Best to present stakeholders with
something to react to
• Need clear boundaries on scope and
purpose
15
16. California - Approach
• Led by the California HealthCare Foundation (work done by SHADAC)
• Development of a set of measures to monitor over time
• Geared toward public
• Focused on the ACA but limited to 3 topic areas:
1. Health insurance coverage (section on public coverage)
2. Affordability and comprehensive of coverage
3. Access to care
• Considerations for measures selection
– Measures that reflect major goals and provisions of the ACA
– Outcomes rather than implementation process
– Relevant/meaningful to policymakers
– Interest in measures available at a sub-state level
– Data availability
• Stakeholders engaged after draft list of measures was developed
http://www.shadac.org/publications/framework-tracking-impacts-affordable-care-
act-in-california
16
17. California - Coverage Measures
17
Uninsured Public Coverage Employer Coverage
Distribution of Insurance Coverage
Health Insurance
Exchange
Point in time
Enrollment as Share of
Nongroup Market
Employer participation
Employees in firms
that offer
% Eligible
Enrollment trend
Employers paying
penalty
Participation rate
Churning
Uninsured for a year
or longer
Uninsured at some
point in past year
Reasons for
uninsurance
Exempt from mandate
Paying penalty
Employers offering
Families with ESI
offer
% Enrolled
All family
members enrolled
18. California - Affordability &
Comprehensiveness of Coverage
Measures
18
Insurance Premiums
Subsidies
Comprehensiveness Financial Burden
% of families with high
cost burden
“Affordable” premium
as % of income
Employer coverage
Total premium
Employee share
Single
Family
Nongroup coverage
Per enrollee
Enrollment by benefit
level
ESI
Nongroup
Deductibles
ESI: single, family
Nongroup: single,
family
Single
Family
# receiving premium
and cost sharing
subsidies in exchange
Average value of
subsidies
19. California - Access to Care Measure
19
Individuals System
Use of services Barriers to care
Has usual
source of care
Did not get
necessary care
(& reasons)
Preventable/
avoidable ER visits
Safety net
Volume and type of
services provided
by safety net clinics
Uncompensated
care
Type of place
for usual source
of care
% of physicians
participating in
public programs
Difficulty finding
provider that
accepts
insurance type
Difficulty finding
provider to take
new patients
Not able to get
timely
appointment
Any doctor visit
in past year
Preventive care
visit in past year
County indigent
care volume and
cost
Ambulatory care
sensitive hospital
admissions
Emergency room
visit rate
% of physicians
accepting new
patients, by payer
20. Maryland - Approach
• Led by the Maryland Health Connection (work done by
SHADAC)
• Development of a set of measures to monitor over time
• Geared toward policy makers and the public
• Focused on the exchange and limited to 5 core
measurement categories:
– Affordability
– Access (includes seamless and non-seamless coverage
transitions)
– Consumer Satisfaction
– Stability
– Health Equity
20
21. Maryland Approach - Continued
• Considerations for measures selection
– Drawn from data currently produced by other state agencies, data
currently collected or analyzed by other state agencies or
generated through exchange
– Highly prioritized, no more than 10 measures in each category
• Exchange board developed measurement categories and
gave feedback throughout the selection of measures
• Public comment period after draft list of measures was
developed
http://marylandhbe.com/wp-
content/uploads/2012/12/Performance-Management.pdf
21
23. Too Daunting? Leverage Available
Resources!
• Leverage federal funding
• Let another agency or division take the lead
– Just make sure to stay engaged
• Consider outside partners to consult on or lead these
efforts
– State universities
– Evaluation consultants
– Local foundations
• No need to remake the whee1
– Look at monitoring/evaluation schemes developed by other states
(ask your NAMD collogues!)
– Utilize data you current collect and use for other purposes (e.g.,
operations, reporting)
23
24. Sign up to receive our newsletter and updates at
www.shadac.org
@shada
c
Contact Information
Elizabeth Lukanen
Senior Research Fellow
elukanen@umn.edu
612.626.1537
Editor's Notes
Framework can be A broad framework helps focus attention on:Big picture goalsHow each component (e.g., Medicaid, exchange) contributes to those goalsOutcomes, not just processesWhere it makes sense, ensure that individual programs are collecting data and measuring consistentlySelect a lead agency or person to be accountable
Scope: Selected your domainsFocus: articulated your policy goalsMeasure: determined the mechanisms for achieving policy goals, keeping in mind near/medium/long termLevel of measurement: Given some thought to the level of measurement – system/population, subpopulation geography, employer size, health provider tpe
overall enrollment, completed applications, transfers from CHIP to Medicaid, spells of enrollment, retention rates, new applicants versus re-entries, reasons for denials, and reasons for disenrollment
Churn: number of program disenrollees in a given month who later reenroll in the program following a gap in coverage of one to six months. (Chris Trenholm)Coverage from admin data – what time frame? Who is excluded (elderly)? Point in time?**Don’t need to remake wheel, use exiting definitions (other data frameworks), SHADAC can helpCoverage: uninsured? Public program – all or aggregate?System/populationSpecific population groups – e.g., age, income, health insurance coverage type, race/ethnicity, immigration statusGeographyEmployer sizeHealth care provider type – e.g., safety net providers
Start with list of priority measures, then do data scanOtherwise:might miss key measures of interest highlight less important measure just because you have the data.Do a lot of work assessing data sources on topics that don’t make the cutSurvey dataAdministrative dataData from health carriers, hospitals, providersOther?Level of Geography Subpopulation analysis , Available benchmarks, Timeliness, Ability to trend, Breadth and depth of topics Methodology, Ease of use and procurementConsider ways of collecting additional data through existing collection efforts : Existing state surveysProvider licensure process State tax returnIdentify data that might come out of new systems/processes Enlist outside support (e.g., state foundations)
51 measuresVery focused on IDing data gapsGoalsInform stakeholders Help CHCF prioritize next steps and resources for filling data gapsBuild coalitions and momentum to move process forwardApproach6 structured group discussions over 3 daysProfessional facilitatorRange of invited participants: advocates, providers, safety net, legislative staff, state and county government, insurers, researchers, foundationsTried to keep groups of “like minded” together
Coverage, continued from previous slide:Health insurance exchange: The insurance exchange plays a key role in the ACA’s coverage reforms, both as a vehicle for subsidies and as a means of organizing the market and making it easier for individuals and employers to shop for coverage. To understand how well the exchanges are performing these functions, trends in enrollment should be tracked over time (both in total and as a percentage of the individual and relevant employer markets, and separately for subsidized vs non-subsidized coverage).AAt a high level, the major gaps in data related to health insurance coverage relate to compliance with the Affordable Care Act’s individual mandate and employer requirements, and to the health insurance exchange. These are the sort of gaps that represent fundamental changes that are specific to the ACA and that can’t be filled until 2014. However, it’s important to be planning now for how to collect this information in a way that will be reliable and useful. REFERENCE TO LYNN’S PRESENTAITON ON REPORTING REQUIREMENTS UNDER THE ACA?One thing to note is that gaps for these recommended measures will of course look very different in differnet states. CA has a pretty rich existing data infrastructure to draw from, and this won’t be the case in all states. For example, we recommend the CHIS and CEHBS in CA for many measures, but in states that don’t conduct their own state specific population and employer based surveys other alternatives, such as the NHIS and MEPS-IC would have to be considered, and in some cases, such as the NHIS in smaller states, data sources may not be available at all.
With regard to affordability and comprehensiveness, there are significant gaps in the information available now, especially with regard to the nongroup health insurance market. We know very little right now about premiums, enrollment, and what kinds of products are being purchased in this market. We know more about the market for employer-sponsored coverage, but will need to track comprehensiveness of coverage in new ways once the exchanges are implemented. And finally, it will also be important to track premium and cost sharing subsidies that are provided through the exchanges beginning in 2014.
With regard to individuals’ access to care, the main gap related to use of services relates to the percentage of people who have preventive care visits; but there are several important gaps related to barriers to care that individuals experience. Specifically, these relate to people being able to get appointments in a timely way, having difficulty finding a provider that will accept new patients, and having difficulty finding a provider that accepts their insurance; it’s probably a good idea to track these measures separately for primary care and specialty care. Key gaps in the information that will be important to track access to care at a system level include the percentage of physicians who are accepting new patients, and the percentage that participate in Medi-Cal and Healthy Families. Again, tracking these measures separately for primary care and specialty care will be useful. And finally, the gaps in information related to the safety net relate mainly to the fact that the safety net system is pretty fragmented, with no data sources that provide a picture of the system as a whole. Many of the California data experts that we talked to mentioned the lack of information from county clinics as a very important gap here, since these clinics play a crucial role in delivering safety net services. There are also gaps in available information about the volume and cost of indigent care provided by counties. And finally, there are also some gaps in the data for hospitals – because not all hospitals report their financial information the same way, the picture of uncompensated care at the state level is currently incomplete.Transition to Barbara
25 measure's in totalState staff drivenDomains were generated based on the state goals of the exchange
Health Equity Number of individuals that attempt to obtain coverage through exchange (Access)Distribution of insurance status (Access)Number of individuals receiving premium subsidies (Affordability)Number of individuals receiving cost sharing subsidies (Affordability)Percentage of adults who cannot afford a doctor visit (Affordability)Percent of families with high cost burden (Affordability)Composite measure of satisfaction (Consumer Satisfaction)
