SHADAC: Overview and Evaluation

Uploaded on

Presentation by Lynn Blewett at the SHAP Grantee Meeting in Arlington, VA, Jan 14 2010.

Presentation by Lynn Blewett at the SHAP Grantee Meeting in Arlington, VA, Jan 14 2010.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. SHADAC Overview and Evaluation
    Lynn Blewett, PhD
    State Health Access Data Assistance Center
    University of Minnesota, Minneapolis, MN
    SHAP Grantee Meeting
    January 14, 2010
    Funded by a grant from the Robert Wood Johnson Foundation
  • 2. Overview
    About SHADAC
    Measuring Health Insurance
    • Survey Assistance
    Data Center
    Evaluation Support
    HRSA Benchmark Areas
    Strategic Considerations
  • 3. About SHADAC
  • 4. 4
    The SHADAC Vision
    Bridging the Gap Between Research and Policy
  • 5. 5
    What is SHADAC
    Independent research center located at the University of Minnesota School of Public Health
    Led by an interdisciplinary team of tenured faculty and supported by research fellows and graduate research assistants
    Primary funding from Robert Wood Johnson Foundation
    Additional project-specific funding from CDC, ASPE, CMS, state-specific contracts, etc.
    New funding from HRSA to provide technical assistance to SHAP grantees
  • 6. 6
    SHADAC Objectives
    Support states in their data, survey, policy and evaluation activities
    Help states monitor rates of insurance coverage and understand factors associated with uninsurance
    Provide assistance to states on policy development, program evaluation and assessment
    Provide support to federal agencies related to conducting health insurance surveys
    • Disseminate research findings in a manner that is meaningful to state and national policy-makers
  • 7
    SHADAC’s Core Activities
    Technical Consultation with States
    State survey design and implementation
    Clarify variance between state estimates from different surveys
    Targeted Policy and Evaluation Work
    Evaluation design and implementation
    Policy analysis of coverage options
    Best Practices for Surveys on the Uninsured
    Best way of asking insurance, income, race/ethnicity questions
    Reviewing and improving estimates from national surveys
    Promoting Use of Sound Data and Methods
    Production of issue briefs, webinars, presentations, web content
    Translating research into useful information
  • 7. 8
    State Health Access Reform Evaluation (SHARE)
    National Program of RWJF
    Supports evaluation of state health reform initiatives
    16 single and multi-study projects covering more than 25 states
    Wide variety of topics including insurance market reforms, outreach and enrollment initiatives, Medicaid/CHIP expansions
    Aim to translate this research to inform other states and the national reform debate
  • 8. SHAP Technical Assistance
    Review grantee evaluation plans
    Provide advice on outcome indicators, data sources, data availability, and evaluation methods
    Help states identify data sources for benchmarks
    Provide technical assistance to grantees in:
    Selecting appropriate metrics to allow measurement of progress toward objectives
    Identifying the types and sources of available data
    Assisting in the use of longitudinal data where feasible
    Survey assistance (as previously described)
    Assessing differences between state and federal survey data
    Resources can be found here:
  • 9. Measuring Health Insurance Coverage
  • 10. Measuring Health Insurance Coverage
    Current Population Survey (CPS)
    American Community Survey (ACS)
    State-Specific Household Surveys
  • 11. Current Population Survey (CPS)
    Currently the most commonly used survey for estimating uninsurance rates at the state and federal level
    Nationally representative household based survey
    Large enough sample for state‐level estimates
    Added an insurance verification question in 2000, which improved accuracy
    Used in SCHIP funding formula – this may be changing soon…..
  • 12. American Community Survey (ACS)
    New source of data for health insurance coverage (2008 is first year)
    Eventually replacing the Decennial Census long form
    Phone survey, mail and in-person follow up
    Large enough sample for state‐level and sub‐state estimates
    Cities, counties, political districts and census tracks
  • 13. ACS – Benefits
    Large sample size
    1.94 million households per year in ACS vs. 75,477 households for CPS
    Ability to drill down to geographic areas
    Geographic areas with at least 100,000 people in public use file
    Counties with populations over 65,000 in restricted Census file (smaller counties added later with multi-year avg.)
    More precision on estimating subpopulations by state
    e.g. low-income uninsured children
    Point-in-time health insurance question
  • 14. ACS – Initial Concerns
    Impact of using mail surveys in addition to telephone and in-person interviews
    Only one health insurance question
    None on health status, access
    Disability-related question only
    Does not include state-specific names for Medicaid and SCHIP
    No verification question for health insurance coverage
  • 15. ACS Question: Is this person CURRENTLY covered by any of the following types of health insurance or health coverage plans?
    a. Insurance through a current or former employer or union;
    b. Insurance purchased directly from an insurance company;
    c. Medicare, for people 65 and older;
    d. Medicaid, Medical Assistance, or any kind of government-assistance plan for those with low incomes or a disability
    (e) VA; (f) TRICARE; (g) Indian Health Service
  • 16. ACS – Different Data Source, Different Data
    U.S. Census FactFinder
    Limited age categories (0-17)
    More variables including counties over 65,000
    SHADAC DataCenter uses the Public Use Microdata
    Age (0-17) or (0-18) uninsurance characteristics
    State-level only
    Ease of access but limited variables
  • 17. ASC - Public Use Microdata Samples (PUMS)
    Public use microdata sample (PUMS) is 1% of the U.S. population
    Single-year file for geographic areas with population of 100,000
    Counties with populations 65,000 and over are included in the FactFinder
    PUMS uses a different geographic area called the
    PUMA - Public Use Microdata Area
  • 18. Households - ACS vs. CPS Sample Size Comparison
    Source: U.S. Census Bureau Current Population Survey Annual Social and Economic Supplement, 2008; and 2007 American Community Survey. Sample counts do not include group quarters or vacant housing units.
  • 19. ACS vs CPS - Uninsurance Rates for Adults and Adults <200% FPL
    Source: U.S. Census Bureau 2008 American Community Survey, Public Use Microdata Sample and CPS-ASEC 2009
    Significance test for difference of ACS and CPS * p<.05 **p<.01 ***p<.001
  • 20. ACS vs CPS - Uninsurance Rates for Kids and Kids <200% FPL
    Source: U.S. Census Bureau 2008 American Community Survey, Public Use Microdata Sample and CPS-ASEC 2009
    Significance test for difference of ACS and CPS * p<.05 **p<.01 ***p<.001 (Children = 0 to 18 years of age)
  • 21. What’s a PUMA?
    Unique geographic areas
    Required to have a minimum population of 100,000
    All PUMA areas exceed the established population threshold (65,000), thus insuring that there will be single-year ACS data for them published each year
    PUMAs provide more state geographic coverage but may be new to many users
  • 22. ACS - Wisconsin Uninsurance Estimates by County for Children 0-17*
    * Summary tables from American Fact Finder contain only fixed age categories.
  • 23. ACS - Wisconsin Uninsurance Estimates by PUMAfor Children 0-18*, 2008
    * Analysis using ACS public use microdata allows user-defined age categories.
  • 24. ACS - Wisconsin Uninsurance by PUMAfor Children 0-18 Under 200% FPL, 2008
  • 25. Survey Assistance
  • 26. Survey Assistance
    State survey design and implementation
    Clarify variance between state estimates from different surveys
    Best way of asking insurance, income, race/ethnicity questions
    Assistance to states using SHADAC's Coordinated State Coverage Survey (CSCS), a survey tool for estimating insurance coverage rates in states -
    Online library of state survey tools
  • 27. Survey Assistance - State Surveys in SHAP States
  • 28. 29
    Survey Assistance - Strengths of State Survey Data
    Typically more sample than national data
    Flexibility in adding policy relevant questions
    Ability to over-sample and drill down to subpopulations
    Children, geographic units, race/ethnicity
    Analysts have data in hand
    Ability to do analysis in-house
    Quick turn-around
    Policy development: Simulation of policy options
    Program design and development, marketing and outreach
  • 29. 30
    Survey Assistance - Weaknesses of State Survey Data
    Lack of comparability across states
    Variability in timing of surveys
    Most are telephone surveys – coverage issues due to large cell-phone coverage
    Inconsistency in data documentation
    Cost concerns limits number of variables
    Discrepancies with other data sources (survey and administrative data)
  • 30. Data Center
  • 31. Data Center
    Online table and chart generator
    Designed to help health policy analysts build policy-relevant tables of health insurance coverage estimates.
    Easy to access.
    Easy to use.
    Estimates available from three sources
    CPS, as published by the Census Bureau.
    CPS, enhanced by SHADAC to account for historical changes in methodology.
    ACS, as published by the Census Bureau (coming soon).
    Trended data
    CPS estimates from survey years 1988 to the present.
    Easy to export
  • 32. Data Center – Available Estimates
    Health insurance coverage
    Uninsured, Insured (private, government, and military)
    Counts, percents, standard error
    Table options
    Household income
    Marital status (individual and family)
    Children in household
    Work status (individual and family)
    Education (individual and family)
    Health status (CPS only)
    Citizenship (ACS only)
  • 33. Data Center - Getting to the Data Center
    Click on
    “Data Center”
  • 34. Evaluation Support
  • 35. Evaluation Support - Resources
    Assistance with developing interview guides, focus group protocols, survey instruments
    Review of qualitative data analysis strategies
    Review of logic models
    Stakeholder analysis
    Information about available evaluation resources
  • 36. Evaluation Support - Plan Review
    Recommend additions or revisions,
    Provide advice on outcome indicators
    Work with grantees to identify data sources
    Help determine data availability and data-sharing agreement requirements
    Advise on evaluation methods
  • 37. Evaluation Support - Assist with Evaluation Design
    Assist states in developing an evaluation plan to meet benchmark and reporting requirements 
    Review evaluation plan in relation to policy objectives and access initiatives.
    Identify areas that are not aligned and other gaps in the evaluation plan, including identifying existing data and data still needing to be collected.
  • 38. HRSA Benchmark Areas
  • 39. Selection Criteria
    Responsive to the needs of HRSA
    Common data are available for all SHAP states allowing a fairly standard comparison of outcomes
    Measures are consistent with the existing grantee evaluation plans and not require additional resources
  • 40. Benchmarks
    Rates of health insurance coverage for target populations
    Generated from the American Community Survey (ACS) and CPS
    Program enrollment of target populations and previous insurance status if possible
    Program costs
    Illustration of funding from all sources
    SHADAC will work with states on these benchmarks and other evaluation needs
  • 41. Strategic considerations
  • 42. Data Acquisition
    Successful evaluations depend on good data
    Data acquisition within and across agencies and between entities can be difficult and time consuming
    HIPAA and IRB process can stall process
    Include data agreements in contracts, legislation and discuss early in the process
    Seek [or consider obtaining] legal review and input on state-specific data sharing and data privacy laws
  • 43. Evaluation Timing
    Many important evaluation measures will only come in year 2 and year 3 of the grant
    Think of ways to track early progress through process measures or interim outcome measures
    Determine implementation milestones and document these to show progress
  • 44. Mid-Course Changes
    Program implementation and evaluation results should be interconnected
    Share evaluation results across the project team
    Use results from the evaluation to inform ongoing project changes, improvements
  • 45. Discussion Questions
    Describe top 3 SHAP plan objectives
    Define target populations
    Discuss 2-3 components of evaluation (hopefully related to #1 and #2)
    Identify challenges – those you have encountered or those you anticipate
  • 46. 47
    Contact Information
    Minnesota SHAP Project Team:
    Lynn Blewett, Ph.D.
    Kelli Johnson, MBA
    Elizabeth Lukanen, MPH
    * Primary contact –, 612-626-1537
    State Health Access Data Assistance Center
    University of Minnesota, Minneapolis, MN
    ©2002-2009 Regents of the University of Minnesota. All rights reserved.The University of Minnesota is an Equal Opportunity Employer