(1) A survey experiment tested different question formats for accurately measuring health insurance coverage and costs. (2) Respondents reported annual deductibles with less "don't know" responses and more confidence when questions included dollar amount ranges rather than open-ended formats. (3) For other costs like out-of-pocket spending and debt, there was no significant difference between question formats, but many respondents felt ranges would improve confidence.

1. CAN WE IMPROVE THE WAY WE ASK ABOUT
HEALTH INSURANCE COVERAGE AND HEALTH
CARE SPENDING?
FI N DING S FROM AN E XPE RIME NT
Susan Sherr & Kathy Langdale | SSRS
Sharon Long | Urban Institute
Zi Zhang & Huong Trieu | CHIA

2. MOTIVATION FOR RESEARCH
• Surveys investigating health insurance coverage often ask respondents to report on the characteristics of
their coverage, including required cost-sharing, which can be difficult to report accurately
• Respondents may be asked about the amount of their insurance premiums, annual deductibles, out-of-
pocket spending and outstanding medical debt
• These can be difficult concepts to ask about in a survey:
− Health insurance is complicated, with terminology that isn’t always easy to understand
− Even if respondents understand the question, they may not know the details of their coverage,
especially in surveys that can’t provide time to look it up
• Nonetheless, accurate estimates are needed in order to evaluate the quality of health insurance
coverage, including the extent to which insurance protects consumers from high health care costs and
underinsurance
2@ssrs_solutions | @urbaninstitute | @Mass_CHIA

3. SURVEY METHODOLOGY
• At the end of the 2015 Massachusetts Health Insurance Survey (MHIS), an RDD telephone survey of
5,000 Massachusetts residents sponsored by the Commonwealth of Massachusetts’ Center for Health
Information and Analysis (CHIA), respondents were asked if they would be willing to be recontacted at a
later date.
• In 2016, on behalf of CHIA and in partnership with the Urban Institute, SSRS recontacted those who said
yes in order to evaluate question formats for use in the 2017 MHIS that can better measure statewide
underinsurance
− Experiments were included to test different formats
− Respondents were asked to report on their own levels of confidence in responding to questions about
their health insurance coverage and health care spending
3@ssrs_solutions | @urbaninstitute | @Mass_CHIA

4. SURVEY METHODOLOGY
• Respondents who had insurance for all of the past twelve months (n=752) were randomly assigned to be
asked questions about the amounts of their deductibles and out-of-pocket spending in either an open-
ended format (n=374) or in dollar amount ranges (n=378)
− Regression analysis was used to determine whether there were statistically significant differences in
confidence levels and nonresponse between these two formats
• Because of space limits in the survey, for premiums and unpaid healthcare bills, respondents were asked
only to report dollar amounts verbatim, but responded to a follow-up question about whether they would
have felt more confident in their response if they were asked to respond based on ranges
4@ssrs_solutions | @urbaninstitute | @Mass_CHIA

5. EXAMPLES OF QUESTIONS:
OUT-OF-POCKET SPENDING
A1. Now, I'd like to ask about "out of pocket" spending for health care.
"Out of pocket" is the amount of money paid for care that is not covered by any insurance or special
assistance that (you/TARGET) might have. It does not include any premiums that are paid for health
insurance or any health care costs that will be reimbursed. Please include costs for prescription medicines,
dental and vision care, and all other medical expenses, including for doctors, hospitals, tests and
equipment. Please include co-pays in your estimate. Finally, please include costs that are covered by a
flexible spending account.
What is the dollar amount of “out of pocket” spending on health care over the past 12 months for
(yourself/TARGET)?
5@ssrs_solutions | @urbaninstitute | @Mass_CHIA

6. EXAMPLES OF QUESTIONS
(ASK HALF SAMPLE A)
(IF NECESSARY: Just your best guess is fine.)
_____________ENTER DOLLAR AMOUNT
($1-$9,997)
0 (DO NOT READ) Zero
7 (DO NOT READ) Did not use care
8 (DO NOT READ) Don’t know
9 (DO NOT READ) Refused
6@ssrs_solutions | @urbaninstitute | @Mass_CHIA
(ASK HALF SAMPLE B)
Was it....?
(IF NECESSARY: Just your best guess is fine.)
(READ LIST. ENTER ONE ONLY)
0 Zero
1 $1 to less than $200
2 $200 to under $500
3 $500 to under $1,000
4 $1,000 to under $3,000
5 $3,000 to under $5,000
6 $5,000 or more
7 (DO NOT READ) Did not use care
8 (DO NOT READ) Don’t know
9 (DO NOT READ) Refused
OPEN-ENDED FORMAT RANGES

7. EXAMPLES OF QUESTIONS:
ANNUAL DEDUCTIBLE
7@ssrs_solutions | @urbaninstitute | @Mass_CHIA
A15. A deductible is the amount of
expenses that must be paid out of
pocket before an insurer will pay health
care expenses. What is the dollar
amount of (your/TARGET’s) yearly
health insurance deductible?

8. ONLY REPORTING ON ANNUAL DEDUCTIBLE SHOWED A CONSISTENT &
SIGNIFICANT DIFFERENCE BETWEEN THE TWO QUESTION FORMATS
8@ssrs_solutions | @urbaninstitute | @Mass_CHIA
35.2
29.6
Verbatim Ranges
Percent Reporting No Annual
Deductible With Current Insurance
37.3
51.7
Verbatim Ranges
Percent Reporting Any Amount Greater
Than Zero for Annual Deductible

9. THOSE REPORTING ANNUAL DEDUCTIBLE IN RANGES ARE LESS LIKELY TO
SAY DON’T KNOW & MORE LIKELY TO BE CONFIDENT IN THEIR RESPONSES
9@ssrs_solutions | @urbaninstitute | @Mass_CHIA
27.6
18.6
Verbatim Ranges
Percent Reporting Don't Know For
Amount of Annual Deductible
32
21.5
Verbatim Ranges
Percent Who Were Not Confident in
Their Deductible Estimate or Who Said
D/K or Refused

10. THERE WERE NO SIGNIFICANT DIFFERENCES
FOR OTHER HEALTH CARE COSTS
• Differences in estimates of the amounts out-of-pocket spending over the past 12 months and
maximum out-of-pocket spending under current health plan are not statistically significant
although they are in the direction of more non-zero responses in the group whose questions
were presented in ranges.
• Differences in confidence in reporting were also not significant for these measures and were not
consistently in the direction of more confidence in the range group.
• Nonresponse combined with lack of confidence was virtually the same between the two groups.
10@ssrs_solutions | @urbaninstitute | @Mass_CHIA

11. 42.1
52.1
5.8
0.0
Premiums
(n=169)
More confident No change in confidence Don't know Refused
THOSE LESS THAN VERY CONFIDENT IN REPORTING DOLLAR AMOUNTS IN OPEN-
END FORMAT REPORTED HOW CONFIDENT THEY WOULD BE IF ASKED IN RANGES
11@ssrs_solutions | @urbaninstitute | @Mass_CHIA
59.1
38.0
2.9
Unpaid Health Care Bills
(N=47)
42.6
57.4
0.0
Deductible
(n=79)

12. CONCLUSIONS
• There is evidence to suggest that presenting dollar amounts in ranges can, with certain types of
spending, provide more accurate estimates from the perspective of the respondent and less nonresponse
• None of the experiments lead to the conclusion that open-ended questions about costs and spending are
preferable
• In each case, at least 40% of those who were not very confident in their estimated costs when reporting
verbatim thought they would be more confident if asked to report in ranges
• Providing the option of reporting in a range format should result in less nonresponse and provide better
estimates than verbatim options only
12@ssrs_solutions | @urbaninstitute | @Mass_CHIA

13. CONTACT US
SUSAN SHERR
SSHERR@SSRS.COM
484-840-4931
@SHERRSUSAN