This document discusses adding cell phone interviews to an existing long-term consumer sentiment study. A test was conducted adding 75 weekly cell phone interviews and 8 Spanish interviews. Results showed no significant differences in overall consumer sentiment index estimates between the traditional landline-only sample and mixed landline/cell phone sample. However, the mixed sample better represented the population demographically. Based on these findings, the document concludes it is time to fully transition the study to a mixed landline/cell phone methodology.

1. Adding Cell Phone Sampling to a
Long-Term Trend Study (Cautiously):
The Bloomberg Consumer Comfort Index
Julie Phelan & Gary Langer
Langer Research Associates
@LangerResearch

2. The CCI
• Continuous weekly tracking of consumer sentiment
since December 1985
• N=250/week summed in a four-week rolling avg.
• Computed as avg. pos-neg on nat’l economy, buying
climate, personal finances; -100 to +100
• Followed in financial markets and by economists and
econometricians
• Traditional design: random digit-dialed landline
telephone interviews

4. Research Questions
• Should cell interviews be added to address growing
noncoverage of cell-only HHs?
• How about Spanish interviews, to better represent the
growing Hispanic population?
• Will these impact trend data, and if so how?
• Will they improve sample representativeness?

5. Operating Principles
• Any change in methodology must be undertaken
cautiously, particularly when the project relies
heavily on trend; but
• Inaction carries its own risks, to the extent that
changes in the nature or accessibility of the target
population themselves represent a de-facto
methodological departure

6. The Cell-Phone Issue
• Do LL samples adequately represent the national
population?
• Weighting to Census corrects demographic discrepancies
• Many papers show minimal diffs. in variables of interest
• But…
• Untested or growing diffs. over time are open questions
• 34% noncoverage raises face validity issues
• LL produces small sample sizes in undercovered groups
• LL has higher design effect due to weighting

7. Test Design
• LL augmented with 75 weekly cell interviews, Jan. to
March 2012, N = 2,750
• Included ~8 Spanish interviews each week, N = 82
overall (43 LL, 39 cell)
• Mixed samples produced for comparison
• Result, eight weeks of full four-week CCI estimates

8. Sample Weights
• LL sample weighted per Census as usual
• Mixed sample: weights added to account for dual-
frame respondents and adjust for phone usage in
post-stratification
• Mixed sample, English-only: weights without the
Spanish interviews

9. CCI Estimates by Sample Type
Aggregated Responses to CCI Questions Over 11-Week Period
LL sample Mixed sample
(all interviews)
Mixed sample (English
interviews)
National economy:
Positive NET 14.8% 13.4% 13.3%
Negative NET 85.2% 86.6% 86.6%
Personal finances:
Positive NET 49.9% 49.1% 49.8%
Negative NET 50.1% 50.9% 50.2%
Buying climate:
Positive NET 29.3% 28.1% 28.5%
Negative NET 70.7% 71.9% 71.5%
CCI -37.3 -39.6 -38.9
Differences in aggregated CCI:
• -1.6 with inclusion of cell phones
• -0.7 with inclusion of Spanish interviews
• No differences are statistically significant

10. Weekly Results by Sample Type
-55
-50
-45
-40
-35
-30
-25
-20
-15
-10
-5
0
2/4/08 2/11/08 2/18/08 2/25/08 3/3/08 3/10/08 3/17/08 3/24/08
Landline-only Mixed (All) Mixed (English-only)

11. Weekly Ratings by Sample
Positive Ratings by Sample and Week
National economy Personal finances Buying climate
LL Mixed Sig? LL Mixed Sig? LL Mixed Sig?
2/5/12 12% 11% N 50% 48% N 26% 27% N
2/12/12 14% 13% N 50% 48% N 26% 27% N
2/19/12 14% 13% N 51% 49% N 29% 27% N
2/26/12 14% 13% N 50% 47% N 28% 28% N
3/4/12 15% 13% N 50% 48% N 30% 27% N
3/11/12 17% 15% N 50% 50% N 33% 31% N
3/18/12 18% 15% N 49% 51% N 31% 30% N
3/25/12 17% 16% N 50% 52% N 32% 29% N

12. Summary: Weekly Ratings by Sample
• No statistically significant differences in any of the
three questions in any week
• Week-to-week variability within each sample type
exceeded average discrepancy between sample types
– Within-sample standard deviations:
• LL = 2.8 points
• Mixed = 2.9 points
– Avg. difference between sample types: 2.1 points

13. Results Among Groups
• Avg. differences by sample type among
demographic groups were greater than the
difference between the full samples (not
surprising given smaller sample sizes).
• But differences were not unidirectional:
– Mixed sample numerically higher in 12 of 36 cases
– LL sample numerically higher in 24 of 36 cases
• In 31 of 36 groups examined, avg. difference
between sample types was smaller than the avg.
variability within each sample type.

14. Groups with Differences
• One was N <100 in LL sample (age 18-34)
• Other four:
– $100,000+ income: 10.2 pt difference
– Westerners: 9.2 pt difference
– High school graduates: 7.2 pt difference
– Political independents: 6.2 pt difference
• We conducted z-tests on each of the three CCI questions to
test whether the differences were statistically significant
– 96 comparisons: 3 questions x 4 response options x 8 weeks
– Just one statistically significant difference: On 3/18/12,
significantly more independents rated the national economy
positively in the LL sample vs. the mixed sample

15. Results of CCI Comparison
• Differences between LL and mixed samples
overall were trivial:
– Responses to the three index questions did not
meaningfully differ by sample type, either across
the full period or by week.
– Overall CCI estimates differed very little, and
results followed the same trajectory.
• Variation by sample type among groups was
no more than would be expected by random
chance.

16. Next Question: Sample Quality
• Absence of a discernible impact of cells and Spanish
can support a decision to switch or not
• What re growing noncoverage of groups most likely
to lack landline service (e.g., young, renters,
minorities, lower-income)?
• We compared the unweighted demographic
composition of the two samples

17. Comparing Sample Quality
• Result: We find statistically significant differences
between the unweighted LL and mixed samples in 13
of 38 variables studied:
– Sex, age (18-34, 55-64, and 65+), race (white and
Hispanic), home ownership, relationship status
(never married and widowed), employment (full-
time and not at all)
• In every case, the mixed frame sample was closer to
Census norms

18. Differences Between Unweighted
Estimates and Census CPS Targets
Benchmarks LL sample Mixed sample
Men 48.6% 43.8% 48.6%
Women 51.4% 56.2% 51.4%
Age: 18-34 30.5% 9.1% 17.1%
Age: 55-64 15.8% 24.4% 21.4%
Age: 65+ 16.7% 35.3% 29.1%
White 66.9% 80.1% 75.0%
Hispanic 13.9% 4.9% 9.1%
Home: Owned 68.8% 79.8% 73.2%
Home: Rented 30.2% 19.3% 25.5%
Never married 27.1% 15.2% 21.2%
Widowed 6.1% 14.9% 12.2%
Not employed 40.1% 52.1% 46.7%
Average absolute differences:
• Benchmarks vs. LL: 11.25 pts
• Benchmarks vs. mixed: 6 pts

19. Sample Quality Summary
• The unweighted sample obtained using the mixed-
frame design is far superior to the sample obtained
using the LL design
– In every case in which demographic estimates differed by
sample type, the mixed sample estimate more closely
matched the true population parameters
– The mixed design provided greater coverage of groups
including young adults, men, lower-income respondents,
non-whites and renters

20. Design Effect
• Design effect = estimate of the impact on
sampling error caused by a survey’s departure
from simple random sampling.
– The larger the weights needed to correct for non-
response and noncoverage, the higher the deff.
– Larger design effects = more difficult to find
statistically significant differences.
• LL deff = 2.3 (MoE for N=1000 is +/- 4.7 pts.)
• Mixed deff = 1.5 (MoE for N=1000 is +/- 3.8 pts.)

21. Conclusion
• No material, consistent impact of including cell-phones
on the Bloomberg CCI.
– week-to-week pattern of results was similar.
– differences by sample type were less than normal weekly
variation.
• Few differences among subgroups, and these may reflect
chance variation, since the CCI among groups typically
differed more within samples than between sample types.
• Better coverage, lower design effect/lower MoE and
greater face validity with no disruption to trend.
• Time to make the switch!