Your SlideShare is downloading. ×
AAPOR 2013 Langer Research: Bloomberg CCI
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

AAPOR 2013 Langer Research: Bloomberg CCI


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • No differences are statistically significant. Tested using a z-test that accounts for the overlap between the samples (the more conservative approach because adjusting for the overlap makes it easier to find statistically significant differences)
  • These and all subsequent analyses compared the full mixed sample (including Spanish interviews) to the LL sample. We used the full mixed sample for two reasons: one, comparisons involving the full mixed sample (hereafter referred to simply as the “mixed sample”) provide a more stringent test of the impact of the sampling shift and two, the Spanish-inclusive estimates are preferable on theoretical grounds.As noted, some of the difference between the full mixed-sample CCI and the LL CCI was due to the inclusion of Spanish language interviews. Therefore if analyses show no differences between the full mixed sample and the landline sample we can be sure there are no differences between the English-only mixed sample and the LL sample.
  • Each of the differences in unweighted estimates between the LL and mixed samples shown in this table are statistically significant, and in every case the unweighted mixed sample more closely matches the census/cps targets.For the sake of simplicity, rather than having separate slightly different targets for each of the samples (which is how I did it in the paper in order to adjust for missing data), I’m just listing one target (averaging the two). I don’t think anyone will say anything, but if they ask why this table is slightly different than the numbers in Tables 5 and 6 in the paper, that’s why.
  • Transcript

    • 1. Adding Cell Phone Sampling to aLong-Term Trend Study (Cautiously):The Bloomberg Consumer Comfort IndexJulie Phelan & Gary LangerLanger Research Associates@LangerResearch
    • 2. The CCI• Continuous weekly tracking of consumer sentimentsince December 1985• N=250/week summed in a four-week rolling avg.• Computed as avg. pos-neg on nat’l economy, buyingclimate, personal finances; -100 to +100• Followed in financial markets and by economists andeconometricians• Traditional design: random digit-dialed landlinetelephone interviews
    • 3. Research Questions• Should cell interviews be added to address growingnoncoverage of cell-only HHs?• How about Spanish interviews, to better represent thegrowing Hispanic population?• Will these impact trend data, and if so how?• Will they improve sample representativeness?
    • 4. Operating Principles• Any change in methodology must be undertakencautiously, particularly when the project reliesheavily on trend; but• Inaction carries its own risks, to the extent thatchanges in the nature or accessibility of the targetpopulation themselves represent a de-factomethodological departure
    • 5. The Cell-Phone Issue• Do LL samples adequately represent the nationalpopulation?• Weighting to Census corrects demographic discrepancies• Many papers show minimal diffs. in variables of interest• But…• Untested or growing diffs. over time are open questions• 34% noncoverage raises face validity issues• LL produces small sample sizes in undercovered groups• LL has higher design effect due to weighting
    • 6. Test Design• LL augmented with 75 weekly cell interviews, Jan. toMarch 2012, N = 2,750• Included ~8 Spanish interviews each week, N = 82overall (43 LL, 39 cell)• Mixed samples produced for comparison• Result, eight weeks of full four-week CCI estimates
    • 7. Sample Weights• LL sample weighted per Census as usual• Mixed sample: weights added to account for dual-frame respondents and adjust for phone usage inpost-stratification• Mixed sample, English-only: weights without theSpanish interviews
    • 8. CCI Estimates by Sample TypeAggregated Responses to CCI Questions Over 11-Week PeriodLL sample Mixed sample(all interviews)Mixed sample (Englishinterviews)National economy:Positive NET 14.8% 13.4% 13.3%Negative NET 85.2% 86.6% 86.6%Personal finances:Positive NET 49.9% 49.1% 49.8%Negative NET 50.1% 50.9% 50.2%Buying climate:Positive NET 29.3% 28.1% 28.5%Negative NET 70.7% 71.9% 71.5%CCI -37.3 -39.6 -38.9Differences in aggregated CCI:• -1.6 with inclusion of cell phones• -0.7 with inclusion of Spanish interviews• No differences are statistically significant
    • 9. Weekly Results by Sample Type-55-50-45-40-35-30-25-20-15-10-502/4/08 2/11/08 2/18/08 2/25/08 3/3/08 3/10/08 3/17/08 3/24/08Landline-only Mixed (All) Mixed (English-only)
    • 10. Weekly Ratings by SamplePositive Ratings by Sample and WeekNational economy Personal finances Buying climateLL Mixed Sig? LL Mixed Sig? LL Mixed Sig?2/5/12 12% 11% N 50% 48% N 26% 27% N2/12/12 14% 13% N 50% 48% N 26% 27% N2/19/12 14% 13% N 51% 49% N 29% 27% N2/26/12 14% 13% N 50% 47% N 28% 28% N3/4/12 15% 13% N 50% 48% N 30% 27% N3/11/12 17% 15% N 50% 50% N 33% 31% N3/18/12 18% 15% N 49% 51% N 31% 30% N3/25/12 17% 16% N 50% 52% N 32% 29% N
    • 11. Summary: Weekly Ratings by Sample• No statistically significant differences in any of thethree questions in any week• Week-to-week variability within each sample typeexceeded average discrepancy between sample types– Within-sample standard deviations:• LL = 2.8 points• Mixed = 2.9 points– Avg. difference between sample types: 2.1 points
    • 12. Results Among Groups• Avg. differences by sample type amongdemographic groups were greater than thedifference between the full samples (notsurprising given smaller sample sizes).• But differences were not unidirectional:– Mixed sample numerically higher in 12 of 36 cases– LL sample numerically higher in 24 of 36 cases• In 31 of 36 groups examined, avg. differencebetween sample types was smaller than the avg.variability within each sample type.
    • 13. Groups with Differences• One was N <100 in LL sample (age 18-34)• Other four:– $100,000+ income: 10.2 pt difference– Westerners: 9.2 pt difference– High school graduates: 7.2 pt difference– Political independents: 6.2 pt difference• We conducted z-tests on each of the three CCI questions totest whether the differences were statistically significant– 96 comparisons: 3 questions x 4 response options x 8 weeks– Just one statistically significant difference: On 3/18/12,significantly more independents rated the national economypositively in the LL sample vs. the mixed sample
    • 14. Results of CCI Comparison• Differences between LL and mixed samplesoverall were trivial:– Responses to the three index questions did notmeaningfully differ by sample type, either acrossthe full period or by week.– Overall CCI estimates differed very little, andresults followed the same trajectory.• Variation by sample type among groups wasno more than would be expected by randomchance.
    • 15. Next Question: Sample Quality• Absence of a discernible impact of cells and Spanishcan support a decision to switch or not• What re growing noncoverage of groups most likelyto lack landline service (e.g., young, renters,minorities, lower-income)?• We compared the unweighted demographiccomposition of the two samples
    • 16. Comparing Sample Quality• Result: We find statistically significant differencesbetween the unweighted LL and mixed samples in 13of 38 variables studied:– Sex, age (18-34, 55-64, and 65+), race (white andHispanic), home ownership, relationship status(never married and widowed), employment (full-time and not at all)• In every case, the mixed frame sample was closer toCensus norms
    • 17. Differences Between UnweightedEstimates and Census CPS TargetsBenchmarks LL sample Mixed sampleMen 48.6% 43.8% 48.6%Women 51.4% 56.2% 51.4%Age: 18-34 30.5% 9.1% 17.1%Age: 55-64 15.8% 24.4% 21.4%Age: 65+ 16.7% 35.3% 29.1%White 66.9% 80.1% 75.0%Hispanic 13.9% 4.9% 9.1%Home: Owned 68.8% 79.8% 73.2%Home: Rented 30.2% 19.3% 25.5%Never married 27.1% 15.2% 21.2%Widowed 6.1% 14.9% 12.2%Not employed 40.1% 52.1% 46.7%Average absolute differences:• Benchmarks vs. LL: 11.25 pts• Benchmarks vs. mixed: 6 pts
    • 18. Sample Quality Summary• The unweighted sample obtained using the mixed-frame design is far superior to the sample obtainedusing the LL design– In every case in which demographic estimates differed bysample type, the mixed sample estimate more closelymatched the true population parameters– The mixed design provided greater coverage of groupsincluding young adults, men, lower-income respondents,non-whites and renters
    • 19. Design Effect• Design effect = estimate of the impact onsampling error caused by a survey’s departurefrom simple random sampling.– The larger the weights needed to correct for non-response and noncoverage, the higher the deff.– Larger design effects = more difficult to findstatistically significant differences.• LL deff = 2.3 (MoE for N=1000 is +/- 4.7 pts.)• Mixed deff = 1.5 (MoE for N=1000 is +/- 3.8 pts.)
    • 20. Conclusion• No material, consistent impact of including cell-phoneson the Bloomberg CCI.– week-to-week pattern of results was similar.– differences by sample type were less than normal weeklyvariation.• Few differences among subgroups, and these may reflectchance variation, since the CCI among groups typicallydiffered more within samples than between sample types.• Better coverage, lower design effect/lower MoE andgreater face validity with no disruption to trend.• Time to make the switch!