No differences are statistically significant. Tested using a z-test that accounts for the overlap between the samples (the more conservative approach because adjusting for the overlap makes it easier to find statistically significant differences)
These and all subsequent analyses compared the full mixed sample (including Spanish interviews) to the LL sample. We used the full mixed sample for two reasons: one, comparisons involving the full mixed sample (hereafter referred to simply as the “mixed sample”) provide a more stringent test of the impact of the sampling shift and two, the Spanish-inclusive estimates are preferable on theoretical grounds.As noted, some of the difference between the full mixed-sample CCI and the LL CCI was due to the inclusion of Spanish language interviews. Therefore if analyses show no differences between the full mixed sample and the landline sample we can be sure there are no differences between the English-only mixed sample and the LL sample.
Each of the differences in unweighted estimates between the LL and mixed samples shown in this table are statistically significant, and in every case the unweighted mixed sample more closely matches the census/cps targets.For the sake of simplicity, rather than having separate slightly different targets for each of the samples (which is how I did it in the paper in order to adjust for missing data), I’m just listing one target (averaging the two). I don’t think anyone will say anything, but if they ask why this table is slightly different than the numbers in Tables 5 and 6 in the paper, that’s why.
AAPOR 2013 Langer Research: Bloomberg CCI
Adding Cell Phone Sampling to aLong-Term Trend Study (Cautiously):The Bloomberg Consumer Comfort IndexJulie Phelan & Gary LangerLanger Research Associates@LangerResearch
The CCI• Continuous weekly tracking of consumer sentimentsince December 1985• N=250/week summed in a four-week rolling avg.• Computed as avg. pos-neg on nat’l economy, buyingclimate, personal finances; -100 to +100• Followed in financial markets and by economists andeconometricians• Traditional design: random digit-dialed landlinetelephone interviews
Research Questions• Should cell interviews be added to address growingnoncoverage of cell-only HHs?• How about Spanish interviews, to better represent thegrowing Hispanic population?• Will these impact trend data, and if so how?• Will they improve sample representativeness?
Operating Principles• Any change in methodology must be undertakencautiously, particularly when the project reliesheavily on trend; but• Inaction carries its own risks, to the extent thatchanges in the nature or accessibility of the targetpopulation themselves represent a de-factomethodological departure
The Cell-Phone Issue• Do LL samples adequately represent the nationalpopulation?• Weighting to Census corrects demographic discrepancies• Many papers show minimal diffs. in variables of interest• But…• Untested or growing diffs. over time are open questions• 34% noncoverage raises face validity issues• LL produces small sample sizes in undercovered groups• LL has higher design effect due to weighting
Test Design• LL augmented with 75 weekly cell interviews, Jan. toMarch 2012, N = 2,750• Included ~8 Spanish interviews each week, N = 82overall (43 LL, 39 cell)• Mixed samples produced for comparison• Result, eight weeks of full four-week CCI estimates
Sample Weights• LL sample weighted per Census as usual• Mixed sample: weights added to account for dual-frame respondents and adjust for phone usage inpost-stratification• Mixed sample, English-only: weights without theSpanish interviews
CCI Estimates by Sample TypeAggregated Responses to CCI Questions Over 11-Week PeriodLL sample Mixed sample(all interviews)Mixed sample (Englishinterviews)National economy:Positive NET 14.8% 13.4% 13.3%Negative NET 85.2% 86.6% 86.6%Personal finances:Positive NET 49.9% 49.1% 49.8%Negative NET 50.1% 50.9% 50.2%Buying climate:Positive NET 29.3% 28.1% 28.5%Negative NET 70.7% 71.9% 71.5%CCI -37.3 -39.6 -38.9Differences in aggregated CCI:• -1.6 with inclusion of cell phones• -0.7 with inclusion of Spanish interviews• No differences are statistically significant
Weekly Ratings by SamplePositive Ratings by Sample and WeekNational economy Personal finances Buying climateLL Mixed Sig? LL Mixed Sig? LL Mixed Sig?2/5/12 12% 11% N 50% 48% N 26% 27% N2/12/12 14% 13% N 50% 48% N 26% 27% N2/19/12 14% 13% N 51% 49% N 29% 27% N2/26/12 14% 13% N 50% 47% N 28% 28% N3/4/12 15% 13% N 50% 48% N 30% 27% N3/11/12 17% 15% N 50% 50% N 33% 31% N3/18/12 18% 15% N 49% 51% N 31% 30% N3/25/12 17% 16% N 50% 52% N 32% 29% N
Summary: Weekly Ratings by Sample• No statistically significant differences in any of thethree questions in any week• Week-to-week variability within each sample typeexceeded average discrepancy between sample types– Within-sample standard deviations:• LL = 2.8 points• Mixed = 2.9 points– Avg. difference between sample types: 2.1 points
Results Among Groups• Avg. differences by sample type amongdemographic groups were greater than thedifference between the full samples (notsurprising given smaller sample sizes).• But differences were not unidirectional:– Mixed sample numerically higher in 12 of 36 cases– LL sample numerically higher in 24 of 36 cases• In 31 of 36 groups examined, avg. differencebetween sample types was smaller than the avg.variability within each sample type.
Groups with Differences• One was N <100 in LL sample (age 18-34)• Other four:– $100,000+ income: 10.2 pt difference– Westerners: 9.2 pt difference– High school graduates: 7.2 pt difference– Political independents: 6.2 pt difference• We conducted z-tests on each of the three CCI questions totest whether the differences were statistically significant– 96 comparisons: 3 questions x 4 response options x 8 weeks– Just one statistically significant difference: On 3/18/12,significantly more independents rated the national economypositively in the LL sample vs. the mixed sample
Results of CCI Comparison• Differences between LL and mixed samplesoverall were trivial:– Responses to the three index questions did notmeaningfully differ by sample type, either acrossthe full period or by week.– Overall CCI estimates differed very little, andresults followed the same trajectory.• Variation by sample type among groups wasno more than would be expected by randomchance.
Next Question: Sample Quality• Absence of a discernible impact of cells and Spanishcan support a decision to switch or not• What re growing noncoverage of groups most likelyto lack landline service (e.g., young, renters,minorities, lower-income)?• We compared the unweighted demographiccomposition of the two samples
Comparing Sample Quality• Result: We find statistically significant differencesbetween the unweighted LL and mixed samples in 13of 38 variables studied:– Sex, age (18-34, 55-64, and 65+), race (white andHispanic), home ownership, relationship status(never married and widowed), employment (full-time and not at all)• In every case, the mixed frame sample was closer toCensus norms
Sample Quality Summary• The unweighted sample obtained using the mixed-frame design is far superior to the sample obtainedusing the LL design– In every case in which demographic estimates differed bysample type, the mixed sample estimate more closelymatched the true population parameters– The mixed design provided greater coverage of groupsincluding young adults, men, lower-income respondents,non-whites and renters
Design Effect• Design effect = estimate of the impact onsampling error caused by a survey’s departurefrom simple random sampling.– The larger the weights needed to correct for non-response and noncoverage, the higher the deff.– Larger design effects = more difficult to findstatistically significant differences.• LL deff = 2.3 (MoE for N=1000 is +/- 4.7 pts.)• Mixed deff = 1.5 (MoE for N=1000 is +/- 3.8 pts.)
Conclusion• No material, consistent impact of including cell-phoneson the Bloomberg CCI.– week-to-week pattern of results was similar.– differences by sample type were less than normal weeklyvariation.• Few differences among subgroups, and these may reflectchance variation, since the CCI among groups typicallydiffered more within samples than between sample types.• Better coverage, lower design effect/lower MoE andgreater face validity with no disruption to trend.• Time to make the switch!