Presented at AAPOR 2017 by Gregory Holyk, Senior Research Analyst at Langer Research Associates

1. A 2016 Election Post-Mortem:
The ABC News/Washington Post
Tracking Poll
Presented at the annual meeting
of the American Association for Public Opinion Research
New Orleans, LA
May 14, 2016
Gregory Holyk, Gary Langer - Langer Research Associates
Scott Clement - The Washington Post

2.  Credibility of the polling industry has been questioned
following Trump’s surprise victory.
 We performed our own post-mortem of the ABC
News/Washington Post tracking poll.
 Abt Associates
 Dual-overlapping RDD, English/Spanish, 65% cell
 18 daily waves, Oct. 22-Nov.6
 Daily n=440, increased to 800 for the last 4 days
 Each estimate (except the first release) based on ~1,100-
1,200 likely voters, increasing the last 4 days to peak at
2,200 for the final estimate
Fallout From the 2016 Election

3.  Final popular vote estimate was 2 pts. off, exactly the mean
error in final estimate going back to 1984.
 Dug around for any possible irregularities and found:
 LV modeling made sense
 No indications of differential non-response by groups
 No improvements using different weighting procedures
 No evidence of “shy” Trump voters
 No consistent interviewer effects
 At the end of the day, the overall narrative based on state-
level polls and aggregator probabilities was off.
 As the AAPOR report concluded, there was no major failure
of high-quality national polls, including the ABC/Post
tracking poll.
Key Takeaways

4. Vote Preference Trendline

5.  Oct. 20-22 data showed 50-38% Clinton-Trump
 At the time, this estimate seemed reasonable given:
 The Access Hollywood tape had come out recently
 Many Republicans publically pulled their support
 He was widely seen as having lost the debates to Clinton
 Enthusiasm among Trump voters was down significantly
 The share of registered leaned Republicans who were certain to
vote also was down
 Narrowed quickly in the next 5 days as new data was rolled
into the estimates (+12, +9, +6, +4, +2, +1)
 Shifts like this occurred in 1996 and 1992 as well.
Variation in Tracking

6. ABC/Post
final est.
Actual
vote
Major party
margin error
Major party
total error
2016 47-43% 48-46% 2 pts. 4 pts.
2012 50-47 51-47 1 1
2008 53-44 53-46 2 2
2004 48-49 48-51 2 2
2000 45-48 48-48 3 3
1996 51-39 49-41 4 4
1992 44-37 43-37 1 1
1988 44-52 46-53 1 3
1984 40-55 41-59 3 5
Average 2 3
ABC/Post Past Performance

7. Predictive Accuracy of Final Polls

8. Evaluating the Tracking Poll

9.  Constructed more than 30 cut-off LV models
 Estimated turnout:
 ABC/Post LV model 61%
 Actual turnout, VEP (highest office) 59%
 Actual turnout, VAP (highest office) 55%
 Nearly all of the models showed a +3-5-point Clinton lead.
 Evaluated a regression-based probabilistic LV model (only
possible in mid-tracking) and it moved estimates 2 pts. on
the margin toward Trump.
 Will explore this approach in the future.
 Advantage: Uses all respondents in the vote estimate, weighted to
their likelihood of voting.
Likely Voter Modeling

10.  Unlike many state-level polls, we properly weighted for
education, as we always do, so that wasn’t a problem.
 Did the weight need to be adjusted for population density?
 We calculated a new weight that took into account
population density.
 This had no notable effect.
 Is there evidence of shy Trump voters among those who
participated in the survey? No, those who refused or DK’d
the vote question didn’t look disproportionately like Trump
voters.
Weighting

11.  Perhaps support for Trump was underestimated because the
likely voter models generally excluded those who were not
certain to vote?
 No. In waves where we asked vote preference among probable,
there was no shift towards Trump.
 The sample of likely voters looked very similar in both cases.
 Did those who refused to answer the vote question end up
supporting Trump?
 It’s not clear this would have helped Trump.
 Refusers were more apt to be women and minorities, but also less
likely to have a college degree and to be younger. (Those who
DK’d also were less apt to have a college degree.)
 Among refusers, leaned Democrats outnumbered leaned
Republicans.
“Shy” Trump Voters

12.  Pre-election polls often overestimate vote for third party
candidates
 This election had two major third party candidates in
Johnson and Stein
 Our final estimate didn’t miss the third party candidates by
much:
 4 percent for Johnson in ABC/Post, vs. 3 percent actual vote
 1 percent for Stein in ABC/Post, vs. 1 percent actual vote
 Johnson voters in our poll looked a lot closer in
demographic characteristics to Trump voters than Clinton
voters, but we were only over by 1 pt.
Third-Party Misestimation?

13. How Did Tracking Compare to
Other Indicators?

14.  No clear benchmark for turnout and vote preferences among
groups.
 Exit polls have problems:
 They’re weighted to actual vote, but cannot be adjusted for size of
group since that’s unknown.
 Research suggests exit polls underestimate less-educated,
younger and nonwhite voters.
 Best comparison for the tracking poll is to multilevel
regression with poststratification (MRP) estimates.
Comparing Tracking, MRP and Exit Poll

15. Tracking and MRP Turnout and Vote
Final tracking Final MRP
Size Cl-Tr Diff. Size Cl-Tr Diff.
All 100% 47-43 +4 100% 47-44 +3
Whites 72 37-53 -16 73 38-54 -16
No degree 59 42-49 -7 63 42-49 -7
Degree 41 54-35 +19 37 54-36 +18
Among whites:
Men, no deg. 18 29-60 -31 21 25-65 -40
Women, no deg. 23 29-64 -35 24 35-58 -23
Men, degree 15 43-42 +1 14 43-46 -3
Women, degree 15 51-39 +12 14 55-37 +18

16. MRP Turnout Predictions

17. MRP Margin Prediction Errors

18. MRP in the Swing States
HuffPost Pollster MRP Estimates
State Actual Average Error Predicted Error State N
Georgia -5.2 -2.4 2.8 -5.6 -0.4 332
Wisconsin -0.8 6.1 6.9 -0.2 0.6 169
Florida -1.2 1.8 3.0 -0.5 0.7 658
Michigan -0.2 6.0 6.2 0.5 0.7 291
Pennsylvania -0.7 4.1 4.8 -2.7 -2.0 375
Colorado 4.9 4.9 0.0 7.0 2.1 139
Minnesota 1.5 6.9 5.4 3.7 2.2 163
Arizona -3.5 -1.6 1.9 -5.8 -2.2 214
Virginia 5.3 5.3 0.0 2.9 -2.4 277
North Carolina -3.7 1.6 5.3 -6.1 -2.5 323
New Hampshire 0.4 3.3 2.9 3.2 2.8 38
Ohio -8.1 -1.0 7.1 -5.2 2.9 367
Iowa -9.4 -3.0 6.4 -5.0 4.4 99

19. Our MRP 538 HuffPo DKos
YouGov
MRP
Clinton % 46.8% 48.5% 45.7% NA 47.9%
Trump % 44.3% 44.9% 40.8% NA 44.1%
Margin 2.5 pts. 3.6 pts. 4.9 pts. NA 3.8 pts.
Correct predictions 50 46 46 46 43
RMSE margin all 5.8% 7.1% 7.1% 7.0% 7.6%
RMSE margin no AK, HI, DC 4.5% 6.7% 7.2% 7.0% 7.6%
RMSE margin battlegrounds 2.5% 3.9% 4.5% 4.7% 5.5%
RMSE 2 party margin (no AK, HI, DC) 4.6% 7.1% 7.1% 6.9% 8.0%
RMSE Clinton % (no AK, HI, DC) 2.3% 3.1% 3.6% 2.7% 3.3%
RMSE Trump % (no AK, HI, DC) 3.5% 4.0% 6.9% 6.6% 4.7%
Our MRP Estimates/RMSEs vs. Others

20. MRP Estimates NEP Exit Poll Estimates
Subgroup Turnout
Turnout
share Clinton Trump Margin
Turnout
share Clinton Trump Margin
18-29 31% 11% 49% 34% 15 pts. 19% 55% 36% 19 pts.
65+ 78 25 44 51 -7 16 45 52 -7
No degree 54 63 42 49 -7 50 44 51 -7
Degree 79 37 54 36 18 50 52 42 10
Whites 69 73 38 54 -16 71 37 57 -20
Among whites:
Men no deg. 60 21 25 66 -41 16 23 71 -48
Women no deg. 65 24 35 58 -23 17 34 61 -27
Men deg. 83 14 43 46 -3 17 39 53 -14
Women deg. 83 14 55 37 18 20 51 44 7
Comparing MRP to Exit Poll

21.  National polls were generally correct and accurate by
historical standards.
 State-level polls showed a competitive, uncertain contest.
 BUT state-level polls clearly underestimated Trump in the
Midwest.
 Reasons for underestimating Trump in state polls:
 Real change in vote preference in the final week
 Over-representation of college-educated whites
 Some “shy” Trump voters in state-level polls in the Midwest, but it
was not the main contributing factor. (Note, we found little evidence
of shy Trump voters in our national sample.)
AAPOR Report on 2016 Polling

22. Predictions by Polling Aggregators
85%
71%
98%
89%
99%
92%
Predicted probability of a
Clinton Electoral College
win

23.  Post-2016 election, for national-level polling, we’re dealing
with a perception problem more than a real problem. (There
were persistent problems with state-level polls and the
estimates based on them.)
 Our final pre-election estimate of the national popular vote
was highly accurate, as were most others’.
 The final estimate of +4 pts for Clinton was 2 points off the
actual margin, average for ABC/Post polls back to ’84.
 To the decimal point, our final estimate was 1.6 pts. off and
the MRP estimate was .4 pts. off.
 These differences are too small to identify any “cause” of
their inaccuracy. As estimates, they weren’t all that
inaccurate.
Conclusions

24.  Much of the problem is the popular vote didn’t match the
Electoral College vote.
 Our results showed no clear leader, but other results
predicted a Clinton victory, including:
 Early exit poll results, weighted to pre-election state polls,
suggested a Clinton win.
 Aggregators’ probabilistic models
 ABC News’ Presidential State Ratings (274-188 Clinton-Trump,
with 76 tossups)
 Our own MRP model based on the tracking poll predicted a
293-245 electoral vote victory for Trump.
Conclusions cont’d

25. A 2016 Election Post-Mortem:
The ABC News/Washington Post
Tracking Poll
Thank you!
gholyk@langerresearch.com