Amelia Showalter_SearchLove London 2013


Published on

Published in: Technology, Education
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Tell story here about the Email Derby
  • Amelia Showalter_SearchLove London 2013

    1. 1. Winning With Data A/B Testing on the Obama Campaign
    2. 2. Introduction
    3. 3. You probably thought we had it easy…
    4. 4. But victory was never assured   Big-spending groups called Super-PACs were largely supporting Romney Obama was usually ahead in the polls, but the advantage was narrow and volatile
    5. 5. You might‟ve heard these stories…
    6. 6. What you didn‟t hear All of that costs money
    7. 7. The fundraising challenge  In 2008, Obama campaign raised $750 million  Would not be enough in 2012 $750 million? Not impressed.
    8. 8. The fundraising challenge  But fundraising was proving more difficult in 2012 than in 2008  President less available for fundraising events  In early campaign, we saw average online donation was half of what it had been in 2008  People were giving less, and less often  We had to be smarter and more innovative
    9. 9. Overview   A/B testing in Obama‟s digital department Lessons learned  Don‟t trust your gut  Foster a culture of testing  Invest in your team  The big picture: make it personal
    10. 10. Winning with A/B Testing
    11. 11. What impact can testing have?
    12. 12. Testing = constant improvement  Little improvements add up  Improving 1% here and 2% there isn‟t a lot at first, but over time it adds up
    13. 13. Testing = many variations
    14. 14. Testing = listening to your audience  Nearly every email was tested, often on multiple levels  We started with message and subject line tests 4 messages, 3 subject lines were tested on every national send (sometimes 6x3)  These tests went to about 20% of the list  After an hour, we would send the winner to the remainder of he list
    15. 15. Example: Draft language
    16. 16. Example: Subject lines Test sends version v1s1 v1s2 v1s3 v2s1 v2s2 v2s3 v3s1 v3s2 v3s3 v4s1 v4s2 v4s3 v5s1 v5s2 v5s3 v6s1 v6s2 v6s3 Subject line Hey Two things: Your turn Hey My opponent You decide Hey Last night Stand with me today Hey This is my last campaign [NAME] Hey There won't be many more of these deadlines What you saw this week Hey Let's win. Midnight deadline  Each draft was tested with three subject lines  One subject line would usually be common across all drafts, to help make comparisons across messages
    17. 17. Example: Best vs. Worst Versions Test sends version v1s1 v1s2 v1s3 v2s1 v2s2 v2s3 v3s1 v3s2 v3s3 v4s1 v4s2 v4s3 v5s1 v5s2 v5s3 v6s1 v6s2 v6s3 Subject line Hey Two things: Your turn Hey My opponent You decide Hey Last night Stand with me today Hey This is my last campaign [NAME] Hey There won't be many more of these deadlines What you saw this week Hey Let's win. Midnight deadline Full send (in millions) donors money 263 $17,646 268 $18,830 276 $22,380 300 $17,644 246 $13,795 222 $27,185 370 $29,976 307 $16,945 381 $25,881 444 $25,643 369 $24,759 514 $34,308 353 $22,190 273 263 363 237 352 $22,405 $21,014 $25,689 $17,154 $23,244 $4 $3 $2 $1 $0 ACTUAL ($3.7m)  IF SENDING AVG IF SENDING WORST $2.2 million additional revenue from sending best draft vs. worst, or $1.5 million additional from sending best vs. average
    18. 18. Test every element  After testing drafts and subject lines, we would split the remaining list and run additional tests  Example: Unsubscribe language Variation Recips Unsubs Unsubs per recipient Significant differences in unsubs per recipient 578,994 105 0.018% None 578,814 79 0.014% Smaller than D4 578,620 86 0.015% Smaller than D4 580,507 115 0.020% Larger than D3 and D4
    19. 19. No, really. Test every element.  We also were always running tests in the background via personalized content
    20. 20. Tests upon tests upon tests Review: Every piece of communication was an opportunity to test  A single email often had many tests attached     Subject & draft tests Full-list tests Background personalization tests
    21. 21. The results  Campaign raised over one billion dollars Raised over half a billion dollars online  Over 4 million Americans donated   Recruited tens of thousands of volunteers, publicized thousands of events and rallies  Did I mention raising >$500 million online?  Conservatively, testing probably resulted in ~$200 million in additional revenue
    22. 22. Lessons
    23. 23. Lesson #1 Don‟t Trust Your Gut
    24. 24. Don‟t trust your gut  We don‟t have all the answers  Conventional wisdom is often wrong  Long-held best practices are often wrong  You are not your audience  There was this thing called the Email Derby…  If even the experts are bad at predicting a winning message, it shows just how important testing is.
    25. 25. Experiments: Ugly vs. Pretty  We tried making our emails prettier  That failed  So we asked: what about ugly?  Ugly yellow highlighting got us better results
    26. 26. Lesson #2 Foster a culture of testing
    27. 27. The culture of testing  Check your ego at the door  Use every opportunity to test something  Compare against yourself, not against your competitors or “the industry”  Are you doing better this month than last month?  Are you doing better than you would have otherwise?
    28. 28. When in doubt, test  In a culture of testing, all questions are answered empirically  Example: With the ugly yellow highlighting, we worried about the novelty factor Maybe highlighting would only work for a short time before people started ignoring it (or being irritated by it).  We decided to do a multi-stage test across three consecutive emails 
    29. 29. The ugly highlighting experiment  Experimental design: First Email Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8  Second Email Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Third Email Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Determined through this test that novelty was indeed a factor
    30. 30. Keep a testing calendar    On the Obama campaign we had short-term and longterm calendars for national emails We added a “tests” column to plan out which tests would be attached to which emails If we saw blank spaces, it would remind us to think of more tests to run!  Important to do frequent brainstorming sessions
    31. 31. Circulate your test results internally  We had an internal listserv entirely for the express purpose of circulating test results  Helped get buy-in and increased familiarity with the testing process  Prompted discussions and generated new ideas for tests
    32. 32. Lesson #3 Invest in your team
    33. 33. OFA Digital Department  Grew from a small team in spring 2011 to a department of 200+ in 2012 Outbound (email, social, mobile, blog)  Ads  Front-End Development  Design  Video  Project management  Digital Analytics 
    34. 34. Hire smart, diverse talent  Many Obama staffers (maybe most) had never worked in politics before  Experience less important than aptitude & passion  Diverse voices led to better content, analysis  Digital analytics: looked for people with strong quantitative skills, willingness to learn  Staff learned politics and programming on the job
    35. 35. The Big Picture Using data to create a more human experience
    36. 36. Big data ≠ big brother  Testing allows you to listen to your user base  Let them tell you what they like  Whether through A/B testing or behavioral segmentation, optimization gives them a better experience  Usually, the interactions that are the most human are the ones that win
    37. 37. Be human!  In general, we founds shorter, less formal emails and subject lines did best.   Classic example: “Hey” When we dropped a mild curse word into a subject line, it usually won “Hell yes, I like Obamacare”  “Let‟s win the damn election”  “Pretty damn cool” 
    38. 38. Behavioral segmentation  Behavioral segmentation makes the experience personal Donor vs. non-donor  High-dollar vs. low-dollar  Volunteer status  What issues do people say they care about?   After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results
    39. 39. Experiments: Personalization   Adding “drop-in sentences” that reference people‟s past behavior can increase conversion rates Example: asking recent donors for more money …it's going to take a lot more of us to match them. Will you donate $25 or more today?  …it's going to take a lot more of us to match them. You stepped up recently to help out -- thank you. We all need to dig a little deeper if we're going to win, so I'm asking you to pitch in again. Will you donate $25 or more today? Added sentence significantly raised donation rate  Confirmed in several similar experiments
    40. 40. Mobilization = Human Interactions From 2012 Campaign Manager Jim Messina: “My favorite story is from a volunteer in Wisconsin 10 days out [from Election Day]. She was knocking on doors on one side of the street and the Romney campaign was knocking on doors on the other side of the street…”
    41. 41. Mobilization = Human Interactions “… [The Obama volunteer] was asked to hit two doors. One was an undecided voter and she knew exactly what to say. The other was an absentee ballot and she was told to make sure they filled it out and returned it. On the other side of the street, the Romney campaign was knocking on every single door. Most of the people weren‟t home, and most of the people that were home were already supporting Barack Obama. She looked at me and said, „You‟re using my time wisely.‟ That‟s what data can do.” - Obama 2012 Campaign Manager Jim Messina
    42. 42. Conclusions
    43. 43. Conclusions  Make it personal  Test everything  Never stop looking for new ideas, new voices, and new innovations
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.