Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me

2,061 views

Published on

The main slide deck from Digital Impact - which goes with the deck uploaded just before this one (with the same cover page)

Published in: Internet
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/EnhPc ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me

  1. 1. Oh Boy! These A/B tests appear to be bullshit!
  2. 2. @OptimiseOrDi e • UX and Analytics (1999) • User Centred Design (2001) • Agile, Startups, No budget (2003) • Funnel optimisation (2004) • Multivariate & A/B (2004) • Conversion Optimisation (2005) • Persuasive Copywriting (2006) • Joined Twitter (2007) • Lean UX (2008) • Holistic Optimisation (2009) Was : Working at Spareroom.co.uk Now : Doing workshops & consulting again!
  3. 3. Oh Boy! These A/B tests appear to be bullshit!
  4. 4. Top F***ups for 2014 1. Testing in the wrong place 2. Your hypothesis inputs are crap 3. No analytics integration 4. Your test will finish after you die 5. You don’t test for long enough 6. You peek before it’s ready 7. No QA for your split test 8. Opportunities are not prioritised 9. Testing cycles are too slow 10. You don’t know when tests are ready 11. Your test fails 12. The test is ‘about the same’ 13. Test flips behaviour 14. Test keeps moving around 15. You run an A/A test and waste time 16. Nobody ‘feels’ the test 17. You forgot you were responsive 18. You forgot you had no traffic 19. You ran the wrong test type 20. You didn’t try all the flavours of testing slidesha.re/1wBbZ9c @OptimiseOrDie
  5. 5. #fail @OptimiseOrDie
  6. 6. @OptimiseOrDie
  7. 7. @OptimiseOrDie
  8. 8. Oppan Gangnam Style! @OptimiseOrDie
  9. 9. @OptimiseOrDie
  10. 10. @OptimiseOrDie
  11. 11. @OptimiseOrDie
  12. 12. The 95% Problem • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable • Read this Nature article : bit.ly/1dwk0if • You can hit 95% early in a test • If you stop, it could be a false positive • Tools need to be smarter about inference • This 95% thingy – it’s last on your list for reasons to stop testing @OptimiseOrDie
  13. 13. Business & Purchase Cycles @OptimiseOrDie Start Test Finish Avg Cycle • Customers change • Your traffic mix changes • Markets, competitors • Be aware of all the waves • Always test whole cycles • Minimum 2 cycles (wk/mo) • Don’t exclude slower buyers
  14. 14. 16
  15. 15. When to stop? • Know the cycles • Run a test length calculator - bit.ly/XqCxuu • Set your test run time • Yup, just like a retail store • Run it • Stop it • Analyse the data • That’s it. No peeking and waiting for the numbers to look good so you can stop. • Go again @OptimiseOrDie
  16. 16. No QA testing for the AB test?
  17. 17. QA Test or Die! • Over 40% of tests have had QA issues. • It’s very easy to break or bias the testing Browser testing www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.saucelabs.com www.multibrowserviewer.com Mobile devices www.appthwack.com www.deviceanywhere.com www.opendevicelab.com Article bit.ly/1wBccsJ @OptimiseOrDie
  18. 18. Gamble the Company AWAY! • I get 60-70% right • UX and Copywriters good at picking! • C level execs are easy marks • Ironically, many decide ‘designs’ • You need collaborative test design • It’s a team game, with customers • Flip a coin, anyone?
  19. 19. WE’RE ALL WINGING IT
  20. 20. 2004 Headspace What I thought I knew in 2004 Reality
  21. 21. 2014 Headspace What I KNOW I know Me, on a good day
  22. 22. Guessaholics Anonymous
  23. 23. Rumsfeldian Space @OptimiseOrDie
  24. 24. Rumsfeldian Space @OptimiseOrDie
  25. 25. 23 : Business Future Testing? “Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy.”
  26. 26. #1 Smart Talented Polymath People The 5 Legged Optimisation Barstool @OptimiseOrDie Flexible and Agile teams
  27. 27. Fittest? Agile! @OptimiseOrDie
  28. 28. #2 : Analytics Investment (tools, people, dev time) @OptimiseOrDie
  29. 29. @OptimiseOrDie #3 : User research and insight
  30. 30. #3 : THE BEST IDEAS COME FROM? @OptimiseOrDie
  31. 31. #4 : GREAT COPYWRITING “On the average, five times as many people read the headline as read the body copy. When you have written your headline, you have spent eighty cents out of your dollar.” David Ogilvy “In 9 years and 40M split tests with visitors, the majority of my testing success came from playing with the words.” @OptimiseOrDie
  32. 32. • Google Content Experiments bit.ly/Ljg7Ds • Optimizely www.optimizely.com • Visual Website Optimizer www.visualwebsiteoptimizer.com • Multi Armed Bandit Explanation bit.ly/Xa80O8 • New Machine Learning Tools www.conductrics.com www.rekko.com @OptimiseOrDie #5 : Split Testing Tools
  33. 33. The 5 Legged Optimisation @OptimiseOrDie Barstool #1 Culture & Team #2 Toolkit & Analytics investment #3 UX, CX, Service Design, Insight #4 Persuasive Copywriting #5 Experimentation (testing) tools
  34. 34. READ STUFF
  35. 35. READ STUFF
  36. 36. READ STUFF
  37. 37. #5 : FIND STUFF @OptimiseOrDie @danbarker Analytics @fastbloke Analytics @timlb Analytics @jamesgurd Analytics @therustybear Analytics @carmenmardiros Analytics @davechaffey Analytics @priteshpatel9 Analytics @cutroni Analytics @avinash Analytics @Aschottmuller Analytics, CRO @cartmetrix Analytics, CRO @Kissmetrics CRO / UX @Unbounce CRO / UX @Morys CRO / Neuro @UXFeeds UX / Neuro @Psyblog Neuro @Gfiorelli1 SEO / Analytics @PeepLaja CRO @TheGrok CRO @UIE UX @LukeW UX / Forms @cjforms UX / Forms @axbom UX @iatv UX @Chudders Photo UX @JeffreyGroks Innovation @StephanieRieger Innovation @BrianSolis Innovation @DrEscotet Neuro @TheBrainLady Neuro @RogerDooley Neuro @Cugelman Neuro @Smashingmag Dev / UX @uxmag UX @Webtrends UX / CRO
  38. 38. #5 : LEARN STUFF @OptimiseOrDie Baymard.com Lukew.com Smashingmagazine.com ConversionXL.com Medium.com Whichtestwon.com Unbounce.com Measuringusability.com RogerDooley.com Kissmetrics.com Uxmatters.com Smartinsights.com Econsultancy.com Cutroni.com www.GetMentalNotes.com
  39. 39. #12 : The Best Companies… • Invest continually in analytics instrumentation, tools, people • Use an Agile, iterative, cross-silo, one team project culture • Prefer collaborative tools to having lots of meetings • Prioritise development based on numbers and insight • Practice real continuous product improvement, not SLEDD* • Are fixing bugs, cruft, bad stuff as well as optimising • Source photos and content that support persuasion and utility • Have cross channel, cross device design, testing and QA • Segment their data for valuable insights, every test or change • Continually reduce cycle (iteration) time in their process • Blend ‘long’ design, continuous improvement AND split tests • Make optimisation the engine of change, not the slave of ego * Single Large Expensive Doomed Developments
  40. 40. THE FUTURE OF TESTING
  41. 41. Thank You! 43 @OptimiseOrDie Email Slides : sullivac@gmail.com : slideshare.com/sullivac : linkd.in/pvrg14

×