Webanalisten .nl, profile picture
Uploaded byWebanalisten .nl
PDF, PPTX4,185 views

Conversion Hotel 2014: Craig Sullivan (UK) keynote

The document discusses critical insights and common pitfalls in A/B testing and conversion rate optimization (CRO). It emphasizes the importance of understanding user behavior, proper data analysis, and the need for careful test design to avoid false conclusions. Various strategies for effective testing and optimization processes are provided, including prioritizing test areas and learning from failed tests.

Marketing◦
Related topics:
A/B Testing•

Embed presentation

Download as PDF, PPTX
Oh Boy! These A/B tests appear to be bullshit!
@OptimiseOrDie • UX, Analytics, Testing and Innovation • Started doing testing & CRO 2004 • Split tested over 40M visitors in 19 languages • 60+ mistakes with AB testing • I’ve made every one of them • Like riding a bike… • Get in touch for workshops, skill transfer, CRO methodology design, training and programme mentoring…
@OptimiseOrDie Hands on!
AB Test Hype Cycle Zen Plumbing @OptimiseOrDie Timeline Tested stupid ideas, lots Most AB or MVT tests are bullshit Discovered AB testing Triage, Triangulation, Prioritisation, Maths
@OptimiseOrDie
@OptimiseOrDie
Oppan Gangnam Style! @OptimiseOrDie
#1 : You’re doing it in the wrong place @OptimiseOrDie
#1 : You’re doing it in the wrong place There are 4 areas a CRO expert always looks at: 1. Inbound a+ri.on (medium, source, landing page, keyword, intent and many more…) 2. Key conversion points (product, basket, registraBon) 3. Processes, lifecycles and steps (forms, logins, registraBon, checkout, onboarding, emails, push) 4. Layers of engagement (search, category, product, add) 5. Use visitor flow reports for aIriBon – very useful. 6. For key conversion points, look at loss rates & interacBons 7. Processes and steps – look at funnels or make your own 8. Layers and engagement – make a ring model @OptimiseOrDie
Examples – Concept Bounce Engage Outcome @OptimiseOrDie
Examples – 16-­‐25Railcard.co.uk Bounce Login to Account Content Engage Start Application Type and Details Eligibility Photo Complete @OptimiseOrDie
Examples – Guide Dogs Bounce Content Engage Donation Pathway Donation Page Starts process Funnel steps Complete @OptimiseOrDie
Within a layer Page 1 Page 2 Page 3 Page 4 Page 5 Exit Deeper Layer Email Contact Like Wishlist Micro Conversions @OptimiseOrDie
#1 : Make a Money Model • Get to know the flow and loss (leaks) inbound, inside and through key processes or conversion points. • Once you know the key steps you’re losing people at and how much traffic you have – make a money model. • 20,000 see the basket page – what’s the basket page to checkout page raBo? • EsBmate how much you think you can shi the key metric (e.g. basket adds, basket -­‐> checkout) • What downstream revenue or profit would that generate? • Sort by the money column • CongratulaBons – you’ve now built the worlds first IT plan for growth with a return on investment esBmate aIached! • I’ll talk more about prioriBsing later – but a good real world analogy for you to use: @OptimiseOrDie
Think like a store owner! If you can’t refurbish the entire store, which floors or departments will you invest in optimising? Wherever there is: • Footfall • Low return • Opportunity @OptimiseOrDie
#2 : Your hypothesis is crap! Insight -­‐ Inputs #FAIL CompeBtor copying Dice rolling Guessing Panic CompeBtor change An arBcle the CEO read Ego Opinion Cherished noBons MarkeBng whims Cosmic rays Not ‘on brand’ enough IT inflexibility Internal company needs Some dumbass consultant Shiny feature Knee jerk blindness reactons @OptimiseOrDie
#2 : These are the inputs you need… Insight -­‐ Inputs Insight Eye tracking SegmentaBon Surveys Sales and Call Centre Customer contact Social analyBcs Session Replay Usability tesBng Forms analyBcs Search analyBcs Voice of Customer Market research A/B and MVT tesBng Big & unstructured data Web analyBcs CompeBtor Customer evals services @OptimiseOrDie
Insight -­‐ Inputs @OptimiseOrDie #2 : Brainstorming the test • Check your inputs • Assemble the widest possible team • Share your data and research • Design Emotive Writing guidelines
Insight -­‐ Inputs @OptimiseOrDie #2 : Emotive Writing - example Customers do not know what to do and need support and advice • Emphasize the fact that you understand that their situation is stressful • Emphasize your expertise and leadership in vehicle glazing and will help them get the best solution for their situation • Explain what they will need to do online and during the call-back so that they know what the next steps will be • Explain that they will be able ask any other questions they might have during the call-back Customers do not feel confident in assessing the damage • Emphasize the fact that you will help them assess the damage correctly online Customers need to understand the benefits of booking online • Emphasize that the online booking system is quick, easy and provides all the information they need in regards with their appointment and general cost information Customers mistrust insurers and find dealing with their insurance situation very frustrating • Where possible communicate the fact that the job is most likely to be free for insured customers, or good value for money for cash customers • Show that you understand the hassle of dealing with insurance companies – emphasise that you will help with their insurance paperwork for them, freeing them of this burden Some customers cannot be bothered to take action to fix their car glass • Emphasize the consequences of not doing anything, e.g. ‘It’s going to cost you more if the chip develops into a crack’
#2 Insight : THE DARK -­‐ Inputs SIDE “Keep your family safe and get back on the road fast with Autoglass.” @OptimiseOrDie
#2 Insight : NOW YOU -­‐ Inputs CAN BEGIN • You should have inputs, research, data, guidelines • Sit down with the team and prompt with 12 questions: – Who is this page (or process) for? – What problem does this solve for the user? – How do we know they need it? – What is the primary action we want people to take? – What might prompt the user to take this action? – How will we know if this is doing what we want it to do? – How do people get to this page? – How long are people here on this page? – What can we remove from this page? – How can we test this solution with people? – How are we solving the users needs in different and better ways than @OptimiseOrDie other places on our site? – If this is a homepage, ask these too (bit.ly/1fX2RAa)
Insight -­‐ Inputs @OptimiseOrDie #2 : PROMPT YOURSELF • Check your UX or Copywriting guidelines. • Use Get Mental Notes • What levers can we apply now? • Create a hypothesis: “WE BELIEVE THAT DOING [A] FOR PEOPLE [B] WILL MAKE OUTCOME [C] HAPPEN. WE'LL KNOW THIS WHEN WE SEE DATA [D] AND FEEDBACK [E]” www.GetMentalNotes.com
Insight -­‐ Inputs @OptimiseOrDie #2 : THE FUN BIT! • Collaborative Sketching • Brainwriting • Refine and Test!
We believe that doing [A] for People [B] will make outcome [C] happen. We’ll know this when we observe data [D] and obtain feedback [E]. (reverse) @OptimiseOrDie
#2 : Solutions • You need multiple tool inputs – Tool decks are here : www.slideshare.net/sullivac • Collaborative, Customer connected team – If you’re not doing this, you’re hosed • Session replay tools provide vital input – Get vital additional customer evidence • Simple page Analytics don’t cut it – Invest in your analytics, especially event tracking • Ego, Opinion, Cherished notions – fill gaps – Fill these vacuums with insights and data • Champion the user – Give them a chair at every meeting @OptimiseOrDie
#2 Insight : HYPOTHESIS -­‐ Inputs DESIGN SUMMARY • Inputs – get the right stuff • Research, Guidelines, Data • Framing the problem(s) • Questions to get you going • Use card prompts for Psychology • Create a hypothesis • Collaborative Sketching • Brainwriting • Refine and Check Hypothesis • Instrument and Test @OptimiseOrDie
#3 : No analytics integration • Investigating problems with tests • Segmentation of results • Tests that fail, flip or move around • Tests that don’t make sense • Broken test setups • What drives the averages you see? @OptimiseOrDie
29 A B B A
These We’re still waiting for our AB tests to Danish porn sites are so hardcore! finish! #4 : The test will finish after you die • Use a test length calculator like this one: • visualwebsiteoptimizer.com/ab-split-test-duration/
@OptimiseOrDie #5 : You get false results
The 95% Stopping Problem • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable • Read this Nature article : bit.ly/1dwk0if • You can hit 95% early in a test • If you stop, it could be a false positive • Tools need to be smarter about inference • This 95% thingy – it’s last on your list for reasons to stop testing • Let me explain @OptimiseOrDie
#5 : When to stop • Self stopping is a huge problem: – “I stopped the test when it looked good” – “It hit 20% on Thursday, so I figured – Bme to cut and run” – “We need test Bme for something else. Looks good to us” – “We’ve got a big sample now so why not finish it today?” • False Posi.ves and Nega.ves – If you cut part of a business cycle, you bias the segments you have in the test. – So if you ignore weekend shoppers by stopping your test on Friday, that will affect results – The other problems is FALSE POSITIVES and FALSE NEGATIVES @OptimiseOrDie
#5 : When to stop Scenario 1 Scenario 2 Scenario 3 Scenario 4 @OptimiseOrDie A"er 200 observa-ons Insignificant Insignificant Significant! Significant! A"er 500 observa-ons Insignificant Significant! Insignificant Significant! End of experiment Insignificant Significant! Insignificant Significant! Scenario 1 Scenario 2 Scenario 3 Scenario 4 A"er 200 observa-ons Insignificant Insignificant Significant! Significant! A"er 500 observa-ons Insignificant Significant! trial stopped trial stopped End of experiment Insignificant Significant! Significant! Significant!
@OptimiseOrDie The 95% Stopping Problem
The 95% Stopping Problem @OptimiseOrDie
The 95% Stopping Problem @OptimiseOrDie abtestguide.com/calc/
62.5cm +/- 1cm @OptimiseOrDie 9.1% Âą 0.5 9.3% Âą 0.5 9.1% Âą 0.2 9.3% Âą 0.2 9.1% Âą 0.1 9.3% Âą 0.1
Graph is a range, not a line: 9.1 Âą 1.9% 9.1 Âą 0.9% 9.1 Âą 0.3%
The 95% Stopping Problem “You should know that stopping a test once it’s significant is deadly sin number 1 in A/B testing land. 77% of A/A tests (testing the same thing as A and B) will reach significance at a certain point.” Ton Wesseling, Online Dialogue “I always tell people that you need a representative sample if your data needs to be valid. What does ‘representative’ mean? First of all you need to include all the weekdays and weekends. You need different weather, because it impacts buyer behaviour. But most important: Your traffic needs to have all traffic sources, especially newsletter, special campaigns, TV,… everything! The longer the test runs, the more insights you get. Andre Morys, Web Arts
Three Articles you MUST read “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP
Business & Purchase Cycles • Customers change • Your traffic mix changes • Markets, competitors • Be aware of all the waves • Always test whole cycles • Minimum 2 cycles (wk/mo) • Don’t exclude slower buyers @OptimiseOrDie Start Test Finish Avg Cycle
When to stop? • MINIMUM two business cycles (week/mo.) • MINIMUM of 1 purchase cycle • MINIMUM 250 outcomes/conversions per creative • MORE if relative difference is low • ALWAYS test full weeks • KNOW what marketing and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time • Run it • Stop it • Analyse the data • When do I run over? Not enough data… @OptimiseOrDie
44 #6 : You peek too early!
#6 : The early stages of a test… • Ignore the graphs. Don’t draw conclusions. Don’t dance. Calm down. • Get a feel for the test but don’t do anything yet! • Remember – in A/B -­‐ 50% of returning visitors will see a new shiny website! • UnBl your test has had at least 2 business cycles and 250+ outcomes, don’t bother even gewng remotely excited! • Watching regularly is good though. You’re looking for anything that looks really odd – if everyone is looking (but not concluding) then oddiBes will get spoIed. • All tests move around or show big swings early in the tesBng cycle. Here is a very high traffic site – it sBll takes 10 days to start seIling. Lower traffic sites will stretch this period further. 45
#7 : No QA testing for the AB test?
#7 – BIG SECRET! • Over 40% of tests have had QA issues. • Over £20M in browser conversion issues! Browser tesBng www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.cloudtesting.com www.multibrowserviewer.com www.saucelabs.com Tablets & Mobiles www.deviceanywhere.com www.perfectomobile.com FREE Device lab! www.opendevicelab.com @OptimiseOrDie
#7 : What other QA testing should I do? • TesBng from several locaBons (office, home, elsewhere) • TesBng the IP filtering is set up • Test tags are firing correctly (analyBcs and the test tool) • Test as a repeat visitor and check session Bmeouts • Cross check figures from 2+ sources • Monitor closely from launch, recheck, watch • WATCH FOR BIAS! @OptimiseOrDie
#8 : Tests are random and not prioritised Once you have a list of potential test areas, rank them by opportunity vs. effort. The common ranking metrics that I use include: • Opportunity (revenue, impact) • Dev resource • Time to market • Risk / Complexity Make yourself a quadrant diagram and plot them
#9 : Velocity or Scope problems 0 6 12 18 Months Conversion @OptimiseOrDie
#9 : Widen the optimisation scope @OptimiseOrDie
#9 : Solutions • Give Priority Boarding for opportunities – The best seats reserved for metric shifters • Release more often to close the gap – More testing resource helps, analytics ‘hawk eye’ • Kaizen – continuous improvement – Others call it JFDI (just f***ing do it) • Make changes AS WELL as tests, basically! – These small things add up as well as compounding effort • Run simultaneous tests – With analytics integration, decoding this becomes easy • Online Hair Booking – over 100 tiny tweaks – No functional changes at all – 37% improvement • Completed in-between product releases – The added lift for 10 days work, worth 360k @OptimiseOrDie
53 #11 : Your test fails @OptimiseOrDie
#11: Your test fails • Learn from the failure! If you can’t learn from the failure, you’ve designed a crap test. • Next Bme you design, imagine all your stuff failing. What would you do? If you don’t know or you’re not sure, get it changed so that a negaBve becomes insigh{ul. • So : failure itself at a creaBve or variable level should tell you something. • On a failed test, always analyse the segmentaBon and analyBcs • One or more segments will be over and under • Check for varied performance • Now add the failure info to your Knowledge Base: • Look at it carefully – what does the failure tell you? Which element do you think drove the failure? • If you know what failed (e.g. making the price bigger) then you have very useful informaBon • You turned the handle the wrong way • Now brainstorm a new test @OptimiseOrDie
#12 : The test is ‘about the same’ • Analyse the segmentaBon • Check the analyBcs and instrumentaBon • One or more segments may be over and under • They may be cancelling out – the average is a lie • The segment level performance will help you (beware of small sample sizes) • If you genuinely have a test which failed to move any segments, it’s a crap test – be bolder • This usually happens when it isn’t bold or brave enough in shiing away from the original design, parBcularly on lower traffic sites • Get tesBng again! @OptimiseOrDie
#13 : The test keeps moving around • There are three reasons it is moving around – Your sample size (outcomes) is sBll too small – The external traffic mix, customers or reacBon has suddenly changed or – Your inbound markeBng driven traffic mix is completely volaBle (very rare) • Check the sample size • Check all your markeBng acBvity • Check the instrumentaBon • If no reason, check segmentaBon @OptimiseOrDie
#14 : The test has flipped on me • Something like this can happen: • Check your sample size. If it’s still small, then expect this until the test settles. • If the test does genuinely flip – and quite severely – then something has changed with the traffic mix, the customer base or your advertising. Maybe the PPC budget ran out? Seriously! • To analyse a flipped test, you’ll need to check your segmented data. This is why you have a split testing package AND an analytics system. • The segmented data will help you to identify the source of the shift in response to your test. I rarely get a flipped one and it’s always something changing on me, without being told. The heartless bastards.
• No – and this is why: – It’s a waste of Bme – It’s easier to test and monitor instead – You are eaBng into test Bme – Also applies to A/A/B/B tesBng – A/B/A running at 25%/50%/25% is the best • Read my post here : hIp://bit.ly/WcI9EZ 58 #15 : Should I run an A/A test first
#16 : Nobody feels the test • You promised a 25% rise in checkouts -­‐ you only see 2% • Traffic, AdverBsing, MarkeBng may have changed • Check they’re using the same precise metrics • Run a calibraBon exercise • I oen leave a 5 or 10% stub running in a test • This tracks old creaBve once new one goes live • If conversion is also down for that one, BINGO! • Remember – the AB test is an esBmate – it doesn’t precisely record future performance • This is why infrequent tesBng is bad • Always be trying a new test instead of basking in the glory of one you ran 6 months ago. You’re only as good as your next test. @OptimiseOrDie
#17 : You forgot about Mobile & Tablet • If you’re AB tesBng a responsive site, pay aIenBon • Content will break differently on many screens • Know thy users and their devices • Use bango or google analyBcs to define a test list • Make sure you test mobile devices & viewports • What looks good on your desk may not be for the user • Harder to design cross device tests • You’ll need to segment mobile, tablet & desktop response in the analyBcs or AB tesBng package • Your personal phone is not a device mix • Ask me about making your device list • Buy core devices, rent the rest from deviceanywhere.com @OptimiseOrDie
#18 : Oh shit – no traffic • If small volumes, contact customers – reach out. • If data volumes aren’t there, there are sBll customers! • Drive design from levers you can apply – game the system • Pick clean and simple clusters of change (hypothesis driven) • Use a goal at an earlier ring stage or funnel step • Beware of using clickthroughs when aIriBon is high on the other side • Try before and aer tesBng on idenBcal Bme periods (measure in analyBcs model) • Be careful about small sample sizes (<100 outcomes) • Are you working automated emails? • Fix JFDI, performance and UX issues too!
#18 : Oh shit – no traffic • Forget MVT or A/B/N tests – run your numbers • Test things with high impact – don’t be a wuss! • Use UX, Session Replay to aid insight • Run a task gap survey (4Q style) • Run a dropped basket survey (LF style) • Run a general survey + check social + other sites • Run sitewide tests that appear on all pages or large clusters of pages – • UVPs (“We are a cool brand”), USPs (“Free returns!”), UCPs (“10% off today”). • Headers, Footers, Nudge Bars, USP bars, footer changes, NavigaBon, Product pages, Delivery info etc.
#19 : I chose the wrong test type • A/B tes.ng – good for: – A single change of content or design layout – A group of related changes (e.g. payment security) – Finding a new and radical shi for a template design – Lower traffic pages or shorter test Bmes • Mul.variate tes.ng – good for: – Higher traffic pages – Groups of unrelated changes (e.g. delivery & security) – MulBple content or design style changes – Finding specific drivers of test lis – TesBng mulBple versions (e.g. click here, book now, go) – Where you need to understand strong and weak cross variable interacBons – Don’t use to seIle arguments or sloppy thinking!
Netherlands A/B Shi Example Previous winner +7.25% +8.19% addiBonal li
#20 – Other flavours of tes.ng • Micro tes.ng (.ny change) – good for: – Proving to the boss that tesBng works – DemonstraBng to IT that it works without impact – Showing the impact of a seemingly Bny change – Proof of concept before larger test • Funnel tes.ng – good for: – Checkouts – Lead gen – Forms processes – QuotaBons – Any mulB-­‐step process with data entry • Fake it and Build it – good for: – TesBng new business ideas – Trying out promoBons on a test sample – EsBmaBng impact before you build – Helps you calculate ROI – You can even split test enBre server farms Vs.
#20 – Other flavours of tes.ng “Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy.”
Top F***ups for 2014 1. TesBng in the wrong place 2. Your hypothesis inputs are crap 3. No analyBcs integraBon 4. Your test will finish aer you die 5. You don’t test for long enough 6. You peek before it’s ready 7. No QA for your split test 8. OpportuniBes are not prioriBsed 9. TesBng cycles are too slow 10. You don’t know when tests are ready 11. Your test fails 12. The test is ‘about the same’ 13. Test flips behaviour 14. Test keeps moving around 15. You run an A/A test and waste Bme 16. Nobody ‘feels’ the test 17. You forgot you were responsive 18. You forgot you had no traffic 19. You ran the wrong test type 20. You didn’t try all the flavours of tesBng @OptimiseOrDie
WE’RE ALL WINGING IT
2004 Headspace What I thought I knew in 2004 Reality
2014 Headspace What I know I know On a good day
Guessaholics Anonymous
Rumsfeldian Space @OptimiseOrDie
Rumsfeldian Space @OptimiseOrDie
The 5 Legged Optimisation Barstool @OptimiseOrDie #1 Smart Talented Polymath People Flexible and Agile teams
Fittest? Agile! @OptimiseOrDie
#2 : Analytics Investment (tools, people, dev time) @OptimiseOrDie
@OptimiseOrDie #3 : User research and insight
@OptimiseOrDie #3 : THE BEST IDEAS COME FROM?
#4 : GREAT COPYWRITING “On the average, five times as many people read the headline as read the body copy. When you have written your headline, you have spent eighty cents out of your dollar.” David Ogilvy “In 9 years and 40M split tests with visitors, the majority of my testing success came from playing with the words.” @OptimiseOrDie
#4 : COPYWRITING TIP “Would you like to…” “I would like to...” http://www.theguardian.com/info/developer-blog/ 2013/jan/02/interactive-button-text-grammar Use the WYLTIWLT test for button copy!
• Google Content Experiments bit.ly/Ljg7Ds • Optimizely www.optimizely.com • Visual Website Optimizer www.visualwebsiteoptimizer.com • Multi Armed Bandit Explanation bit.ly/Xa80O8 • New Machine Learning Tools www.conductrics.com www.rekko.com @OptimiseOrDie #5 : Split Testing Tools
The 5 Legged Optimisation Barstool #1 Culture & Team #2 Toolkit & Analytics investment #3 UX, CX, Service Design, Insight #4 Persuasive Copywriting #5 Experimentation (testing) tools @OptimiseOrDie
READ STUFF
READ STUFF
READ STUFF
#5 : FIND STUFF @OptimiseOrDie @danbarker Analytics @fastbloke Analytics @timlb Analytics @jamesgurd Analytics @therustybear Analytics @carmenmardiros Analytics @davechaffey Analytics @priteshpatel9 Analytics @cutroni Analytics @avinash Analytics @Aschottmuller Analytics, CRO @cartmetrix Analytics, CRO @Kissmetrics CRO / UX @Unbounce CRO / UX @Morys CRO / Neuro @UXFeeds UX / Neuro @Psyblog Neuro @Gfiorelli1 SEO / Analytics @PeepLaja CRO @TheGrok CRO @UIE UX @LukeW UX / Forms @cjforms UX / Forms @axbom UX @iatv UX @Chudders Photo UX @JeffreyGroks Innovation @StephanieRieger Innovation @BrianSolis Innovation @DrEscotet Neuro @TheBrainLady Neuro @RogerDooley Neuro @Cugelman Neuro @Smashingmag Dev / UX @uxmag UX @Webtrends UX / CRO
#5 : LEARN STUFF @OptimiseOrDie Baymard.com Lukew.com Smashingmagazine.com ConversionXL.com Medium.com Whichtestwon.com Unbounce.com Measuringusability.com RogerDooley.com Kissmetrics.com Uxmatters.com Smartinsights.com Econsultancy.com Cutroni.com www.GetMentalNotes.com
#12 : The Best Companies… • Invest con.nually in analyBcs instrumentaBon, tools, people • Use an Agile, itera.ve, cross-­‐silo, one team project culture • Prefer collabora.ve tools to having lots of meeBngs • Priori.se development based on numbers and insight • PracBce real con.nuous product improvement, not SLEDD* • Are fixing bugs, cru, bad stuff as well as opBmising • Source photos and content that support persuasion and uBlity • Have cross channel, cross device design, tesBng and QA • Segment their data for valuable insights, every test or change • Con.nually reduce cycle (iteraBon) Bme in their process • Blend ‘long’ design, conBnuous improvement AND split tests • Make op.misa.on the engine of change, * Single Large Expensive Doomed Developments not the slave of ego
THE FUTURE OF TESTING
Get your quesBons in! Mail : sullivac@gmail.com Deck : slideshare.com/sullivac Linkedin : linkd.in/pvrg14 @OptimiseOrDie

More Related Content

PDF
Conversion Hotel 2014: Brian Massey (US) keynote
byWebanalisten .nl
 
PDF
Conversion Hotel 2014: Karl Gilis (BE) keynote
byWebanalisten .nl
 
PDF
Conversion Hotel 2014: Angie Schotmuller (US) keynote
byWebanalisten .nl
 
PDF
Conversion Hotel 2014: Michael Aagaard (DK) keynote
byWebanalisten .nl
 
PPT
Startup Metrics for Pirates (Startonomics Beijing, June 2009)
byDave McClure
 
PDF
[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...
byCXL
 
PPTX
20 Ways to Shaft your Split Tesring : Conversion Conference
byCraig Sullivan
 
PPTX
Mark Thomas - 10 Step Technical SEO Game Plan (annotated edition)
byAnna Morrison
 
Conversion Hotel 2014: Brian Massey (US) keynote
byWebanalisten .nl
 
Conversion Hotel 2014: Karl Gilis (BE) keynote
byWebanalisten .nl
 
Conversion Hotel 2014: Angie Schotmuller (US) keynote
byWebanalisten .nl
 
Conversion Hotel 2014: Michael Aagaard (DK) keynote
byWebanalisten .nl
 
Startup Metrics for Pirates (Startonomics Beijing, June 2009)
byDave McClure
 
[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...
byCXL
 
20 Ways to Shaft your Split Tesring : Conversion Conference
byCraig Sullivan
 
Mark Thomas - 10 Step Technical SEO Game Plan (annotated edition)
byAnna Morrison
 

What's hot

PDF
SearchLove London 2016 | Stephen Pavlovich | Habits of Advanced Conversion Op...
byDistilled
 
PPTX
The Top 5 Ecommerce Tests You Should Run Right Now
byOptimizely
 
PDF
[Elite Camp 2016] Thomas Barker - From Zero to In-House Optimisation Superstars
byCXL
 
PPTX
Humanizing The Serp, One Word at a Time
byWil Reynolds
 
PPTX
How to Compete on Google Shopping (PLAs)
byKoozai
 
PDF
Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...
byTuring Fest
 
PDF
Google Analytics 101 for Business - How to Get Started With Google Analytics
byJeff Sauer
 
PPTX
SearchLove London 2016 | Amy Harrison | Stand out to YOUR Crowd: A Simple Fra...
byDistilled
 
PDF
[Elite Camp 2016] Martijn Scheijbeler - Optimization for Content Sites
byCXL
 
PDF
Save a Floundering Digital Marketing Campaign
byEffin Amazing
 
PPT
Making Websites Work - Retail is Detail
byDistilled
 
PDF
[Elite Camp 2016] Yehoshua Coren - Strategic And Tactical Implementation And ...
byCXL
 
PDF
Lean Analytics: A short summary
byJan KĂśnig
 
PDF
[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...
byCXL
 
PPTX
Conversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
byWeb Marketing ROI
 
PDF
SearchLove London 2016 | Tom Anthony | SEO Split-Testing - How You can Run Te...
byDistilled
 
PDF
Why the digital skills gap is good for you
bySteve Linney
 
PDF
[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and Tactics
byCXL
 
PDF
Conversion Optimization Tools
byAlessandro Martin
 
PPTX
Janna Bastow — The Power of Product Focus (Turing Festival 2016)
byTuring Fest
 
SearchLove London 2016 | Stephen Pavlovich | Habits of Advanced Conversion Op...
byDistilled
 
The Top 5 Ecommerce Tests You Should Run Right Now
byOptimizely
 
[Elite Camp 2016] Thomas Barker - From Zero to In-House Optimisation Superstars
byCXL
 
Humanizing The Serp, One Word at a Time
byWil Reynolds
 
How to Compete on Google Shopping (PLAs)
byKoozai
 
Patrick Campbell — How to Build Actual Customer-Driven Product (Turing Festiv...
byTuring Fest
 
Google Analytics 101 for Business - How to Get Started With Google Analytics
byJeff Sauer
 
SearchLove London 2016 | Amy Harrison | Stand out to YOUR Crowd: A Simple Fra...
byDistilled
 
[Elite Camp 2016] Martijn Scheijbeler - Optimization for Content Sites
byCXL
 
Save a Floundering Digital Marketing Campaign
byEffin Amazing
 
Making Websites Work - Retail is Detail
byDistilled
 
[Elite Camp 2016] Yehoshua Coren - Strategic And Tactical Implementation And ...
byCXL
 
Lean Analytics: A short summary
byJan KĂśnig
 
[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...
byCXL
 
Conversion Rate Optimisation: The Science Behind Turning Visitors Into Customers
byWeb Marketing ROI
 
SearchLove London 2016 | Tom Anthony | SEO Split-Testing - How You can Run Te...
byDistilled
 
Why the digital skills gap is good for you
bySteve Linney
 
[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and Tactics
byCXL
 
Conversion Optimization Tools
byAlessandro Martin
 
Janna Bastow — The Power of Product Focus (Turing Festival 2016)
byTuring Fest
 

Similar to Conversion Hotel 2014: Craig Sullivan (UK) keynote

PPTX
20 top AB testing mistakes and how to avoid them
byCraig Sullivan
 
PPTX
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
byCraig Sullivan
 
PPTX
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
byCraig Sullivan
 
PPTX
John lewis lego craig sullivan 1335
byCharlie Lines
 
PDF
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
byMarketing Festival
 
PDF
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...
byWebanalisten .nl
 
PPTX
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
byCraig Sullivan
 
PPTX
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
byCraig Sullivan
 
PPTX
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
byCraig Sullivan
 
PDF
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
byNorthern User Experience
 
PPTX
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
byCraig Sullivan
 
PPTX
Toolkits and tips for UX analytics CRO by Craig Sullivan
byUXPA UK
 
PPTX
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
byCraig Sullivan
 
PPTX
Surviving the hype cycle Shortcuts to split testing success
byCraig Sullivan
 
PPTX
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
byCraig Sullivan
 
PPTX
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
byCraig Sullivan
 
PPTX
eMetrics London - The AB Testing Hype Cycle
byCraig Sullivan
 
PPTX
VWO Webinar: How To Plan Your Optimisation Roadmap
byVWO
 
KEY
Marketing Institute | Optimisation course
byLar Veale
 
PDF
Conversion Optimisation
byLar Veale
 
20 top AB testing mistakes and how to avoid them
byCraig Sullivan
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
byCraig Sullivan
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
byCraig Sullivan
 
John lewis lego craig sullivan 1335
byCharlie Lines
 
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
byMarketing Festival
 
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel ...
byWebanalisten .nl
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
byCraig Sullivan
 
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
byCraig Sullivan
 
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
byCraig Sullivan
 
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
byNorthern User Experience
 
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
byCraig Sullivan
 
Toolkits and tips for UX analytics CRO by Craig Sullivan
byUXPA UK
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
byCraig Sullivan
 
Surviving the hype cycle Shortcuts to split testing success
byCraig Sullivan
 
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
byCraig Sullivan
 
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
byCraig Sullivan
 
eMetrics London - The AB Testing Hype Cycle
byCraig Sullivan
 
VWO Webinar: How To Plan Your Optimisation Roadmap
byVWO
 
Marketing Institute | Optimisation course
byLar Veale
 
Conversion Optimisation
byLar Veale
 

More from Webanalisten .nl

PDF
DDTT11: Hans Willemse - 21-01-20
byWebanalisten .nl
 
PDF
DDTT11: Ton Wesseling - 21-01-20
byWebanalisten .nl
 
PDF
CH2019 keynote: Guy Yalif - AI and personalization demystified
byWebanalisten .nl
 
PDF
CH2019 keynote: AurĂŠlie Pols - Data governance
byWebanalisten .nl
 
PDF
CH2019 keynote: Lucia van den Brink - Keynote pitch winner
byWebanalisten .nl
 
PDF
CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments
byWebanalisten .nl
 
PDF
CH2019 keynote: Roger Dooley - Friction
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Ton Wesseling
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Nir Eyal
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Erin Weigel
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Lea Pica
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: TU Delft Solar Team
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Karl Gilis
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Lisa Pierson
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Chad Sanderson
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Aleksander Fabijan
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Lizzie Eardley
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Michael Aagaard
byWebanalisten .nl
 
PDF
Conversion Hotel 2018 Keynote: Annemarie Steen
byWebanalisten .nl
 
PDF
#DDTT meetup #7 Amsterdam - Results on the state of the Dutch CRO Industry su...
byWebanalisten .nl
 
DDTT11: Hans Willemse - 21-01-20
byWebanalisten .nl
 
DDTT11: Ton Wesseling - 21-01-20
byWebanalisten .nl
 
CH2019 keynote: Guy Yalif - AI and personalization demystified
byWebanalisten .nl
 
CH2019 keynote: AurĂŠlie Pols - Data governance
byWebanalisten .nl
 
CH2019 keynote: Lucia van den Brink - Keynote pitch winner
byWebanalisten .nl
 
CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments
byWebanalisten .nl
 
CH2019 keynote: Roger Dooley - Friction
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Ton Wesseling
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Nir Eyal
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Erin Weigel
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Lea Pica
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: TU Delft Solar Team
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Karl Gilis
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Lisa Pierson
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Chad Sanderson
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Aleksander Fabijan
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Lizzie Eardley
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Michael Aagaard
byWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Annemarie Steen
byWebanalisten .nl
 
#DDTT meetup #7 Amsterdam - Results on the state of the Dutch CRO Industry su...
byWebanalisten .nl
 

Recently uploaded

PDF
Rochester Homes Product Guide January 2026
byJimWilliams206540
 
PDF
Learn About Buying Verified Twitter (X) Accounts (Every User Needs to Learn).pdf
bygsghhgf8
 
PDF
What is the Difference Between Unacademy Referral Code and Promo Code
byvayomeg600
 
PDF
SFMC User Group CPH - January 2026 - From Campaigns to Conversations
byLukas Lunow
 
PDF
Lions Club Presentation on How to Use Facebook, Instagram and META products
byTalking Finger, social media marketing agency
 
PPTX
SORTIQSOLUTIONS. pptx ppt on ppc a topic from digital marketing
bysortiqsolutionss
 
PDF
2026 Google Ads Playbook: 5 Strategies Behind $200M+ in Profitable Ad Spend
bySearch Engine Journal
 
PDF
Latest SEO Trends Followed by Top SEO Agencies in Noida - Manojnegi.in.pdf
byManojNegi
 
PDF
ROAD safety SOLUTIONS brochure ltek NEW (1).pdf
bydgtlmktlyncotek
 
PDF
AI In Advertising 2026 - Optimization Expectation and Framework
byAvinash Chandra
 
PPTX
Affiliate Summit West - Lily Ray Keynote 2026
byLily Ray
 
PPTX
seo-complete-guide-2026-for-rankings-pro
byprabhakarpandav2
 
PDF
How To Buy Verified Wise Accounts – for a secure,& fully prepared profile.pdf
byapasanilion108
 
PDF
TopMate ES11 3-Wheel Electric Scooter: A Compact Mobility Solution for Everyd...
byTopmate
 
PDF
biosecuritymeasuresinshrimpfarming-190830124902.pdf
byNeospark
 
PDF
Purchase Twitter Accounts with Full Access and Secure Setup.pdf
bytopusapro.com
 
DOCX
Top Factors That Make Old Gmail Accounts More Valuable
bypvasmmpath
 
PDF
Your Website Is Invisible to 800 Million People: The ChatGPT Visibility Crisis
byUdit Goenka
 
PDF
Karin Gargus' Marketing Portfolio - DSGN 253
byKarin Gargus
 
PDF
Top 123 Place to USA Binance Accounts Online-Uk.pdf
bybimif63078
 
Rochester Homes Product Guide January 2026
byJimWilliams206540
 
Learn About Buying Verified Twitter (X) Accounts (Every User Needs to Learn).pdf
bygsghhgf8
 
What is the Difference Between Unacademy Referral Code and Promo Code
byvayomeg600
 
SFMC User Group CPH - January 2026 - From Campaigns to Conversations
byLukas Lunow
 
Lions Club Presentation on How to Use Facebook, Instagram and META products
byTalking Finger, social media marketing agency
 
SORTIQSOLUTIONS. pptx ppt on ppc a topic from digital marketing
bysortiqsolutionss
 
2026 Google Ads Playbook: 5 Strategies Behind $200M+ in Profitable Ad Spend
bySearch Engine Journal
 
Latest SEO Trends Followed by Top SEO Agencies in Noida - Manojnegi.in.pdf
byManojNegi
 
ROAD safety SOLUTIONS brochure ltek NEW (1).pdf
bydgtlmktlyncotek
 
AI In Advertising 2026 - Optimization Expectation and Framework
byAvinash Chandra
 
Affiliate Summit West - Lily Ray Keynote 2026
byLily Ray
 
seo-complete-guide-2026-for-rankings-pro
byprabhakarpandav2
 
How To Buy Verified Wise Accounts – for a secure,& fully prepared profile.pdf
byapasanilion108
 
TopMate ES11 3-Wheel Electric Scooter: A Compact Mobility Solution for Everyd...
byTopmate
 
biosecuritymeasuresinshrimpfarming-190830124902.pdf
byNeospark
 
Purchase Twitter Accounts with Full Access and Secure Setup.pdf
bytopusapro.com
 
Top Factors That Make Old Gmail Accounts More Valuable
bypvasmmpath
 
Your Website Is Invisible to 800 Million People: The ChatGPT Visibility Crisis
byUdit Goenka
 
Karin Gargus' Marketing Portfolio - DSGN 253
byKarin Gargus
 
Top 123 Place to USA Binance Accounts Online-Uk.pdf
bybimif63078
 

Conversion Hotel 2014: Craig Sullivan (UK) keynote

  • 1.
    Oh Boy! TheseA/B tests appear to be bullshit!
  • 2.
    @OptimiseOrDie • UX,Analytics, Testing and Innovation • Started doing testing & CRO 2004 • Split tested over 40M visitors in 19 languages • 60+ mistakes with AB testing • I’ve made every one of them • Like riding a bike… • Get in touch for workshops, skill transfer, CRO methodology design, training and programme mentoring…
  • 3.
  • 5.
    AB Test HypeCycle Zen Plumbing @OptimiseOrDie Timeline Tested stupid ideas, lots Most AB or MVT tests are bullshit Discovered AB testing Triage, Triangulation, Prioritisation, Maths
  • 6.
  • 7.
  • 8.
    Oppan Gangnam Style! @OptimiseOrDie
  • 9.
    #1 : You’redoing it in the wrong place @OptimiseOrDie
  • 10.
    #1 : You’redoing it in the wrong place There are 4 areas a CRO expert always looks at: 1. Inbound a+ri.on (medium, source, landing page, keyword, intent and many more…) 2. Key conversion points (product, basket, registraBon) 3. Processes, lifecycles and steps (forms, logins, registraBon, checkout, onboarding, emails, push) 4. Layers of engagement (search, category, product, add) 5. Use visitor flow reports for aIriBon – very useful. 6. For key conversion points, look at loss rates & interacBons 7. Processes and steps – look at funnels or make your own 8. Layers and engagement – make a ring model @OptimiseOrDie
  • 11.
    Examples – Concept Bounce Engage Outcome @OptimiseOrDie
  • 12.
    Examples – 16-­‐25Railcard.co.uk Bounce Login to Account Content Engage Start Application Type and Details Eligibility Photo Complete @OptimiseOrDie
  • 13.
    Examples – Guide Dogs Bounce Content Engage Donation Pathway Donation Page Starts process Funnel steps Complete @OptimiseOrDie
  • 14.
    Within a layer Page 1 Page 2 Page 3 Page 4 Page 5 Exit Deeper Layer Email Contact Like Wishlist Micro Conversions @OptimiseOrDie
  • 15.
    #1 : Makea Money Model • Get to know the flow and loss (leaks) inbound, inside and through key processes or conversion points. • Once you know the key steps you’re losing people at and how much traffic you have – make a money model. • 20,000 see the basket page – what’s the basket page to checkout page raBo? • EsBmate how much you think you can shi the key metric (e.g. basket adds, basket -­‐> checkout) • What downstream revenue or profit would that generate? • Sort by the money column • CongratulaBons – you’ve now built the worlds first IT plan for growth with a return on investment esBmate aIached! • I’ll talk more about prioriBsing later – but a good real world analogy for you to use: @OptimiseOrDie
  • 16.
    Think like a store owner! If you can’t refurbish the entire store, which floors or departments will you invest in optimising? Wherever there is: • Footfall • Low return • Opportunity @OptimiseOrDie
  • 17.
    #2 : Yourhypothesis is crap! Insight -­‐ Inputs #FAIL CompeBtor copying Dice rolling Guessing Panic CompeBtor change An arBcle the CEO read Ego Opinion Cherished noBons MarkeBng whims Cosmic rays Not ‘on brand’ enough IT inflexibility Internal company needs Some dumbass consultant Shiny feature Knee jerk blindness reactons @OptimiseOrDie
  • 18.
    #2 : Theseare the inputs you need… Insight -­‐ Inputs Insight Eye tracking SegmentaBon Surveys Sales and Call Centre Customer contact Social analyBcs Session Replay Usability tesBng Forms analyBcs Search analyBcs Voice of Customer Market research A/B and MVT tesBng Big & unstructured data Web analyBcs CompeBtor Customer evals services @OptimiseOrDie
  • 19.
    Insight -­‐ Inputs @OptimiseOrDie #2 : Brainstorming the test • Check your inputs • Assemble the widest possible team • Share your data and research • Design Emotive Writing guidelines
  • 20.
    Insight -­‐ Inputs @OptimiseOrDie #2 : Emotive Writing - example Customers do not know what to do and need support and advice • Emphasize the fact that you understand that their situation is stressful • Emphasize your expertise and leadership in vehicle glazing and will help them get the best solution for their situation • Explain what they will need to do online and during the call-back so that they know what the next steps will be • Explain that they will be able ask any other questions they might have during the call-back Customers do not feel confident in assessing the damage • Emphasize the fact that you will help them assess the damage correctly online Customers need to understand the benefits of booking online • Emphasize that the online booking system is quick, easy and provides all the information they need in regards with their appointment and general cost information Customers mistrust insurers and find dealing with their insurance situation very frustrating • Where possible communicate the fact that the job is most likely to be free for insured customers, or good value for money for cash customers • Show that you understand the hassle of dealing with insurance companies – emphasise that you will help with their insurance paperwork for them, freeing them of this burden Some customers cannot be bothered to take action to fix their car glass • Emphasize the consequences of not doing anything, e.g. ‘It’s going to cost you more if the chip develops into a crack’
  • 21.
    #2 Insight :THE DARK -­‐ Inputs SIDE “Keep your family safe and get back on the road fast with Autoglass.” @OptimiseOrDie
  • 22.
    #2 Insight :NOW YOU -­‐ Inputs CAN BEGIN • You should have inputs, research, data, guidelines • Sit down with the team and prompt with 12 questions: – Who is this page (or process) for? – What problem does this solve for the user? – How do we know they need it? – What is the primary action we want people to take? – What might prompt the user to take this action? – How will we know if this is doing what we want it to do? – How do people get to this page? – How long are people here on this page? – What can we remove from this page? – How can we test this solution with people? – How are we solving the users needs in different and better ways than @OptimiseOrDie other places on our site? – If this is a homepage, ask these too (bit.ly/1fX2RAa)
  • 23.
    Insight -­‐ Inputs @OptimiseOrDie #2 : PROMPT YOURSELF • Check your UX or Copywriting guidelines. • Use Get Mental Notes • What levers can we apply now? • Create a hypothesis: “WE BELIEVE THAT DOING [A] FOR PEOPLE [B] WILL MAKE OUTCOME [C] HAPPEN. WE'LL KNOW THIS WHEN WE SEE DATA [D] AND FEEDBACK [E]” www.GetMentalNotes.com
  • 24.
    Insight -­‐ Inputs @OptimiseOrDie #2 : THE FUN BIT! • Collaborative Sketching • Brainwriting • Refine and Test!
  • 25.
    We believe that doing [A] for People [B] will make outcome [C] happen. We’ll know this when we observe data [D] and obtain feedback [E]. (reverse) @OptimiseOrDie
  • 26.
    #2 : Solutions • You need multiple tool inputs – Tool decks are here : www.slideshare.net/sullivac • Collaborative, Customer connected team – If you’re not doing this, you’re hosed • Session replay tools provide vital input – Get vital additional customer evidence • Simple page Analytics don’t cut it – Invest in your analytics, especially event tracking • Ego, Opinion, Cherished notions – fill gaps – Fill these vacuums with insights and data • Champion the user – Give them a chair at every meeting @OptimiseOrDie
  • 27.
    #2 Insight :HYPOTHESIS -­‐ Inputs DESIGN SUMMARY • Inputs – get the right stuff • Research, Guidelines, Data • Framing the problem(s) • Questions to get you going • Use card prompts for Psychology • Create a hypothesis • Collaborative Sketching • Brainwriting • Refine and Check Hypothesis • Instrument and Test @OptimiseOrDie
  • 28.
    #3 : Noanalytics integration • Investigating problems with tests • Segmentation of results • Tests that fail, flip or move around • Tests that don’t make sense • Broken test setups • What drives the averages you see? @OptimiseOrDie
  • 29.
    29 A BB A
  • 30.
    These We’re still waiting for our AB tests to Danish porn sites are so hardcore! finish! #4 : The test will finish after you die • Use a test length calculator like this one: • visualwebsiteoptimizer.com/ab-split-test-duration/
  • 31.
    @OptimiseOrDie #5 :You get false results
  • 32.
    The 95% StoppingProblem • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable • Read this Nature article : bit.ly/1dwk0if • You can hit 95% early in a test • If you stop, it could be a false positive • Tools need to be smarter about inference • This 95% thingy – it’s last on your list for reasons to stop testing • Let me explain @OptimiseOrDie
  • 33.
    #5 : When to stop • Self stopping is a huge problem: – “I stopped the test when it looked good” – “It hit 20% on Thursday, so I figured – Bme to cut and run” – “We need test Bme for something else. Looks good to us” – “We’ve got a big sample now so why not finish it today?” • False Posi.ves and Nega.ves – If you cut part of a business cycle, you bias the segments you have in the test. – So if you ignore weekend shoppers by stopping your test on Friday, that will affect results – The other problems is FALSE POSITIVES and FALSE NEGATIVES @OptimiseOrDie
  • 34.
    #5 : When to stop Scenario 1 Scenario 2 Scenario 3 Scenario 4 @OptimiseOrDie A"er 200 observa-ons Insignificant Insignificant Significant! Significant! A"er 500 observa-ons Insignificant Significant! Insignificant Significant! End of experiment Insignificant Significant! Insignificant Significant! Scenario 1 Scenario 2 Scenario 3 Scenario 4 A"er 200 observa-ons Insignificant Insignificant Significant! Significant! A"er 500 observa-ons Insignificant Significant! trial stopped trial stopped End of experiment Insignificant Significant! Significant! Significant!
  • 35.
    @OptimiseOrDie The 95%Stopping Problem
  • 36.
    The 95% StoppingProblem @OptimiseOrDie
  • 37.
    The 95% StoppingProblem @OptimiseOrDie abtestguide.com/calc/
  • 38.
    62.5cm +/- 1cm @OptimiseOrDie 9.1% Âą 0.5 9.3% Âą 0.5 9.1% Âą 0.2 9.3% Âą 0.2 9.1% Âą 0.1 9.3% Âą 0.1
  • 39.
    Graph is a range, not a line: 9.1 Âą 1.9% 9.1 Âą 0.9% 9.1 Âą 0.3%
  • 40.
    The 95% StoppingProblem “You should know that stopping a test once it’s significant is deadly sin number 1 in A/B testing land. 77% of A/A tests (testing the same thing as A and B) will reach significance at a certain point.” Ton Wesseling, Online Dialogue “I always tell people that you need a representative sample if your data needs to be valid. What does ‘representative’ mean? First of all you need to include all the weekdays and weekends. You need different weather, because it impacts buyer behaviour. But most important: Your traffic needs to have all traffic sources, especially newsletter, special campaigns, TV,… everything! The longer the test runs, the more insights you get. Andre Morys, Web Arts
  • 41.
    Three Articles youMUST read “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP
  • 42.
    Business & PurchaseCycles • Customers change • Your traffic mix changes • Markets, competitors • Be aware of all the waves • Always test whole cycles • Minimum 2 cycles (wk/mo) • Don’t exclude slower buyers @OptimiseOrDie Start Test Finish Avg Cycle
  • 43.
    When to stop? • MINIMUM two business cycles (week/mo.) • MINIMUM of 1 purchase cycle • MINIMUM 250 outcomes/conversions per creative • MORE if relative difference is low • ALWAYS test full weeks • KNOW what marketing and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time • Run it • Stop it • Analyse the data • When do I run over? Not enough data… @OptimiseOrDie
  • 44.
    44 #6 :You peek too early!
  • 45.
    #6 : Theearly stages of a test… • Ignore the graphs. Don’t draw conclusions. Don’t dance. Calm down. • Get a feel for the test but don’t do anything yet! • Remember – in A/B -­‐ 50% of returning visitors will see a new shiny website! • UnBl your test has had at least 2 business cycles and 250+ outcomes, don’t bother even gewng remotely excited! • Watching regularly is good though. You’re looking for anything that looks really odd – if everyone is looking (but not concluding) then oddiBes will get spoIed. • All tests move around or show big swings early in the tesBng cycle. Here is a very high traffic site – it sBll takes 10 days to start seIling. Lower traffic sites will stretch this period further. 45
  • 46.
    #7 : NoQA testing for the AB test?
  • 47.
    #7 – BIG SECRET! • Over 40% of tests have had QA issues. • Over £20M in browser conversion issues! Browser tesBng www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.cloudtesting.com www.multibrowserviewer.com www.saucelabs.com Tablets & Mobiles www.deviceanywhere.com www.perfectomobile.com FREE Device lab! www.opendevicelab.com @OptimiseOrDie
  • 48.
    #7 : Whatother QA testing should I do? • TesBng from several locaBons (office, home, elsewhere) • TesBng the IP filtering is set up • Test tags are firing correctly (analyBcs and the test tool) • Test as a repeat visitor and check session Bmeouts • Cross check figures from 2+ sources • Monitor closely from launch, recheck, watch • WATCH FOR BIAS! @OptimiseOrDie
  • 49.
    #8 : Testsare random and not prioritised Once you have a list of potential test areas, rank them by opportunity vs. effort. The common ranking metrics that I use include: • Opportunity (revenue, impact) • Dev resource • Time to market • Risk / Complexity Make yourself a quadrant diagram and plot them
  • 50.
    #9 : Velocityor Scope problems 0 6 12 18 Months Conversion @OptimiseOrDie
  • 51.
    #9 : Widenthe optimisation scope @OptimiseOrDie
  • 52.
    #9 : Solutions • Give Priority Boarding for opportunities – The best seats reserved for metric shifters • Release more often to close the gap – More testing resource helps, analytics ‘hawk eye’ • Kaizen – continuous improvement – Others call it JFDI (just f***ing do it) • Make changes AS WELL as tests, basically! – These small things add up as well as compounding effort • Run simultaneous tests – With analytics integration, decoding this becomes easy • Online Hair Booking – over 100 tiny tweaks – No functional changes at all – 37% improvement • Completed in-between product releases – The added lift for 10 days work, worth 360k @OptimiseOrDie
  • 53.
    53 #11 :Your test fails @OptimiseOrDie
  • 54.
    #11: Your testfails • Learn from the failure! If you can’t learn from the failure, you’ve designed a crap test. • Next Bme you design, imagine all your stuff failing. What would you do? If you don’t know or you’re not sure, get it changed so that a negaBve becomes insigh{ul. • So : failure itself at a creaBve or variable level should tell you something. • On a failed test, always analyse the segmentaBon and analyBcs • One or more segments will be over and under • Check for varied performance • Now add the failure info to your Knowledge Base: • Look at it carefully – what does the failure tell you? Which element do you think drove the failure? • If you know what failed (e.g. making the price bigger) then you have very useful informaBon • You turned the handle the wrong way • Now brainstorm a new test @OptimiseOrDie
  • 55.
    #12 : Thetest is ‘about the same’ • Analyse the segmentaBon • Check the analyBcs and instrumentaBon • One or more segments may be over and under • They may be cancelling out – the average is a lie • The segment level performance will help you (beware of small sample sizes) • If you genuinely have a test which failed to move any segments, it’s a crap test – be bolder • This usually happens when it isn’t bold or brave enough in shiing away from the original design, parBcularly on lower traffic sites • Get tesBng again! @OptimiseOrDie
  • 56.
    #13 : Thetest keeps moving around • There are three reasons it is moving around – Your sample size (outcomes) is sBll too small – The external traffic mix, customers or reacBon has suddenly changed or – Your inbound markeBng driven traffic mix is completely volaBle (very rare) • Check the sample size • Check all your markeBng acBvity • Check the instrumentaBon • If no reason, check segmentaBon @OptimiseOrDie
  • 57.
    #14 : Thetest has flipped on me • Something like this can happen: • Check your sample size. If it’s still small, then expect this until the test settles. • If the test does genuinely flip – and quite severely – then something has changed with the traffic mix, the customer base or your advertising. Maybe the PPC budget ran out? Seriously! • To analyse a flipped test, you’ll need to check your segmented data. This is why you have a split testing package AND an analytics system. • The segmented data will help you to identify the source of the shift in response to your test. I rarely get a flipped one and it’s always something changing on me, without being told. The heartless bastards.
  • 58.
    • No – and this is why: – It’s a waste of Bme – It’s easier to test and monitor instead – You are eaBng into test Bme – Also applies to A/A/B/B tesBng – A/B/A running at 25%/50%/25% is the best • Read my post here : hIp://bit.ly/WcI9EZ 58 #15 : Should I run an A/A test first
  • 59.
    #16 : Nobodyfeels the test • You promised a 25% rise in checkouts -­‐ you only see 2% • Traffic, AdverBsing, MarkeBng may have changed • Check they’re using the same precise metrics • Run a calibraBon exercise • I oen leave a 5 or 10% stub running in a test • This tracks old creaBve once new one goes live • If conversion is also down for that one, BINGO! • Remember – the AB test is an esBmate – it doesn’t precisely record future performance • This is why infrequent tesBng is bad • Always be trying a new test instead of basking in the glory of one you ran 6 months ago. You’re only as good as your next test. @OptimiseOrDie
  • 60.
    #17 : Youforgot about Mobile & Tablet • If you’re AB tesBng a responsive site, pay aIenBon • Content will break differently on many screens • Know thy users and their devices • Use bango or google analyBcs to define a test list • Make sure you test mobile devices & viewports • What looks good on your desk may not be for the user • Harder to design cross device tests • You’ll need to segment mobile, tablet & desktop response in the analyBcs or AB tesBng package • Your personal phone is not a device mix • Ask me about making your device list • Buy core devices, rent the rest from deviceanywhere.com @OptimiseOrDie
  • 61.
    #18 : Ohshit – no traffic • If small volumes, contact customers – reach out. • If data volumes aren’t there, there are sBll customers! • Drive design from levers you can apply – game the system • Pick clean and simple clusters of change (hypothesis driven) • Use a goal at an earlier ring stage or funnel step • Beware of using clickthroughs when aIriBon is high on the other side • Try before and aer tesBng on idenBcal Bme periods (measure in analyBcs model) • Be careful about small sample sizes (<100 outcomes) • Are you working automated emails? • Fix JFDI, performance and UX issues too!
  • 62.
    #18 : Ohshit – no traffic • Forget MVT or A/B/N tests – run your numbers • Test things with high impact – don’t be a wuss! • Use UX, Session Replay to aid insight • Run a task gap survey (4Q style) • Run a dropped basket survey (LF style) • Run a general survey + check social + other sites • Run sitewide tests that appear on all pages or large clusters of pages – • UVPs (“We are a cool brand”), USPs (“Free returns!”), UCPs (“10% off today”). • Headers, Footers, Nudge Bars, USP bars, footer changes, NavigaBon, Product pages, Delivery info etc.
  • 63.
    #19 : Ichose the wrong test type • A/B tes.ng – good for: – A single change of content or design layout – A group of related changes (e.g. payment security) – Finding a new and radical shi for a template design – Lower traffic pages or shorter test Bmes • Mul.variate tes.ng – good for: – Higher traffic pages – Groups of unrelated changes (e.g. delivery & security) – MulBple content or design style changes – Finding specific drivers of test lis – TesBng mulBple versions (e.g. click here, book now, go) – Where you need to understand strong and weak cross variable interacBons – Don’t use to seIle arguments or sloppy thinking!
  • 64.
    Netherlands A/B Shi Example Previous winner +7.25% +8.19% addiBonal li
  • 65.
    #20 – Other flavours of tes.ng • Micro tes.ng (.ny change) – good for: – Proving to the boss that tesBng works – DemonstraBng to IT that it works without impact – Showing the impact of a seemingly Bny change – Proof of concept before larger test • Funnel tes.ng – good for: – Checkouts – Lead gen – Forms processes – QuotaBons – Any mulB-­‐step process with data entry • Fake it and Build it – good for: – TesBng new business ideas – Trying out promoBons on a test sample – EsBmaBng impact before you build – Helps you calculate ROI – You can even split test enBre server farms Vs.
  • 66.
    #20 – Other flavours of tes.ng “Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy.”
  • 67.
    Top F***ups for 2014 1. TesBng in the wrong place 2. Your hypothesis inputs are crap 3. No analyBcs integraBon 4. Your test will finish aer you die 5. You don’t test for long enough 6. You peek before it’s ready 7. No QA for your split test 8. OpportuniBes are not prioriBsed 9. TesBng cycles are too slow 10. You don’t know when tests are ready 11. Your test fails 12. The test is ‘about the same’ 13. Test flips behaviour 14. Test keeps moving around 15. You run an A/A test and waste Bme 16. Nobody ‘feels’ the test 17. You forgot you were responsive 18. You forgot you had no traffic 19. You ran the wrong test type 20. You didn’t try all the flavours of tesBng @OptimiseOrDie
  • 68.
  • 69.
    2004 Headspace WhatI thought I knew in 2004 Reality
  • 70.
    2014 Headspace WhatI know I know On a good day
  • 71.
  • 72.
  • 73.
  • 74.
    The 5 LeggedOptimisation Barstool @OptimiseOrDie #1 Smart Talented Polymath People Flexible and Agile teams
  • 75.
  • 76.
    #2 : AnalyticsInvestment (tools, people, dev time) @OptimiseOrDie
  • 77.
    @OptimiseOrDie #3 :User research and insight
  • 78.
    @OptimiseOrDie #3 :THE BEST IDEAS COME FROM?
  • 79.
    #4 : GREATCOPYWRITING “On the average, five times as many people read the headline as read the body copy. When you have written your headline, you have spent eighty cents out of your dollar.” David Ogilvy “In 9 years and 40M split tests with visitors, the majority of my testing success came from playing with the words.” @OptimiseOrDie
  • 80.
    #4 : COPYWRITINGTIP “Would you like to…” “I would like to...” http://www.theguardian.com/info/developer-blog/ 2013/jan/02/interactive-button-text-grammar Use the WYLTIWLT test for button copy!
  • 81.
    • Google ContentExperiments bit.ly/Ljg7Ds • Optimizely www.optimizely.com • Visual Website Optimizer www.visualwebsiteoptimizer.com • Multi Armed Bandit Explanation bit.ly/Xa80O8 • New Machine Learning Tools www.conductrics.com www.rekko.com @OptimiseOrDie #5 : Split Testing Tools
  • 82.
    The 5 LeggedOptimisation Barstool #1 Culture & Team #2 Toolkit & Analytics investment #3 UX, CX, Service Design, Insight #4 Persuasive Copywriting #5 Experimentation (testing) tools @OptimiseOrDie
  • 83.
  • 84.
  • 85.
  • 86.
    #5 : FINDSTUFF @OptimiseOrDie @danbarker Analytics @fastbloke Analytics @timlb Analytics @jamesgurd Analytics @therustybear Analytics @carmenmardiros Analytics @davechaffey Analytics @priteshpatel9 Analytics @cutroni Analytics @avinash Analytics @Aschottmuller Analytics, CRO @cartmetrix Analytics, CRO @Kissmetrics CRO / UX @Unbounce CRO / UX @Morys CRO / Neuro @UXFeeds UX / Neuro @Psyblog Neuro @Gfiorelli1 SEO / Analytics @PeepLaja CRO @TheGrok CRO @UIE UX @LukeW UX / Forms @cjforms UX / Forms @axbom UX @iatv UX @Chudders Photo UX @JeffreyGroks Innovation @StephanieRieger Innovation @BrianSolis Innovation @DrEscotet Neuro @TheBrainLady Neuro @RogerDooley Neuro @Cugelman Neuro @Smashingmag Dev / UX @uxmag UX @Webtrends UX / CRO
  • 87.
    #5 : LEARNSTUFF @OptimiseOrDie Baymard.com Lukew.com Smashingmagazine.com ConversionXL.com Medium.com Whichtestwon.com Unbounce.com Measuringusability.com RogerDooley.com Kissmetrics.com Uxmatters.com Smartinsights.com Econsultancy.com Cutroni.com www.GetMentalNotes.com
  • 88.
    #12 : TheBest Companies… • Invest con.nually in analyBcs instrumentaBon, tools, people • Use an Agile, itera.ve, cross-­‐silo, one team project culture • Prefer collabora.ve tools to having lots of meeBngs • Priori.se development based on numbers and insight • PracBce real con.nuous product improvement, not SLEDD* • Are fixing bugs, cru, bad stuff as well as opBmising • Source photos and content that support persuasion and uBlity • Have cross channel, cross device design, tesBng and QA • Segment their data for valuable insights, every test or change • Con.nually reduce cycle (iteraBon) Bme in their process • Blend ‘long’ design, conBnuous improvement AND split tests • Make op.misa.on the engine of change, * Single Large Expensive Doomed Developments not the slave of ego
  • 89.
  • 90.
    Get your quesBons in! Mail : sullivac@gmail.com Deck : slideshare.com/sullivac Linkedin : linkd.in/pvrg14 @OptimiseOrDie