SlideShare a Scribd company logo
1 of 89
20 ways to Shaft your Split testing 
@OptimiseOrDie
@OptimiseOrDi 
e 
• UX, Analytics, Testing and Innovation 
• Started doing testing & CRO 2004 
• Split tested over 40M visitors in 19 
languages 
• 60+ mistakes with AB testing 
• I’ve made every one of them 
• Like riding a bike… 
• Get in touch for workshops, skill transfer, 
CRO methodology design, training and 
programme mentoring…
@OptimiseOrDie 
Hands on!
AB Test Hype Cycle 
Zen Plumbing 
@OptimiseOrDie 
Timeline 
Tested stupid ideas, 
lots 
Most AB or MVT tests are 
bullshit 
Discovered AB 
testing 
Triage, 
Triangulation, 
Prioritisation, Maths
@OptimiseOrDie
@OptimiseOrDie
Oppan Gangnam Style! 
@OptimiseOrDie
#1 : You’re doing it in the wrong 
place 
@OptimiseOrDie
#1 : You’re doing it in the wrong place 
There are 4 areas a CRO expert always looks at: 
1. Inbound attrition (medium, source, landing page, 
keyword, intent and many more…) 
2. Key conversion points (product, basket, registration) 
3. Processes, lifecycles and steps (forms, logins, 
registration, checkout, onboarding, emails, push) 
4. Layers of engagement (search, category, product, add) 
1. Use visitor flow reports for attrition – very useful. 
2. For key conversion points, look at loss rates & 
interactions 
3. Processes and steps – look at funnels or make your own 
4. Layers and engagement – make a ring model 
@OptimiseOrDie
Examples – Concept 
Bounce 
Engage 
Outcome 
@OptimiseOrDie
Examples – 16-25Railcard.co.uk 
Bounce 
Login to 
Account 
Content 
Engage 
Start 
Application 
Type and 
Details 
Eligibility 
Photo 
Complete 
@OptimiseOrDie
Examples – Guide Dogs 
Bounce 
Content 
Engage 
Donation 
Pathway 
Donation 
Page 
Starts 
process 
Funnel 
steps 
Complete 
@OptimiseOrDie
Within a layer 
Page 1 
Page 2 
Page 3 
Page 4 Page 5 
Exit 
Deeper 
Layer 
Email 
Wishlist 
Contact Like 
Micro 
Conversions 
@OptimiseOrDie
#1 : Make a Money Model 
• Get to know the flow and loss (leaks) inbound, inside and 
through key processes or conversion points. 
• Once you know the key steps you’re losing people at and how 
much traffic you have – make a money model. 
• 20,000 see the basket page – what’s the basket page to 
checkout page ratio? 
• Estimate how much you think you can shift the key metric 
(e.g. basket adds, basket -> checkout) 
• What downstream revenue or profit would that generate? 
• Sort by the money column 
• Congratulations – you’ve now built the worlds first IT plan for 
growth with a return on investment estimate attached! 
• I’ll talk more about prioritising later – but a good real world 
analogy for you to use: 
@OptimiseOrDie
Think like a 
store owner! 
If you can’t 
refurbish the 
entire store, 
which floors or 
departments will 
you invest in 
optimising? 
Wherever there 
is: 
• Footfall 
• Low return 
@OptimiseOrDie
#2 : Your hypothesis is 
crap! 
Insight - Inputs 
#FAIL 
Competitor 
copying 
Guessing 
Dice rolling 
Panic 
Competitor 
change 
An article 
the CEO 
read 
Ego 
Opinion 
Cherished 
notions 
Marketing 
whims Cosmic rays 
Not ‘on 
brand’ 
enough 
IT 
inflexibility 
Internal 
company 
needs 
Some 
dumbass 
consultant 
Shiny 
feature 
blindness 
Knee jerk 
reactons 
@OptimiseOrDie
#2 : These are the inputs you 
need… 
Insight - Inputs 
Insight 
Eye tracking 
Segmentation 
Surveys 
Sales and 
Call Centre 
Customer 
contact 
Social 
analytics 
Session 
Replay 
Usability 
testing 
Forms 
analytics 
Search 
analytics Voice of 
Customer 
Market 
research 
A/B and 
MVT testing 
Big & 
unstructured 
data 
Web 
analytics 
Competitor 
Customer evals 
services 
@OptimiseOrDie
Insight - Inputs 
@OptimiseOrDie 
#2 : Brainstorming the test 
• Check your inputs 
• Assemble the widest possible team 
• Share your data and research 
• Design Emotive Writing guidelines
Insight - Inputs 
@OptimiseOrDie 
#2 : Emotive Writing - example 
Customers do not know what to do and need support and advice 
• Emphasize the fact that you understand that their situation is stressful 
• Emphasize your expertise and leadership in vehicle glazing and will help them get the best 
solution for their situation 
• Explain what they will need to do online and during the call-back so that they know what the 
next steps will be 
• Explain that they will be able ask any other questions they might have during the call-back 
Customers do not feel confident in assessing the damage 
• Emphasize the fact that you will help them assess the damage correctly online 
Customers need to understand the benefits of booking online 
• Emphasize that the online booking system is quick, easy and provides all the information 
they need in regards with their appointment and general cost information 
Customers mistrust insurers and find dealing with their insurance situation very frustrating 
• Where possible communicate the fact that the job is most likely to be free for insured 
customers, or good value for money for cash customers 
• Show that you understand the hassle of dealing with insurance companies – emphasise that 
you will help with their insurance paperwork for them, freeing them of this burden 
Some customers cannot be bothered to take action to fix their car glass 
• Emphasize the consequences of not doing anything, 
e.g. ‘It’s going to cost you more if the chip develops into a crack’
Insight - Inputs 
@OptimiseOrDie 
#2 : THE DARK SIDE 
“Keep your family safe and get back on the 
road fast with Autoglass.”
Insight - Inputs 
@OptimiseOrDie 
#2 : NOW YOU CAN BEGIN 
• You should have inputs, research, data, guidelines 
• Sit down with the team and prompt with 12 questions: 
– Who is this page (or process) for? 
– What problem does this solve for the user? 
– How do we know they need it? 
– What is the primary action we want people to take? 
– What might prompt the user to take this action? 
– How will we know if this is doing what we want it to do? 
– How do people get to this page? 
– How long are people here on this page? 
– What can we remove from this page? 
– How can we test this solution with people? 
– How are we solving the users needs in different and better ways than other 
places on our site? 
– If this is a homepage, ask these too (bit.ly/1fX2RAa)
Insight - Inputs 
@OptimiseOrDie 
#2 : PROMPT YOURSELF 
• Check your UX or Copywriting 
guidelines. 
• Use Get Mental Notes 
• What levers can we apply now? 
• Create a hypothesis: 
“WE BELIEVE THAT DOING [A] 
FOR PEOPLE [B] WILL MAKE 
OUTCOME [C] HAPPEN. 
WE'LL KNOW THIS WHEN WE 
SEE DATA [D] AND FEEDBACK 
[E]” 
www.GetMentalNotes.com
Insight - Inputs 
@OptimiseOrDie 
#2 : THE FUN BIT! 
• Collaborative Sketching 
• Brainwriting 
• Refine and Test!
We believe that doing [A] for 
People [B] will make 
outcome [C] happen. 
We’ll know this when we 
observe data [D] and obtain 
feedback [E]. (reverse) 
@OptimiseOrDie
#2 : Solutions 
• You need multiple tool inputs 
– Tool decks are here : www.slideshare.net/sullivac 
• Collaborative, Customer connected team 
– If you’re not doing this, you’re hosed 
• Session replay tools provide vital input 
– Get vital additional customer evidence 
• Simple page Analytics don’t cut it 
– Invest in your analytics, especially event 
tracking 
• Ego, Opinion, Cherished notions – fill gaps 
– Fill these vacuums with insights and data 
• Champion the user 
– Give them a chair at every meeting @OptimiseOrDie
#2 : HYPOTHESIS DESIGN SUMMARY 
Insight - Inputs 
@OptimiseOrDie 
• Inputs – get the right stuff 
• Research, Guidelines, Data 
• Framing the problem(s) 
• Questions to get you going 
• Use card prompts for Psychology 
• Create a hypothesis 
• Collaborative Sketching 
• Brainwriting 
• Refine and Check Hypothesis 
• Instrument and Test
#3 : No analytics integration 
• Investigating problems with tests 
• Segmentation of results 
• Tests that fail, flip or move around 
• Tests that don’t make sense 
• Broken test setups 
• What drives the averages you see? 
@OptimiseOrDie
29 
A B B A
These Danish 
porn sites are 
so hardcore! 
We’re still 
waiting for our 
AB tests to 
finish! 
#4 : The test will finish after you die 
• Use a test length calculator like this one: 
• visualwebsiteoptimizer.com/ab-split-test-duration/
@OptimiseOrDie 
#5 : You get false results
The 95% Stopping Problem 
• Many people use 95, 99% ‘confidence’ to stop 
• This value is unreliable 
• Read this Nature article : bit.ly/1dwk0if 
• You can hit 95% early in a test 
• If you stop, it could be a false positive 
• Tools need to be smarter about inference 
• This 95% thingy – it’s last on your list for reasons to 
stop testing 
• Let me explain 
@OptimiseOrDie
#5 : When to stop 
• Self stopping is a huge problem: 
– “I stopped the test when it looked good” 
– “It hit 20% on Thursday, so I figured – time to cut and run” 
– “We need test time for something else. Looks good to us” 
– “We’ve got a big sample now so why not finish it today?” 
• False Positives and Negatives 
– If you cut part of a business cycle, you bias the segments you have in 
the test. 
– So if you ignore weekend shoppers by stopping your test on Friday, that 
will affect results 
– The other problems is FALSE POSITIVES and FALSE NEGATIVES 
@OptimiseOrDie
#5 : When to stop 
Scenario 1 Scenario 2 Scenario 3 Scenario 4 
@OptimiseOrDie 
After 200 
observations 
Insignificant Insignificant Significant! Significant! 
After 500 
observations 
Insignificant Significant! Insignificant Significant! 
End of 
experiment 
Insignificant Significant! Insignificant Significant! 
Scenario 1 Scenario 2 Scenario 3 Scenario 4 
After 200 
observations 
Insignificant Insignificant Significant! Significant! 
After 500 
observations 
Insignificant Significant! trial stopped trial stopped 
End of 
experiment 
Insignificant Significant! Significant! Significant!
@OptimiseOrDie 
The 95% Stopping Problem
The 95% Stopping Problem 
@OptimiseOrDie
The 95% Stopping Problem 
@OptimiseOrDie 
abtestguide.com/calc/
62.5cm 
+/- 1cm 
@OptimiseOrDie 
9.1% 
± 0.5 
9.3% 
± 0.5 
9.1% 
± 0.2 
9.3% 
± 0.2 
9.1% 
± 0.1 
9.3% 
± 0.1
Graph is a range, not a line: 
9.1 ± 1.9% 9.1 ± 0.9% 9.1 ± 0.3%
The 95% Stopping Problem 
“You should know that stopping a test once it’s significant is 
deadly sin number 1 in A/B testing land. 77% of A/A tests (testing 
the same thing as A and B) will reach significance at a certain 
point.” 
Ton Wesseling, Online Dialogue 
“I always tell people that you need a representative sample if 
your data needs to be valid. What does ‘representative’ mean? 
First of all you need to include all the weekdays and weekends. 
You need different weather, because it impacts buyer behaviour. 
But most important: Your traffic needs to have all traffic sources, 
especially newsletter, special campaigns, TV,… everything! The 
longer the test runs, the more insights you get. 
Andre Morys, Web Arts
Three Articles you MUST read 
“Statistical Significance does not equal Validity” 
http://bit.ly/1wMfmY2 
“Why every Internet Marketer should be a Statistician” 
http://bit.ly/1wMfs1G 
“Understanding the Cycles in your site” 
http://mklnd.com/1pGSOUP
Business & Purchase Cycles 
@OptimiseOrDie 
Start Test Finish Avg Cycle 
• Customers change 
• Your traffic mix changes 
• Markets, competitors 
• Be aware of all the waves 
• Always test whole cycles 
• Minimum 2 cycles (wk/mo) 
• Don’t exclude slower buyers
When to stop? 
• MINIMUM two business cycles (week/mo.) 
• MINIMUM of 1 purchase cycle 
• MINIMUM 250 outcomes/conversions per creative 
• MORE if relative difference is low 
• ALWAYS test full weeks 
• KNOW what marketing and cycles are doing 
• RUN a test length calculator - bit.ly/XqCxuu 
• SET your test run time 
• Run it 
• Stop it 
• Analyse the data 
• When do I run over? Not enough data… 
@OptimiseOrDie
44 
#6 : You peek too early!
#6 : The early stages of a test… 
• Ignore the graphs. Don’t draw conclusions. Don’t dance. Calm down. 
• Get a feel for the test but don’t do anything yet! 
• Remember – in A/B - 50% of returning visitors will see a new shiny website! 
• Until your test has had at least 2 business cycles and 250+ outcomes, don’t bother 
even getting remotely excited! 
• Watching regularly is good though. You’re looking for anything that looks really 
odd – if everyone is looking (but not concluding) then oddities will get spotted. 
• All tests move around or show big swings early in the testing cycle. Here is a very 
high traffic site – it still takes 10 days to start settling. Lower traffic sites will 
stretch this period further. 
45
#7 : No QA 
testing for the AB 
test?
#7 – BIG SECRET! 
• Over 40% of tests have had QA issues. 
• Over £20M in browser conversion issues! 
Browser testing www.crossbrowsertesting.com 
www.browserstack.com 
www.spoon.net 
www.cloudtesting.com 
www.multibrowserviewer.com 
www.saucelabs.com 
Tablets & Mobiles www.deviceanywhere.com 
www.perfectomobile.com 
FREE Device lab! www.opendevicelab.com 
@OptimiseOrDie
#7 : What other QA testing should I do? 
• Testing from several locations (office, home, elsewhere) 
• Testing the IP filtering is set up 
• Test tags are firing correctly (analytics and the test tool) 
• Test as a repeat visitor and check session timeouts 
• Cross check figures from 2+ sources 
• Monitor closely from launch, recheck, watch 
• WATCH FOR BIAS! 
@OptimiseOrDie
#8 : Tests are random and not 
prioritised 
Once you have a list of 
potential test areas, rank 
them by opportunity vs. 
effort. 
The common ranking 
metrics that I use include: 
•Opportunity (revenue, 
impact) 
•Dev resource 
•Time to market 
•Risk / Complexity 
Make yourself a quadrant
#9 : Velocity or Scope problems 
0 6 12 18 
Months 
Conversio 
n 
@OptimiseOrDie
#9 : Widen the optimisation 
scope 
@OptimiseOrDie
#9 : Solutions 
• Give Priority Boarding for opportunities 
– The best seats reserved for metric shifters 
• Release more often to close the gap 
– More testing resource helps, analytics ‘hawk eye’ 
• Kaizen – continuous improvement 
– Others call it JFDI (just f***ing do it) 
• Make changes AS WELL as tests, basically! 
– These small things add up as well as compounding effort 
• Run simultaneous tests 
– With analytics integration, decoding this becomes easy 
• Online Hair Booking – over 100 tiny 
tweaks 
– No functional changes at all – 37% improvement 
• Completed in-between product releases 
– The added lift for 10 days work, worth 360k @OptimiseOrDie
53 
#11 : Your test 
fails 
@OptimiseOrDie
#11: Your test fails 
• Learn from the failure! If you can’t learn from the failure, you’ve 
designed a crap test. 
• Next time you design, imagine all your stuff failing. What would 
you do? If you don’t know or you’re not sure, get it changed so 
that a negative becomes insightful. 
• So : failure itself at a creative or variable level should tell you 
something. 
• On a failed test, always analyse the segmentation and analytics 
• One or more segments will be over and under 
• Check for varied performance 
• Now add the failure info to your Knowledge Base: 
• Look at it carefully – what does the failure tell you? Which 
element do you think drove the failure? 
• If you know what failed (e.g. making the price bigger) then you 
have very useful information 
• You turned the handle the wrong way 
• Now brainstorm a new test 
@OptimiseOrDie
#12 : The test is ‘about the same’ 
• Analyse the segmentation 
• Check the analytics and instrumentation 
• One or more segments may be over and under 
• They may be cancelling out – the average is a lie 
• The segment level performance will help you (beware of 
small sample sizes) 
• If you genuinely have a test which failed to move any 
segments, it’s a crap test – be bolder 
• This usually happens when it isn’t bold or brave enough in 
shifting away from the original design, particularly on 
lower traffic sites 
• Get testing again! 
@OptimiseOrDie
#13 : The test keeps moving 
around 
• There are three reasons it is moving around 
– Your sample size (outcomes) is still too small 
– The external traffic mix, customers or reaction has 
suddenly changed or 
– Your inbound marketing driven traffic mix is 
completely volatile (very rare) 
• Check the sample size 
• Check all your marketing activity 
• Check the instrumentation 
• If no reason, check segmentation 
@OptimiseOrDie
#14 : The test has flipped on me 
• Something like this can happen: 
• Check your sample size. If it’s still small, then expect this until the test 
settles. 
• If the test does genuinely flip – and quite severely – then something has 
changed with the traffic mix, the customer base or your advertising. 
Maybe the PPC budget ran out? Seriously! 
• To analyse a flipped test, you’ll need to check your segmented data. This 
is why you have a split testing package AND an analytics system. 
• The segmented data will help you to identify the source of the shift in 
response to your test. I rarely get a flipped one and it’s always something
• No – and this is why: 
– It’s a waste of time 
– It’s easier to test and monitor instead 
– You are eating into test time 
– Also applies to A/A/B/B testing 
– A/B/A running at 25%/50%/25% is the best 
• Read my post here : 
http://bit.ly/WcI9EZ 
58 
#15 : Should I run an A/A test 
first
#16 : Nobody feels the 
test 
• You promised a 25% rise in checkouts - you only see 2% 
• Traffic, Advertising, Marketing may have changed 
• Check they’re using the same precise metrics 
• Run a calibration exercise 
• I often leave a 5 or 10% stub running in a test 
• This tracks old creative once new one goes live 
• If conversion is also down for that one, BINGO! 
• Remember – the AB test is an estimate – it doesn’t 
precisely record future performance 
• This is why infrequent testing is bad 
• Always be trying a new test instead of basking in the 
glory of one you ran 6 months ago. You’re only as good 
as your next test. 
@OptimiseOrDie
#17 : You forgot about Mobile & 
Tablet 
• If you’re AB testing a responsive site, pay attention 
• Content will break differently on many screens 
• Know thy users and their devices 
• Use bango or google analytics to define a test list 
• Make sure you test mobile devices & viewports 
• What looks good on your desk may not be for the user 
• Harder to design cross device tests 
• You’ll need to segment mobile, tablet & desktop response 
in the analytics or AB testing package 
• Your personal phone is not a device mix 
• Ask me about making your device list 
• Buy core devices, rent the rest from deviceanywhere.com 
@OptimiseOrDie
#18 : Oh shit – no traffic 
• If small volumes, contact customers – reach out. 
• If data volumes aren’t there, there are still customers! 
• Drive design from levers you can apply – game the system 
• Pick clean and simple clusters of change (hypothesis driven) 
• Use a goal at an earlier ring stage or funnel step 
• Beware of using clickthroughs when attrition is high on the 
other side 
• Try before and after testing on identical time periods 
(measure in analytics model) 
• Be careful about small sample sizes (<100 outcomes) 
• Are you working automated emails? 
• Fix JFDI, performance and UX issues too!
#18 : Oh shit – no traffic 
• Forget MVT or A/B/N tests – run your numbers 
• Test things with high impact – don’t be a wuss! 
• Use UX, Session Replay to aid insight 
• Run a task gap survey (4Q style) 
• Run a dropped basket survey (LF style) 
• Run a general survey + check social + other sites 
• Run sitewide tests that appear on all pages or large clusters 
of pages – 
• UVPs (“We are a cool brand”), USPs (“Free returns!”), UCPs 
(“10% off today”). 
• Headers, Footers, Nudge Bars, USP bars, footer changes, 
Navigation, Product pages, Delivery info etc.
#19 : I chose the wrong test 
type 
• A/B testing – good for: 
– A single change of content or design layout 
– A group of related changes (e.g. payment security) 
– Finding a new and radical shift for a template design 
– Lower traffic pages or shorter test times 
• Multivariate testing – good for: 
– Higher traffic pages 
– Groups of unrelated changes (e.g. delivery & security) 
– Multiple content or design style changes 
– Finding specific drivers of test lifts 
– Testing multiple versions (e.g. click here, book now, go) 
– Where you need to understand strong and weak cross variable 
interactions 
– Don’t use to settle arguments or sloppy thinking!
Netherlands A/B Shift Example 
Previous winner 
+7.25% 
+8.19% additional lift
#20 – Other flavours of testing 
• Micro testing (tiny change) – good for: 
– Proving to the boss that testing works 
– Demonstrating to IT that it works without impact 
– Showing the impact of a seemingly tiny change 
– Proof of concept before larger test 
• Funnel testing – good for: 
– Checkouts 
– Lead gen 
– Forms processes 
– Quotations 
– Any multi-step process with data entry 
• Fake it and Build it – good for: 
– Testing new business ideas 
– Trying out promotions on a test sample 
– Estimating impact before you build 
– Helps you calculate ROI 
– You can even split test entire server farms 
Vs.
#20 – Other flavours of testing 
“Congratulations! 
Today you’re the 
lucky winner of our 
random awards 
programme. You 
get all these extra 
features for free, 
on us. Enjoy.”
Top F***ups for 2014 
1. Testing in the wrong place 
2. Your hypothesis inputs are crap 
3. No analytics integration 
4. Your test will finish after you die 
5. You don’t test for long enough 
6. You peek before it’s ready 
7. No QA for your split test 
8. Opportunities are not prioritised 
9. Testing cycles are too slow 
10. You don’t know when tests are ready 
11. Your test fails 
12. The test is ‘about the same’ 
13. Test flips behaviour 
14. Test keeps moving around 
15. You run an A/A test and waste time 
16. Nobody ‘feels’ the test 
17. You forgot you were responsive 
18. You forgot you had no traffic 
19. You ran the wrong test type 
20. You didn’t try all the flavours of testing 
@OptimiseOrDie
WE’RE ALL WINGING IT
2004 Headspace 
What I thought 
I knew in 2004 
Reality
2014 Headspace 
What I 
know I 
know 
On a 
good day
Guessaholics Anonymous
Rumsfeldian 
Space 
@OptimiseOrDie
Rumsfeldian 
Space 
@OptimiseOrDie
#1 Smart Talented Polymath People 
The 5 Legged Optimisation 
Barstool 
@OptimiseOrDie 
Flexible and Agile teams
Fittest? Agile! 
@OptimiseOrDie
#2 : Analytics Investment (tools, people, dev 
time) 
@OptimiseOrDie
@OptimiseOrDie 
#3 : User research and 
insight
#3 : THE BEST IDEAS COME FROM? 
@OptimiseOrDie
#4 : GREAT COPYWRITING 
“On the average, five times as many people 
read the headline as read the body copy. When 
you have written your headline, you have spent 
eighty cents out of your dollar.” 
David Ogilvy 
“In 9 years and 40M split tests with visitors, the 
majority of my testing success came from 
playing with the words.” 
@OptimiseOrDie
• Google Content Experiments 
bit.ly/Ljg7Ds 
• Optimizely 
www.optimizely.com 
• Visual Website Optimizer 
www.visualwebsiteoptimizer.com 
• Multi Armed Bandit Explanation 
bit.ly/Xa80O8 
• New Machine Learning Tools 
www.conductrics.com 
www.rekko.com 
@OptimiseOrDie 
#5 : Split Testing Tools
The 5 Legged Optimisation 
@OptimiseOrDie 
Barstool 
#1 Culture & Team 
#2 Toolkit & Analytics investment 
#3 UX, CX, Service Design, Insight 
#4 Persuasive Copywriting 
#5 Experimentation (testing) tools
READ STUFF
READ STUFF
READ STUFF
#5 : FIND STUFF 
@OptimiseOrDie 
@danbarker Analytics 
@fastbloke Analytics 
@timlb Analytics 
@jamesgurd Analytics 
@therustybear Analytics 
@carmenmardiros Analytics 
@davechaffey Analytics 
@priteshpatel9 Analytics 
@cutroni Analytics 
@avinash Analytics 
@Aschottmuller Analytics, CRO 
@cartmetrix Analytics, 
CRO 
@Kissmetrics CRO / UX 
@Unbounce CRO / UX 
@Morys CRO / Neuro 
@UXFeeds UX / Neuro 
@Psyblog Neuro 
@Gfiorelli1 SEO / Analytics 
@PeepLaja CRO 
@TheGrok CRO 
@UIE UX 
@LukeW UX / Forms 
@cjforms UX / Forms 
@axbom UX 
@iatv UX 
@Chudders Photo UX 
@JeffreyGroks Innovation 
@StephanieRieger Innovation 
@BrianSolis Innovation 
@DrEscotet Neuro 
@TheBrainLady Neuro 
@RogerDooley Neuro 
@Cugelman Neuro 
@Smashingmag Dev / UX 
@uxmag UX 
@Webtrends UX / 
CRO
#5 : LEARN STUFF 
@OptimiseOrDie 
Baymard.com 
Lukew.com 
Smashingmagazine.com 
ConversionXL.com 
Medium.com 
Whichtestwon.com 
Unbounce.com 
Measuringusability.com 
RogerDooley.com 
Kissmetrics.com 
Uxmatters.com 
Smartinsights.com 
Econsultancy.com 
Cutroni.com 
www.GetMentalNotes.com
#12 : The Best Companies… 
• Invest continually in analytics instrumentation, tools, people 
• Use an Agile, iterative, cross-silo, one team project culture 
• Prefer collaborative tools to having lots of meetings 
• Prioritise development based on numbers and insight 
• Practice real continuous product improvement, not SLEDD* 
• Are fixing bugs, cruft, bad stuff as well as optimising 
• Source photos and content that support persuasion and utility 
• Have cross channel, cross device design, testing and QA 
• Segment their data for valuable insights, every test or change 
• Continually reduce cycle (iteration) time in their process 
• Blend ‘long’ design, continuous improvement AND split tests 
• Make optimisation the engine of change, not the slave of ego 
* Single Large Expensive Doomed Developments
THE FUTURE OF TESTING
Projects? Questions? Mail me! 
Mail : sullivac@gmail.com 
Deck : slideshare.com/sullivac 
Linkedin : linkd.in/pvrg14 
@OptimiseOrDie

More Related Content

What's hot

12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...Craig Sullivan
 
Myths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split TestingMyths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split TestingCraig Sullivan
 
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014Craig Sullivan
 
Conversion Research in One Hour
Conversion Research in One HourConversion Research in One Hour
Conversion Research in One HourCraig Sullivan
 
Product Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieProduct Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieCraig Sullivan
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeCraig Sullivan
 
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB testsCraig Sullivan
 
Onboard like a juggernaut - Elite camp 2015
Onboard like a juggernaut - Elite camp 2015Onboard like a juggernaut - Elite camp 2015
Onboard like a juggernaut - Elite camp 2015Conversionista
 
Why do my AB tests suck? measurecamp
Why do my AB tests suck?   measurecampWhy do my AB tests suck?   measurecamp
Why do my AB tests suck? measurecampCraig Sullivan
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROCraig Sullivan
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meCraig Sullivan
 
Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Mobile presentation - Sydney Online Retailer - 26 Sep 2011Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Mobile presentation - Sydney Online Retailer - 26 Sep 2011Craig Sullivan
 
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB TestingCraig Sullivan
 
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...PRWD
 
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...Craig Sullivan
 
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Marketing Festival
 
eMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype CycleeMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype CycleCraig Sullivan
 
Natural born conversion killers - Conversion Jam
Natural born conversion killers - Conversion JamNatural born conversion killers - Conversion Jam
Natural born conversion killers - Conversion JamCraig Sullivan
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9Craig Sullivan
 
OrangeHills flyer
OrangeHills flyerOrangeHills flyer
OrangeHills flyerorangehills
 

What's hot (20)

12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
 
Myths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split TestingMyths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split Testing
 
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
 
Conversion Research in One Hour
Conversion Research in One HourConversion Research in One Hour
Conversion Research in One Hour
 
Product Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieProduct Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to die
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
 
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
 
Onboard like a juggernaut - Elite camp 2015
Onboard like a juggernaut - Elite camp 2015Onboard like a juggernaut - Elite camp 2015
Onboard like a juggernaut - Elite camp 2015
 
Why do my AB tests suck? measurecamp
Why do my AB tests suck?   measurecampWhy do my AB tests suck?   measurecamp
Why do my AB tests suck? measurecamp
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
 
Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Mobile presentation - Sydney Online Retailer - 26 Sep 2011Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Mobile presentation - Sydney Online Retailer - 26 Sep 2011
 
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
 
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
 
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
 
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
 
eMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype CycleeMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype Cycle
 
Natural born conversion killers - Conversion Jam
Natural born conversion killers - Conversion JamNatural born conversion killers - Conversion Jam
Natural born conversion killers - Conversion Jam
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
 
OrangeHills flyer
OrangeHills flyerOrangeHills flyer
OrangeHills flyer
 

Viewers also liked

Joost de Valk - SES Amsterdam
Joost de Valk - SES AmsterdamJoost de Valk - SES Amsterdam
Joost de Valk - SES AmsterdamJoost de Valk
 
Infographic cmssystemen - Seminar Overheid Online
Infographic cmssystemen - Seminar Overheid OnlineInfographic cmssystemen - Seminar Overheid Online
Infographic cmssystemen - Seminar Overheid OnlineYouwe Drecomm OOIP
 
It's not all about top tasks - lightning talk @ Confab 2013 London
It's not all about top tasks - lightning talk @ Confab 2013 LondonIt's not all about top tasks - lightning talk @ Confab 2013 London
It's not all about top tasks - lightning talk @ Confab 2013 Londondaphneshinn
 
Tim Manders | Embedding a thesaurus ContentCafé #10
Tim Manders | Embedding a thesaurus ContentCafé #10Tim Manders | Embedding a thesaurus ContentCafé #10
Tim Manders | Embedding a thesaurus ContentCafé #10ContentCafé
 
Presenter kennissessie 16 november 2010
Presenter kennissessie 16 november 2010Presenter kennissessie 16 november 2010
Presenter kennissessie 16 november 2010Tim van Waard
 
Friends of search 2016 - een samenvatting in beeld
Friends of search 2016 - een samenvatting in beeldFriends of search 2016 - een samenvatting in beeld
Friends of search 2016 - een samenvatting in beeldJudith Verhoef
 
Conversion Hotel 2014: Michael Aagaard (DK) keynote
Conversion Hotel 2014: Michael Aagaard (DK) keynoteConversion Hotel 2014: Michael Aagaard (DK) keynote
Conversion Hotel 2014: Michael Aagaard (DK) keynoteWebanalisten .nl
 
Top 100 merken whitepaper greenberry 2012
Top 100 merken whitepaper greenberry 2012Top 100 merken whitepaper greenberry 2012
Top 100 merken whitepaper greenberry 2012Greenberry
 
Conversion Hotel 2014: Bart Schutz (NL) keynote
Conversion Hotel 2014: Bart Schutz (NL) keynoteConversion Hotel 2014: Bart Schutz (NL) keynote
Conversion Hotel 2014: Bart Schutz (NL) keynoteWebanalisten .nl
 
Conversion Hotel 2014: Brian Massey (US) keynote
Conversion Hotel 2014: Brian Massey (US) keynoteConversion Hotel 2014: Brian Massey (US) keynote
Conversion Hotel 2014: Brian Massey (US) keynoteWebanalisten .nl
 
"Contentmarketing als cultuurverandering" door Tristan Lavender van Deloitte ...
"Contentmarketing als cultuurverandering" door Tristan Lavender van Deloitte ..."Contentmarketing als cultuurverandering" door Tristan Lavender van Deloitte ...
"Contentmarketing als cultuurverandering" door Tristan Lavender van Deloitte ...Content Club Nederland
 
Welches Content Management System für welche Fragestellung?
Welches Content Management System für welche Fragestellung? Welches Content Management System für welche Fragestellung?
Welches Content Management System für welche Fragestellung? Dominic Brander
 

Viewers also liked (17)

Joost de Valk - SES Amsterdam
Joost de Valk - SES AmsterdamJoost de Valk - SES Amsterdam
Joost de Valk - SES Amsterdam
 
Infographic cmssystemen - Seminar Overheid Online
Infographic cmssystemen - Seminar Overheid OnlineInfographic cmssystemen - Seminar Overheid Online
Infographic cmssystemen - Seminar Overheid Online
 
It's not all about top tasks - lightning talk @ Confab 2013 London
It's not all about top tasks - lightning talk @ Confab 2013 LondonIt's not all about top tasks - lightning talk @ Confab 2013 London
It's not all about top tasks - lightning talk @ Confab 2013 London
 
Tim Manders | Embedding a thesaurus ContentCafé #10
Tim Manders | Embedding a thesaurus ContentCafé #10Tim Manders | Embedding a thesaurus ContentCafé #10
Tim Manders | Embedding a thesaurus ContentCafé #10
 
Confab MN
Confab MNConfab MN
Confab MN
 
Presenter kennissessie 16 november 2010
Presenter kennissessie 16 november 2010Presenter kennissessie 16 november 2010
Presenter kennissessie 16 november 2010
 
Conversion hotel 2015 de kracht van content
Conversion hotel 2015 de kracht van contentConversion hotel 2015 de kracht van content
Conversion hotel 2015 de kracht van content
 
Schrijven voor het web
Schrijven voor het webSchrijven voor het web
Schrijven voor het web
 
Friends of search 2016 - een samenvatting in beeld
Friends of search 2016 - een samenvatting in beeldFriends of search 2016 - een samenvatting in beeld
Friends of search 2016 - een samenvatting in beeld
 
Conversion Hotel 2014: Michael Aagaard (DK) keynote
Conversion Hotel 2014: Michael Aagaard (DK) keynoteConversion Hotel 2014: Michael Aagaard (DK) keynote
Conversion Hotel 2014: Michael Aagaard (DK) keynote
 
Top 100 merken whitepaper greenberry 2012
Top 100 merken whitepaper greenberry 2012Top 100 merken whitepaper greenberry 2012
Top 100 merken whitepaper greenberry 2012
 
Conversion Hotel 2014: Bart Schutz (NL) keynote
Conversion Hotel 2014: Bart Schutz (NL) keynoteConversion Hotel 2014: Bart Schutz (NL) keynote
Conversion Hotel 2014: Bart Schutz (NL) keynote
 
Conversion Hotel 2014: Brian Massey (US) keynote
Conversion Hotel 2014: Brian Massey (US) keynoteConversion Hotel 2014: Brian Massey (US) keynote
Conversion Hotel 2014: Brian Massey (US) keynote
 
'Nog vragen'? De klant in beeld. Gastcollege UU 25 oktober 2016
'Nog vragen'? De klant in beeld. Gastcollege UU 25 oktober 2016'Nog vragen'? De klant in beeld. Gastcollege UU 25 oktober 2016
'Nog vragen'? De klant in beeld. Gastcollege UU 25 oktober 2016
 
"Contentmarketing als cultuurverandering" door Tristan Lavender van Deloitte ...
"Contentmarketing als cultuurverandering" door Tristan Lavender van Deloitte ..."Contentmarketing als cultuurverandering" door Tristan Lavender van Deloitte ...
"Contentmarketing als cultuurverandering" door Tristan Lavender van Deloitte ...
 
Presentatie Succesvolle contentmigratie in 5 stappen
Presentatie Succesvolle contentmigratie in 5 stappenPresentatie Succesvolle contentmigratie in 5 stappen
Presentatie Succesvolle contentmigratie in 5 stappen
 
Welches Content Management System für welche Fragestellung?
Welches Content Management System für welche Fragestellung? Welches Content Management System für welche Fragestellung?
Welches Content Management System für welche Fragestellung?
 

Similar to 20 Ways to Shaft your Split Tesring : Conversion Conference

Conversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteConversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteWebanalisten .nl
 
Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...
Why does my Mobile Conversion rate suck?  19 Sep 2013 @ Conversion Thursday #...Why does my Mobile Conversion rate suck?  19 Sep 2013 @ Conversion Thursday #...
Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...Craig Sullivan
 
Toolkits and tips of the conversion pros v 1.6
Toolkits and tips of the conversion pros v 1.6Toolkits and tips of the conversion pros v 1.6
Toolkits and tips of the conversion pros v 1.6Craig Sullivan
 
Toolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanToolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanUXPA UK
 
John lewis lego craig sullivan 1335
John lewis lego craig sullivan  1335John lewis lego craig sullivan  1335
John lewis lego craig sullivan 1335Charlie Lines
 
Conversion Optimization Willa Fogarty
Conversion Optimization Willa FogartyConversion Optimization Willa Fogarty
Conversion Optimization Willa FogartyWilla Fogarty
 
Product Management - Successful products, Strategy, Finding a job, B2B vs B2C...
Product Management - Successful products, Strategy, Finding a job, B2B vs B2C...Product Management - Successful products, Strategy, Finding a job, B2B vs B2C...
Product Management - Successful products, Strategy, Finding a job, B2B vs B2C...Rahul Deshpande
 
Lean startup & customer development
Lean startup & customer developmentLean startup & customer development
Lean startup & customer developmentEmanuele Musa
 
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...Northern User Experience
 
Conversion Optimization Webninar with Peep Laja
Conversion Optimization Webninar with Peep Laja Conversion Optimization Webninar with Peep Laja
Conversion Optimization Webninar with Peep Laja Optimizely
 
Conversion Optimization with Peep Laja
Conversion Optimization with Peep LajaConversion Optimization with Peep Laja
Conversion Optimization with Peep LajaElla Quivooij
 
Velocity Campus - Customer Discovery 2
Velocity Campus - Customer Discovery 2Velocity Campus - Customer Discovery 2
Velocity Campus - Customer Discovery 2Henry S
 
Planning a Successful Online Business
Planning a Successful Online BusinessPlanning a Successful Online Business
Planning a Successful Online Businesslisabaadeseo
 
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...Invest Northern Ireland
 
12 reasons your site sucks - InvestNI
12 reasons your site sucks - InvestNI12 reasons your site sucks - InvestNI
12 reasons your site sucks - InvestNICraig Sullivan
 
Is Growth Hacking Bullshit?
Is Growth Hacking Bullshit?Is Growth Hacking Bullshit?
Is Growth Hacking Bullshit?Matt Lerner
 
Inside eCommerce - Micksgarage Masterclass - April 15th 2014.
Inside eCommerce - Micksgarage Masterclass - April 15th 2014.Inside eCommerce - Micksgarage Masterclass - April 15th 2014.
Inside eCommerce - Micksgarage Masterclass - April 15th 2014.John Walsh
 
Growth hacking 101 - Matt Lerner
Growth hacking 101 - Matt LernerGrowth hacking 101 - Matt Lerner
Growth hacking 101 - Matt LernerMatt Lerner
 
Customer Feedback: the missing piece of the Agile puzzle
Customer Feedback: the missing piece of the Agile puzzleCustomer Feedback: the missing piece of the Agile puzzle
Customer Feedback: the missing piece of the Agile puzzleskierkowski
 
Lean Analytics & Analytics Dashboards
Lean Analytics & Analytics DashboardsLean Analytics & Analytics Dashboards
Lean Analytics & Analytics DashboardsYves Ferket
 

Similar to 20 Ways to Shaft your Split Tesring : Conversion Conference (20)

Conversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteConversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynote
 
Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...
Why does my Mobile Conversion rate suck?  19 Sep 2013 @ Conversion Thursday #...Why does my Mobile Conversion rate suck?  19 Sep 2013 @ Conversion Thursday #...
Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...
 
Toolkits and tips of the conversion pros v 1.6
Toolkits and tips of the conversion pros v 1.6Toolkits and tips of the conversion pros v 1.6
Toolkits and tips of the conversion pros v 1.6
 
Toolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanToolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig Sullivan
 
John lewis lego craig sullivan 1335
John lewis lego craig sullivan  1335John lewis lego craig sullivan  1335
John lewis lego craig sullivan 1335
 
Conversion Optimization Willa Fogarty
Conversion Optimization Willa FogartyConversion Optimization Willa Fogarty
Conversion Optimization Willa Fogarty
 
Product Management - Successful products, Strategy, Finding a job, B2B vs B2C...
Product Management - Successful products, Strategy, Finding a job, B2B vs B2C...Product Management - Successful products, Strategy, Finding a job, B2B vs B2C...
Product Management - Successful products, Strategy, Finding a job, B2B vs B2C...
 
Lean startup & customer development
Lean startup & customer developmentLean startup & customer development
Lean startup & customer development
 
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
 
Conversion Optimization Webninar with Peep Laja
Conversion Optimization Webninar with Peep Laja Conversion Optimization Webninar with Peep Laja
Conversion Optimization Webninar with Peep Laja
 
Conversion Optimization with Peep Laja
Conversion Optimization with Peep LajaConversion Optimization with Peep Laja
Conversion Optimization with Peep Laja
 
Velocity Campus - Customer Discovery 2
Velocity Campus - Customer Discovery 2Velocity Campus - Customer Discovery 2
Velocity Campus - Customer Discovery 2
 
Planning a Successful Online Business
Planning a Successful Online BusinessPlanning a Successful Online Business
Planning a Successful Online Business
 
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
 
12 reasons your site sucks - InvestNI
12 reasons your site sucks - InvestNI12 reasons your site sucks - InvestNI
12 reasons your site sucks - InvestNI
 
Is Growth Hacking Bullshit?
Is Growth Hacking Bullshit?Is Growth Hacking Bullshit?
Is Growth Hacking Bullshit?
 
Inside eCommerce - Micksgarage Masterclass - April 15th 2014.
Inside eCommerce - Micksgarage Masterclass - April 15th 2014.Inside eCommerce - Micksgarage Masterclass - April 15th 2014.
Inside eCommerce - Micksgarage Masterclass - April 15th 2014.
 
Growth hacking 101 - Matt Lerner
Growth hacking 101 - Matt LernerGrowth hacking 101 - Matt Lerner
Growth hacking 101 - Matt Lerner
 
Customer Feedback: the missing piece of the Agile puzzle
Customer Feedback: the missing piece of the Agile puzzleCustomer Feedback: the missing piece of the Agile puzzle
Customer Feedback: the missing piece of the Agile puzzle
 
Lean Analytics & Analytics Dashboards
Lean Analytics & Analytics DashboardsLean Analytics & Analytics Dashboards
Lean Analytics & Analytics Dashboards
 

More from Craig Sullivan

Elite Camp 2013 - Estonia
Elite Camp 2013 - EstoniaElite Camp 2013 - Estonia
Elite Camp 2013 - EstoniaCraig Sullivan
 
Conversionista : Conversion manager course - Stockholm 20 march 2013
Conversionista : Conversion manager course  - Stockholm 20 march 2013Conversionista : Conversion manager course  - Stockholm 20 march 2013
Conversionista : Conversion manager course - Stockholm 20 march 2013Craig Sullivan
 
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 20133 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013Craig Sullivan
 
Measure camp tools of the cro rabble
Measure camp   tools of the cro rabbleMeasure camp   tools of the cro rabble
Measure camp tools of the cro rabbleCraig Sullivan
 
5 cro tools that i can't live without
5 cro tools that i can't live without5 cro tools that i can't live without
5 cro tools that i can't live withoutCraig Sullivan
 
eMetrics Stockholm - What the F*** is wrong with my conversion?
eMetrics Stockholm - What the F*** is wrong with my conversion?eMetrics Stockholm - What the F*** is wrong with my conversion?
eMetrics Stockholm - What the F*** is wrong with my conversion?Craig Sullivan
 

More from Craig Sullivan (6)

Elite Camp 2013 - Estonia
Elite Camp 2013 - EstoniaElite Camp 2013 - Estonia
Elite Camp 2013 - Estonia
 
Conversionista : Conversion manager course - Stockholm 20 march 2013
Conversionista : Conversion manager course  - Stockholm 20 march 2013Conversionista : Conversion manager course  - Stockholm 20 march 2013
Conversionista : Conversion manager course - Stockholm 20 march 2013
 
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 20133 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
 
Measure camp tools of the cro rabble
Measure camp   tools of the cro rabbleMeasure camp   tools of the cro rabble
Measure camp tools of the cro rabble
 
5 cro tools that i can't live without
5 cro tools that i can't live without5 cro tools that i can't live without
5 cro tools that i can't live without
 
eMetrics Stockholm - What the F*** is wrong with my conversion?
eMetrics Stockholm - What the F*** is wrong with my conversion?eMetrics Stockholm - What the F*** is wrong with my conversion?
eMetrics Stockholm - What the F*** is wrong with my conversion?
 

Recently uploaded

『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书rnrncn29
 
Top 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxTop 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxDyna Gilbert
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119APNIC
 
Company Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxCompany Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxMario
 
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书rnrncn29
 
Film cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasaFilm cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasa494f574xmv
 
TRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxTRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxAndrieCagasanAkio
 
SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predieusebiomeyer
 
Unidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxUnidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxmibuzondetrabajo
 
ETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxNIMMANAGANTI RAMAKRISHNA
 
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书zdzoqco
 

Recently uploaded (11)

『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
『澳洲文凭』买拉筹伯大学毕业证书成绩单办理澳洲LTU文凭学位证书
 
Top 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxTop 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptx
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119
 
Company Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptxCompany Snapshot Theme for Business by Slidesgo.pptx
Company Snapshot Theme for Business by Slidesgo.pptx
 
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
『澳洲文凭』买詹姆士库克大学毕业证书成绩单办理澳洲JCU文凭学位证书
 
Film cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasaFilm cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasa
 
TRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptxTRENDS Enabling and inhibiting dimensions.pptx
TRENDS Enabling and inhibiting dimensions.pptx
 
SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predi
 
Unidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptxUnidad 4 – Redes de ordenadores (en inglés).pptx
Unidad 4 – Redes de ordenadores (en inglés).pptx
 
ETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptxETHICAL HACKING dddddddddddddddfnandni.pptx
ETHICAL HACKING dddddddddddddddfnandni.pptx
 
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
 

20 Ways to Shaft your Split Tesring : Conversion Conference

  • 1. 20 ways to Shaft your Split testing @OptimiseOrDie
  • 2. @OptimiseOrDi e • UX, Analytics, Testing and Innovation • Started doing testing & CRO 2004 • Split tested over 40M visitors in 19 languages • 60+ mistakes with AB testing • I’ve made every one of them • Like riding a bike… • Get in touch for workshops, skill transfer, CRO methodology design, training and programme mentoring…
  • 4.
  • 5. AB Test Hype Cycle Zen Plumbing @OptimiseOrDie Timeline Tested stupid ideas, lots Most AB or MVT tests are bullshit Discovered AB testing Triage, Triangulation, Prioritisation, Maths
  • 8. Oppan Gangnam Style! @OptimiseOrDie
  • 9. #1 : You’re doing it in the wrong place @OptimiseOrDie
  • 10. #1 : You’re doing it in the wrong place There are 4 areas a CRO expert always looks at: 1. Inbound attrition (medium, source, landing page, keyword, intent and many more…) 2. Key conversion points (product, basket, registration) 3. Processes, lifecycles and steps (forms, logins, registration, checkout, onboarding, emails, push) 4. Layers of engagement (search, category, product, add) 1. Use visitor flow reports for attrition – very useful. 2. For key conversion points, look at loss rates & interactions 3. Processes and steps – look at funnels or make your own 4. Layers and engagement – make a ring model @OptimiseOrDie
  • 11. Examples – Concept Bounce Engage Outcome @OptimiseOrDie
  • 12. Examples – 16-25Railcard.co.uk Bounce Login to Account Content Engage Start Application Type and Details Eligibility Photo Complete @OptimiseOrDie
  • 13. Examples – Guide Dogs Bounce Content Engage Donation Pathway Donation Page Starts process Funnel steps Complete @OptimiseOrDie
  • 14. Within a layer Page 1 Page 2 Page 3 Page 4 Page 5 Exit Deeper Layer Email Wishlist Contact Like Micro Conversions @OptimiseOrDie
  • 15. #1 : Make a Money Model • Get to know the flow and loss (leaks) inbound, inside and through key processes or conversion points. • Once you know the key steps you’re losing people at and how much traffic you have – make a money model. • 20,000 see the basket page – what’s the basket page to checkout page ratio? • Estimate how much you think you can shift the key metric (e.g. basket adds, basket -> checkout) • What downstream revenue or profit would that generate? • Sort by the money column • Congratulations – you’ve now built the worlds first IT plan for growth with a return on investment estimate attached! • I’ll talk more about prioritising later – but a good real world analogy for you to use: @OptimiseOrDie
  • 16. Think like a store owner! If you can’t refurbish the entire store, which floors or departments will you invest in optimising? Wherever there is: • Footfall • Low return @OptimiseOrDie
  • 17. #2 : Your hypothesis is crap! Insight - Inputs #FAIL Competitor copying Guessing Dice rolling Panic Competitor change An article the CEO read Ego Opinion Cherished notions Marketing whims Cosmic rays Not ‘on brand’ enough IT inflexibility Internal company needs Some dumbass consultant Shiny feature blindness Knee jerk reactons @OptimiseOrDie
  • 18. #2 : These are the inputs you need… Insight - Inputs Insight Eye tracking Segmentation Surveys Sales and Call Centre Customer contact Social analytics Session Replay Usability testing Forms analytics Search analytics Voice of Customer Market research A/B and MVT testing Big & unstructured data Web analytics Competitor Customer evals services @OptimiseOrDie
  • 19. Insight - Inputs @OptimiseOrDie #2 : Brainstorming the test • Check your inputs • Assemble the widest possible team • Share your data and research • Design Emotive Writing guidelines
  • 20. Insight - Inputs @OptimiseOrDie #2 : Emotive Writing - example Customers do not know what to do and need support and advice • Emphasize the fact that you understand that their situation is stressful • Emphasize your expertise and leadership in vehicle glazing and will help them get the best solution for their situation • Explain what they will need to do online and during the call-back so that they know what the next steps will be • Explain that they will be able ask any other questions they might have during the call-back Customers do not feel confident in assessing the damage • Emphasize the fact that you will help them assess the damage correctly online Customers need to understand the benefits of booking online • Emphasize that the online booking system is quick, easy and provides all the information they need in regards with their appointment and general cost information Customers mistrust insurers and find dealing with their insurance situation very frustrating • Where possible communicate the fact that the job is most likely to be free for insured customers, or good value for money for cash customers • Show that you understand the hassle of dealing with insurance companies – emphasise that you will help with their insurance paperwork for them, freeing them of this burden Some customers cannot be bothered to take action to fix their car glass • Emphasize the consequences of not doing anything, e.g. ‘It’s going to cost you more if the chip develops into a crack’
  • 21. Insight - Inputs @OptimiseOrDie #2 : THE DARK SIDE “Keep your family safe and get back on the road fast with Autoglass.”
  • 22. Insight - Inputs @OptimiseOrDie #2 : NOW YOU CAN BEGIN • You should have inputs, research, data, guidelines • Sit down with the team and prompt with 12 questions: – Who is this page (or process) for? – What problem does this solve for the user? – How do we know they need it? – What is the primary action we want people to take? – What might prompt the user to take this action? – How will we know if this is doing what we want it to do? – How do people get to this page? – How long are people here on this page? – What can we remove from this page? – How can we test this solution with people? – How are we solving the users needs in different and better ways than other places on our site? – If this is a homepage, ask these too (bit.ly/1fX2RAa)
  • 23. Insight - Inputs @OptimiseOrDie #2 : PROMPT YOURSELF • Check your UX or Copywriting guidelines. • Use Get Mental Notes • What levers can we apply now? • Create a hypothesis: “WE BELIEVE THAT DOING [A] FOR PEOPLE [B] WILL MAKE OUTCOME [C] HAPPEN. WE'LL KNOW THIS WHEN WE SEE DATA [D] AND FEEDBACK [E]” www.GetMentalNotes.com
  • 24. Insight - Inputs @OptimiseOrDie #2 : THE FUN BIT! • Collaborative Sketching • Brainwriting • Refine and Test!
  • 25. We believe that doing [A] for People [B] will make outcome [C] happen. We’ll know this when we observe data [D] and obtain feedback [E]. (reverse) @OptimiseOrDie
  • 26. #2 : Solutions • You need multiple tool inputs – Tool decks are here : www.slideshare.net/sullivac • Collaborative, Customer connected team – If you’re not doing this, you’re hosed • Session replay tools provide vital input – Get vital additional customer evidence • Simple page Analytics don’t cut it – Invest in your analytics, especially event tracking • Ego, Opinion, Cherished notions – fill gaps – Fill these vacuums with insights and data • Champion the user – Give them a chair at every meeting @OptimiseOrDie
  • 27. #2 : HYPOTHESIS DESIGN SUMMARY Insight - Inputs @OptimiseOrDie • Inputs – get the right stuff • Research, Guidelines, Data • Framing the problem(s) • Questions to get you going • Use card prompts for Psychology • Create a hypothesis • Collaborative Sketching • Brainwriting • Refine and Check Hypothesis • Instrument and Test
  • 28. #3 : No analytics integration • Investigating problems with tests • Segmentation of results • Tests that fail, flip or move around • Tests that don’t make sense • Broken test setups • What drives the averages you see? @OptimiseOrDie
  • 29. 29 A B B A
  • 30. These Danish porn sites are so hardcore! We’re still waiting for our AB tests to finish! #4 : The test will finish after you die • Use a test length calculator like this one: • visualwebsiteoptimizer.com/ab-split-test-duration/
  • 31. @OptimiseOrDie #5 : You get false results
  • 32. The 95% Stopping Problem • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable • Read this Nature article : bit.ly/1dwk0if • You can hit 95% early in a test • If you stop, it could be a false positive • Tools need to be smarter about inference • This 95% thingy – it’s last on your list for reasons to stop testing • Let me explain @OptimiseOrDie
  • 33. #5 : When to stop • Self stopping is a huge problem: – “I stopped the test when it looked good” – “It hit 20% on Thursday, so I figured – time to cut and run” – “We need test time for something else. Looks good to us” – “We’ve got a big sample now so why not finish it today?” • False Positives and Negatives – If you cut part of a business cycle, you bias the segments you have in the test. – So if you ignore weekend shoppers by stopping your test on Friday, that will affect results – The other problems is FALSE POSITIVES and FALSE NEGATIVES @OptimiseOrDie
  • 34. #5 : When to stop Scenario 1 Scenario 2 Scenario 3 Scenario 4 @OptimiseOrDie After 200 observations Insignificant Insignificant Significant! Significant! After 500 observations Insignificant Significant! Insignificant Significant! End of experiment Insignificant Significant! Insignificant Significant! Scenario 1 Scenario 2 Scenario 3 Scenario 4 After 200 observations Insignificant Insignificant Significant! Significant! After 500 observations Insignificant Significant! trial stopped trial stopped End of experiment Insignificant Significant! Significant! Significant!
  • 35. @OptimiseOrDie The 95% Stopping Problem
  • 36. The 95% Stopping Problem @OptimiseOrDie
  • 37. The 95% Stopping Problem @OptimiseOrDie abtestguide.com/calc/
  • 38. 62.5cm +/- 1cm @OptimiseOrDie 9.1% ± 0.5 9.3% ± 0.5 9.1% ± 0.2 9.3% ± 0.2 9.1% ± 0.1 9.3% ± 0.1
  • 39. Graph is a range, not a line: 9.1 ± 1.9% 9.1 ± 0.9% 9.1 ± 0.3%
  • 40. The 95% Stopping Problem “You should know that stopping a test once it’s significant is deadly sin number 1 in A/B testing land. 77% of A/A tests (testing the same thing as A and B) will reach significance at a certain point.” Ton Wesseling, Online Dialogue “I always tell people that you need a representative sample if your data needs to be valid. What does ‘representative’ mean? First of all you need to include all the weekdays and weekends. You need different weather, because it impacts buyer behaviour. But most important: Your traffic needs to have all traffic sources, especially newsletter, special campaigns, TV,… everything! The longer the test runs, the more insights you get. Andre Morys, Web Arts
  • 41. Three Articles you MUST read “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP
  • 42. Business & Purchase Cycles @OptimiseOrDie Start Test Finish Avg Cycle • Customers change • Your traffic mix changes • Markets, competitors • Be aware of all the waves • Always test whole cycles • Minimum 2 cycles (wk/mo) • Don’t exclude slower buyers
  • 43. When to stop? • MINIMUM two business cycles (week/mo.) • MINIMUM of 1 purchase cycle • MINIMUM 250 outcomes/conversions per creative • MORE if relative difference is low • ALWAYS test full weeks • KNOW what marketing and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time • Run it • Stop it • Analyse the data • When do I run over? Not enough data… @OptimiseOrDie
  • 44. 44 #6 : You peek too early!
  • 45. #6 : The early stages of a test… • Ignore the graphs. Don’t draw conclusions. Don’t dance. Calm down. • Get a feel for the test but don’t do anything yet! • Remember – in A/B - 50% of returning visitors will see a new shiny website! • Until your test has had at least 2 business cycles and 250+ outcomes, don’t bother even getting remotely excited! • Watching regularly is good though. You’re looking for anything that looks really odd – if everyone is looking (but not concluding) then oddities will get spotted. • All tests move around or show big swings early in the testing cycle. Here is a very high traffic site – it still takes 10 days to start settling. Lower traffic sites will stretch this period further. 45
  • 46. #7 : No QA testing for the AB test?
  • 47. #7 – BIG SECRET! • Over 40% of tests have had QA issues. • Over £20M in browser conversion issues! Browser testing www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.cloudtesting.com www.multibrowserviewer.com www.saucelabs.com Tablets & Mobiles www.deviceanywhere.com www.perfectomobile.com FREE Device lab! www.opendevicelab.com @OptimiseOrDie
  • 48. #7 : What other QA testing should I do? • Testing from several locations (office, home, elsewhere) • Testing the IP filtering is set up • Test tags are firing correctly (analytics and the test tool) • Test as a repeat visitor and check session timeouts • Cross check figures from 2+ sources • Monitor closely from launch, recheck, watch • WATCH FOR BIAS! @OptimiseOrDie
  • 49. #8 : Tests are random and not prioritised Once you have a list of potential test areas, rank them by opportunity vs. effort. The common ranking metrics that I use include: •Opportunity (revenue, impact) •Dev resource •Time to market •Risk / Complexity Make yourself a quadrant
  • 50. #9 : Velocity or Scope problems 0 6 12 18 Months Conversio n @OptimiseOrDie
  • 51. #9 : Widen the optimisation scope @OptimiseOrDie
  • 52. #9 : Solutions • Give Priority Boarding for opportunities – The best seats reserved for metric shifters • Release more often to close the gap – More testing resource helps, analytics ‘hawk eye’ • Kaizen – continuous improvement – Others call it JFDI (just f***ing do it) • Make changes AS WELL as tests, basically! – These small things add up as well as compounding effort • Run simultaneous tests – With analytics integration, decoding this becomes easy • Online Hair Booking – over 100 tiny tweaks – No functional changes at all – 37% improvement • Completed in-between product releases – The added lift for 10 days work, worth 360k @OptimiseOrDie
  • 53. 53 #11 : Your test fails @OptimiseOrDie
  • 54. #11: Your test fails • Learn from the failure! If you can’t learn from the failure, you’ve designed a crap test. • Next time you design, imagine all your stuff failing. What would you do? If you don’t know or you’re not sure, get it changed so that a negative becomes insightful. • So : failure itself at a creative or variable level should tell you something. • On a failed test, always analyse the segmentation and analytics • One or more segments will be over and under • Check for varied performance • Now add the failure info to your Knowledge Base: • Look at it carefully – what does the failure tell you? Which element do you think drove the failure? • If you know what failed (e.g. making the price bigger) then you have very useful information • You turned the handle the wrong way • Now brainstorm a new test @OptimiseOrDie
  • 55. #12 : The test is ‘about the same’ • Analyse the segmentation • Check the analytics and instrumentation • One or more segments may be over and under • They may be cancelling out – the average is a lie • The segment level performance will help you (beware of small sample sizes) • If you genuinely have a test which failed to move any segments, it’s a crap test – be bolder • This usually happens when it isn’t bold or brave enough in shifting away from the original design, particularly on lower traffic sites • Get testing again! @OptimiseOrDie
  • 56. #13 : The test keeps moving around • There are three reasons it is moving around – Your sample size (outcomes) is still too small – The external traffic mix, customers or reaction has suddenly changed or – Your inbound marketing driven traffic mix is completely volatile (very rare) • Check the sample size • Check all your marketing activity • Check the instrumentation • If no reason, check segmentation @OptimiseOrDie
  • 57. #14 : The test has flipped on me • Something like this can happen: • Check your sample size. If it’s still small, then expect this until the test settles. • If the test does genuinely flip – and quite severely – then something has changed with the traffic mix, the customer base or your advertising. Maybe the PPC budget ran out? Seriously! • To analyse a flipped test, you’ll need to check your segmented data. This is why you have a split testing package AND an analytics system. • The segmented data will help you to identify the source of the shift in response to your test. I rarely get a flipped one and it’s always something
  • 58. • No – and this is why: – It’s a waste of time – It’s easier to test and monitor instead – You are eating into test time – Also applies to A/A/B/B testing – A/B/A running at 25%/50%/25% is the best • Read my post here : http://bit.ly/WcI9EZ 58 #15 : Should I run an A/A test first
  • 59. #16 : Nobody feels the test • You promised a 25% rise in checkouts - you only see 2% • Traffic, Advertising, Marketing may have changed • Check they’re using the same precise metrics • Run a calibration exercise • I often leave a 5 or 10% stub running in a test • This tracks old creative once new one goes live • If conversion is also down for that one, BINGO! • Remember – the AB test is an estimate – it doesn’t precisely record future performance • This is why infrequent testing is bad • Always be trying a new test instead of basking in the glory of one you ran 6 months ago. You’re only as good as your next test. @OptimiseOrDie
  • 60. #17 : You forgot about Mobile & Tablet • If you’re AB testing a responsive site, pay attention • Content will break differently on many screens • Know thy users and their devices • Use bango or google analytics to define a test list • Make sure you test mobile devices & viewports • What looks good on your desk may not be for the user • Harder to design cross device tests • You’ll need to segment mobile, tablet & desktop response in the analytics or AB testing package • Your personal phone is not a device mix • Ask me about making your device list • Buy core devices, rent the rest from deviceanywhere.com @OptimiseOrDie
  • 61. #18 : Oh shit – no traffic • If small volumes, contact customers – reach out. • If data volumes aren’t there, there are still customers! • Drive design from levers you can apply – game the system • Pick clean and simple clusters of change (hypothesis driven) • Use a goal at an earlier ring stage or funnel step • Beware of using clickthroughs when attrition is high on the other side • Try before and after testing on identical time periods (measure in analytics model) • Be careful about small sample sizes (<100 outcomes) • Are you working automated emails? • Fix JFDI, performance and UX issues too!
  • 62. #18 : Oh shit – no traffic • Forget MVT or A/B/N tests – run your numbers • Test things with high impact – don’t be a wuss! • Use UX, Session Replay to aid insight • Run a task gap survey (4Q style) • Run a dropped basket survey (LF style) • Run a general survey + check social + other sites • Run sitewide tests that appear on all pages or large clusters of pages – • UVPs (“We are a cool brand”), USPs (“Free returns!”), UCPs (“10% off today”). • Headers, Footers, Nudge Bars, USP bars, footer changes, Navigation, Product pages, Delivery info etc.
  • 63. #19 : I chose the wrong test type • A/B testing – good for: – A single change of content or design layout – A group of related changes (e.g. payment security) – Finding a new and radical shift for a template design – Lower traffic pages or shorter test times • Multivariate testing – good for: – Higher traffic pages – Groups of unrelated changes (e.g. delivery & security) – Multiple content or design style changes – Finding specific drivers of test lifts – Testing multiple versions (e.g. click here, book now, go) – Where you need to understand strong and weak cross variable interactions – Don’t use to settle arguments or sloppy thinking!
  • 64. Netherlands A/B Shift Example Previous winner +7.25% +8.19% additional lift
  • 65. #20 – Other flavours of testing • Micro testing (tiny change) – good for: – Proving to the boss that testing works – Demonstrating to IT that it works without impact – Showing the impact of a seemingly tiny change – Proof of concept before larger test • Funnel testing – good for: – Checkouts – Lead gen – Forms processes – Quotations – Any multi-step process with data entry • Fake it and Build it – good for: – Testing new business ideas – Trying out promotions on a test sample – Estimating impact before you build – Helps you calculate ROI – You can even split test entire server farms Vs.
  • 66. #20 – Other flavours of testing “Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy.”
  • 67. Top F***ups for 2014 1. Testing in the wrong place 2. Your hypothesis inputs are crap 3. No analytics integration 4. Your test will finish after you die 5. You don’t test for long enough 6. You peek before it’s ready 7. No QA for your split test 8. Opportunities are not prioritised 9. Testing cycles are too slow 10. You don’t know when tests are ready 11. Your test fails 12. The test is ‘about the same’ 13. Test flips behaviour 14. Test keeps moving around 15. You run an A/A test and waste time 16. Nobody ‘feels’ the test 17. You forgot you were responsive 18. You forgot you had no traffic 19. You ran the wrong test type 20. You didn’t try all the flavours of testing @OptimiseOrDie
  • 69. 2004 Headspace What I thought I knew in 2004 Reality
  • 70. 2014 Headspace What I know I know On a good day
  • 74. #1 Smart Talented Polymath People The 5 Legged Optimisation Barstool @OptimiseOrDie Flexible and Agile teams
  • 76. #2 : Analytics Investment (tools, people, dev time) @OptimiseOrDie
  • 77. @OptimiseOrDie #3 : User research and insight
  • 78. #3 : THE BEST IDEAS COME FROM? @OptimiseOrDie
  • 79. #4 : GREAT COPYWRITING “On the average, five times as many people read the headline as read the body copy. When you have written your headline, you have spent eighty cents out of your dollar.” David Ogilvy “In 9 years and 40M split tests with visitors, the majority of my testing success came from playing with the words.” @OptimiseOrDie
  • 80. • Google Content Experiments bit.ly/Ljg7Ds • Optimizely www.optimizely.com • Visual Website Optimizer www.visualwebsiteoptimizer.com • Multi Armed Bandit Explanation bit.ly/Xa80O8 • New Machine Learning Tools www.conductrics.com www.rekko.com @OptimiseOrDie #5 : Split Testing Tools
  • 81. The 5 Legged Optimisation @OptimiseOrDie Barstool #1 Culture & Team #2 Toolkit & Analytics investment #3 UX, CX, Service Design, Insight #4 Persuasive Copywriting #5 Experimentation (testing) tools
  • 85. #5 : FIND STUFF @OptimiseOrDie @danbarker Analytics @fastbloke Analytics @timlb Analytics @jamesgurd Analytics @therustybear Analytics @carmenmardiros Analytics @davechaffey Analytics @priteshpatel9 Analytics @cutroni Analytics @avinash Analytics @Aschottmuller Analytics, CRO @cartmetrix Analytics, CRO @Kissmetrics CRO / UX @Unbounce CRO / UX @Morys CRO / Neuro @UXFeeds UX / Neuro @Psyblog Neuro @Gfiorelli1 SEO / Analytics @PeepLaja CRO @TheGrok CRO @UIE UX @LukeW UX / Forms @cjforms UX / Forms @axbom UX @iatv UX @Chudders Photo UX @JeffreyGroks Innovation @StephanieRieger Innovation @BrianSolis Innovation @DrEscotet Neuro @TheBrainLady Neuro @RogerDooley Neuro @Cugelman Neuro @Smashingmag Dev / UX @uxmag UX @Webtrends UX / CRO
  • 86. #5 : LEARN STUFF @OptimiseOrDie Baymard.com Lukew.com Smashingmagazine.com ConversionXL.com Medium.com Whichtestwon.com Unbounce.com Measuringusability.com RogerDooley.com Kissmetrics.com Uxmatters.com Smartinsights.com Econsultancy.com Cutroni.com www.GetMentalNotes.com
  • 87. #12 : The Best Companies… • Invest continually in analytics instrumentation, tools, people • Use an Agile, iterative, cross-silo, one team project culture • Prefer collaborative tools to having lots of meetings • Prioritise development based on numbers and insight • Practice real continuous product improvement, not SLEDD* • Are fixing bugs, cruft, bad stuff as well as optimising • Source photos and content that support persuasion and utility • Have cross channel, cross device design, testing and QA • Segment their data for valuable insights, every test or change • Continually reduce cycle (iteration) time in their process • Blend ‘long’ design, continuous improvement AND split tests • Make optimisation the engine of change, not the slave of ego * Single Large Expensive Doomed Developments
  • 88. THE FUTURE OF TESTING
  • 89. Projects? Questions? Mail me! Mail : sullivac@gmail.com Deck : slideshare.com/sullivac Linkedin : linkd.in/pvrg14 @OptimiseOrDie

Editor's Notes

  1. You’re just doing it the wrong way. But it’s not a surprise to me that many companies fail to hit pay dirt on testing programmes. Although I admire AB testing companies - all of them - for championing the right to test and making it easy for anyone to implement - there's a problem. Democratisation of testing brings with it a large chunk of stupidity too. When YouTube first appeared, did anyone think "Oh boy, there's only ever going to be high quality content to see on here. Seriously. No”
  2. And here’s a boring slide about me – and where I’ve been driving over 400M of additional revenue in the last few years. For the sharp eyed amongst you, you’ll see that Lean UX hasn’t been around since 2008. Many startups and teams were doing this stuff before it got a new name, even if the approach was slightly different. For the last 4 years, I’ve been optimising sites using a blend of these techniques.
  3. And here are some of the clients I’ve been working for. Dull bit is now officially over.
  4. And this crappy AB testing is basically the equivalent of funny cat videos
  5. People taking videos of themselves playing video games
  6. And like, wow, there are 8.1 million Gangnam Style videos. Just incredible. But hidden in those big numbers, YouTube will always have a tiny percentage of really great stuff, very little good stuff and a long tail of absolute bollocks. And the same is true of split testing - there's some really well run stuff, getting very good results and there's a lot of air guitar going on.
  7. This is actually a card on an e-card apology site. I’m assuming this is not on their top 10 list, that’s for sure. But what about false positives? Well this is a big problem for a lot of people. You think you’ve got a winning test but actually it’s not conclusive – it could easily be a different result. Some testing tools will even send you an email after 3 days of testing, cheerfully telling you it thinks it’s finished. You haven’t even done a weekly cycle yet! I see (and hear) about this kind of thing all the time. It means people are calling the wrong results on their tests? So why do people get itchy trigger fingers. Because the tools don’t do a good enough job of explaining things like how long to run your test for and most importantly, when to stop. People trust the tools implicitly but there’s also a responsibility for the tool user too, to not do dumbass stuff. Juggle with chainsaws at your own risk.
  8. Copywriting skills, every time, to get good AB testing. Most people completely under invest here. 60-70% of all my test lifts came not from design tweaks or images - but the wording in headlines, call to action buttons, subheads and body copy in the tests.
  9. Copywriting skills, every time, to get good AB testing. Most people completely under invest here. 60-70% of all my test lifts came not from design tweaks or images - but the wording in headlines, call to action buttons, subheads and body copy in the tests.
  10. I once explained to my daughter – you know, when adults like look really in control and making decisions and appearing not to suffer from indecision? Don’t believe it for a minute – we’re just better at winging it cause we’re older. And this is the huge hole that’s gnawing at the hear of many digital operations. The inability to understand what you can and can’t be confident about – but nobody wants to admit they’re guessing a lot of the time.
  11. And this was the state of my head in 2004. The inability to understand what you can and can’t be confident about – but nobody wants to admit they’re fucking guessing a lot of the time. And it took me a long time to figure out I didn’t know anything really – it was all assumptions and cherished notions. It was pretty crushing to test my way to this realisation but MUCH I’m happier now.
  12. Now I think I know this much - but I might know a wee bit more than I think I do – but I’m erring on the side of caution. That’s because I'm always questioning everything I do through the lens of that consumer insight and testing. Without customers and data driven insights, you can’t shape revenue and delight. They’ll give you the very psychological insights you need to apply levers to influence them, if you only ask questions. Everything else is just a fucking guess. Even with tests, if the only inputs you’ve got are ego and opinion, they’re going to be lousy guesses and you’re wasting your experiments.
  13. There is one answer to this trap I call taking a visit to Guessaholics Anonymous - to surrender to the higher power of testing and innovation by using consumer psychological insight and data to guide your hand. To recognise you’re powerless at deciding what’s best or second guessing what will win. It's actually liberating to not be sitting in a meeting room, arguing about the wording of a bloody button for 4 fucking hours, ever again.
  14. And now a bit about something I call Rumsfeldian Space – exploring the unknowns.
  15. If you were an organism that awoke and found you had 8 limbs but no sight and only feeling. What would you do? You'd explore the environment around you. You'd reach out and experience pain when you found something uncomfortable but also pleasant surprises when you found some food you could grab. If your company isn't innovating by failing and exploring (along with successful tests) then you'll never map out the possible models, pricing, setup or potentiality of your product. It's an elasticity that's inside every product, begging to be realised. However, most companies never explore this rumsfeldian space as they can be risk averse or fail to invest – and starve themselves for fear of pain or failure. And this is a mistake that companies are making. They’re testing themselves into a corner, like an ever decreasing spiral. How do you break free?
  16. So to explore this space, it's like a barstool - the more foundations you have, the more stable and useful it becomes. It's my opinion you need: 1 - Smart Polymath people, working in a flexible and agile product development environment. AO uses small teams dedicated to platforms but glues them together. I've split product and dev teams into smaller units and acted as the go-between - far, far more flexible and adaptable than bigger team units.
  17. Darwin did NOT say 'survival of the fittest' – that was actually another guy called Herbert Spencer. What Darwin actually pushed was that the key ingredients were heritability of traits, variation and selection based on survival. If only your marketing programme was quite as ruthless eh? And if you want variation and innovation, the survival of good ideas in favour of bad and knowledge that you pass on – you need a culture of adaptability, improvement and change. Agile is about a shared mind-set across managers, leaders and everyone in the team. There’s a Harvard survey about how the *most* productive teams communicate. Not in meetings but all the time - deskside, IM, phone, skype, GitHub, agile tools, apps - these are the telegraph wires of the collaborative, participative and mission oriented teams. My key insight of the last 10 years in building and leading teams is that agile, open, flat, cross-silo, participative, flexible and collaborative environments produce customer connected products of high quality. Autoglass NPS higher than Apple.
  18. Copywriting skills, every time, to get good AB testing. Most people completely under invest here. 60-70% of all my test lifts came not from design tweaks or images - but the wording in headlines, call to action buttons, subheads and body copy in the tests.
  19. You need to inhabit the contextual and emotional landscape of the consumer to really shape product or service experience. The only way to do this is have teams and cultures that create a direct and meaningful connection between teams and the customer, in the impact that every change has on the outcome. Every atom of every piece of copy, design, error message, email, website, support, help content, absolutely bloody everything you do - has to be framed within knowledge and empathy with the consumer fears, worries, barriers, pain but also the real problems we solve by designing products not as features but as life enhancing. And this is the best marketing of all, like the IBM ad. Business Model Optimisation requires a watchmakers eye – a complete understanding of the watch from macro to micro - the flow of delight and money that can be shaped inside every customer experience, website, and interaction - at a component and a service design level. Most people have 1 or 2 legs at most. The best companies I've worked with are doing all of these.
  20. So – what’s driving this change then? Well there have been great books on selling and persuading people – all the way back to ‘Scientific Advertising’ in 1923. And my favourite here is the Cialdini work – simply because it’s a great help for people to find practical uses for these techniques. I’ve also included some analytics and testing books here – primarily because they help so MUCH in augmenting our customer insight, testing and measurement efforts. There are lots of books with really cool examples, great stories and absolutely no fucking useful information you can use on your website – if you’ve read some of these, you’ll know exactly what I mean. These are the tomes I got most practical use from and I’d recommend you buy the whole lot – worth every penny.
  21. So – what’s driving this change then? Well there have been great books on selling and persuading people – all the way back to ‘Scientific Advertising’ in 1923. And my favourite here is the Cialdini work – simply because it’s a great help for people to find practical uses for these techniques. I’ve also included some analytics and testing books here – primarily because they help so MUCH in augmenting our customer insight, testing and measurement efforts. There are lots of books with really cool examples, great stories and absolutely no fucking useful information you can use on your website – if you’ve read some of these, you’ll know exactly what I mean. These are the tomes I got most practical use from and I’d recommend you buy the whole lot – worth every penny.
  22. So – what’s driving this change then? Well there have been great books on selling and persuading people – all the way back to ‘Scientific Advertising’ in 1923. And my favourite here is the Cialdini work – simply because it’s a great help for people to find practical uses for these techniques. I’ve also included some analytics and testing books here – primarily because they help so MUCH in augmenting our customer insight, testing and measurement efforts. There are lots of books with really cool examples, great stories and absolutely no fucking useful information you can use on your website – if you’ve read some of these, you’ll know exactly what I mean. These are the tomes I got most practical use from and I’d recommend you buy the whole lot – worth every penny.
  23. These are all people on twitter who cover hybrid stuff – where usability, psychology, analytics and persuasive writing collide. If you follow this lot, you’ll be much smarter within a month, guaranteed.
  24. And here are the most useful resources I regularly use or share with people. They have the best and most practical advice – cool insights but with practical applications.
  25. In my opinion, these are the attributes of companies doing great things with optimisation and continuous improvement.
  26. This is the future of testing. A machine learning system that will test out variants and tell you what’s driving response to all your experiments. Know if that offer works because of someone’s age or past spending patterns – let the tool explain to you where the value is and let it exploit these patterns as an intelligent agent under your control.
  27. I hope you enjoyed it as much as I did writing it. All my details are here and slides will be uploaded shortly. Thank you for your time today.