SlideShare a Scribd company logo
1 of 28
Stop Running Random Tests
Validate experiments, improve results
Theo van der Zee - ConversionReview @ Conversion Hotel 2014
“Split testing is not about throwing all ideas
at users to see which ones make sense.”
Helping visitors
To achieve their goals
Experiments
They don’t know
Faster horses?
Quantitative Qualitative
Google Analytics
Customer data
Quantitative Qualitative
Expert review
Variations
Research
“Split testing doesn’t give you any excuses
not to do your research first.”
Theo van der Zee
ConversionReview.com

More Related Content

More from Theo van der Zee

More from Theo van der Zee (13)

Sustainable Marketing
Sustainable MarketingSustainable Marketing
Sustainable Marketing
 
The Secret Life of Your Users
The Secret Life of Your UsersThe Secret Life of Your Users
The Secret Life of Your Users
 
Uncommon Ways to Speed up your Website
Uncommon Ways to  Speed up your WebsiteUncommon Ways to  Speed up your Website
Uncommon Ways to Speed up your Website
 
WordCamp NL 2016
WordCamp NL 2016WordCamp NL 2016
WordCamp NL 2016
 
Meet Magento 2016
Meet Magento 2016Meet Magento 2016
Meet Magento 2016
 
Emerce Conversion 2016
Emerce Conversion 2016Emerce Conversion 2016
Emerce Conversion 2016
 
SEO Meetup Utrecht
SEO Meetup UtrechtSEO Meetup Utrecht
SEO Meetup Utrecht
 
WooCommerce Meetup Utrecht
WooCommerce Meetup UtrechtWooCommerce Meetup Utrecht
WooCommerce Meetup Utrecht
 
Search Marketing Thursday 2015
Search Marketing Thursday 2015Search Marketing Thursday 2015
Search Marketing Thursday 2015
 
Joomla!Days 2015
Joomla!Days 2015Joomla!Days 2015
Joomla!Days 2015
 
Emerce Conversion 2015
Emerce Conversion 2015Emerce Conversion 2015
Emerce Conversion 2015
 
Stop Building Links, Start Earning Them
Stop Building Links, Start Earning ThemStop Building Links, Start Earning Them
Stop Building Links, Start Earning Them
 
Joomla! SEO (With Super Mario)
Joomla! SEO (With Super Mario)Joomla! SEO (With Super Mario)
Joomla! SEO (With Super Mario)
 

Recently uploaded

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligenceRevolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Precisely
 

Recently uploaded (20)

How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligenceRevolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 

Stop Running Random Tests

Editor's Notes

  1. Tim Ash, author of best-seller LPO, best picture I could find of him ;-) Important message. [HANDS] who has run test with less than 10 mins pre-research?
  2. Even in the light of testing quotas, performance-based work, getting paid by the hour, being on a payroll from 9-5, or just starting out with testing
  3. Is our proposed experiment fixing something that actually bothers visitors? Or do we just THINK it bothers them? Is there data (quantitative or qualitative) to back this up?
  4. “A lot of times, people don't know what they want until you show it to them.” — Steve Jobs
  5. “If I had asked people what they wanted, they would have said faster horses.” — Henry Ford
  6. People as numbers, data points, learnings derived via statistical modeling. Learn the ‘what’ is happening, no why given
  7. Find conversion bottlenecks. Landing pages, bounce rates, dashboards, demographics, interests, segments?
  8. Log events: is really nobody clicking button X? Which of the new dynamic landing pages aren’t converting? [QUESTION] Who uses it? how?
  9. From CRM, sales data, surveys, etc. Who is buying, what, why, from where, when? Patterns? Predictions? [EXAMPLES]
  10. PMA, backend of database. SalesForce / CRM? Use wisely and ethically. Which item (im)popular for which user? Useful for segmentation! [QUESTION] Who uses it? how?
  11. People as people, n=1 at a time, asking questions. Learn the ‘Why’ behind the ‘What’
  12. Remote or real life, usertesting.com. Scenarios, tasks, demographic selection. [QUESTION] who used user testing? results?
  13. 5 minutes (often more), English websites, 3 per month, valuable insights.
  14. Heuristic analysis of user scenarios: what do users want as a given time? How to give that to them optimally? (Check out ConversionReview.com)
  15. Contrast analysis tool, lightest setting, shows part of the text is unreadable
  16. Algorithmically generated heatmap shows menu gets zero attention (doesn’t have to be a problem though)
  17. Is there a real problem? Then look for proper solutions in the form of variations.
  18. What information can we find about (solving this) problem? Articles, blogs, journals
  19. Feedly, I’ll publish a blog with CRO resources I’m following. [QUESTION] Which sources do you recommend?
  20. Can the variations work for visitors? Do they understand what is going on? Choose from: 5 second, click, navigation or preference test.
  21. Some of the usability tests I’ve done. Important to focus on strengths of each method
  22. Do the variations work for real visitors (and why)? Currently in public beta, gonna disrupt the market if the play it right
  23. Let me know if you want a little demo
  24. No need for any credits for this quote right? ;-)
  25. Get a free review, check it out!