Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Implementing
an
A/B & Usability testing
process
@guido | userlegion.com
@guido userlegion.com
Guido?
2
@guido userlegion.com3
Netherlands: May 27 & 28, Utrecht
@guido userlegion.com
Usability?
5
@guido userlegion.com
Usability?
6
Learnability
Efficiency
Memorability
Errors
Satisfaction
@guido userlegion.com
Usability versus Persuasion
7
@guido userlegion.com
A/B Testing?
8
A B C D E
A B C D E
A B C D E
Multivariate Testing (MvT)
@guido userlegion.com
So why even bother…?
9
@guido userlegion.com
Selling online is tough!
10
@guido userlegion.com
Selling online is tough!
11
Average conversion rates
Offline: 20 - 25%
Online: 5 - 15%
@guido userlegion.com
Selling online is tough!
12
Average conversion rates by product category
Source:	
  MarketingSherpa	...
@guido userlegion.com
Selling online is tough!
13
Some shop examples
Source:	
  MarketingSherpa	
  Benchmark	
  2014
8% - ...
@guido userlegion.com
Selling online is tough!
14
Some shop examples
Source:	
  Marketingcharts.com
15.8%
@guido userlegion.com
Selling online is tough!
15
Some shop examples
Source:	
  Marketingcharts.com
http://www.google.nl/i...
@guido userlegion.com
Selling online is tough!
16
Some shop examples
Source:	
  Marketingcharts.com
40.6%
But Still…
What are users missing out on?
@guido userlegion.com
What are online users missing out on?
19
Smell
Feedback Commitment
Personalization
@guido userlegion.com
2 x 5 tips
20
My experience with testing process
Direct optimization tips
5 Testing Tips
Learnings from
the process
behind
optimization
@guido userlegion.com
Learnings first
22
Proper hypotheses
Time & resources are limited
Brute force vs smart testing
1
@guido userlegion.com
Get yourself a team
23
2
@guido userlegion.com
Optimization Team
24
Create a well balanced team
CM
Web/ Business Analist
Functional Tester
(Fronten...
@guido userlegion.com
Supporting roles
25
Your team will need support
Team
Marketing
Customer support
(Backend) Developmen...
@guido userlegion.com
Optimize for the journey
26
Not just single elements
3
Experience Ordering
ShippingInspiration Detai...
@guido userlegion.com
Optimize for the journey
27
Not just single elements
Communicate consistently
Segmentation
Offsite b...
@guido userlegion.com
Optimization Workflow
28
4
@guido userlegion.com
Optimization Workflow
29
@guido userlegion.com
Optimization Workflow
30
Input Hypotheses Analysis
TestingPriority Results
@guido userlegion.com
Optimization Workflow
31
Don’t forget:
Share results
Get employees and management involved
@guido userlegion.com
Involve your users
32
5
@guido userlegion.com
When to test?
33
Competitors
Old/ New Design Sketch
Test Preparation
Prototype
Live site
@guido userlegion.com
What to test?
34
Setting priorities with the PIE Framework
Potential
Heuristics
Voice of Customer
P
...
@guido userlegion.com
What to test?
35
Setting priorities with the PIE Framework
PIE Scoring Example
Idea	
  A Idea	
  B I...
@guido userlegion.com
How many users to include?
36
Test Preparation
Source:	
  nngroup.com
@guido userlegion.com
What kind of users to include?
37
Test Preparation
Source:	
  nngroup.com
Everyone
@guido userlegion.com
Where to test?
38
Test Preparation
Remote On-site
Test Day
@guido userlegion.com
Let the user do the talking
40
Help getting comfortable
Help explaining the case
Help thinking aloud...
@guido userlegion.com
Review day
41
Lots of ideas > A/B testing
Lots of bugs > fix it
Better estimate A/B test potentials
@guido userlegion.com
Sidenote: Toolbox
42
Project management
Web analytics
BI
A/B testingDevelopment
Raw data
Visualisati...
5 Direct Optimization Tips
Learnings about users
from watching Usability and A/B testing
@guido userlegion.com
Get customers to return
44
Small next steps to the next micro conversion
1
Almost 50 % of users do n...
@guido userlegion.com
Negative search / filter results
45
2
@guido userlegion.com
Discontinued products
46
3
@guido userlegion.com
Sidestep: context matters!
47
Perceptual contrast / framing
Web only
Print only
Web & print
$59
$125...
@guido userlegion.com
Direct checkout
48
+ 4,77% CTR
4
Source:	
  Kieskeurig
@guido userlegion.com
Mark visited products
49
or even make it a checklist
5
Source:	
  Kieskeurig
@guido userlegion.com
Mark visited products
50
or even make it a checklist
Checkmark in button
+ 3,97% CTR
Source:	
  Kies...
@guido userlegion.com
That’s it!
51
@guido userlegion.com
So… will it make you money?
52
Conclusion
@guido userlegion.com
Case: UX sprint
53
@guido userlegion.com
Conversion is not a stable factor
54
@guido userlegion.com
Important off-site factors
55
Weather
Calendar
Stock/ Interest rates
(National) Holidays
Sport event...
Thanks For Listening
Guido Jansen
guido@userlegion.com
+31 6 42093976
@guido
@guido userlegion.com
Simple, Affordable Usability Testing in your Local Language
57
Business card = Free 10 user trial :)...
Upcoming SlideShare
Loading in …5
×

Implementing an A/B & Usability testing process

2,300 views

Published on

Presented by me at Emerce Conversion 2015, Meet Magento Germany and Belgento. THis talk is about 5 internal/process tips and 5 external/ on-site tips for applying A/B testing and usability improvements to your website or webshop.

Published in: Internet
  • Be the first to comment

Implementing an A/B & Usability testing process

  1. 1. Implementing an A/B & Usability testing process @guido | userlegion.com
  2. 2. @guido userlegion.com Guido? 2
  3. 3. @guido userlegion.com3
  4. 4. Netherlands: May 27 & 28, Utrecht
  5. 5. @guido userlegion.com Usability? 5
  6. 6. @guido userlegion.com Usability? 6 Learnability Efficiency Memorability Errors Satisfaction
  7. 7. @guido userlegion.com Usability versus Persuasion 7
  8. 8. @guido userlegion.com A/B Testing? 8 A B C D E A B C D E A B C D E Multivariate Testing (MvT)
  9. 9. @guido userlegion.com So why even bother…? 9
  10. 10. @guido userlegion.com Selling online is tough! 10
  11. 11. @guido userlegion.com Selling online is tough! 11 Average conversion rates Offline: 20 - 25% Online: 5 - 15%
  12. 12. @guido userlegion.com Selling online is tough! 12 Average conversion rates by product category Source:  MarketingSherpa  Benchmark  2014 0% 5% 10% 15% 20% 25%
  13. 13. @guido userlegion.com Selling online is tough! 13 Some shop examples Source:  MarketingSherpa  Benchmark  2014 8% - 11% Average order value: €430,-
  14. 14. @guido userlegion.com Selling online is tough! 14 Some shop examples Source:  Marketingcharts.com 15.8%
  15. 15. @guido userlegion.com Selling online is tough! 15 Some shop examples Source:  Marketingcharts.com http://www.google.nl/imgres?imgurl=http://cdn.bargainez.com/womanwithin/store/600/woman logo&imgrefurl=http://bargainez.com/ womanwithin&h=168&w=600&tbnid=Md9NfBl9JPFALM:&zoom=1&docid=uCe3emUjdu4dkM&ei= PY6cgbAE&tbm=isch&ved=0CDYQMygCMAI 25.3%
  16. 16. @guido userlegion.com Selling online is tough! 16 Some shop examples Source:  Marketingcharts.com 40.6%
  17. 17. But Still…
  18. 18. What are users missing out on?
  19. 19. @guido userlegion.com What are online users missing out on? 19 Smell Feedback Commitment Personalization
  20. 20. @guido userlegion.com 2 x 5 tips 20 My experience with testing process Direct optimization tips
  21. 21. 5 Testing Tips Learnings from the process behind optimization
  22. 22. @guido userlegion.com Learnings first 22 Proper hypotheses Time & resources are limited Brute force vs smart testing 1
  23. 23. @guido userlegion.com Get yourself a team 23 2
  24. 24. @guido userlegion.com Optimization Team 24 Create a well balanced team CM Web/ Business Analist Functional Tester (Frontend) Developer Content (UX/UI) Designer Psychologist
  25. 25. @guido userlegion.com Supporting roles 25 Your team will need support Team Marketing Customer support (Backend) Development Traffic (SEO/SEA) Business Sponsor (shit umbrella)
  26. 26. @guido userlegion.com Optimize for the journey 26 Not just single elements 3 Experience Ordering ShippingInspiration Detailed research Service
  27. 27. @guido userlegion.com Optimize for the journey 27 Not just single elements Communicate consistently Segmentation Offsite behaviour
  28. 28. @guido userlegion.com Optimization Workflow 28 4
  29. 29. @guido userlegion.com Optimization Workflow 29
  30. 30. @guido userlegion.com Optimization Workflow 30 Input Hypotheses Analysis TestingPriority Results
  31. 31. @guido userlegion.com Optimization Workflow 31 Don’t forget: Share results Get employees and management involved
  32. 32. @guido userlegion.com Involve your users 32 5
  33. 33. @guido userlegion.com When to test? 33 Competitors Old/ New Design Sketch Test Preparation Prototype Live site
  34. 34. @guido userlegion.com What to test? 34 Setting priorities with the PIE Framework Potential Heuristics Voice of Customer P I Importance Business Traffic E Ease Development Cost
  35. 35. @guido userlegion.com What to test? 35 Setting priorities with the PIE Framework PIE Scoring Example Idea  A Idea  B Idea  C Potential .50 .20 .80 Importance .20 1 .50 Ease .40 .80 .20 PIE  Score   x  100 4 16 8
  36. 36. @guido userlegion.com How many users to include? 36 Test Preparation Source:  nngroup.com
  37. 37. @guido userlegion.com What kind of users to include? 37 Test Preparation Source:  nngroup.com Everyone
  38. 38. @guido userlegion.com Where to test? 38 Test Preparation Remote On-site
  39. 39. Test Day
  40. 40. @guido userlegion.com Let the user do the talking 40 Help getting comfortable Help explaining the case Help thinking aloud Other than that: shut up :)
  41. 41. @guido userlegion.com Review day 41 Lots of ideas > A/B testing Lots of bugs > fix it Better estimate A/B test potentials
  42. 42. @guido userlegion.com Sidenote: Toolbox 42 Project management Web analytics BI A/B testingDevelopment Raw data Visualisation Feedback UX Testing
  43. 43. 5 Direct Optimization Tips Learnings about users from watching Usability and A/B testing
  44. 44. @guido userlegion.com Get customers to return 44 Small next steps to the next micro conversion 1 Almost 50 % of users do not come back if they found it hard to find relevant information on the web site (Forrester research)
  45. 45. @guido userlegion.com Negative search / filter results 45 2
  46. 46. @guido userlegion.com Discontinued products 46 3
  47. 47. @guido userlegion.com Sidestep: context matters! 47 Perceptual contrast / framing Web only Print only Web & print $59 $125 $125 16% 0% 84% 68% 32% Source:  Dan  Ariely
  48. 48. @guido userlegion.com Direct checkout 48 + 4,77% CTR 4 Source:  Kieskeurig
  49. 49. @guido userlegion.com Mark visited products 49 or even make it a checklist 5 Source:  Kieskeurig
  50. 50. @guido userlegion.com Mark visited products 50 or even make it a checklist Checkmark in button + 3,97% CTR Source:  Kieskeurig
  51. 51. @guido userlegion.com That’s it! 51
  52. 52. @guido userlegion.com So… will it make you money? 52 Conclusion
  53. 53. @guido userlegion.com Case: UX sprint 53
  54. 54. @guido userlegion.com Conversion is not a stable factor 54
  55. 55. @guido userlegion.com Important off-site factors 55 Weather Calendar Stock/ Interest rates (National) Holidays Sport events Competitors
  56. 56. Thanks For Listening Guido Jansen guido@userlegion.com +31 6 42093976 @guido
  57. 57. @guido userlegion.com Simple, Affordable Usability Testing in your Local Language 57 Business card = Free 10 user trial :) Agency accounts available

×