0
Would you bet your job on
your A/B test results?

@QubitGroup #QubitABtest
Friday, 28 February 14
Agenda
• 5 digital experience technology trends that will guide
your technology investment
— Anjali Yakkundi, Forrester Re...
5 Digital Experience Technology Trends
To Guide Your Investment
Anjali Yakkundi, Analyst
February 27, 2014

Friday, 28 Feb...
Burberry brings digital into the store
© 2013 Forrester Research, Inc. Reproduction Prohibited
4

Friday, 28 February 14
Room
Number

Address &
directions

Starwood’s Preferred Guest Streamlines The
Check-In Process
© 2013 Forrester Research, ...
Agenda

› Why digital experience technology matters
› Key digital customer experience
technologies
› Five trends that will...
Agenda

› Why digital experience technology matters
› Key digital customer experience
technologies
› Five trends that will...
Age of the customer

Source: October 10, 2013, “Competitive Strategy In The Age Of The Customer” Forrester report
© 2013 F...
Your customers are always connected…

Source: September 19, 2013, “Build Seamless Experiences Now” Forrester report
© 2013...
…often on multiple devices

Source: Google - The New Multi-screen World: Understanding Cross-platform Consumer Behavior, A...
You need to sustain your customer’s attention
with engaging experiences.

Friday, 28 February 14
Engaging experiences requires strong
technology management

© 2013 Forrester Research, Inc. Reproduction Prohibited
12

Fr...
Agenda

› Why digital experience technology matters
› Key digital customer experience
technologies
› Five trends that will...
Digital experience reference architecture

Friday, 28 February 14
Many solutions make up the digital
experience ecosystem
Testing

Site search

Commerce platform

Product content
managemen...
Testing and optimization solutions are key to
engagement and measurement
Testing

Site search

Commerce platform

Product ...
Agenda

› Why digital experience technology matters
› Key digital customer experience
technologies
› Five trends that will...
#1- This is the beginning of the end of the allin-one-suite

9%

Only
prefer a
suite approach

Base: 233 Digital Experienc...
#2- Tools must create business value out of
data

Solutions must effectively harness customer data,
product data, social m...
#3- Tools must support personalized
experiences
•
•
•
•

• Likely next steps
Device / Channel • Historical past
• Immediat...
#4- Look for more than software: look for
solutions

› Look for solutions that help marketers and business
users quickly a...
#5- Solutions must prioritize multichannel

© 2013 Forrester Research, Inc. Reproduction Prohibited
22

Friday, 28 Februar...
Agenda

› Why digital experience technology matters
› Key digital customer experience
technologies
› Five trends that will...
Recommendations

› You can’t do it all at once — prioritize key customer
touchpoints
› Assess current technology gaps– mos...
Thank you
Anjali Yakkundi
ayakkundi@forrester.com
@Ayakkundi

Friday, 28 February 14
5 fundamental
A/B testing mistakes
and how to avoid them
Will Browne, Qubit
will.browne@qubitdigital.com
@QubitGroup #Qubi...
Why do I have to test?

Friday, 28 February 14
A/B testing is an essential tool

@TheGrahamCooke
Friday, 28 February 14
A/B testing is an essential tool

• It is the best, simplest way to prove the business value of
your digital technology

@...
A/B testing is an essential tool

• It is the best, simplest way to prove the business value of
your digital technology

•...
But there are costs associated with A/B testing

Friday, 28 February 14
But there are costs associated with A/B testing

• The monthly cost of your testing platform

@QubitGroup #QubitABtest
Fri...
But there are costs associated with A/B testing

• The monthly cost of your testing platform

• The salaries and time of y...
But there are costs associated with A/B testing

• The monthly cost of your testing platform

• The salaries and time of y...
We’re here to show you how to
reduce these costs, and increase ROI

@QubitGroup #QubitABtest
Friday, 28 February 14
We’re here to show you how to
reduce these costs, and increase ROI

• Find the ideas that are likely to have a real
positi...
We’re here to show you how to
reduce these costs, and increase ROI

• Find the ideas that are likely to have a real
positi...
We’re here to show you how to
reduce these costs, and increase ROI

• Find the ideas that are likely to have a real
positi...
Find the ideas that are likely to have a real positive effect
on user behavior

39

@QubitGroup #QubitABtest

Friday, 28 F...
Size does matter

Friday, 28 February 14
Detect as many true winners as possible

We’ve found that tests based on data are

7.7x more
effective
than tests based on...
Imagine testing for weighted coins

We’vemany coin flips would be on datato prove that one
found that tests based enough ar...
Imagine testing for weighted coins

We’vemany coin flips would be on datato prove that one
found that tests based enough ar...
Imagine testing for weighted coins

We’vemany coin flips would be on datato prove that one
found that tests based enough ar...
Minimize the chance of finding tests that look like winners,
but have no true effect

We’ve found that tests based on data...
Minimize the chance of finding tests that look like winners,
but have no true effect

•We’ve found that tests based on dat...
Minimize the chance of finding tests that look like winners,
but have no true effect

•We’ve found that tests based on dat...
Minimize the chance of finding tests that look like winners,
but have no true effect

•We’ve found that tests based on dat...
Splitting tests too much can lead to false positives

Friday, 28 February 14
Get a second opinion

Friday, 28 February 14
Validate your results

We recommend a

95/5 split
to confirm your winning variation
is still winning

“Most winning A/B te...
A worked example: Imagine Mr Bean’s jelly beans shop

“Most winning A/B test results are illusory”, Qubit 2014

@QubitGrou...
A worked example: Imagine Mr Bean’s jelly beans shop

• Mr Bean has a 1% conversion rate
• He runs 20 different tests in on...
Truth

@QubitGroup #QubitABtest
Friday, 28 February 14

Correct
method

Naive
method
Truth

Correct
method

Naive
method

No. of tests

20

20

20

Reported
winners

2

3

9

@QubitGroup #QubitABtest
Friday,...
Truth

Correct
method

Naive
method

No. of tests

20

20

20

Reported
winners

2

3

9

After
validation

2

2

9

@Qubi...
Truth

Correct
method

Naive
method

No. of tests

20

20

20

Reported
winners

2

3

9

After
validation

2

2

9

2 x ~...
Correct method

Naive method

Reported uplift

2 x 5%

9 x (5-40)%

Delivered uplift

2 x 5%

2 x 5% + (? x -5%)

@QubitGr...
Correct method

Naive method

Reported uplift

2 x 5%

9 x (5-40)%

Delivered uplift

2 x 5%

2 x 5% + (? x -5%)

None

7 ...
Correct method

Naive method

Reported uplift

2 x 5%

9 x (5-40)%

Delivered uplift

2 x 5%

2 x 5% + (? x -5%)

None

7 ...
Correct method

Naive method

Reported uplift

2 x 5%

9 x (5-40)%

Delivered uplift

2 x 5%

2 x 5% + (? x -5%)

None

7 ...
So, to conclude, those
5 mistakes are:
• Mistake 1: Not A/B testing
• Mistake 2: Not prioritizing tests properly
• Mistake...
Questions you should ask your
testing provider
• How do you prioritize tests?
• How do you calculate the sample
size requi...
Thank you
If you have any detailed questions please contact
research@qubitdigital.com

Will Browne, Qubit
will.browne@qubi...
How to run a successful A/B testing program

Melanie Kyrklund
Senior Ecommerce Analyst

Friday, 28 February 14
Testing programs have the potential to drive the
online customer experience
 Understand your customers
and respond to the...
The reality of running a testing program
For many organizations, testing represents a cultural change with
many political ...
A lot of testing is required to deliver results

 1/5 tests or 1/10 tests will deliver statistically significant wins
 A ...
Two factors are critical in building a successful
testing program

Approach

Speed

Methodology
Idea Generation

Operation...
The right approach to optimization
Be inspired – every visit is an opportunity to establish a relationship
with a custome...
Idea Generation

71
Friday, 28 February 14
Idea Generation

71
Friday, 28 February 14
Idea Generation

71
Friday, 28 February 14
Idea Generation

71
Friday, 28 February 14
Speed – Operational Processes
Testing Tool selection : Map out the most efficient
implementation process
 Operational eff...
Conclusion: A rapid, data-driven approach is required
 Testing programs have the potential to drive the online
customer e...
Thank you
Any questions?

Friday, 28 February 14
More A/B testing material from Qubit

bit.ly/qABresearch

75

@QubitGroup
#QubitABtests

Friday, 28 February 14

bit.ly/qA...
Upcoming SlideShare
Loading in...5
×

Would you bet your job on your A/B test results?

550

Published on

It is possible that 72% of your successful A/B tests may not be driving any business benefit, or may actually be harming your bottom line. This is because bad methodology is preventing you from detecting the truly successful ideas.

In this webinar, Qubit explains how you can avoid this problem. With special guests from Forrester who explain how A/B testing fits into the digital landscape, and Staples, who give practical advice for how to set up an A/B testing campaign, this webinar will change the way you think about A/B testing forever

Published in: Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
550
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Would you bet your job on your A/B test results?"

  1. 1. Would you bet your job on your A/B test results? @QubitGroup #QubitABtest Friday, 28 February 14
  2. 2. Agenda • 5 digital experience technology trends that will guide your technology investment — Anjali Yakkundi, Forrester Research, Inc. • 5 fundamental A/B testing mistakes and how to avoid them —Will Browne, Qubit • How to run a successful A/B testing program —Melanie Kyrklund, Staples • Q&A @QubitGroup #QubitABtest Friday, 28 February 14
  3. 3. 5 Digital Experience Technology Trends To Guide Your Investment Anjali Yakkundi, Analyst February 27, 2014 Friday, 28 February 14
  4. 4. Burberry brings digital into the store © 2013 Forrester Research, Inc. Reproduction Prohibited 4 Friday, 28 February 14
  5. 5. Room Number Address & directions Starwood’s Preferred Guest Streamlines The Check-In Process © 2013 Forrester Research, Inc. Reproduction Prohibited 5 Friday, 28 February 14
  6. 6. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 6 Friday, 28 February 14
  7. 7. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 7 Friday, 28 February 14
  8. 8. Age of the customer Source: October 10, 2013, “Competitive Strategy In The Age Of The Customer” Forrester report © 2013 Forrester Research, Inc. Reproduction Prohibited 8 Friday, 28 February 14
  9. 9. Your customers are always connected… Source: September 19, 2013, “Build Seamless Experiences Now” Forrester report © 2013 Forrester Research, Inc. Reproduction Prohibited 9 Friday, 28 February 14
  10. 10. …often on multiple devices Source: Google - The New Multi-screen World: Understanding Cross-platform Consumer Behavior, August 2012 © 2013 Forrester Research, Inc. Reproduction Prohibited 10 Friday, 28 February 14
  11. 11. You need to sustain your customer’s attention with engaging experiences. Friday, 28 February 14
  12. 12. Engaging experiences requires strong technology management © 2013 Forrester Research, Inc. Reproduction Prohibited 12 Friday, 28 February 14
  13. 13. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 13 Friday, 28 February 14
  14. 14. Digital experience reference architecture Friday, 28 February 14
  15. 15. Many solutions make up the digital experience ecosystem Testing Site search Commerce platform Product content management Optimization Portals Web content management Digital asset management Analytics (mobile, social, Web) Recommendation engine Online video platform Campaign management Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report © 2013 Forrester Research, Inc. Reproduction Prohibited 15 Friday, 28 February 14
  16. 16. Testing and optimization solutions are key to engagement and measurement Testing Site search Commerce platform Product content management Optimization Portals Web content management Digital asset management Analytics (mobile, social, Web) Recommendation engine Online video platform Campaign management Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report © 2013 Forrester Research, Inc. Reproduction Prohibited 16 Friday, 28 February 14
  17. 17. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 17 Friday, 28 February 14
  18. 18. #1- This is the beginning of the end of the allin-one-suite 9% Only prefer a suite approach Base: 233 Digital Experience Professionals Source: March 2013, “Digital Experience Delivery Online Survey” © 2013 Forrester Research, Inc. Reproduction Prohibited 18 Friday, 28 February 14
  19. 19. #2- Tools must create business value out of data Solutions must effectively harness customer data, product data, social media data, and other information to create and optimize experiences. © 2013 Forrester Research, Inc. Reproduction Prohibited 19 Friday, 28 February 14
  20. 20. #3- Tools must support personalized experiences • • • • • Likely next steps Device / Channel • Historical past • Immediate past Location • Current session Time of day Weather Situation Real-time optimization Prediction © 2013 Forrester Research, Inc. Reproduction Prohibited 20 Friday, 28 February 14
  21. 21. #4- Look for more than software: look for solutions › Look for solutions that help marketers and business users quickly adapt to changing customer needs. This can be achieved by: •Keeping IT out of critical publishing paths •Prioritizing user interface needs © 2013 Forrester Research, Inc. Reproduction Prohibited 21 Friday, 28 February 14
  22. 22. #5- Solutions must prioritize multichannel © 2013 Forrester Research, Inc. Reproduction Prohibited 22 Friday, 28 February 14
  23. 23. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 23 Friday, 28 February 14
  24. 24. Recommendations › You can’t do it all at once — prioritize key customer touchpoints › Assess current technology gaps– most organizations tend to have gaps around: •Optimization •Personalization •Multichannel delivery › When sourcing new solutions, look to vendors that can help you connect the dots. © 2013 Forrester Research, Inc. Reproduction Prohibited 24 Friday, 28 February 14
  25. 25. Thank you Anjali Yakkundi ayakkundi@forrester.com @Ayakkundi Friday, 28 February 14
  26. 26. 5 fundamental A/B testing mistakes and how to avoid them Will Browne, Qubit will.browne@qubitdigital.com @QubitGroup #QubitABtest Friday, 28 February 14
  27. 27. Why do I have to test? Friday, 28 February 14
  28. 28. A/B testing is an essential tool @TheGrahamCooke Friday, 28 February 14
  29. 29. A/B testing is an essential tool • It is the best, simplest way to prove the business value of your digital technology @QubitGroup @TheGrahamCooke Friday, 28 February 14 #QubitABtest
  30. 30. A/B testing is an essential tool • It is the best, simplest way to prove the business value of your digital technology • It is the only way to get close to proving cause and effect in what you do @QubitGroup @TheGrahamCooke Friday, 28 February 14 #QubitABtest
  31. 31. But there are costs associated with A/B testing Friday, 28 February 14
  32. 32. But there are costs associated with A/B testing • The monthly cost of your testing platform @QubitGroup #QubitABtest Friday, 28 February 14
  33. 33. But there are costs associated with A/B testing • The monthly cost of your testing platform • The salaries and time of your developers and team @QubitGroup #QubitABtest Friday, 28 February 14
  34. 34. But there are costs associated with A/B testing • The monthly cost of your testing platform • The salaries and time of your developers and team • The lost revenue from implementing changes that do nothing for your site (or actively harm it) @QubitGroup #QubitABtest Friday, 28 February 14
  35. 35. We’re here to show you how to reduce these costs, and increase ROI @QubitGroup #QubitABtest Friday, 28 February 14
  36. 36. We’re here to show you how to reduce these costs, and increase ROI • Find the ideas that are likely to have a real positive effect on user behavior (true winners) @QubitGroup #QubitABtest Friday, 28 February 14
  37. 37. We’re here to show you how to reduce these costs, and increase ROI • Find the ideas that are likely to have a real positive effect on user behavior (true winners) • Detect as many true winners as possible @QubitGroup #QubitABtest Friday, 28 February 14
  38. 38. We’re here to show you how to reduce these costs, and increase ROI • Find the ideas that are likely to have a real positive effect on user behavior (true winners) • Detect as many true winners as possible • Minimize the chance of finding tests that look like winners, but have no true effect @QubitGroup #QubitABtest Friday, 28 February 14
  39. 39. Find the ideas that are likely to have a real positive effect on user behavior 39 @QubitGroup #QubitABtest Friday, 28 February 14
  40. 40. Size does matter Friday, 28 February 14
  41. 41. Detect as many true winners as possible We’ve found that tests based on data are 7.7x more effective than tests based on intuition alone (Qubit, 2013) Friday, 28 February 14
  42. 42. Imagine testing for weighted coins We’vemany coin flips would be on datato prove that one found that tests based enough are • How coin was weighted? 7.7x more effective than tests based on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  43. 43. Imagine testing for weighted coins We’vemany coin flips would be on datato prove that one found that tests based enough are • How coin was weighted? 7.7x more effective • It is impossible to be 100% certain, but the more data you have, the closer you can get than tests based on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  44. 44. Imagine testing for weighted coins We’vemany coin flips would be on datato prove that one found that tests based enough are • How coin was weighted? 7.7x more effective • It is impossible to be 100% certain, but the more data you have, the closer you can get • We follow scientific standards and wait until there is an than tests based on intuition alone 80% chance of detecting a difference (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  45. 45. Minimize the chance of finding tests that look like winners, but have no true effect We’ve found that tests based on data are 7.7x more effective than tests based on intuition alone (Qubit, 2013) Friday, 28 February 14
  46. 46. Minimize the chance of finding tests that look like winners, but have no true effect •We’ve found that tests based on data are there is a 5% We generally assume that with good testing false positive rate 7.7x more effective than tests based on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  47. 47. Minimize the chance of finding tests that look like winners, but have no true effect •We’ve found that tests based on data are there is a 5% We generally assume that with good testing false positive rate 7.7x more effective • We can test this using A/A tests (showing both groups the exact thing) than tests based on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  48. 48. Minimize the chance of finding tests that look like winners, but have no true effect •We’ve found that tests based on data are there is a 5% We generally assume that with good testing false positive rate 7.7x more effective • We can test this using A/A tests (showing both groups the exact thing) • So out of 100 A/A tests, 5 would be winners, despite than tests based nothing doing absolutely on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  49. 49. Splitting tests too much can lead to false positives Friday, 28 February 14
  50. 50. Get a second opinion Friday, 28 February 14
  51. 51. Validate your results We recommend a 95/5 split to confirm your winning variation is still winning “Most winning A/B test results are illusory”, Qubit 2014 @QubitGroup #QubitABtest Friday, 28 February 14
  52. 52. A worked example: Imagine Mr Bean’s jelly beans shop “Most winning A/B test results are illusory”, Qubit 2014 @QubitGroup Friday, 28 February 14
  53. 53. A worked example: Imagine Mr Bean’s jelly beans shop • Mr Bean has a 1% conversion rate • He runs 20 different tests in one year • There are 2 good tests with uplifts of 5% • There are also 2 bad tests with uplifts of -5% “Most winning A/B test results are illusory”, Qubit 2014 @QubitGroup #QubitABtest @QubitGroup Friday, 28 February 14
  54. 54. Truth @QubitGroup #QubitABtest Friday, 28 February 14 Correct method Naive method
  55. 55. Truth Correct method Naive method No. of tests 20 20 20 Reported winners 2 3 9 @QubitGroup #QubitABtest Friday, 28 February 14
  56. 56. Truth Correct method Naive method No. of tests 20 20 20 Reported winners 2 3 9 After validation 2 2 9 @QubitGroup #QubitABtest Friday, 28 February 14
  57. 57. Truth Correct method Naive method No. of tests 20 20 20 Reported winners 2 3 9 After validation 2 2 9 2 x ~5% 9x (5-40)% Reported uplift 2 x 5% @QubitGroup #QubitABtest Friday, 28 February 14
  58. 58. Correct method Naive method Reported uplift 2 x 5% 9 x (5-40)% Delivered uplift 2 x 5% 2 x 5% + (? x -5%) @QubitGroup #QubitABtest Friday, 28 February 14
  59. 59. Correct method Naive method Reported uplift 2 x 5% 9 x (5-40)% Delivered uplift 2 x 5% 2 x 5% + (? x -5%) None 7 unnecessary tests Wasted resource @QubitGroup #QubitABtest Friday, 28 February 14
  60. 60. Correct method Naive method Reported uplift 2 x 5% 9 x (5-40)% Delivered uplift 2 x 5% 2 x 5% + (? x -5%) None 7 unnecessary tests Improved Degraded Improved Confused Wasted resource Customer experience Your understanding @QubitGroup #QubitABtest Friday, 28 February 14
  61. 61. Correct method Naive method Reported uplift 2 x 5% 9 x (5-40)% Delivered uplift 2 x 5% 2 x 5% + (? x -5%) None 7 unnecessary tests Improved Degraded Improved Confused Happy Angry Wasted resource Customer experience Your understanding Your manager @QubitGroup #QubitABtest Friday, 28 February 14
  62. 62. So, to conclude, those 5 mistakes are: • Mistake 1: Not A/B testing • Mistake 2: Not prioritizing tests properly • Mistake 3: Not waiting for significance and sample size • Mistake 4: Testing too many things at once • Mistake 5: Not validating the tests after implementation @QubitGroup #QubitABtest Friday, 28 February 14
  63. 63. Questions you should ask your testing provider • How do you prioritize tests? • How do you calculate the sample size required? • How do you account for the multiple testing problem? • How do you validate winning tests? @QubitGroup #QubitABtest Friday, 28 February 14
  64. 64. Thank you If you have any detailed questions please contact research@qubitdigital.com Will Browne, Qubit will.browne@qubitdigital.com @QubitGroup #QubitABtest Friday, 28 February 14
  65. 65. How to run a successful A/B testing program Melanie Kyrklund Senior Ecommerce Analyst Friday, 28 February 14
  66. 66. Testing programs have the potential to drive the online customer experience  Understand your customers and respond to their needs  Optimize the user journey Customer experience  Create personalized experiences  Quickly respond to market pressures Personalization Persuasion Targeted offers User experience, functionality & content  Rapidly align marketing & onsite experience  Validate development items  Every visit represents an opportunity to establish a relationship with a customer 66 Friday, 28 February 14
  67. 67. The reality of running a testing program For many organizations, testing represents a cultural change with many political and operational barriers to be overcome:  Resource & operational factors  Buy-in / Multiple stakeholders  Alignment with other business priorities and development plans  Instilling a data-driven culture 67 Friday, 28 February 14
  68. 68. A lot of testing is required to deliver results  1/5 tests or 1/10 tests will deliver statistically significant wins  A rapid pace of testing is required to build a program that is delivering consistent results month over month 68 Friday, 28 February 14
  69. 69. Two factors are critical in building a successful testing program Approach Speed Methodology Idea Generation Operational Process Prioritization 69 Friday, 28 February 14
  70. 70. The right approach to optimization Be inspired – every visit is an opportunity to establish a relationship with a customer Use data to understand how visitors are using your website, what is working and what isn’t Customer feedback to understand why visitors are not purchasing on your website If working over multiple websites & countries, look for insight at local level first and then identify commonalities at a later stage Persuasion research, understand the psychology of your customers Supplement with quick UX testing Thoroughly research & quantify each opportunity (visitors reached, expected uplift and incremental revenue) UX optimization Personalization Persuasion 70 Friday, 28 February 14
  71. 71. Idea Generation 71 Friday, 28 February 14
  72. 72. Idea Generation 71 Friday, 28 February 14
  73. 73. Idea Generation 71 Friday, 28 February 14
  74. 74. Idea Generation 71 Friday, 28 February 14
  75. 75. Speed – Operational Processes Testing Tool selection : Map out the most efficient implementation process  Operational efficiency  Pick managed services (analysis, implementation) as required Prioritization Build a test pipeline logging:  Idea & hypothesis  Expected uplift & incremental revenue  Implementation estimates  Test duration  Prioritize tests weekly, to keep up momentum of testing and focus on biggest wins 72 Friday, 28 February 14
  76. 76. Conclusion: A rapid, data-driven approach is required  Testing programs have the potential to drive the online customer experience, however a rapid data-driven approach is required to deliver results  Plan the required resources upfront  Consider ease of use, test time to market and managed services in tool selection  Map out the most efficient end to end process to get tests live  Process supported by data-driven idea generation & prioritization 73 Friday, 28 February 14
  77. 77. Thank you Any questions? Friday, 28 February 14
  78. 78. More A/B testing material from Qubit bit.ly/qABresearch 75 @QubitGroup #QubitABtests Friday, 28 February 14 bit.ly/qABinfographic
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×