Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
#SMX #15B1 @ayat
9 lessons to succeed at A/B testing
Advanced Testing:
What 4,000
Successful A/B Tests
Can Teach You
#SMX #15B1 @ayat
9 lessons to succeed at A/B testing
Advanced Testing:
What 4,000
Successful A/B Tests
Can Teach You
#SMX #15B1 @ayat
• 65%	average	uplift
• 4,000+	Successful	Tests
• 400+	CRO	Projects
• 11	different	countries
THE	NUMBERS
#SMX #15B1 @ayat
SOME	OF	OUR	CLIENTS…
#SMX #15B1 @ayat
1.	We	focused	on	the	wrong	thing	all	along
#SMX #15B1 @ayat
#SMX #15B1 @ayat
We	focused	too	much	on	
the	short	term	– and	lost	
focus	on	the	long	term,	
and	as	a	result	most	CRO	
com...
#SMX #15B1 @ayat
Even the way we looked
at funnels was wrong
Bowtie loop
Design thinking
Via Beintheknow.co
#SMX #15B1 @ayat
#SMX #15B1 @ayat
2.	Create	a	compelling	website	narrative
#SMX #15B1 @ayat
Element	level	testing
#SMX #15B1 @ayat
Page	level	testing
#SMX #15B1 @ayat
Checkout Checkout
Visitor	flow	testing
#SMX #15B1 @ayat
#SMX #15B1 @ayat
#SMX #15B1 @ayat
#SMX #15B1 @ayat
#SMX #15B1 @ayat
What’s	your	narrative
• What	are	you	trying	to	communicate	to	your	customers?
• What	do	you	want	visitors...
#SMX #15B1 @ayat
The	five	second	
test	gives	visitors	
first	impressions	
of	your	design
#SMX #15B1 @ayat
3.	Don’t	assume	you	live	in	a	vacuum
#SMX #15B1 @ayat
This might	be	a	conversion	problem,	
There	is	a	real	
reputation	problem
#SMX #15B1 @ayat
#SMX #15B1 @ayat
Via Smashing Magazine
Understand every	
touch	point	and	its	
impact	on	the	customer
#SMX #15B1 @ayat
4.	Don’t	assume	you	know	everything	customers
#SMX #15B1 @ayat
“People	tell	you	who	they	are,	but	we	ignore	it	-
because	we	want	them	to	be	who	we	want	them	to	
be”
#SMX #15B1 @ayat
Qualitative	
Research
#SMX #15B1 @ayat
Think like a customer
CUSTOMER PAIN POINTS
NEEDS PROBLEMS DESIRES CHALLENGES
#SMX #15B1 @ayat
Polls	and	surveys
§ Motivators	
§ Barriers	
§ Hooks	
§ Satisfaction	(NPS,	page	level)
§ Demographical	inf...
#SMX #15B1 @ayat
5.	Get	complete	buy-in	from	everyone
#SMX #15B1 @ayat
We use to
think it is a
problem
with the
HIPPO
Via undelay.io
#SMX #15B1 @ayat
But you should get
buy in from
everyone to
succeed
Via DogTime
#SMX #15B1 @ayat
6.	Embrace	Failure
#SMX #15B1 @ayat
Failed
AB tests
All
ABTests
87%
Only 13% of AB tests
report significant uplift
#SMX #15B1 @ayat
Rules	on	failure
1. “Failure	is	an	option	here.	If	things	are	not	failing,	you	are	not	innovating	
enough...
#SMX #15B1 @ayat
7.	Align	your	website	objectives	with	business	KPIs
#SMX #15B1 @ayat
A	tool	is	only	as	
useful	as	the	
actionable insights	
you	are	able	to	derive	
from	it
#SMX #15B1 @ayat
Ayat’s	7	questions	for	analytics	report
1. So,	what?
2. What	KPI	does	this	measurement	support?	
3. How	d...
#SMX #15B1 @ayat
Increase online
revenue by 10%
Increase
website
revenue
Higher
conversion rate
Cart checkout Add to cart
...
#SMX #15B1 @ayat
invespcro.com/s17
#SMX #15B1 @ayat
8.	Understand	AB	Test	Statistics
#SMX #15B1 @ayat
A/B	testing	statistics
Confidence level
Factors that Affect Confidence Intervals (CI)
1. Population size:...
#SMX #15B1 @ayat
Statistical	power
• The	probability	that	a	
statistical	test	will	detect	
a	difference	between	two	
value...
#SMX #15B1 @ayat
If	you	run	100	tests	(95%	confidence	&	80%	power)
1. Let	the	tests	run	long	enough	to	achieve	confidence
...
#SMX #15B1 @ayat
If	you	run	100	tests	(95%	confidence	&	30%	power)
1. Let	the	tests	run	long	enough	to	achieve	confidence
...
#SMX #15B1 @ayat
When	segmenting	test	results	in	GA
• Run	your	tests	until	you	achieve	an	80%	to	90%	power	(well	powered	t...
#SMX #15B1 @ayat
9.	Be	careful	with	polluting		A/B	test	results
#SMX #15B1 @ayat
A/B	testing	results’	pollution
• Not	running	the	test	long	enough/too	long
• Recommended	a	minimum	of	wee...
#SMX #15B1 @ayat
If you would like to get a free
trial account, email
convert@invesp.com
#SMX #15B1 @ayat
How	can	I	help		you?
@ayat
ayat@invesp.com
www.invespcro.com
Upcoming SlideShare
Loading in …5
×

Advanced Testing: What 4,000 Successful A/B Tests Can Teach You By Ayat Shukairy

8,336 views

Published on

From the SMX Advanced Conference in Seattle, Washington, June 12-14, 2017. SESSION: Advanced Testing: What 4,000 A/B Tests Can Teach You. PRESENTATION: Advanced Testing: What 4,000 Successful A/B Tests Can Teach You - Given by Ayat Shukairy, @ayat - Invesp, VP of Client Solutions and Co-Founder. #SMX #15B1

Published in: Marketing

Advanced Testing: What 4,000 Successful A/B Tests Can Teach You By Ayat Shukairy

  1. 1. #SMX #15B1 @ayat 9 lessons to succeed at A/B testing Advanced Testing: What 4,000 Successful A/B Tests Can Teach You
  2. 2. #SMX #15B1 @ayat 9 lessons to succeed at A/B testing Advanced Testing: What 4,000 Successful A/B Tests Can Teach You
  3. 3. #SMX #15B1 @ayat • 65% average uplift • 4,000+ Successful Tests • 400+ CRO Projects • 11 different countries THE NUMBERS
  4. 4. #SMX #15B1 @ayat SOME OF OUR CLIENTS…
  5. 5. #SMX #15B1 @ayat 1. We focused on the wrong thing all along
  6. 6. #SMX #15B1 @ayat
  7. 7. #SMX #15B1 @ayat We focused too much on the short term – and lost focus on the long term, and as a result most CRO companies are nothing more than A/B testing companies
  8. 8. #SMX #15B1 @ayat Even the way we looked at funnels was wrong Bowtie loop Design thinking Via Beintheknow.co
  9. 9. #SMX #15B1 @ayat
  10. 10. #SMX #15B1 @ayat 2. Create a compelling website narrative
  11. 11. #SMX #15B1 @ayat Element level testing
  12. 12. #SMX #15B1 @ayat Page level testing
  13. 13. #SMX #15B1 @ayat Checkout Checkout Visitor flow testing
  14. 14. #SMX #15B1 @ayat
  15. 15. #SMX #15B1 @ayat
  16. 16. #SMX #15B1 @ayat
  17. 17. #SMX #15B1 @ayat
  18. 18. #SMX #15B1 @ayat What’s your narrative • What are you trying to communicate to your customers? • What do you want visitors to think from the minute they land on your website? • Are you wowing your visitors? • What are you doing to bring customers back to your website? • What are you doing for your visitors so that they refer their friends and colleagues to your website?
  19. 19. #SMX #15B1 @ayat The five second test gives visitors first impressions of your design
  20. 20. #SMX #15B1 @ayat 3. Don’t assume you live in a vacuum
  21. 21. #SMX #15B1 @ayat This might be a conversion problem, There is a real reputation problem
  22. 22. #SMX #15B1 @ayat
  23. 23. #SMX #15B1 @ayat Via Smashing Magazine Understand every touch point and its impact on the customer
  24. 24. #SMX #15B1 @ayat 4. Don’t assume you know everything customers
  25. 25. #SMX #15B1 @ayat “People tell you who they are, but we ignore it - because we want them to be who we want them to be”
  26. 26. #SMX #15B1 @ayat Qualitative Research
  27. 27. #SMX #15B1 @ayat Think like a customer CUSTOMER PAIN POINTS NEEDS PROBLEMS DESIRES CHALLENGES
  28. 28. #SMX #15B1 @ayat Polls and surveys § Motivators § Barriers § Hooks § Satisfaction (NPS, page level) § Demographical information § Feedback § Scoping competitive space http://bit.ly/polls101
  29. 29. #SMX #15B1 @ayat 5. Get complete buy-in from everyone
  30. 30. #SMX #15B1 @ayat We use to think it is a problem with the HIPPO Via undelay.io
  31. 31. #SMX #15B1 @ayat But you should get buy in from everyone to succeed Via DogTime
  32. 32. #SMX #15B1 @ayat 6. Embrace Failure
  33. 33. #SMX #15B1 @ayat Failed AB tests All ABTests 87% Only 13% of AB tests report significant uplift
  34. 34. #SMX #15B1 @ayat Rules on failure 1. “Failure is an option here. If things are not failing, you are not innovating enough." 2. You cannot let the feel of failure stop you from doing things that can have a direct impact on your success. You will and cannot succeed without the possibility of failure. 3. Failure should generate the meaningful research opportunities
  35. 35. #SMX #15B1 @ayat 7. Align your website objectives with business KPIs
  36. 36. #SMX #15B1 @ayat A tool is only as useful as the actionable insights you are able to derive from it
  37. 37. #SMX #15B1 @ayat Ayat’s 7 questions for analytics report 1. So, what? 2. What KPI does this measurement support? 3. How does this measurement support this KPI? 4. What does the measurement tell me? 5. What the dollar impact of that measurement? 6. What actions I must take based on the measurement? 7. What would drilling deeper tell me?
  38. 38. #SMX #15B1 @ayat Increase online revenue by 10% Increase website revenue Higher conversion rate Cart checkout Add to cart Product view ration Product viewed/session Product relevance Product page landing Site Search Sub category Filter interaction Product interaction Higher number of visitors Higher AOV Increase customer retention Increase brand reference by 5% Establish stronger brand Uncovering problems is driven by data as it relates to business objectives
  39. 39. #SMX #15B1 @ayat invespcro.com/s17
  40. 40. #SMX #15B1 @ayat 8. Understand AB Test Statistics
  41. 41. #SMX #15B1 @ayat A/B testing statistics Confidence level Factors that Affect Confidence Intervals (CI) 1. Population size: this does not usually affect the CI but can be a factor if you are working with small and known groups of people. 2. Sample Size: the smaller your sample, the less likely it is you can be confident the results reflect the true population value. 3. Percentage: higher uplifts come with better accuracy.
  42. 42. #SMX #15B1 @ayat Statistical power • The probability that a statistical test will detect a difference between two values when there is an underlying difference
  43. 43. #SMX #15B1 @ayat If you run 100 tests (95% confidence & 80% power) 1. Let the tests run long enough to achieve confidence 2. 10 out of our 100 variations will be effective and we expect to detect 80%= 8 tests 3. If we use a significance level cutoff of 5% we also expect to see 5 false positives. 4. So, on average, we will see 8 + 5 = 13 winning results from 100 A/B tests. 5. 38% of your winning tests are false positives
  44. 44. #SMX #15B1 @ayat If you run 100 tests (95% confidence & 30% power) 1. Let the tests run long enough to achieve confidence 2. 10 out of our 100 variations will be effective and we expect to detect 30%= 3 tests 3. If we use a significance level cutoff of 5% we also expect to see 5 false positives. 4. So, on average, we will see 3 + 5 = 8 winning results from 100 A/B tests. 5. 63% of your winning tests are false positives
  45. 45. #SMX #15B1 @ayat When segmenting test results in GA • Run your tests until you achieve an 80% to 90% power (well powered tests) • Run your tests until you’ve reached validity (beyond significance) • Each variation/segment must have statistical significance, otherwise you might identify false positives
  46. 46. #SMX #15B1 @ayat 9. Be careful with polluting A/B test results
  47. 47. #SMX #15B1 @ayat A/B testing results’ pollution • Not running the test long enough/too long • Recommended a minimum of week – best practice of two weeks • Recommended max run time of 4 weeks • Simpson paradox • There are factors that you cannot control • Device pollution • Browser pollution • Cookies pollution http://bit.ly/2qX6py4
  48. 48. #SMX #15B1 @ayat If you would like to get a free trial account, email convert@invesp.com
  49. 49. #SMX #15B1 @ayat How can I help you? @ayat ayat@invesp.com www.invespcro.com

×