1. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
How To Win Friends &
Influence People With
A/B Testing
@
Charles Hua
Director of Optimization & AB Testing

2. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
Real-Life Test: Which is True About Charles?
I have 2 kids:
(older kid’s birthday
month/day/year) x 2 =
(younger kid’s birthday
month/day/year)
Version A
At one point, I
considered a career
as an accountant
Version B
How good is your gut?

3. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
Web Analyst
Project Manager
QA
UX Designer
Tagging Eng
Frontend Developer
Statistician
Politician
Mom
Experiment Designer

4. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
I perform experiments on people…
#TLE2015

5. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
What the heck is RetailMeNot ?
#TLE2015
Multinational coupon company with the largest
coupon sites in US and UK
We are here to save the world… some money!

6. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
30
100
700+
2013 2014
Testing Volume
Veg CQMS

7. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
+12M Annualized incremental
revenue with just one algorithm test
#TLE2015

8. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Making sound
business decisions with
DATA
What Is The Purpose Of Testing @RMN?
#TLE2015

9. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Build a solid foundation for success
with Tools Process and People
#TLE2015
Get your House in order
1. Involve Others
Know what They are thinking

10. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Step 1: Get The House
In Order
#TLE2015

11. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Right Tools For The Right Job
#TLE2015
Testing Tool Project Management

12. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
If You Don’t Do These, You Are
Heading For Trouble
#TLE2015

13. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Calibrate The Testing Tool Regularly
#TLE2015
AA or Blank Test

14. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Filter Out Bots & Other Traffic Noise
#TLE2015
Keep the IP filter list up to date

15. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Create A Framework For Outliers Then
Monitor Them
#TLE2015
3 Standard Deviations

16. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Check page load speed
#TLE2015
100mS increase in load time ~2% increase in bounce

17. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Create a Process
#TLE2015
1 Define roles and responsibility
2 Set scope and cadence
3 Create test threshold
4 Know when to stop a test
5 Retrospective

18. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Set Expectations w/ Test Spec
#TLE2015
1 Hypothesis
2 Decision KPI
3 Sample size needed
4 Experiment design
5 What to do with win?

19. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Importance Of Both Quantitative And Qualitative Kpis
#TLE2015
-18% in NPS due to
Drop in Promoter

20. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
How To Monitor A Test In Flight?
#TLE2015
1. Create a Threshold
2. Don’t panic

21. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Do You Have The Confidence & Power?
#TLE2015
90%
Confident
80%
Power

22. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Why Not Peek Before Hitting Sample Size?
#TLE2015
Control Test
Order 1 2
Visits 5 5
Conversion 20% 40%
Lift 100%
Confidence 90%
Power 16%
64
samples
needed

23. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
How Is It Trending?
#TLE2015
Look for diverging or
parallel lines

24. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Congrats! 15% of your work is done.
Now you have to deal with
PEOPLE…
#TLE2015

25. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Step 2: Involve Others
#TLE2015

26. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Testing
Product
Eng
UX /
Design
Ops /
Merch
Finance
Analyst
SEO
Informed Decision Making
Form Core Testing A-Team & Educate The Process

27. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
LIFT Brainstorm Sessions
#TLE2015
Structured and Focused

28. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Turn Ideas Into Experiment Design
#TLE2015
Combine ideas into MVT
Element 1 Element 2
Variation 1 Variation 1
Variation 2 Variation 2
Variation 3 Variation 3
2 X 3 = 6 Variations

29. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Build An Efficient Roadmap
#TLE2015
1 EVFPIE (Potential Importance Ease)
2 Room for the unexpected and HiPPOs

30. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Be Transparent
#TLE2015
Communicate
Communicate
Communicate

31. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Over-communicate, Share & Be Transparent
#TLE2015
WIKI
Email
Notifications

32. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Discuss the results before making
decisions…
#TLE2015

33. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Promoting A Testing Culture – Make It Fun
#TLE2015

34. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Step 3: Know What
They Are Thinking
#TLE2015

35. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Who are they and what
are they thinking?
Decision Makers?
#TLE2015

36. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Questions To Ask
#TLE2015
1 What is the purpose of testing?
2 Who are the supporters?
3 How is the incentive structure set up?
4 Where is the testing team in the org?

37. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Common Reasons People Push Back On
Testing
#TLE2015
It slows me
down
I’m afraid to
know
I’m
confident it’s
a win

38. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Prep The Sirens
#TLE2015

39. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Homepage Test
#TLE2015
Merchandising
Team (Offers)
Design Team
(Aesthetics, UX)
Sales Team
($$$)
Operations Team
(No Issues)
Marketing Team
(Traffic)

40. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Measure ROI
#TLE2015
1 Make friends at Finance Dept
2 Define measurement framework
3 Set up goals
4 Show sequential performance

41. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Show them you care and make them
feel important
#TLE2015

42. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
What’s HOT?
Get your HOUSE in order
Involve OTHERS
Know what THEY are thinking
#TLE2015

43. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Continuous
Innovation
#TLE2015

44. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Sequential Likelihood Ratio Test – Bayesian Stats To complement
Student T-Test Stats
#TLE2015
Test End (T-Test)Test End (SS)

45. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Flexible Testing Platform
#TLE2015
Easy
Integration
With Any
Testing
Tools

46. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Questions?
#TLE2015

47. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
Charles Hua
RetailMeNot
Director of Optimization & AB Testing
chua@rmn.com

48. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
Real-Life Test: Which is True About Charles?
I have 2 kids:
(older kid’s birthday
month/day/year) x 2 =
(younger kid’s birthday
month/day/year)
Version A
At one point, I
considered a career
as an accountant
Version B
How good is your gut?