A/B testing #eMetrics London

155 views

Published on

product development, a/b testing, case studies

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
155
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Insert images of our charaters
  • $ 7.5 bn in 2013$ 5.5 bn USA by 2015
  • If you have tests that show positive improvements, and when switched live they have little or a negative effect…. This might be the cause
  • We’re going now into gamification territory
  • The difference in the two groups is about 40-50,000 users.
  • The difference is between – 0.4 and 0.8 percentage points at 95% confidence.Also looks like the gap is widening…. And the other retention measure showed a similar behaviour.
  • Running the opportunity cost calculations I discovered that we could be making on average $1,500 / day more switching to the expensive group.
  • A/B testing #eMetrics London

    1. 1. A/B Testing Alexandra Paun @ToastingIdeas
    2. 2. Pre Test Significances A/A Testing
    3. 3. User Base Control Test 75% 25%
    4. 4. Revenue Test Control
    5. 5. Revenue Test Control
    6. 6. Pre test significances results biased
    7. 7. The Viral Gate Test
    8. 8. Revenue Control Test
    9. 9. Daily Active Users 1,500,000 1,450,000 1,400,000 1,350,000 1,300,000 Control Test 1,250,000 1,200,000 1,150,000 1,100,000
    10. 10. Daily Active Users 7 Day Moving Average 1,440,000 1,420,000 1,400,000 1,380,000 1,360,000 1,340,000 Control Test 1,320,000 1,300,000 1,280,000 1,260,000
    11. 11. 3 Day Retention 18.60 18.40 18.20 18.00 17.80 17.60 17.40 Control Test 17.20 17.00 16.80 16.60 16.40
    12. 12. Viral gate
    13. 13. Too good to be true?
    14. 14. The Opportunity Cost Test Could This Be Making More Money?
    15. 15. User Base Cheap Plasma Expensiv e Plasma 10% 90%
    16. 16. 3/6/13 3/8/13 3/10/13 3/12/13 3/14/13 3/16/13 3/18/13 3/20/13 3/22/13 3/24/13 3/26/13 3/28/13 3/30/13 4/1/13 4/3/13 4/5/13 4/7/13 4/9/13 4/11/13 4/13/13 4/15/13 4/17/13 4/19/13 4/21/13 4/23/13 4/25/13 4/27/13 4/29/13 5/1/13 5/3/13 5/5/13 5/7/13 5/9/13 5/11/13 5/13/13 Revenue Cheap Plasma Expensive Plasma
    17. 17. Daily Active Users 700,000 600,000 500,000 400,000 Cheap Plasma Expensive Plasma 300,000 200,000 100,000 0 2/28/13 3/7/13 3/14/13 3/21/13 3/28/13 4/4/13 4/11/13 4/18/13 4/25/13 5/2/13 5/9/13
    18. 18. Plasma bubbles sold 200,000 180,000 160,000 140,000 120,000 100,000 Cheap Plasma Expensive Plasma 80,000 60,000 40,000 20,000 0 3/6/13 3/13/13 3/20/13 3/27/13 4/3/13 4/10/13 4/17/13 4/24/13 5/1/13 5/8/13
    19. 19. Fire bubbles sold 100,000 90,000 80,000 70,000 60,000 50,000 Cheap Plasma Expensive Plasma 40,000 30,000 20,000 10,000 0 3/6/13 3/13/13 3/20/13 3/27/13 4/3/13 4/10/13 4/17/13 4/24/13 5/1/13 5/8/13
    20. 20. Question everything
    21. 21. Long Run Testing
    22. 22. How long should I keep this test running for? Long enough
    23. 23. 4/29/13 4/27/13 4/25/13 4/23/13 4/21/13 4/19/13 4/17/13 4/15/13 4/13/13 4/11/13 4/9/13 4/7/13 4/5/13 4/3/13 4/1/13 3/30/13 3/28/13 3/26/13 3/24/13 3/22/13 3/20/13 3/18/13 3/16/13 3/14/13 3/12/13 3/10/13 3/8/13 3/6/13 3/4/13 3/2/13 2/28/13 Daily Active Users 1,400,000 1,200,000 1,000,000 800,000 600,000 Test Control 400,000 200,000 0
    24. 24. 4/29/13 4/27/13 4/25/13 4/23/13 4/21/13 4/19/13 4/17/13 4/15/13 4/13/13 4/11/13 4/9/13 4/7/13 4/5/13 4/3/13 4/1/13 3/30/13 3/28/13 3/26/13 3/24/13 3/22/13 3/20/13 3/18/13 3/16/13 3/14/13 3/12/13 3/10/13 3/8/13 3/6/13 1 Day Retention 22.5 22 21.5 21 20.5 20 Test Control 19.5 19 18.5 18
    25. 25. 4/29/13 4/27/13 4/25/13 4/23/13 4/21/13 4/19/13 4/17/13 4/15/13 4/13/13 4/11/13 4/9/13 4/7/13 4/5/13 4/3/13 4/1/13 3/30/13 3/28/13 3/26/13 3/24/13 3/22/13 3/20/13 3/18/13 3/16/13 3/14/13 3/12/13 3/10/13 3/8/13 3/6/13 Revenue Test Control
    26. 26. 4/29/13 4/27/13 4/25/13 4/23/13 4/21/13 4/19/13 4/17/13 4/15/13 4/13/13 4/11/13 4/9/13 4/7/13 4/5/13 4/3/13 4/1/13 3/30/13 3/28/13 3/26/13 3/24/13 3/22/13 3/20/13 3/18/13 3/16/13 3/14/13 3/12/13 3/10/13 3/8/13 3/6/13 Revenue Test Control
    27. 27. Could this be better?
    28. 28. Why did I not test this?
    29. 29. Daily Active Users 1,200,000 1,000,000 800,000 600,000 400,000 200,000 0
    30. 30. 1 Day Retention 25.00% 20.00% 15.00% 10.00% 5.00% 0.00%
    31. 31. New Users 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0
    32. 32. New Users Segementation 25,000 20,000 15,000 Featuring 10,000 5,000 0 Requests
    33. 33. Make your life easy. Run an A/B test
    34. 34. • A/A Testing • Too good to be true (?) • Could this be better? • Long run tests • Test everything
    35. 35. Questions? @ToastingIdea s

    ×