Did you ever work on a feature where you had multiple options on how to display it to the user but you did not know which one would work best? Why speculate about what would work best if you can actually measure it? Welcome to the world of a/b testing: Implement multiple variations of the same thing, and let different users experience different variations. Measure which variant works best and continue with that one. At Wooga we extensively use a/b testing to polish features to perfection. This helped us becoming one of the biggest providers of social games on Facebook with about 40 million players every month. We would like to show you what a/b testing is good for, how you do it and what pitfalls you need to avoid.

1. Metrics
Driven
Design.

3. @chiefofspace

4. @jrirei

11. 500.000
0
1.000.000
1.500.000
2.000.000
2.500.000
2010
W16
2010
W19
2010
W22
2010
W25
2010
W28
2010
W31
2010
W34
2010
W37
2010
W40
2010
W43
2010
W46
2010
W49
2010
W52
2011
W03
DAU Monster World
2011
W06
2011
W09
2011
W12
2011
W15
2011
W18
2011
W21
2011
W24
2011
W27
2011
W30
2011
W33
2011
W36
2011
W39
2011
W42
2011
W45
2011
W48
2011
W51
2012
W02
2012
W05
2012
W08
2012
W11
2012
W14
2012
W17
2012
W20

13. >> Retention
>> Virality
>> Monetization

14. Retention

15. Retention
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins 100%
Step 2 - Loading complete 85%
Step - 3 70%
Step - 4 68%
Step - 5 66%
Step - 6 50%
Level 2 reached 48%
Step - 7 46%
Step - 8 44%

16. Virality

17. Virality

18. Monetization

19. ~2%
~ 12 %
~ 10 % ~3%

22. h/ps://mw.t.wooga.com/k/pgr/?
i=0&
language=de_DE&
s=100000107880707&
ts=1347085569&
w_friends=238

23. Tracking
Service

31. KaNa
DBs
…

39. >> Day 1
Retention

40. Day 1 Retention
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins 100%
Step 2 - Loading complete 85%
Step - 3 70%
Step - 4 68%
Step - 5 66%
Step - 6 50%
Level 2 reached 48%
Step - 7 46%
Step - 8 44%

41. Is it for me?
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins 100%
Step 2 - Loading complete 85%
Step - 3 70%
Step - 4 68%
Step - 5 66%
Step - 6 50%
Level 2 reached 48%
Step - 7 46%
Step - 8 44%

43. Do I get it?
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins 100%
Step 2 - Loading complete 85%
Step - 3 70%
Step - 4 68%
Step - 5 66%
Step - 6 50%
Level 2 reached 48%
Step - 7 46%
Step - 8 44%

44. Do I get it?

45. Do I get it?

46. On initial design solutions

47. Further, even further
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins 100%
Step 2 - Loading complete 85%
Step - 3 70%
Step - 4 68%
Step - 5 66%
Step - 6 50%
Level 2 reached 48%
Step - 7 46%
Step - 8 44%

48. Growth Time
7 day ret
3 day ret
1 day ret
Reached lvl 4
3 min
Reached lvl 3
5 min
0% 10% 20% 30% 40% 50% 60%

49. Beautification

50. Beautification
73,2 % 73,0 %

51. Plateau

52. What's next?
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins 100%
Step 2 - Loading complete 85%
Step - 3 70%
Step - 4 68%
Step - 5 66%
Step - 6 50%
Level 2 reached 48%
Step - 7 50%
Step - 8 48%

56. >> Day to Day
Retention

62. 0
50.000
100.000
150.000
200.000
250.000
'2010-09-13'
'2010-09-14'
'2010-09-15'
'2010-09-16'
'2010-09-17'
'2010-09-18' Launch
'2010-09-19'
'2010-09-20'
'2010-09-21'
'2010-09-22'
Post responses
'2010-09-23'
'2010-09-24'
'2010-09-25'
'2010-09-26'
'2010-09-27'
'2010-09-28'
'2010-09-29'
'2010-09-30'
'2010-10-01'
'2010-10-02'
2nd
Toy
'2010-10-03'
'2010-10-04'
'2010-10-05'
'2010-10-06'
'2010-10-07'
'2010-10-08'
'2010-10-09'
'2010-10-10'
'2010-10-11'
'2010-10-12'
'2010-10-13'
'2010-10-14'
'2010-10-15'
'2010-10-16'
'2010-10-17'
'2010-10-18'
'2010-10-19'
'2010-10-20'

63. >> Understanding User
Behaviour

65. 0
5.000
10.000
15.000
20.000
25.000
30.000
8/30/2011
9/6/2011
9/13/2011
9/20/2011
9/27/2011
10/4/2011
10/11/2011
10/18/2011
10/25/2011
11/1/2011
11/8/2011
11/15/2011
11/22/2011
Users without missions
12/28/2011
1/4/2012
1/11/2012
1/18/2012
1/25/2012
2/1/2012
2/8/2012
2/15/2012
2/22/2012
2/29/2012
3/7/2012
3/14/2012
3/21/2012
3/28/2012
4/4/2012
4/11/2012
4/18/2012

67. >> Next
try

68. Mr. Beazzy

73. >> + Engaged

74. >> + Engaged
>> + $

75. >> Virality

78. >>
Monetization

79. Magic Wand Shop
Old
New

80. Magic Wand Shop
Old
New
+ 24 %

81. Magic Wand Shop

82. Magic Wand Shop

83. Magic Wand Shop
+ 23 %

85. Remember the user

86. Match user to test group

87. Match user to test group

88. Feature switch

89. Wait for data to become relevant

90. Compare results
7 day ret
3 day ret
1 day ret
Reached lvl 4
3 min
Reached lvl 3
5 min
0% 10% 20% 30% 40% 50% 60%

91. Compare results

92. Side Effects

93. Side Effects

96. >> Endless
Love

97. Meet the bad guy:
Robert the robot

99. Meet his new
date:
Roberta the robot

106. Stephanie Kaiser
stephanie.kaiser@wooga.net
@chiefofspace
Jesper Richter-Reichhelm
jesper.richter-
reichhelm@wooga.net
@jrirei