Your SlideShare is downloading. ×
0
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Jesse Hull - LOGIN Conference 2012

252

Published on

Jesse Hull - Building A Culture of A/B Testing - LOGIN Conference 2012 - 9/26/2012

Jesse Hull - Building A Culture of A/B Testing - LOGIN Conference 2012 - 9/26/2012

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
252
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. Building a Culture of A/B Testing Jesse HullExecutive Director of Technology, Social Games GSN Digital
  2. 2
  3. 3
  4. What is A/B Testing? - Randomly Assign Users We are the same Group A Group B Group A Group B 4
  5. What is A/B Testing? – Show Different Content A B Group A Group B 5
  6. What is A/B Testing? – Measure Result A B 6
  7. “”A/B testing is like sandpaper. Youcan use it to smooth out details, but you cant actually create anything with it.” 7
  8. This Is Wrong! 8
  9. You Can Do A Lot More Then Just Smooth Out The Details With…Holistic A/B Testing 9
  10. What is Holistic A/B Testing? Principle #1Everyone is Testing 10
  11. Most Organizations Use These People To Test 11
  12. And They Come Up With Tests Like These A B (2.4% more revenue) 12
  13. But You Want These People Testing 13
  14. What is Holistic A/B Testing? Principle #2Test Big and Small Ideas 14
  15. Why Small Ideas?10 Ideas That Each Improve Revenue 4% +50% 15
  16. Why Big Ideas? 16
  17. Example:Casino Games 17
  18. Game DesignerTested Wildly Different Casino Game Mechanics 18
  19. Scratch Madness 19
  20. Roulette 20
  21. Video Bingo 21
  22. Blackjack 22
  23. Video Bingo Won• Time playing: +356% (over closest competition)• Next day return rate: +42%• Revenue: +238% 23
  24. That Was Just The Beginning Engineers Improved Load Time 24
  25. • Time spent playing: +12.9%• Next day return rate: +6.8%• Revenue: +3.6% 25
  26. ArtistsTested Completely New UI 26
  27. 27
  28. 28
  29. • Revenue: +12.3%• Bounce Rate: -25.8% 29
  30. Enterprising Developer Tested Various Power-Ups 30
  31. Three Extra Balls At The End Revenue: +5.9% 31
  32. U Pick’emRevenue: +4.7% 32
  33. • Revenue: +23.6%!• Next day return: unchanged 33
  34. How Do You Build An Organization That Can Holistically A/B Test? In 2 Easy Steps… 34
  35. Organization: Step 1Build A FrameworkThat Everyone Can Use 35
  36. A/B Testing Framework Variations Keep Common Testing Infrastructure 36
  37. Organization: Step 2Build A/B Testing Into Your Culture 37
  38. Set Team Goals In Terms Of Tests Perform 10 A/B tests focused on improving daily active users.The combination of these tests should improve DAUs by 20%. 38
  39. Everyone Is Testing: Game Designers 39
  40. Everyone Is Testing: Game Designers 40
  41. Everyone Is Testing: Game Designers 41
  42. Everyone Is Testing: Artists 42
  43. Everyone Is Testing: Artists 89.3% 4.3% 23.5% 42.0% 8.9%click rate install rate click rate click rate install rate 12,039 130.2% 0.4% 34,084 41.2%publishes click rate install rate publishes click rate 43
  44. Everyone Is Testing: Engineers• Performance improvements• Architectural changes 44
  45. Everyone Is Testing: Marketing• Test sending emails to only 50% of your user base• Cross-promotion 45
  46. Deciding What To Test• Hire great people• Improve their intuition by testing 46
  47. When Not to A/B TestAw, come on, baby. I wasn’t really cheating on you I was A/B testing. 47
  48. When Not to A/B Test• Technology limitations• No users• Long-term effects 48
  49. Everyone in theorganization should be testing all the time because... 49
  50. the impact of your testing is limited only by the breadth and creativity of the ideas you test. 50
  51. Our Company• Contact Me: jhull@gsn.com We Are Hiring! www.gsngames.com 51
  52. "Do not believe in anything simply because you have heard it. Do not believe in anything simply because it is spoken and rumored by many.... But after observation and analysis, when you find that anything agrees with reason and is conducive to the good and benefit of one and all, then accept it and live up to it." -- Buddha 52
  53. Questions? 53

×