Your SlideShare is downloading. ×
  • Like
  • Save
Jesse Hull - LOGIN Conference 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Jesse Hull - LOGIN Conference 2012

  • 222 views
Published

Jesse Hull - Building A Culture of A/B Testing - LOGIN Conference 2012 - 9/26/2012

Jesse Hull - Building A Culture of A/B Testing - LOGIN Conference 2012 - 9/26/2012

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
222
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Building a Culture of A/B Testing Jesse HullExecutive Director of Technology, Social Games GSN Digital
  • 2. 2
  • 3. 3
  • 4. What is A/B Testing? - Randomly Assign Users We are the same Group A Group B Group A Group B 4
  • 5. What is A/B Testing? – Show Different Content A B Group A Group B 5
  • 6. What is A/B Testing? – Measure Result A B 6
  • 7. “”A/B testing is like sandpaper. Youcan use it to smooth out details, but you cant actually create anything with it.” 7
  • 8. This Is Wrong! 8
  • 9. You Can Do A Lot More Then Just Smooth Out The Details With…Holistic A/B Testing 9
  • 10. What is Holistic A/B Testing? Principle #1Everyone is Testing 10
  • 11. Most Organizations Use These People To Test 11
  • 12. And They Come Up With Tests Like These A B (2.4% more revenue) 12
  • 13. But You Want These People Testing 13
  • 14. What is Holistic A/B Testing? Principle #2Test Big and Small Ideas 14
  • 15. Why Small Ideas?10 Ideas That Each Improve Revenue 4% +50% 15
  • 16. Why Big Ideas? 16
  • 17. Example:Casino Games 17
  • 18. Game DesignerTested Wildly Different Casino Game Mechanics 18
  • 19. Scratch Madness 19
  • 20. Roulette 20
  • 21. Video Bingo 21
  • 22. Blackjack 22
  • 23. Video Bingo Won• Time playing: +356% (over closest competition)• Next day return rate: +42%• Revenue: +238% 23
  • 24. That Was Just The Beginning Engineers Improved Load Time 24
  • 25. • Time spent playing: +12.9%• Next day return rate: +6.8%• Revenue: +3.6% 25
  • 26. ArtistsTested Completely New UI 26
  • 27. 27
  • 28. 28
  • 29. • Revenue: +12.3%• Bounce Rate: -25.8% 29
  • 30. Enterprising Developer Tested Various Power-Ups 30
  • 31. Three Extra Balls At The End Revenue: +5.9% 31
  • 32. U Pick’emRevenue: +4.7% 32
  • 33. • Revenue: +23.6%!• Next day return: unchanged 33
  • 34. How Do You Build An Organization That Can Holistically A/B Test? In 2 Easy Steps… 34
  • 35. Organization: Step 1Build A FrameworkThat Everyone Can Use 35
  • 36. A/B Testing Framework Variations Keep Common Testing Infrastructure 36
  • 37. Organization: Step 2Build A/B Testing Into Your Culture 37
  • 38. Set Team Goals In Terms Of Tests Perform 10 A/B tests focused on improving daily active users.The combination of these tests should improve DAUs by 20%. 38
  • 39. Everyone Is Testing: Game Designers 39
  • 40. Everyone Is Testing: Game Designers 40
  • 41. Everyone Is Testing: Game Designers 41
  • 42. Everyone Is Testing: Artists 42
  • 43. Everyone Is Testing: Artists 89.3% 4.3% 23.5% 42.0% 8.9%click rate install rate click rate click rate install rate 12,039 130.2% 0.4% 34,084 41.2%publishes click rate install rate publishes click rate 43
  • 44. Everyone Is Testing: Engineers• Performance improvements• Architectural changes 44
  • 45. Everyone Is Testing: Marketing• Test sending emails to only 50% of your user base• Cross-promotion 45
  • 46. Deciding What To Test• Hire great people• Improve their intuition by testing 46
  • 47. When Not to A/B TestAw, come on, baby. I wasn’t really cheating on you I was A/B testing. 47
  • 48. When Not to A/B Test• Technology limitations• No users• Long-term effects 48
  • 49. Everyone in theorganization should be testing all the time because... 49
  • 50. the impact of your testing is limited only by the breadth and creativity of the ideas you test. 50
  • 51. Our Company• Contact Me: jhull@gsn.com We Are Hiring! www.gsngames.com 51
  • 52. "Do not believe in anything simply because you have heard it. Do not believe in anything simply because it is spoken and rumored by many.... But after observation and analysis, when you find that anything agrees with reason and is conducive to the good and benefit of one and all, then accept it and live up to it." -- Buddha 52
  • 53. Questions? 53