Shared my experiences with A/B testing and lessons learned to make developers’ lives easier at #360iDev

1. Buildingan A/BTesting Framework
thatWorks for Developers
Jenny Chang Ho
Senior iOS Engineer @ GRUBHUB
Twitter: @TipTopGS

2. Aboutme
Software Engineer @ Google
Consultant @ IBM
Tech Lead @ Royal Bank of
Canada
Senior iOS Engineer @ GRUBHUB
2

3. A/Btesting
3

4. A/B testing is
testingahypothesis

5. A/B testing is
feature flagging(flows and design variations)
@TipTopGS 5

6. A/B testing is
controlling rollouts of features
@TipTopGS 6

7. A/B testing is
innovatingwhile minimizing risks
@TipTopGS

8. A/Btesting is messy
@TipTopGS <<Photo by Ricardo Viana on Unsplash>>

9. Dead code
@TipTopGS <<Photo by Mathew MacQuarrie on Unsplash>>

10. Increased in
developmentefforts
@TipTopGS <<Photo by #WOCinTech Chat>>

11. Increasedtesting
scope
@TipTopGS 11

12. A/Btesting is notfor everyone
@TipTopGS <<Photo by Charles Deluvio
! "
on Unsplash>> 12

13. Who drives experimentation?
@TipTopGS <<Photo by rawpixel on Unsplash>> 13

14. A/BTesting
» Feature flagging
» Innovation
» Increase development/testing scope
» Not for everyone
@TipTopGS 14

15. A/B Experiments Examples
@TipTopGS 15

16. Coach bubbles
First losing experiment
@TipTopGS 16

17. @TipTopGS 17

18. Hackathon Experiments
@TipTopGS 18

19. Schedule order entrypoint
@TipTopGS 19

20. Haptic feedbacks
@TipTopGS 20

21. Flow example:
Defaulttabs onapplaunches
@TipTopGS 21

22. @TipTopGS 22

23. Multiple iterations examples:
GrubHub reviewswith menu items
@TipTopGS 23

24. @TipTopGS. GrubHub reviews with menu items 24

25. RestaurantPromotions
@TipTopGS 25

26. @TipTopGS. Restaurant promotions 26

27. Testing restaurant promotions carousel
enableRTPCarouselVariant enableRTPCarouselWithCuisines App behavior
OFF OFF Standard RTP carousel
is shown
ON OFF Carousel variant is
shown (without cuisine)
OFF ON Carousel variant is
shown (with cuisine)
ON ON Carousel variant is
shown (without cuisine)
ON or OFF does not exist in feed App listen to
enableRTPCarouselVarian
t toggle
@TipTopGS 27

28. Experimentsthattakealongtime
@TipTopGS 28

29. Navigation(drawer vs tabs)
@TipTopGS 29

30. @TipTopGS 30

31. Redesign or style changes
@TipTopGS 31

32. @TipTopGS 32

33. A/B Experiments
» Winning is not guaranteed
» User observations
» Iterations
» Time
@TipTopGS 33

34. Keys forA/B testing success
» Objectives must be defined
» Thinking about problems from your user's
perspective
» Learning from results
@TipTopGS 34

35. “Failearly, failoften,
butalways fail
forward.”
John C. Maxwell
@TipTopGS 35

36. Howto beagood loser?
Deleting feature work hurts
@TipTopGS <<Photo by Matthew Henry on Unsplash>> 36

37. Lessons learned
@TipTopGS 37

38. Lesson #1
Understandwhatcould
gowrongand
question
assumptions
@TipTopGS 38

39. Lesson #2
Evaluatethetools:
A/BTesting
Framework Setupand
Customizations
@TipTopGS 39

40. Lesson #3
Should everyfeature
have afeature flag?
Why not a kill switch?
@TipTopGS 40

41. Lesson #4
Create Debug
Settings
Local and remote toggles
@TipTopGS 41

42. 42

43. Lesson #5
Automatetests!Experiments can take a long time.
@TipTopGS 43

44. Don'tforgetto...
Unittestyour default
feature flagvalues
@TipTopGS 44

45. Lesson #6
Write code easierto
make permanent
or delete
@TipTopGS 45

46. Lesson #7
Setupaclean upprocess
» Product Manager
» Variables Used
» Start/Ended Dates
» Release App Store Version
» Winning Variant
@TipTopGS <<Photo by Perry Grone on Unsplash>> 46

47. Lesson #8
Usethe results from
the other platform!!!
@TipTopGS 47

48. What's in it for productfolks?
@TipTopGS 48

49. 49

50. TakeAways
» Check the assumptions of the experiments
» Set up clean up process
» Decide feature flag vs kill switch
» Automate tests
50

51. JennyChang Ho
jchangho@grubhub.com
Twitter: @tiptopgs
https://medium.com/@Tiptopgs
51

52. Questions &Answers
52