Experimentation, it seems easy but it's difficult to execute properly. We started from the beginning with some basic concepts, what testing looks like in action, and how you can utilize testing to get an early indication of what kind of impact a feature will make on your product. MaryKate Guidry, the Associate Product Manager for Experimentation at Jet.com, talked about the basic concepts of experimentation, what does success look like, how to best leverage experimentation and how to engage with your resources.

1. Experimentation Concepts & How to Leverage
Them with Jet.com’s Product Manager
www.productschool.com

2. FREE INVITE
Join 12,000+ Product Managers on
productschool.com/slack-community

3. Product
Management
2-month part-time Courses

4. Coding
for Managers
2-month part-time Courses

5. Data Analytics
for Managers
2-month part-time Courses

6. Include @productschool and #prodmgmt at the
end of your tweet
Tweet to get a free ticket
for our next Event!

7. MaryKate Guidry
Tonight’s Speaker

8. Experimentation Concepts
and How to Leverage Them
MaryKate Guidry
Associate Product Manager, Testing & Optimization
8

9. Ground Rules 9
- Feel free to ask questions as
we go along (or wait, you
choose!)
- Hear something?
Say something!
- Laughing at jokes is
encouraged, but not required

10. Let’s Start with a Game

11. Product Recommendations

12. Product Recommendations 12
Variant A
Image

13. Product Recommendations 13
Variant B
Image +
Title

14. Product Recommendations 14
Variant C
Image +
Price

15. Product Recommendations 15
Variant D
Image +
Attributes

16. Product Recommendations 16
Variant E
Image +
Title +
Price

17. Product Recommendations 17
Variant F
Image +
Title +
Attributes

18. Product Recommendations 18
Variant G
Image +
Title +
Price +
Attributes

19. Experiment Design
Image
A Image
Price
C Image
Attributes
DImage
Title
B Image
Title
Price
E Image
Title
Price
Attributes
GImage
Title
Attributes
F

20. Experiment Winner
Variant A
20

21. Experimentation Types

22. Experiment Design Types: A/B 22

23. Experiment Design Types: Multivariate 23

24. Experiment Design Types: Full Factorial 24

25. Designing an Experiment

26. Remember the Scientific Method? 26
Your science fairs and labs of
yore are back!
1. Observation
2. Hypothesis
3. Prediction
4. Experiment
5. Analysis
6. Conclusion

27. Objective 27

28. Analytics 28

29. Qualification & Targeting 29

30. Variants 30

31. Timing 31

32. Sample Sizing 32

33. 33

34. Statistical Significance 34

35. Some Parting Thoughts… 35
• Culture of testing
• Find your tribe
• Pre-release testing vs.
optimization of existing site

36. All Experiments are
Good Experiments

39. Questions?

40. Part-time Product Management Courses in
San Francisco, Silicon Valley, Los Angeles
and New York
www.productschool.com