Experimentation allows product managers to make decisions based on data rather than mere intuition. But too many teams don't know what to test, which leads to poorly designed experiments and unclear results. How can a product manager be certain they’re making effective decisions when it comes to experimentation? Join Holly Hester-Reilly, Founder and Product Management Coach at H2R Product Science, as she shares her approach to high-impact experimentation. She’ll walk us through the entire process, from deciding what to test to sharing the results with stakeholders, to illustrate what strong experimentation practices look like and how they can be implemented in every organization.

1. The Science of High-Impact
Experimentation
Holly Hester-Reilly Brittany Shear
With: Moderated by:
TO USE YOUR COMPUTER'S AUDIO:
When the webinar begins, you will be connected to audio using
your computer's microphone and speakers (VoIP). A headset is
recommended.
Webinar will begin:
9:00 am, PT
TO USE YOUR TELEPHONE:
If you prefer to use your phone, you must select "Use Telephone"
after joining the webinar and call in using the numbers below.
United States: +1 (213) 929-4232
Access Code: 648-540-186
Audio PIN: Shown after joining the webinar
--OR--
Experiment Your Way to Product Success
Webinar Series

2. 2
Experiment Your Way to Product Success
Webinar Series
Optimizely is the world’s leading experimentation platform, enabling
businesses to deliver continuous experimentation and personalization
across websites, mobile apps and connected devices. Optimizely
enables businesses to experiment deeply into their technology stack
and broadly across the entire customer experience. To learn more,
visit https://www.optimizely.com/

3. 3
Experiment Your Way to Product Success
Webinar Series
Click on the Questions panel to
interact with the presenters
https://www.productmanagementtoday.com/webinar-s
eries/experiment-your-way-to-product-success/

4. 4
Experiment Your Way to Product Success
Webinar Series
About Holly Hester-Reilly
Holly Hester-Reilly is the Founder and CEO of H2R Product Science, a product management coaching and consulting
firm that teaches the science of high-growth product development. Holly is a former Columbia University research
scientist and has led over a dozen successful digital product initiatives at startups, high-growth companies, and
enterprises like MediaMath, Shutterstock, The Lean Startup Co, and Weight Watchers. With those experiences she
has developed the Product Science Method, a framework to discover the strongest product opportunities and lay the
foundations for high-growth products, teams, and businesses.
Her team at H2R Product Science partners with startup founders and product leaders to share this framework, helping
them to figure out which product growth opportunities they should pursue and build the product management skill to
deliver on their goals.
Holly also teaches public and private workshops and has spoken about building high-growth products for events such
as Lean Startup Summit Europe, ProductTankNYC, Parsons School of Design, and the Product School.
About Brittany Shear
Brittany went to Emmanuel College, where she majored in Writing, Editing, and Publishing and minored in Marketing,
Psychology, and Communications. She now works with Aggregage as an editor and webinar host on sites including CTO
Universe, Product Management Today, and Connected Health Pulse.
Experiment Your Way to Product Success
Webinar Series

5. 5
Experiment Your Way to Product Success
Webinar Series
The Science of High-Impact Experimentation
Webinar Agenda
1. Determining what and when to test
2. Designing a high-impact experiment
3. Synthesizing results and determining next steps
4. Sharing experimentation for buy-in and alignment
5. Q&A

6. 6
Experiment Your Way to Product Success
Webinar Series
The Science of High-Impact Experimentation
Webinar Agenda
1. Determining what and when to test
2. Designing a high-impact experiment
3. Synthesizing results and determining next steps
4. Sharing experimentation for buy-in and alignment
5. Q&A

8. New Price Tiers
New Channels
New Products
New Features

12. profitwell.com/blog/customer-research-benchmarks

13. profitwell.com/blog/customer-research-benchmarks

14. profitwell.com/blog/customer-research-benchmarks

16. •2-sided marketplace for online education
•$1M annual revenue
•~1 year old
•Not profitable

17. “We wanted to create another
high-volume marketplace, to take what
we learned from Shutterstock and
create something else that had the
same sort of network effect.”

20. Pain
Outcome
Solution
Market
Product
Discovery
Loop

21. Pain
Outcome
Solution
Market
$$$

22. Pain
Outcome
Solution
Market

24. •Image Editor for non-professionals
•Prototype to public beta in 3 months
•At 1 year, 10x product usage
•At 3 years, 5 million people have tried the
product
“Shutterstock Editor was created to help
make non-designers' jobs easier and
significantly reduce the time it takes for
them to find, edit and use an image."

25. Pain
Outcome
Solution
Market
Product
Discovery
Loop

28. Pre-Mortem Risk
Assessment
●
●
●
●
●

30. 30
Experiment Your Way to Product Success
Webinar Series
The Science of High-Impact Experimentation
Webinar Agenda
1. Determining what and when to test
2. Designing a high-impact experiment
3. Synthesizing results and determining next steps
4. Sharing experimentation for buy-in and alignment
5. Q&A

31. Answers WHY and HOW
●
●
●
Answers HOW MUCH and HOW MANY
●
●

32. Org Position Decision Types Testing Methods
Leadership Vision & Target Market Quantitative & Qualitative
Behavior-Based Research
Competitive Landscape
Market Trends Analysis
Management Pains & Outcomes for each Target Market User Interviews
Competitive Product Reviews
User Surveys
Usage Data
Teams Solutions
(Designs, Features, Technologies)
Design Research
User Interviews
Usability Testing
Feasibility Research
Usage Data
A/B Tests

34. ●
○
○
What do the most active users like about Skillfeed?
What courses are most popular?
How many users have completed several courses?

35. 35
Experiment Your Way to Product Success
Webinar Series
The Science of High-Impact Experimentation
Webinar Agenda
1. Determining what and when to test
2. Designing a high-impact experiment
3. Synthesizing results and determining next steps
4. Sharing experimentation for buy-in and alignment
5. Q&A

37. We experiment at every step of the loop.
All the way up to launch of a new feature

38. 38
Experiment Your Way to Product Success
Webinar Series
The Science of High-Impact Experimentation
Webinar Agenda
1. Determining what and when to test
2. Designing a high-impact experiment
3. Synthesizing results and determining next steps
4. Sharing experimentation for buy-in and alignment
5. Q&A

43. 43
Experiment Your Way to Product Success
Webinar Series
Q&A
Brittany Shear
With: Moderated by:
CEO and Founder, H2R Product Science
Linkedin page: /in/hollyhesterreilly/
Twitter ID: @H2Rproductsci
Website: https://www.h2rproductscience.com
Holly Hester-Reilly
Editor, Product Management Today
Linkedin page: /in/brittanyshear
Twitter ID: @prodmgmttoday
Website: productmanagementtoday.com
https://www.productmanagementtoday.com/webinar-series/exp
eriment-your-way-to-product-success/