The document outlines a masterclass on forming strong hypotheses for product experimentation, led by Jason G'sell, who has extensive experience in customer education and training. Key points include using direct and indirect data to identify user problems, formulating clear problem statements, and validating these with data. The webinar aims to teach participants how to craft hypotheses that address defined user issues and leverage experimentation as a problem-solving framework.

1. 1
Product Experimentation
Masterclass:
Forming Strong Experiment
Hypotheses

2. Jason has been a member of Optimizely’s
Customer Success, Education, and Services
teams for over three years, enabling hundreds
of businesses to build their experimentation
programs, increase their testing and
personalization skills, and derive meaningful
insights from their experiment data.
OPTIMIZELYSERVICES
2018
Jason G’Sell
Lead Training Consultant,
EMEA
Opticon Training Day
Managed five training workshops tailored to all
skill levels delivered at the annual Opticon
conference.
Launch Customer Education in Europe
Expanded Optimizely Customer Education
training and support from US to EMEA region
Selected experience
Customers
Customized Training Programs
Build and deliver bespoke virtual and
in-person training programs for large
enterprises to effectively onboard
hundreds of teams globally
Education & Certification
University of Notre Dame, USA
Bachelor of Arts in Sociology, Minor in Gender
Studies
Association of Talent Development
ATD Training Certificate
Sr. Training Consultant, Optimizely
Training customers on high-value skills across
Web Experimentation, Personalization, and Full
Stack
Launch Manager, Optimizely
Onboarding Web Experimentation customers
at-scale, helping them achieve value as quickly
as possible
Customer Success Manager – Implementation,
ClearSlide
Developed train-the-trainer program to enable
onboarding for thousands of sales reps
nationally
Areas of expertise & industry

3. 1. Use direct and indirect data to develop
experiment ideas
2. Craft hypotheses using a defined
framework
3. Identify the differences between
optimization and discovery
Agenda

4. Housekeeping
➔ This webinar will last around 60 minutes.
➔ Feel free to ask questions in the chat console at
any time!
➔ We are recording this session and will email a link
to the recording.

5. Audiences
Areas
KPIs
Goal
Levers
Experiments
What problem are we trying to solve?
How will we know if we’ve solved it?
Where is the problem?
Who does it impact and how?
What solutions could exist?
Which is the best solution?
Experimentation is a Problem-Solving Framework

6. ● Web or Product analytics
● Experiment results
● Voice of the customer
● Heatmaps
● User personas
● Usability tests
Indirect Data is collected about the industry, your
competitors, and external trends
Direct & Indirect Data
Direct Data is directly focused on your site experience
and can tell you about a specific problem in your site
experience.
● Competitor sites
● Best-in-class experiences
● Blogs and webinars
● User groups
● Academic literature

7. Revisit the Customer’s
Journey
Put Yourself in Your
User’s Shoes

8. Start by Identifying User Problems

9. Formulate a
Problem Statement
A problem statement is a clear concise description of the issue(s) that
need(s) to be addressed.

10. Exercise: Good or Bad Problem Statement?
1. Our Android app experience is bad.
2. Users are confused by our pricing tiers.
3. We should transition to a single-page app.
4. Users don’t understand our value proposition.
5. Users exit between Step 3 and 4.
6. Users don’t trust our marketing.

11. Exercise: Good or Bad Problem Statement?
1. Our Android app experience is bad.
2. Users are confused by our pricing tiers.
3. We should transition to a single-page app.
4. Users don’t understand our value proposition.
5. Users exit between Step 3 and 4.
6. Users don’t trust our marketing.
X

12. Exercise: Good or Bad Problem Statement?
1. Our Android app experience is bad.
2. Users are confused by our pricing tiers.
3. We should transition to a single-page app.
4. Users don’t understand our value proposition.
5. Users exit between Step 3 and 4.
6. Users don’t trust our marketing.
X

13. Exercise: Good or Bad Problem Statement?
1. Our Android app experience is bad.
2. Users are confused by our pricing tiers.
3. We should transition to a single-page app.
4. Users don’t understand our value proposition.
5. Users exit between Step 3 and 4.
6. Users don’t trust our marketing.
X
X

14. Exercise: Good or Bad Problem Statement?
1. Our Android app experience is bad.
2. Users are confused by our pricing tiers.
3. We should transition to a single-page app.
4. Users don’t understand our value proposition.
5. Users exit between Step 3 and 4.
6. Users don’t trust our marketing.
X
X

15. Exercise: Good or Bad Problem Statement?
1. Our Android app experience is bad.
2. Users are confused by our pricing tiers.
3. We should transition to a single-page app.
4. Users don’t understand our value proposition.
5. Users exit between Step 3 and 4.
6. Users don’t trust our marketing.
X
? Symptom of a Problem
X

16. Exercise: Good or Bad Problem Statement?
1. Our Android app experience is bad.
2. Users are confused by our pricing tiers.
3. We should transition to a single-page app.
4. Users don’t understand our value proposition.
5. Users exit between Step 3 and 4.
6. Users don’t trust our marketing.
X
? Symptom of a Problem
X

17. The #1 comment from our sales team is that our pricing tiers are confusing.
78% of paid/campaign users bounce from the landing page.
88% of users opt-out of receiving our promotional marketing messaging upon signing up.
1. Users are confused by our pricing tiers.
2. Users don’t understand our value proposition.
3. Users don’t trust our marketing.
Validate Problems with Data Points

18. 1. Poor: Customers want to be able to pay bills using their smartphone
camera.
2. Good: All mobile banking customers want the convenience of easily paying
bills on their mobile device but are required to enter payee details for every
single transaction.
3. Great! All mobile banking customers want the convenience of easily paying
bills on their mobile device but are required to enter payee details for every
single transaction. We see a 45% drop off on the first form field entry and
an average time on page that is 70% site average.
Problem Statements from Poor to Great

19. 1. Poor: Customers want to be able to pay bills using their smartphone
camera.
2. Good: All mobile banking customers want the convenience of easily paying
bills on their mobile device but are required to enter payee details for every
single transaction.
3. Great! All mobile banking customers want the convenience of easily paying
bills on their mobile device but are required to enter payee details for every
single transaction. We see a 45% drop off on the first form field entry and
an average time on page that is 70% site average.
Problem Statements from Poor to Great

20. 1. Poor: Customers want to be able to pay bills using their smartphone
camera.
2. Good: All mobile banking customers want the convenience of easily paying
bills on their mobile device but are required to enter payee details for every
single transaction.
3. Great! All mobile banking customers want the convenience of easily paying
bills on their mobile device but are required to enter payee details for every
single transaction. We see a 45% drop off on the first form field entry and
an average time on page that is 70% site average.
Problem Statements from Poor to Great

21. Audiences
Areas
KPIs
Goal
Levers
Experiments
What problem are we trying to solve?
How will we know if we’ve solved it?
Where is the problem?
Who does it impact and how?
What solutions could exist?
Which is the best solution?
Experimentation is a Problem-Solving Framework

22. Problem-Solution-Result Framework

27. 32
Discovery vs Delivery

28. 33
Discovery vs Delivery

29. Discovery MVP Build
Feature
Refinement
Experiment
Launched
MVP
Go-Live
Decision
MVP
Feature
Definition
Experiment
Definition
Incremental
Feature
Definition
Discovery to Optimization

30. 1. Ensure your hypotheses solve for
defined and validated user problems
2. Identify the primary, secondary, and
monitoring metrics that will give you the
full story of how your experiment
impacts user behavior
3. Determine where experimentation can be
leveraged in both discovery and
optimization/refinement
Key Takeaways