Taking a narrow view of experimentation data and looking solely at basic conversion metrics like clicks, lead generation or purchases may actually be limiting your growth strategy. In this session, you’ll learn how to draw confident conclusions about your experiments and their revenue impact, and avoid common pitfalls. ClassPass optimization expert, Nina Bayatti will share her experience approaching every customer acquisition effort as a hypothesis to be tested and how even when presented with positive outcomes she takes a critical eye to the results. By leveraging down-funnel data to evaluate the full story, Nina is able to identify which experiments are truly serving her goals beyond a single conversion metric.

1. Congratulations to our
Outperform Award Winners!
Most Dramatic
Business Impact
Most Customer-
Obsessed Company
Culture
Most Transformative
Innovation
Most Inspiring Social
Impact

2. 2
 Welcome to Day 2 of Opticon!
 Sessions will be recorded and available after Opticon
 Share your feedback and rate sessions on the mobile app

3. Nina Bayatti
Conversion Marketing Manager, ClassPass
Driving Customer Acquisition &
Growth with Experimentation

4. 4
About me

5. 5
Agenda
 ClassPass’ iterative, minimum viable product (MVP)
approach to testing
 Demystifying conversion rate optimization (CRO)
 Applying CRO principles company-wide
 Q&A

6. 6
ClassPass is the
largest studio
fitness aggregator

7. 7
We’re not just testing
new fitness classes.
In 2018, we ran over
150 optimization tests
company-wide.

8. 8
How did we run 150
optimization tests
company-wide?

9. 9
Build minimum viable
product (MVP)
• How can we test this
idea in a low-lift, high-
impact way?
• De-risk resource
investment
• Maintain high test
velocity
Boost test velocity & de-risk resources with an
iterative, MVP approach

10. 10
Build minimum viable
product (MVP)
• How can we test this
idea in a low-lift, high-
impact way?
• De-risk resource
investment
• Maintain high test
velocity
Launch & monitor
• Ask CX team to flag
test-related outreach
• Monitor web behavior
& analytics
Boost test velocity & de-risk resources with an
iterative, MVP approach

11. 11
Build minimum viable
product (MVP)
• How can we test this
idea in a low-lift, high-
impact way?
• De-risk resource
investment
• Maintain high test
velocity
Launch & monitor
• Ask CX team to flag
test-related outreach
• Monitor web behavior
& analytics
Iterate &
re-launch
• Anticipate 2-3
iterations ahead
• Continue to utilize
MVP-first approach
• Utilize initial test data,
customer feedback &
web analytics
Boost test velocity & de-risk resources with an
iterative, MVP approach

12. 12
Build minimum viable
product (MVP)
• How can we test this
idea in a low-lift, high-
impact way?
• De-risk resource
investment
• Maintain high test
velocity
Launch & monitor
• Ask CX team to flag
test-related outreach
• Monitor web behavior
& analytics
Iterate &
re-launch
• Anticipate 2-3
iterations ahead
• Continue to utilize
MVP-first approach
• Utilize initial test data,
customer feedback &
web analytics
Productize
• Share learnings
• Consider impact
when allocating
resourcing for
productization build
Boost test velocity & de-risk resources with an
iterative, MVP approach

13. 13
Demystifying CRO: Beyond the CTA color test

14. 14
Understanding down-
funnel impact of your
CRO initiatives

15. 15
• Bounce rate
• Click through rate
• Pageviews
• Lead captures
• Purchase conversion rate
Your primary KPI is not
telling the full story

16. 16
Reach a confident
conclusion with down-
funnel data
• Quality of leads
• Duration of sales cycle
• Customer acquisition cost
• Operational cost
• Customer retention rate
• Bounce rate
• Click through rate
• Pageviews
• Lead captures
• Purchase conversion rate
Your primary KPI is not
telling the full story

17. 17
Measuring the down-
funnel impact of our
re-engagement
optimization

18. 18
We tested extending our
winback offer
• Winback offer serves as a re-trial period for
inactive members
• Inactive members can experience new
product features & newly added classes
• Tested extending winback offer period from
1 to 2 weeks

19. 19
We tested extending our
winback offer
Hypothesis: If we extend our winback offer from 1 to 2
weeks, then our reactivation rates will increase
because members will have more time to experience
new features and classes. And the lift in reactivations
will offset the costs of the increased duration of the
free offer.

20. 20
Extending our winback offer lifted our
reactivation rate by 25%
Reactivations +25%
ReactivationCVR
B: Extended OfferA: Control

21. 21
Despite the lift in reactivations, our marginal
acquisition was not cost-effective
Control Extended Offer Delta
Reactivations* 100 125 +25% CVR
Cost of Goods (COGs) $15,000 $24,375
+30%
(increased usage)
Retained Members 80 95 -5% CVR
Customer Acquisition Cost $187.50
Marginal CAC $625 > ⅓ Lifetime Value (LTV)
*Reactivation counts are illustrative to better highlight the impact of the purchase CVR, retention CVR and increased cost of
the offer.

22. 22
Identify & iterate down-funnel levers
How can we maintain the 25%
reactivation CVR lift while . . .
• reducing COGs incurred
• maintaining retention rate
so that we can ultimately reach a cost-
effectively increase acquisitions?
Delta
Reactivations* +25% CVR
Cost of Goods (COGs)
+30%
(increased usage)
Retained Members -5% CVR
Customer Acquisition Cost
Marginal CAC $625 > ⅓ LTV

23. 23
Marketing
• Find optimal channels of acquisition
• Personalize experiences across touchpoints
• Optimize web to help educate, inspire & convert
Finance
Apply CRO principles company-wide

24. 24
Marketing
• Find optimal channels of acquisition
• Personalize experiences across touchpoints
• Optimize web to help educate, inspire & convert
Customer Experience
• Find methods of reducing customer confusion
• Determine optimal methods of communication with customers
Apply CRO principles company-wide

25. 25
Marketing
• Find optimal channels of acquisition
• Personalize experiences across touchpoints
• Optimize web to help educate, inspire & convert
Customer Experience
• Find methods of reducing customer confusion
• Determine optimal methods of communication with customers
• Merchandise effectively
• Develop optimal subscription plans
• Understand acquisition curve
Finance
Apply CRO principles company-wide

26. 26
Marketing
• Find optimal channels of acquisition
• Personalize experiences across touchpoints
• Optimize web to help educate, inspire & convert
Customer Experience
• Find methods of reducing customer confusion
• Determine optimal methods of communication with customers
• Merchandise effectively
• Develop optimal subscription plans
• Understand acquisition curve
Product
• De-risk negative impact of new product feature
• Determine acquisition gains
Finance
Apply CRO principles company-wide

27. Q&A

28. Thank you!

29.  Head to the Expo Hall for morning refreshments
 Up next…Ravi Parikh, co-founder of Heap at 11:15am!