Here’s the playbook Intuit uses to increase its experimentation velocity — even when they face traffic limitations. Mike Loveridge is not new to running experimentation teams. Before Intuit, he built out programs at Ancestry.com, GE, Humana, and CheapOair. He's an expert at making experimentation work at high velocity, even in traffic-challenged situations. In this webinar, Mike Loveridge shared his best practices for making CRO work at high velocity, key lessons from scaling multiple teams, and why he's bullish on the future of "test and learn".

1. FIRST CALL 1

2. OPTIMIZELY.COM/DIGITAL-CHANGE-AGENTS

3. How to Scale Your
Experimentation Program
WEBINAR
Mike Loveridge
Intuit
The playbook Intuit uses to increase its experimentation
velocity — even when they face traffic limitations.
Wes McCabe
Optimizely

4. Agenda
! Introduction
! What is Optimizely?
! Lessons from Intuit
! Q&A

5. Digital Experience Optimization:
Digital Products, Commerce & Campaigns
Up to 5X Increase in Yield:
Revenue, Share of wallet, Funnel conversion,
Risk mitigation, Ops efficiency
Partner of Choice:
Work with leading global enterprises & “digital
disruptors” including 26 of F100
OUR COMPANY
Digital Experimentation Platform:
Next gen “Test and Learn” System
Replaces Digital Guesswork:
with Evidence-based optimization
Speeds Innovation & Optimization:
Single platform for marketing & product teams
Best in class stats & machine learning
Consumer grade usability
Enterprise program management & prof. services
OUR SOLUTION

6. “Our success is a function
of how many experiments
we do per year, per month,
per week, per day.”
“Instead of saying ‘I have an
idea,’ what if you said
‘I have a new hypothesis,
let’s go test it.’
“Our company culture
encourages
experimentation and
free flow of ideas.”
“One of the things I’m
most proud of, and I think
what is the key to our
success, is this testing
framework we’ve built.”
Experimentation is the
Next Great Business
Transformation
Jeff Bezos Larry Page
Mark Zuckerberg Satya Nadella
The Surprising
Power of Online
Experiments

7. 10x more experiments
Consumer-grade usability
Open data integration
Maximum yield of business value
UX and feature-level experiments
and personalization at every digital
touch point
Enterprise-wide
management & governance
Captures, governs and shares
ideation, analyses & results
Optimizely X Unlocks the Experimentation Best Practices
of the World’s Greatest Digital Companies
World’s most trusted outcomes
BestinclassStatsEngine
FasttimetoresultsviaML
Accelerates digital innovation
Speeds devops&deployment
De-riskscontinuous feature delivery
Ensuressuccessofnewfeatures
Unifying flagging & experiments
enablescontrolledtestingof new
features while maintaining high
performance
Ideate
Manage
StoreGovern
Analyze
Share
Open Data
Integration
Security &
compliance
Stats
Engine
Stats
Accelerator
Consumer-
grade
usability
APIs &
Developer
Tools
Feature
Flags
Open source
SDKs
X-Channel
Full Stack
Experimentation
Personali-
zation
Recommen-
dations
Web
Experimentation
PRODUCTS
COMMERCE
CAMPAIGNS

8. We’re Proud to Work With
Great Global Enterprises
21 of the Fortune 100 have Chosen Optimizely to Drive Their Digital Experience

9. B U S I N E S S V A L U E
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
Our Products and Services Take
You on Your Experimentation Journey

10. ROI
CHALLENGE
NEXT STEPS
MATURITY
BENCHMARK
optimizely.com/maturity-model

11. Mike Loveridge
How to Scale Your
Experimentation Program

12. See the speaker notes for
instructions on replacing this
photo

13. High velocity
Traffic-challenged situations
Scaling multiple teams
The future of “test and learn”
Agenda

14. High velocity
Traffic-challenged situations
Scaling multiple teams
The future of “test and learn”
Agenda

15. If everything seems
under control, you
just aren’t going fast
enough.
Mario Andretti

16. 1. More learnings
• More tests*
• Testing in more places
• Testing through additional mediums (usability, surveys, etc.)
• Efficient designs (MVT)
2. Reduced time to launch = cost savings
3. Timely analysis & recommendations = quicker time to market
4. Quicker implementation of winners = cost savings
*Without sacrificing quality
How can velocity be defined?

17. 1. Bottom line impact
2. Number of tests
3. Development time by t-shirt size (S, M, L, XL)
4. Analysis & recommendation time
5. Implementation time
6. Error rate
7. Win rate
8. Number of learnings
9. Test quality score
Supporting KPIs

18. “If one day the speed kills me, do not cry
because I was smiling.”
Paul Walker (1973 – 2013)

19. VELOCITY PRINCIPLE Fully
utilize traffic with ‘always
on’ testing

20. HOW?
1. Calculate maximum tests by
page or channel
2. Run back-to-back tests. Never
have a gap.

22. # PAGE WEEKS TESTS/MO URL
1 Homepage 1 4 /
2 Homepage 1 4 /
3 Signup Complete 2 2 /signup-complete.php/business
4 Signup 2 2 /signup?type=business&s=mobile_app
5 PPC (twopart) 4 1 /pages/timesheets-twopart
6 PPC (time tracker) 4 1 /pages/time-tracking-app-twopart
7 Quickbooks 4 1 /app-marketplace/quickbooks-time-tracker
8 Pricing 4 1 /online-time-clock-software-pricing
9 App 8 .5 /mobile-time-tracking
10 Other Pages (7) 8 3.5 Various
Velocity Calculations – Front End Pages

23. # CHANNEL TESTS/MO
1 Front End 20
2 Mobile Web 10
3 Mobile App 2
4 PPC 20
5 Email 8
6 Product 10
7 Surveys 2
8 Usability 4
Velocity Calculations – All Channels

24. VELOCITY PRINCIPLE
Iterate; “fast-follow”
testing.

25. Value prop messages were tested in the banner text on the homepage

26. VARIABLES
Length
Features
Benefits
Adjectives
Order
Awards

27. Everything you need to track & schedule employee time
Time tracking and scheduling that’s accurate to the second
Time tracking and scheduling that’s full-featured & easy to use
Time tracking that’s full-featured & easy to use
Baseline
+3.7%
+26.9%
+9.2%
Homepage Test Results

28. More Homepage Testing Results - AWARDS
● Control: Time tracking with over 8,000 five-star reviews
● T1: 250 million timesheets processed and counting
● T2: 246,897,755 timesheets processed and counting
● T3: Time tracking with 500,000 active users and counting
● T4: PC Magazine Editors’ Choice “Best Time Tracking Software”
● T5: G2 Crowd Winner for “Best Time Tracking Software”
● T6: TSheets, the highest-rated time tracking app on top review hubs

29. ● Control: Time tracking with over 8,000 five-star reviews
● T1: 250 million timesheets processed and counting (+5.5%)
● T2: 246,897,755 timesheets processed and counting
● T3: Time tracking with 500,000 active users and counting (+8%)
● T4: PC Magazine Editors’ Choice “Best Time Tracking Software”
● T5: G2 Crowd Winner for “Best Time Tracking Software”
● T6: TSheets, the highest-rated time tracking app on top review hubs (+6.4%)
More Homepage Testing Results - AWARDS

30. This is where we are at today

31. How much will your
velocity increase in a
year?

32. High velocity
Traffic-challenged situations
Scaling multiple teams
The future of “test and learn”
Agenda

33. Don’t be afraid to run tests when a lower sample size makes it difficult to
imagine getting statistical significance. You will still learn things!
We test to generate learnings.
We also test to obtain statistical significance.

35. • Usability
• Online surveys
• A/B tests with rich data sets
• Heatmaps, clickmaps, scrollmaps, confetti reports
• Video recordings
• Track all clickable items the page being tested
• Multiple KPIs… do they all align?
• Have a strong hypothesis
• Does the data you get back make sense in explaining the hypothesis?
• Why or why not?
• Alternative statistical models
• Patience. Just let it run… for a LONG time.
• Be conservative with mission-critical pages
• Test higher impact changes
Ways to experiment in low traffic situations

36. Homepage Pricing
Signup
Buy Now
Signup
Complete
Signup
Free Trial
Control
Sign up for a 2-week free trial. Then, buy it!
KPI 1: Free trial conversion rate (How many start a trial)
KPI 2: Activation rate (How many buy after the trial)
Treatment
Sign up for a 2-week free trial. Then, buy it! OR… Just buy it right now and get 50% off for the first 3 months.
KPI 1: Free trial conversion rate (How many start a trial)
KPI 2: Activation rate (How many buy right now or buy after the trial)
KPI 3: Revenue per user
KPI 4: Cancellation rate

37. Test Results (WEB.039.18 - Homepage Buy Now) TEST DATES: JULY 17 - AUG 1, 2018
LEARNINGS
● FREE TRIAL SIGNUPS DECREASED BY 30%
● WHEN MOBILE WAS REMOVED, THE DECREASE WAS 10%
● USERS ARE CHOOSING BUY NOW WITH 50/50 INTENT. SIGNIFICANT FALLOUT IN THE BILLING FUNNEL.
● LARGE FALLOUT ON THE PRICING PAGE, DESPITE HUGE GAINS IN ENGAGEMENT
● ACTIVATIONS WERE FLAT
Control Treatment
RESULTS
ACTIVATIONS WERE FLAT

38. Learnings
+106%
+100%
50/50

39. Learnings
Usability
Heatmaps
Videos
Clickstream data

40. Crazy Egg Data
Readability stalls at the first CTA but
low click-through

41. Trend Lines
Not this
Better!

42. Customer Service Surveys – Value Prop Statements
Survey 1 Survey 2
What is most important for great customer service?

43. High velocity
Traffic-challenged situations
Scaling multiple teams
The future of “test and learn”
Agenda

44. • Dedicated (“Center of Excellence” executes all tests)
• Specialists are efficient
• Head of Experimentation has more control
• Dispersed (Each product team does their own testing)
• Product owners OWN performance
• Hybrid (“Center of Excellence” teaches a man to fish)
Dedicated vs Dispersed Teams

45. • Repeatable, written process
• Executive sponsorship
• Having a PM is nice… especially for complex, cross functional tests
Key Elements for ANY Model

46. High velocity
Traffic-challenged situations
Scaling multiple teams
The future of “test and learn”
Agenda

47. Experiment = Explore

48. Never Stop Exploring