To develop winning customer experiences - ones that resonate with your customers and help drive your business forward - you need to build experimentation into everything you do. But managing an experimentation program at scale can bring its own challenges. How do you get executive buy-in? Get more teams and operationalize your processes? Effectively share learnings and insights across your organization? Join Optimizely’s Senior Vice President of Product Management, Claire Vo, and Experimentation Program Manager, Rebecca Bruggman to learn best practices for running an experimentation program at scale from the team running Optimizely’s program. During this webinar, learn how to: - Get executive buy-in for your experimentation program - Manage the execution of your program across all your teams - Increase the velocity and effectiveness of your program by utilizing the right people, processes and technology

1. 1
Optimizely Under the Hood
Managing
Experimentation at
Scale
Claire Vo & Becca Bruggman
December 5, 2018

2. Housekeeping
● This session is being recorded
● You will receive a copy of the
recording and slides
● We’ll answer questions at the
end of the presentation

3. 3
Presenters
Becca Bruggman
Technical Program
Manager,
Experimentation
@bexcitement
Claire Vo
SVP, Product
@clairevo

4. 4
Agenda
1. Executive Sponsorship
2. Program Management
3. Measure & Share
4. Takeaways & Questions

5. Digital Experience Optimization:
Digital Products, commerce & campaigns
Up to 5X Increase in Yield:
Revenue, share of wallet, funnel conversion,
risk mitigation, ops efficiency
Partner of Choice:
Work with leading global enterprises & “digital
disruptors” including 26 of F100
OUR COMPANY
Digital Experimentation Platform:
Next gen “Test and Learn” system
Replaces Digital Guesswork:
with evidence-based optimization
Speeds Innovation & Optimization:
Single platform for marketing & product teams
Best-in-class stats & machine learning
Consumer-grade usability
Enterprise program management & prof. services
OUR SOLUTION

6. “Our success is a function
of how many experiments
we do per year, per month,
per week, per day.”
“Instead of saying ‘I have an
idea,’ what if you said
‘I have a new hypothesis,
let’s go test it.’”
“Our company culture
encourages
experimentation and
free flow of ideas.”
“One of the things I’m
most proud of, and I think
what is the key to our
success, is this testing
framework we’ve built.”
Experimentation is the
Next Great Business
Transformation
Jeff Bezos Larry Page
Mark Zuckerberg
Satya Nadella
The Surprising
Power of Online
Experiments

7. 7
Executive Sponsorship

8. 8
What is an executive sponsor?
Measure & ReportPrioritize & Focus Enable & Empower
Experimentation Teams
Executive Leaders

9. 9
Who should be your exec sponsor?
Title
Business
Goals
VP Marketing /
Marketing Analytics
Deliver better product experiences
Prioritize effectively
Increase testing velocity
Track ROI of investment in
experimentation
Ensure effective governance and
oversight
VP Product/ Engineering,
Head of UX
Develop a culture of
experimentation
Measure and improve results over
time and across channels
CEO / CMO / CPO

10. 10
Set Goals
Velocity
• Set basic experiments per
year / quarter / month
target
• Velocity goal should be
achievable
• At first, don’t focus too much
on which experiments but
the total number.
Center of Excellence
• Make list of improvements to
experimentation
infrastructure
• Set goals for putting in place
key roles and processes
• Measure enablement and
engagement by team

11. 11
Creating a Culture of
Experimentation

12. 12
Driving toward Goals
Provide Company-
Wide Reporting on
the Program
Assign Roles &
Responsibilities / Hold
Team Accountable
Know When to Push vs.
Pull on Scaling the
Program

13. 13
Program Management

14. 14
What is a Program Manager?

15. 15
Squads
Enterprise
Experiences
Data Platform Experimentation

16. 16
We Believe in ‘Drinking Our Own Champagne’
Photo by Tristan Gassert on Unsplash

17. 17
Keys to Running a Great Program
1. Clearly define your program goals
2. Empower your team
3. Get the whole company involved
4. Measure the performance of your program
5. Share experimentation learnings broadly

18. 18
Goal Setting

19. 19
Empowering: Strategy Sessions

20. 20
Empowering: Experiment Review

21. 21
Empowering: Program
Management in Experiment Review

22. 22
Empowering: Resources
link.optimizely.com/ideatehypolink.optimizely.com/optimethod

23. 23

24. 24
Involvement: Submit ideas in
Program Management

25. 25
Involvement: Commenting in
Program Management

26. 26
Involvement: Scoring in Program
Management

27. 27
Measure and Share

28. 28

29. 29
Measure: Program Management
Tags

30. 30
Measure: Physical Boards

31. 31
Visibility: Company Presentation

32. 32
Visibility: Encourage Experimenters
to Present their Work

33. 33
Visibility: Slack

34. 34
Visibility: Program Reporting

35. 35
Visibility: Experiment Review for
Results Sharing

36. 36
Putting it all together!

37. 37
Takeaways

38. 38

39. 39
Key takeaways
1. Executive Sponsors and Program Managers work hand-in-hand to
build a culture of experimentation
2. Empower everyone in your organization to get involved
3. Measure how your program impacts the business
4. Experiment with your experimentation program

40. 40
Questions?