The document outlines 10 tactics for building a successful experimentation culture focused on data-driven decision making. Key tactics include establishing a clear organizational structure, identifying the right talent, engaging stakeholders, prioritizing tests, and sharing results. It emphasizes the importance of involving everyone in the testing process and balancing learning with achieving wins.

1. 10 Tactics for Building an Optimization Culture
OPTIMIZELY & BROOKS BELL
3.4.2014

2. Welcome!
Gregory Ng
Chief Marketing Officer
Brooks Bell
@GregoryNg
Kyle Rush
Head of Optimization
Optimizely
@kylerush

3. 10 Tactics for Developing a Successful
Experimentation Culture

4. What we’ll cover ...
1. Best practices from top testing
organizations
2. The people behind testing
3. How to organize your program
4. How to keep the process
manageable
5. How to include others and build
excitement

5. Successful experimentation culture
must be data-driven.

6. What does it mean to
be data-driven?
Progress is compelled by data, rather than by intuition or
personal experience.
Evidence-based decision making.
Using data to inform key decisions.

7. How data-driven are you, actually?
•
Do you dictate strategy without referencing data?
•
Do you rely on the person with the most experience to
make the ultimate call?
•
Do you use words like “think” and “feel” when talking
about strategy?
•
Does your team trust the data?

8. How do you establish a successful
testing culture?

9. 10 Tactics
1. Clear organizational structure
2. Identify the right talent
3. Establish stakeholders
4. Solicit test ideas from everyone
5. Prioritize tests
6. Share results and successes
7. Build a knowledge base
8. Don’t rush into MVT tests
9. Balance learning and “wins”
10.Go beyond CRO

10. #1 – Choose the right
organization structure
• No “one-size-fits-all”
• One or multi-person
team
• Centralized or
decentralized?
Members of the Obama for America Team

11. #1 – Choose the right
organization structure
• Army of One
• Ringleader
• Fellowship
• Kingdom

12. #2 – Pick the Right Talent
• Qualitative and quantitative
combination
• Curiosity, creativity
• People skills
• Analytical mindset
VP of Marketing
E-Commerce
Director
Program Manager,
Retail
Optimizely’s customers come from a
variety of backgrounds
Product Manager

13. #3 – Establish Stakeholders Early On
• Identify the correct
stakeholders
• Secure an executive sponsor
• Develop an approval process
for tests
• Communicate and share
successes

14. #4 – Solicit Test Ideas from
Everyone
• Make it easy to submit
ideas
• Capture interest after
sharing results
• Follow up for every idea
• Create a scoreboard
• Attribute experiments to
team members
Example: Optimizely Scorecard
Vote on predicted winning variations

15. #5 - Know how to Prioritize Tests
ESTABLISH A SCORECARD
•Relevance to business goals
(KPIs)
•Potential impact
•Potential for learning
•Level of effort
•Time to significance and
available traffic
•Likelihood of implementation

16. #6 – Share Results and
Successes
•
•
•
•
•
Make it easy to understand
Clarify the primary goal
Share 100 percent of
results, regardless of
outcome
Standing meetings
Keep score
Presenting A/B test findings at company all-hands

17. #7 – Build a Knowledge Base
• Gain broader insights
• Lay a foundation for growth
• Eliminate single point of failure
• Increase accountability
• Standardize processes and
documentation

18. #8 – Don’t Rush Into MVT Tests
• Ideal for fine-tuning
optimization
• More complicated but less
effective
• Fewer learnings

19. #9 – Balance Learning and “Wins”
•
•
•
•
Always start with a hypothesis
Never settle for a flat result
Invitation for deep analysis
Segment results, target
accordingly

20. #10 – Go Beyond CRO
Optimization doesn’t need to occur in isolation.
Almost anyone can test.
For instance ...

21. #10 – Go Beyond CRO
• Designers
Variation A
Variation B
Walk designers through A/B tests so they can inform their designs with data

22. #10 – Go Beyond CRO
• Engineers
Launch of OFA 2012 New Donation API
A/B testing the new platform revealed an
architectural bug that was missed in QA

23. #10 – Go Beyond CRO
• Email and Content Marketers
A/B Tested Emails and Landing Pages for Subscribers
#dataculture

24. 1. Clear organizational structure
2. Identify the right talent
3. Establish stakeholders
4. Solicit test ideas from everyone
In summary ...
5. Prioritize tests
6. Share results and successes
7. Build a knowledge base
8. Don’t rush into MVT tests
9. Balance learning and “wins”
10.Go beyond CRO

25. Thank you for reading!