This document provides an overview of how to build a successful experimentation program within an organization. It discusses the importance of experimentation for innovation and outlines key areas to assess when developing an experimentation capability, including organizational culture. It then describes attributes of mature experimentation cultures and how most organizations are currently at an early stage. The rest of the document focuses on operationalizing experimentation through identifying a team, following an experimentation methodology, understanding metrics, and getting started with small changes.

1. Building a Winning Experimentation
Team
December 2017
Managing Director, Australia & New Zealand
dan.ross@optimizely.com
/danross9
Dan Ross

2. By the end of today’s session, you will
have learned:
• Why experimentation is central to competing and innovating
• Areas to assess when building your experimentation capability
• How organisational culture helps scale an experimentation program

3. The world’s leading companies utilise
experimentation to build a culture that fosters
innovation and agility

5. Optimization Is A Journey

6. 2015 2017

8. “If you want to gain a competitive advantage, master the science of
conducting online tests. The returns you reap – in cost savings, new
revenue, and improved user experience – can be huge.”
“…experiment with everything” approach has
surprisingly large payoffs.”
“…rigorous online experiments should be standard
operating procedure.”

10. “Our success
at Amazon is a function of
how many experiments we do
per year, per month, per week, per day…”
J E F F B E Z O S

11. “Every decision is an experiment.”
M E . R I G H T N O W

12. Soon after Ron Johnson left Apple to become the CEO of J.C. Penney, in
2011, his team implemented a bold plan that eliminated coupons and
clearance racks, filled stores with branded boutiques, and used technology to
eliminate cashiers, cash registers, and checkout counters. Yet just 17
months after Johnson joined Penney, sales had plunged, losses had
soared, and Johnson had lost his job. The retailer then did an about-face.
Had J.C. Penney done thorough experiments on its CEO’s
proposed changes, the company might have discovered
that customers would probably reject them.
Source: Harvard Business Review,
“The Discipline of Business Experimentation” December 2014
EVERY CHANGE IS AN EXPERIMENT...CONTROLLED
OR NOT
Stefan Thomke ,
Harvard Business School

13. Fortune 1000
2012 2013 2014 2015 2016 2017
0
100
200
300
400
500
600
700
50
150
+56%

14. 100
200
300
400
500
600
700
Fortune 1000 Culture of Experimentation Index
50
150
2012 2013 2014 2015 2016 2017
0
+56%
+756
%

15. 1,000,000+
E X P E R I M E N T S

17. CULTURE OF EXPERIMENTATION

18. 0
10
100
1000
VELOCITY
MATURITY
10000

19. VELOCITY
EXPERIMENTATION
HERO
MATURITY
0
10
100
1000
10000

20. VELOCITY
EXPERIMENTATION
PROGRAM
EXPERIMENTATION
HERO
MATURITY
0
10
100
1000
10000

21. VELOCITY
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION
MATURITY
0
10
100
1000
10000

22. The key to experimentation is to have both the
right tools (software) in combination with the right
people and processes

23. Experimentation Maturity Model

24. So how do you bring this concept to your organisation?
The first step is to understand what we see as the
common attributes within these market leading
organisations.
These attributes when combined together represent a
“Culture of Experimentation”

25.  Experimentation is deeply
embedded into company
culture
 Dedicated, customised technology
conducts experiments and surfaces
insights
Attributes of
Mature “Culture of Experimentation”
Organisations
 Enfranchised teams work on
independent optimisation
roadmaps
 Central groups manage data
collection and analysis,
empowers individuals with
resources and process
 Strategy and metrics for digital
performance directly linked to
company success and widely
circulated
 Collective expertise creates virtuous
cycle of learning and best practices

26. These Culture of Experimentation Organisations
would be considered “best-in-class” globally.
So if that is what the end state of building an
experimentation program looks like, where do most
organisations stand today?
Most organisations would be considered
Experimentation Hero Organisations

27.  Goals and processes poorly
formalised, lack executive
sponsorship
 Experimentation are focused
marketing and sales channels
only versus every business
department
Attributes of
Early Stage “Experimentation Hero”
Organisations
 Consumer intentions and digital
behavior poorly understood
 Experimentation treated as series of
tactics to ‘improve conversion rate’
 Skill sets and tools divided across
functional silos or not available at all
 Lone individuals or small teams
working through design, production,
and analysis despite limited experience

28. The vast majority of organisations fall somewhere
between Mature “Cultures of Experimentation”
and Early Stage “Experimentation Hero”
Organisations
…but most exemplify more criteria from the 2nd set of
attributes

29. Mature Cultures of
Experimentation
Early Stage
Experimentation
Organisations
 Embedded into company
culture
 Enfranchised teams
 Empowered individuals
 Dedicated technology
conducts experiments
 Metrics linked to company
success
 Collective expertise
 Lack executive sponsorship
 Customer digital behaviors
poorly understood
 Experimentation seen only as
Conversion Rate optimisation
 Focused solely on sales and
marketing funnels
 Tools and skilled people in
silos
 Limited expertise available

30. This is the called the Alignment Gap
The gap between these two sets of attributes
represents the key opportunity to create and sustain
a competitive advantage in experimentation
Mature Cultures of
Experimentation
Early Stage
Experimentation
Organisations

31. This gap has numerous root causes for why an
organisation may be on one side or the other including
business model, organisational structure, technology
investments, and available trained staff.
Yet, there are measures that any organisation can take
to efficiently leapfrog these hurdles to the front of their
competitive group.

32. So how do you bridge The Alignment Gap?

33. To bridge the Alignment Gap your
organisation must operationalise
your experimentation program

34. How
Operationalising
Experimentation
Identify Team and Support Their Growth
Embrace Experimentation Methodology
Understand Operational Metrics
Understand Success Factors
Getting Started Today

35. Identify Your Team

36. CORE EXPERIMENTATION TEAM
SKILLSETS & TEAM ROLE
Executive Sponsor Project Manager Technical Lead Developer Content

37. Invest in your most valuable assets:
Identify a Team and Support Professional Growth
Certify Measure Socialise

38. You can also jumpstart your
experimentation program or enhance
your existing team’s capabilities by
utilising a best-in-class partner

39. Once you have a team in place we
move to the next key concept

40. The practice of experimentation follows a
defined set of repeatable actions called the
Experimentation Methodology

41. Ideation
Generate ideas for experiments
Experimentation Methodology

42. Planning
Design
individual
experiments
and
campaigns
Experimentation Methodology

43. Developmen
t
Build your experiment and launch it
Experimentation Methodology

44. Analysis Evaluate the results
of experiment
Experimentation Methodology

45. Experimentation Methodology
Analysis
Developmen
t
Planning
Ideation

46. The Experimentation Methodology is universal
framework for any sized organisation, but you have
to map it for your business.
To learn more about mapping the Experimentation
Methodology please visit the Optimizely Knowledge
Base for Optimisation Methodology: Running your
Optimisation Program

47. Experimentation Maturity Model

48. Most team’s don’t effectively measure their
of use of the Experimentation
Methodology.
This is a missed opportunity.
Structured measurement gives you
leading indicators of performance called
Operational Metrics.

49. These Operational Metrics are the
Success Factors for having a world-class
experimentation program

50. Experimentation Success
Factors
VelocityAgility
Efficiency
Quality
Throughpu
t
Operational Metrics
Impact Rate
Drag
Iteration=
=
=
=

51. Exp. Methodology Planning Development Ideation Analysis
Operational
Metrics
Throughput Drag Impact Rate Iteration
Why it’s
Important
Large percentage of
experiments don’t win.
Improving the quantity of
experiments will ultimately
generate more success.
Knowing how long it takes to
both build an experiment and
how long it takes for each
stage of the experimentation
methodology is important for
measuring overall program
improvement.
To give your team the best
chance of creating a
winning experiment you
need to gather insightful
data to inform hypothesis
generation.
The process of aggregating,
interpreting, and sharing the
experiment results within the
organisation to inform the
next experiment.
Description
(What You Measure)
1.) Overall number of
experiments per property
2.) Number of experiments
per month and week for
each property.
1.) Number of hours spent
redeveloping experiments due
to QA or troubleshooting
issues compared to
‘productive’ time.
2.) Length in hours and days
of the entire experiment
production cycle.
1.) Percentage of
experiments generating
meaningful result (with a
statistically significant
winner or loser).
1.) Percentage of
experiments put into
production and iterated upon.
2.) Number of days from
current experiment to iterated
upon experiment.
Success Factors Velocity Efficiency Quality Agility
Description Quantity of experiments
being ran.
Rate at which tests move
through both Development
stage and the overall
Experimentation Methodology.
Average likelihood that a
given experiment will
produce business impact
(either positive or negative)
The quantity and speed the
experimentation program acts
on experiment results and
iterates upon them.

52. It is critical to create a habit of optimizing your
experimentation program toward these
Operational Metrics to ensure you are
continually measuring and improving on your
Success Factors.

53. Now that we know the Experimentation
Methodology, Operational Metrics, and
Success Factors how do we get started
today?

54. The most successful experimentation programs result in business
transformation. Get on this journey today!
Executives
 Prioritise and evangelise for experimentation within your business
 Identify your team and invest in their success by giving them the time
availability, resources, and goals for the overall experimentation program that
align with business objectives
Executives and Practitioners
 Begin mapping the Experimentation Methodology to your organisation
Practitioners
 Learn the best practices recommended by Optimizely for starting your
experimentation program
Getting Started Today

55. Thank You
Dan Ross
Optimizely ANZ
dan.ross@optimizely.com
/danross9