Culture of experimentation: A conversation with Microsoft

1
Chris Goward
CEO, Widerfunnel
July 23, 2020
Culture of experimentation:
how organizational culture can
make or break your testing program
WEBINAR
Beth Foster
Principal Program Manager, Experimentation,
Microsoft

2
© Copyright 2007-2020 Widerfunnel. All rights reserved.
Before we get started...
• You'll receive a recording of this webinar
within the next two days.
• Use the Q&A button to send your questions
to the presenters. We'll answer them at the
end of the presentation.

3
● 13+ years in industry
● Author of industry bestseller
You Should Test That!
● Creator of the LIFT®
Model, PIE
Framework®
, PACET®
, and Inﬁnity
Experimentation Process®
● Global keynote speaker at 300+ events
Chris Goward: Behavioral Design and
Experimentation Thought-Leader
Chris Goward
Founder & CEO, Widerfunnel

4

5
Experimentation
is a core strategy
for market leaders.
Experimentation Insight

6
© Copyright 2007-2020 Widerfunnel. All rights reserved. Download the full research report at: Widerfunnel.com/research
State of Experimentation Research
We've been studying the world's best
experimentation programs since 2007.

7
Growing the experimentation competency

8
5 Components of Experimentation Success: PACET®
The way people coordinate their eﬀorts
How the program is measured
Shared beliefs that support experimentation principles
Tools that enable customer insights and testing at scale
The right talent and experience capacity

9
Insight-driven businesses with experimentation at their core
grow 8x faster than GDP
Source: The Insight-Driven Business, Forrester
3.5%
27%
40%
8x GDP

10

11
Beth Foster
Principal Program Manager,
Experimentation

12
Executive
support for
experimentation
is critical.
Research Insight

13
80%
of fast growth executives see
the value or are fully bought
into experimentation.
Source: "Harnessing the Experimentation Mindset to Drive Growth, 2019, Widerfunnel.
Research Insight

14
What's worked at Microsoft to gain
leadership buy-in?
B

15
How do you mobilize that across the
organization?
B

16
How do you Inspire enthusiasm?
B

17
The fastest growth
companies are more
likely to report a strong
knowledge-sharing
culture.
What good are customer
insights if no one knows about
them?
Source: "Harnessing the Experimentation Mindset to
Drive Growth, 2019, Widerfunnel.

18
Shared customer insight database
Request beta access at www.Liftmap.com

19
Goal
alignment
inspires leadership
trust.
Research Insight

20
Align experiment and program metrics with company goals
Experiment
Metrics
Experimentation
Program
Metrics
Company or
Business Unit
Goals Compete with Amazon for
sportswear category
Launch 12 Experiments
60% insight target
Launch 4 community
focused experiments
Increased AOV
Increased Purchases
Increased Revenue

21
The spectrum of experimentation opportunities
Explore Exploit

22
Explore Exploit
OptimizationDisruption

23
Explore Exploit
How should we
do it best?
(Optimization)
What’s the next
new idea?
(Disruption)
Should we do it?
(Validation)

24
You also need
program evaluation
metrics
Experimentation Insight

25
Productivity metrics
Experiment Velocity - number of experiments launched / time period
Variant Velocity - number of variations launched / time period
Quality metrics
Actionable Rate - the proportion of experiments that produce actionable results
Hardcode Rate - the percentage of positive conclusive experiments that end up as deployed code
Impact metrics
Uncovered Revenue Ratio - total revenue lift / time period
Test Coverage - the percentage of visitors who are included in a test
Program evaluation criteria

26
Request beta access at www.Liftmap.com

27
What if you can't test everything?
B

28
Intake form and PIE prioritization

29
Taking action:
Improve your organization's
experimentation competency

30
The goal: scaling experimentation

31
Stage 1:
Expertise constraint
Goal:
Improve expertise
Stage 2:
Process constraint
Goal:
Improve process

32

33

34
Architecting insights.
Delivering growth. ™
Questions? Email: iwant@widerfunnel.com

35
The best companies
design an Overall
Evaluation Criterion
(OEC) with the
executive team
Download the full research report at: Widerfunnel.com/research2019
Slow growth companies do
not involve executives in
goal-setting.

36
What makes a great Overall Evaluation Criterion (OEC)?
It's a blended, weighted combination of metrics that aligns experiment
results to long-term business goals.
It should:
● Be indicative of long-term gain.
● Be diﬃcult to manipulate.
● Incentivize the right set of actions.
● Be sensitive to detect aﬀect.

37
How do you make experimentation core to
the business?
B

38
Fast growth
companies reported
the strongest culture
of ownership and
accountability.
Download the full research report at: Widerfunnel.com/research
An ownership culture means
more people with ﬁrst-hand
knowledge solve problems.

39
Inform
Communicate ongoing
insights and updates.
Inspire
Create awareness of new
possibilities.
Educate
Train and provide support and
supervised practice.

40

Culture of experimentation: A conversation with Microsoft

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Culture of experimentation: A conversation with Microsoft

Similar to Culture of experimentation: A conversation with Microsoft (20)

More from Chris Goward

More from Chris Goward (20)

Recently uploaded

Recently uploaded (20)

Culture of experimentation: A conversation with Microsoft