Why Some Companies Are More Innovative Than Others

Copyright © 2018 by Stefan Thomke
Stefan Thomke
Professor of Business
Administration
Harvard Business School
Ashish
Chopra
Director, Product Marketing
Optimizely
Business Experimentation:
The Engine of Innovation

Housekeeping •  We are recording this webinar
•  You’ll receive the recording and slides
•  We’ll answer all questions at the end

Stefan Thomke
William Barclay Harding Professor of Business Administration

Copyright © 2018 by Stefan Thomke 4
Business Experimentation:
The Engine of Innovation
Stefan Thomke
William Barclay Harding Professor of Business Administration

What Is Innovation?
33,528 Number of times “innovation” was mentioned
in quarterly & annual reports (2011)*
255 Books published in last 90 days with
“innovation” in title*
Source: * Wall Street Journal, May 23, 2012

What Is Innovation?
Innovation
(Novelty + Value)
•  Incremental (can be powerful)
•  Breakthrough
•  Disruptive
Products & Services
Channels & Markets
Processes
Technologies
Business Models
What is Innovation @ Your Organization?
Invention
(Patents)
6

Why And How Do Some Companies
Develop Great Innovations?
7

…And Others Do NOT?

…While Some Firms Have Surprising
Solutions To Customer Problems

Growing Through Innovation:
Management’s Dream
•  A steady and predictable stream of breakthrough
products, services, and businesses.
•  High margins and growth due to first-mover
advantages.
•  High speed and productivity.
•  Investments in disruptive technologies address
new markets and customer needs.
•  …

Why Is Innovation So Difficult?

Growing Through Innovation:
Management’s Reality
•  Management faces strong incentives to focus on
short term.
•  Innovation activities can be highly variable and
difficult to plan and predict.
•  Customer behavior is very difficult to predict.
•  Organizational incentives encourage discourage
risk taking, experimentation and change.
•  Core Challenge: the ability to manage uncertainty
is fundamental to innovation.

Why Business Experimentation
Drives Innovation
Source of uncertainty How can uncertainty
be resolved?
•  R&D
•  Does “it” work as intended?
•  Production/Scale-Up
•  Can “it” be effectively produced?
•  Customer Needs
•  Does “it” address actual needs?
•  Market/Business
•  Does the opportunity justify the
(resource) investment?
1. Experience?

How can uncertainty
be resolved?
•  R&D
1. Experience
2. Data?
Source of uncertainty
Drives Innovation

But What Can We Learn From Data?
More Novelty = Less Data
(Is It Really Novel If Someone Has Done It Before?)
Context Matters
(Does Experience Transfer to Other Contexts?)
Correlation ≠ Causation
(Does Management Action Lead to Desired Outcome?)

Famous Correlations:
Don’t Trust Observational Studies
Palm size correlates with life expectancy
Ice cream sales correlates with drowning deaths
More cleaning by men correlates with shorter lives

Famous Correlations:
Don’t Trust Observational Studies
Palm size correlates with life expectancy
(Common cause: women have smaller palms and live longer)
Ice cream sales correlates with drowning deaths
(Common cause: warmer temperatures)
More cleaning by men correlates with shorter lives
(Common cause: unknown, many factors possible)

Drives Innovation
Source of uncertainty How can uncertainty
be resolved?
•  R&D
1. Experience
2. Data
3. Experiments

Recipes for Innovation:
Past and Present
“ The real measure of success is the number of experiments
that can be crowded into twenty-four hours. ”
– Thomas Alva Edison, Inventor
“ Our success at Amazon is a function of how many
experiments we do per year, per month, per week, per
day...”
– Jeff Bezos, CEO, Amazon
“ We try things … we celebrate our failures. This is a company
where it’s absolutely okay to try something that’s very hard,
have it not be successful, and take the learning from that. ”
– Eric Schmidt, ex-CEO, Google

The Digital Opportunity:
Competing Through Online Experiments
20

Experimentation Organization Needed:
How Microsoft (And Others) Do It
Key Data at Bing
•  > 15,000 online
experiments per year.
•  Experiments run on
millions of users (billions
of variations).
•  Only 10-20% of
experiments generate
positive results.
•  80% of proposed changes
are tested as controlled
experiments.
Source: The Surprising Power of Online Experiments (Harvard Business Review, 2017).
21

Scale and Velocity Pays
0
100
200
300
400
500
600
2013 2014 2015 2016 2017
01/01/2013 = 100
Experimenters S&P 500
Source: Bloomberg (01/02/2018)
Experimenters: Amazon, ETSY, Facebook, Google, Microsoft, Netflix, Priceline (Booking.com)
22

Advice 1: Scale and Velocity
Could You Run >10x Experiments?
23
•  Capacity: How many experiments can you run
simultaneously (infrastructure, user population)?
•  Creativity: Can you generate enough hypotheses
that are worth testing?
•  Communication: Can you manage large test
programs and absorb the results? Do they really
impact decision-making?

Experimentation Capacity
Infrastructure Should Not Be the Bottleneck
when people are not working 100% of the time—and
therefore, a busy development organization will be
faster and more efficient than one that is not as good
at utilizing its people.
But in practice that logic doesn’t hold up. We
have seen that projects’ speed, efficiency, and out-
put quality inevitably decrease when managers com-
take 5% more time to complete.
Processes with high variability behave very dif-
ferently. As utilization increases, delays lengthen
dramatically. (See the exhibit “High Utilization
Leads to Delays.”) Add 5% more work, and complet-
ing it may take 100% longer. But few people under-
stand this effect. In our experience with hundreds
High Utilization
Leads to Delays
The curve below is calculated using Queuing
Theory, the mathematical study of waiting
lines. It shows that with variable processes,
the amount of time projects spend on hold,
waiting to be worked on, rises steeply as
utilization of resources increases. Though
the curve changes slightly depending on the
project work, it always turns sharply upward
as utilization nears 100%.
30x
0
RESOURCE UTILIZATION
WAITINGTIME
Waiting times more
than double as
utilization moves
from 80% to 90%
and double again as
it moves from 90%
to 95%.
100%20% 40% 60% 80%
1802 Harvard Business Review May 2012From: “Six Myths of Product Development”, Harvard Business Review, May 2012. 24

Actions That Can Increase Velocity
•  Invest in capacity: non-linear returns (speed)
•  Align objectives: focus on response time
•  No idle time: make “deviations” easier to see
•  Expectations: Create strategic slack (3M, Google)
25

Advice 2: Culture
Create Learning Organization
•  Learning from failure is central to innovation.
•  Failure ≠ mistakes; mistakes produce no new or
useful information and are without value.
•  Not winning ≠ losing; failed experiments are the
source of new hypotheses and iterations.
•  Biases, incentives, and governance can slow down
experimentation.
26

Advice 3: Quality
Pay Attention to Discipline and Rigor
•  Invest in testable hypotheses that have well-
defined, measureable variables
•  Strong: “Opening stores one hour later to reduce
operating costs will not lead to a significant drop in sales”
•  Weak: “We can extend our brand upmarket”
•  Build trust in the data
•  Safeguards: A/A tests, outlier detection, simplicity, etc.
•  Understand the statistics
•  Follow Twyman’s law: “Any figure that looks interesting or
different is usually wrong”
27

Disciplined Business Experiments:
Some Essential Questions
•  Does the experiment have a clear purpose?
•  Specific management decision, clear learning objective.
•  Will the stakeholders abide by the results?
•  Agreed changes/actions, results aren’t ignored.
•  Is the experiment doable?
•  Testable prediction, required sample size, minimum duration.
•  How can we ensure reliable results?
•  Control group, systemic biases, randomization.
•  Have we maximized the experiment’s value?
•  Targeted rollout, components with highest ROI, causality.

Management Decisions by Experiment
“It doesn’t matter how beautiful your
theory is, it doesn’t matter how smart
you are. If it doesn’t agree with
experiment(s), it’s wrong.”
Richard Feynman
Scientist, Teacher, Storyteller, Musician
and Nobel Laureate (1918-1988)

To Learn More
Omnichannel Digital

optimize.ly/opticon18
Promo code: THOMKE550

Questions?

Why Some Companies Are More Innovative Than Others

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Why Some Companies Are More Innovative Than Others

Similar to Why Some Companies Are More Innovative Than Others (20)

More from Optimizely

More from Optimizely (20)

Recently uploaded

Recently uploaded (20)

Why Some Companies Are More Innovative Than Others