B5: The 8 Deadly Maladies of Rapid Experimentation, Lukas Oberhuber

Who are we?
Simply Business is an online insurance broker
with more than 600,000 micro-business
customers, covering over 1,000 types of
business
Our disruption? Serving customers forgotten by
time-pressed insurance agents
© Simply Business 2019

Small business
Big impact
SMEs account for 99% of
all UK and US businesses
£1.9tn annual revenue
in the UK
$8.3tn annual revenue
in the US

Award-winning culture © Simply Business 2019

Exited to Travelers Insurance
(2017)

● Group CTO @SB for last 9 years (+Agile, +CI/CD, +++)
● Lean Startup(like) techniques for well more than a decade
● Current heavy focus on Architecture, Innovation and Strategy
(and where those things block us)
Who am I?

What is this talk about?
➔ disorders I’ve healed in SB
➔ teams often suffer from more than one
➔ I have cures, but no vaccines
This is going to be sick!

What will I help you with?
Using the
maladies to
identify when
a team/org
losing its way
Give you
a language
for the
challenges

It is possible to fail
in many ways...while
to succeed is
possible
only in one way.
Aristotle
‘‘
’’

But first, the roadmap to a malady
1. Patients: one or more examples
2. Symptoms: what to watch out for
3. Diagnosis: naming the malady
4. Consequences: what happens when symptoms go unchecked
5. Root causes: the emotional angle
6. Coaching: questions to ask teams
7. Cure

PATIENT
CASE01
It did
But PayPal is more expensive than current provider
We would never switch just because people would use it
.
Tested to see if PayPal would get click-throughs

❏ Everything must be statistically significant
❏ Doing experiments without success criteria or way forward✓
✓
❏ Choosing tests based familiarity with the technique
CASE01
✓

CASE01

➔ Time wasted on experiments which don’t matter
➔ Experimenting because we should
➔ Lack of accountability
➔ While A/B tests are critical for incremental changes, they can’t
be the only tool in the arsenal
A/BTESTOSTROPHY

➔ Fear (of damaging something) / lack of courage
A/BTESTOSTROPHY
➔ Doing “what has been asked”
➔ No deep understanding of goals
➔ Intolerance of failure
.

A/BTESTOSTROPHY
Q: Is this valuable?
Q: Are we moving fast enough?
Q: What does success
look like?
.

➔ Decide what you need to learn, then design experiment
A/BTESTOSTROPHY
➔ Teams accountable for results, not experiments
➔ Customer behaviour before treatment
➔ Don’t test if cost to test more than likely worth
.

Fixed numerous small niggles in back office for call center
PATIENT
CASE02
To prove value of product team
Progress would be next quarter
In the end, team’s performance was questioned
.

❏ Continuous quick wins to prove we are worth it✓
❏ Marginal metric improvements✓
❏ Big change is coming (but not yet)✓
CASE02

CASE02

Low results
LOW-HANGING-FRUIT-ARRHEA

➔ No vision
➔ Fear of not proving value
➔ Expectation of immediate results
.

Q: Are you making progress?
Q: What is it?
Q: What does success look like?
.

➔ Start with a valuable goal and don’t be distracted
. © Simply Business 2019

CASE03

PATIENT
We A/B tested an improved quote
comparison page (toggling)
CASE03

PATIENT
We A/B tested an improved quote
comparison page (toggling)
Then did not act on positive results
CASE03
Because people questioned: “are we sure?”

❏ Few ideas on how to move forward✓
❏ “Nothing different will happen so don’t try”✓
❏ Having to prove every step, an inch at a time. Results not accepted.
Like a court case against a powerful person
✓
❏ Major questions unresolved in spite of much activity✓
CASE03

➔ Business questions do not move forward,
or at slow pace
➔ Business failure
CAGED-INSYNDROME

➔ Don’t want to look bad (in one’s field of expertise)
CAGED-INSYNDROME
➔ Lack of understanding of what change is needed
➔ Listening to others, but no point of view
➔ External pressure
➔ Anchored in current world, rather than desired future
.

➔ What are your riskiest assumptions?
CAGED-INSYNDROME
➔ What happens if you are wrong about
the way things work?
.

➔ Decide / negotiate ahead of time the burden of proof.
Then meet that proof
CAGED-INSYNDROME
➔ Test the riskiest assumptions, regardless of sensitivities
➔ Once proven, questions/learnings are not permitted to be
reopened
➔ “You can question the methodology, but you can’t question
the outcome”
.

CASE03

PATIENT A
Needed to speak to insurance partners
about our proposition
CASE04
Kept enhancing and enhancing our deck
But without testing the pitch on ‘customers’
When we finally reached out, we didn’t use the deck
Led to weeks of delay
.

PATIENT B
CASE04
After choosing to improve product for higher value customers
Experimented around which segment could be best served
Kept testing opportunity after opportunity
Couldn’t decide which one to go after
The playing field remained wide open
.

❏ Continuously asking: “is this the best opportunity?”
✓
✓
❏ No choices made while more tests are run
CASE04

CASE04

Hunt for “the best thing”
leads to paralysis
THE-BESTPOX

➔ Dipping toes in the water, never jumping in
THE-BESTPOX
➔ Fear of mistakes
➔ Questioning of value from senior leadership
.

Q: Is what you know good enough to move forward?
THE-BESTPOX
Q: How will you decide the best
path? What are the criteria?
How hard will it be to get
the answer?
.

➔ Finding the “best” is hard/expensive. Choose good and go for it
THE-BESTPOX
➔ Choosing which opportunity will likely be a heuristic or gut feel
➔ Make the most of that good opportunity
➔ Over time, optimise
.

PATIENT
CASE05
Team executed a series of experiments
Always comfortable that they would pan out
With a clear path forward
They hadn’t solved the tricky problems
Having slipped into delivery mode

❏ No emotional highs and lows as difficult truths learned or
overcome
✓
CASE06

CASE06

Building something truly new is never easy.
If it appears to be, something is likely wrong.
The feeling of running a startup is
similar to a roller coaster.
Expect that
IT’S-SO-EASYDISEASY

➔ Going to comfort is easy
➔ Lack of time pressure (metered funding)
➔ “Positivity” culture (bad news is punishable)
.

Q: What could mess you up?
Q: What concrete data do you have
that this will work? What would
you do to get it?
.

➔ Ensure actual progress on riskiest assumptions
(not excessive positivity)
➔ Regular pivot / persevere / kill meetings
➔ Insist on strong (and improving) evidence
.

PATIENT
Experiments proved gap in the market
CASE06
Learnings not diligently documented
So didn’t stop continued speculation on
the existence of good enough products
.

❏ Debate is never closed✓
❏ Weak documentation of learning (hard to find, if documented)✓
❏ Opinions trump data✓
❏ Tests run forever, or poorly designed✓
❏ No innovation accounting metrics (or not up to date)✓
❏ Lack of governance (missing: standups, experiment design,
documentation standards, pivot/persevere/kill meetings, prioritisation,
retrospectives, communication)
✓
CASE06

➔ What little progress that is made, is quickly unravelled
➔ Learning stuck in individuals’ heads
➔ Learning not in priority order
NO-RIGOUR-MORTIS

➔ People on team fear owning the result (mercenaries)
NO-RIGOUR-MORTIS
➔ Lack of leadership
➔ No accountability given or requested
➔ Too many non-dedicated team members
.

Q: Where can I see what you’ve learned?
NO-RIGOUR-MORTIS
Q: When did you last finish an experiment?
Q: Do you have the resources to succeed?
.

● Bi-weekly cadence in which riskiest assumptions
are reduced. Failure to make progress is pivot
situation (accountability)
NO-RIGOUR-MORTIS
● Track governance carefully. May be only way to know
if a team is making progress (innovation accounting)
● If needed, team capability must be addressed
.

Spent a year researching most valuable way
to serve high value customers
PATIENT
CASE07
Led to us not choosing a way forward
More and more tests to run, none landing killer
conclusion
Kept asking: is this an opportunity?
.

❏ Understanding from first principles (we know nothing)✓
❏ Many tests, few results✓
❏ No courage or conviction✓
❏ Few sharp puzzle pieces but no overall picture✓
CASE07

CASE07

Lots of time passes, and the team is on the
verge of moving forward, but then…
BOIL-THE-OCEANOMA

➔ Fear of making a call
BOIL-THE-OCEANOMA
➔ Must know everything to
decide what to do
➔ Waiting for the spreadsheet
to give the answer
.

BOIL-THE-OCEANOMA
Q: What progress have you made in the last two weeks?
Q: If you had to, what would you build right now?
Q: What are the riskiest assumptions still to be identified?
.

➔ Make small iterative steps in the right direction, rather
than giant leaps
BOIL-THE-OCEANOMA
➔ Track the reduction or change in riskiest assumptions carefully
➔ Expect changes on the order of weeks, not months or years
.

We launched a new product for commercial landlords
PATIENT A
CASE08
Before it had traction, we pulled the funding because
we had higher short-term value opportunities
Pulling the funding meant that the product never
reached our target performance metrics
.

The Board provided money to investigate a new market
PATIENT B
CASE08
But it was unclear what a positive result would be for the
Board
When asked, the answer was...
“Nothing would convince us to move forward”
We gave the money back

❏ Even if the experiment is a success, there are no resources
to follow up the success
✓
❏ Funding gets pulled early
CASE08
✓

CASE08

Experiments have no value since they
won’t be followed up or get the support they need
INVESTAPHOBIA

➔ Leadership never committed to project in the first place
INVESTAPHOBIA
➔ People expect the numbers to look good right away
(no innovation metrics)
.

Q: How important is this to
the business?
INVESTAPHOBIA
Q: If the tests succeed,
are the funds/people
ready to move forward?
.

➔ This is not a go/no-go situation. Funding should be ring-fenced
for projects where opportunity is proven
.© Simply Business 2019
INVESTAPHOBIA

If Simply Business is this bad, how did we do so well?
We identified and cured these painful diseases
We are never satisfied (self-critical)
Our culture is about learning and being a great place
for the right kind of people
.

It’s all about the people/team. The more entrepreneurial, the better
Fear of messing up is powerful – courage is needed
Culture, and culture of experimentation are crucial (you can’t fake it)
Must be ready to invest in ambiguity
Good and bad experimentation can be hard to tease apart.
Maladies provide strong early signals
.
Things to remember for your own teams

Takeaways
➔ A/B TESTOSTROPHY – many similar tests, little learning
Do tests that matter
➔ LOW-HANGING-FRUIT-ARRHEA – constant quick wins to prove
worth
Stop trying to prove you are worth it. Do the important
➔ CAGED-IN SYNDROME – nothing can be changed, so don’t try
You have to change the world to succeed
➔ THE-BEST POX – always asking “Is this the BEST opportunity?”
Pick a good opportunity. Finding the best takes too long
Camera Icon by https://icons8.com

Takeaways
➔ IT’S-SO-EASY DISEASY – “We got this, don’t worry”
Innovation is a roller-coaster, expect that
➔ NO-RIGOUR-MORTIS – No innovation accounting or regular
progress on risky assumptions
Make sure you measure progress on a bi-weekly basis
➔ BOIL-THE-OCEANOMA – Need to understand it all before any action
Small iterations towards your big goal
➔ INVESTAPHOBIA – No funds available even if the test succeeds
Without funding, don’t bother with a test

B5: The 8 Deadly Maladies of Rapid Experimentation, Lukas Oberhuber

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to B5: The 8 Deadly Maladies of Rapid Experimentation, Lukas Oberhuber

Similar to B5: The 8 Deadly Maladies of Rapid Experimentation, Lukas Oberhuber (20)

More from Lean Startup Co.

More from Lean Startup Co. (20)

Recently uploaded

Recently uploaded (20)

B5: The 8 Deadly Maladies of Rapid Experimentation, Lukas Oberhuber

Editor's Notes