By Andy Pulkstenis, Director - Advanced Analytics - State Farm Insurance

1. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization
Do You Know or Do You Think You Know?
Incorporating Business Experiments Into
Strategy Development
Andy Pulkstenis
Director, Advanced Analytics
Andy.Pulkstenis.re6i@StateFarm.com

2. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 1
• Setting the context
• State Farm examples, from simple to sophisticated
• Blueprint for getting testing started (or growing testing) at your firm
• Responding to resistance
• Setting the context
• State Farm examples, from simple to sophisticated
• Blueprint for getting testing started (or growing testing) at your firm
• Responding to resistance
What are we doing today?

3. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 2
What is a “designed experiment?”
Designed testing is a way of treating the business environment like a controlled
experimental setting, randomly assigning strategies to customers as in a clinical
drug study or lab
Benefits
• Clear, accurate insights about causation
• Ultra-clean data for model development = more accurate prediction
• Enables strategy optimization at the customer level

4. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 3
Innovators, on Testing
“By thorough experimentation and trial-and-
error in the market, [successful companies]
iterate towards a strategy and a business
model that really is viable.”
- Harvard Business School Professor
Clayton Christensen, a recognized
authority on “disruptive innovation”

5. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 4
- Rich Fairbank, Chairman
& CEO Capital One
“What we’ve done is to create an
innovation machine. This is not
just a credit card strategy, testing
is a revolution that can be applied
to many businesses.”
Innovators, on Testing

6. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 5
- Jeff Bezos, Amazon Founder
“If you double the number of
experiments you do per year,
you’re going to double your
inventiveness.”
Innovators, on Testing
Fun fact: Amazon once fired a team for implementing
strategic change without proper statistical testing
first!

7. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 6
How do businesses use testing?
Strategy
Optimization
Targeted
Messaging
Pricing
Real-time Website
Design/Optimization
Customer-level
Decisioning
Retention & Save
Strategies
Product & Strategy
Development
Operations
Improvement

8. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 7
Specific business examples
Sources: “How to Design Smart Business Experiments” by Tom Davenport, HBR 2/2009
“The Discipline of Business Experimentation” by Stefan Thomke and Jim Manzi, HBR 12/2014
Do visible lobster tanks
increase lobster sales?
Are higher bids
driven by CC
payment option?
Will opening stores an
hour later lead to a drop
in sales?
Nearly all aspects
of business.

9. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 8
Typical Testing Evolution
None A/B
“one factor at a time”
(OFAT)
Multivariate
(MVT)
MVT w/
covariate analysis
Pick “one winner for all”
Increased business insight while progressing left to right
Enables customer or segment-
level strategy/message
optimization
Companies typically flow through an evolution when embarking on testing agendas
Continuous testing
A pervading culture of
evolving, continuous, and
cyclical testing provides
strategic foundation
Many stop at A/B or OFAT through lack of knowledge or lack of skillset, missing out
on benefits that increase with each successive milestone.

10. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 9
• Setting the context
• State Farm examples, from simple to sophisticated
• Blueprint for getting testing started (or growing testing) at your firm
• Responding to resistance
What are we doing today?

11. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 10
Which banner led to the most visits to the
“Agent Search” page?
Version B:
Version A:
(Click leads to “Drive
Safe & Save” page)
(Click leads to “Agent
Search” page)

12. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization
Answer: Version B led more people to the
Agent Search page

13. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 12
Which statefarm.com banner led to the most
visits to the “Drive Safe & Save” page?
Version B:
Version A:
(Click leads to “Drive
Safe & Save” page)
(Click leads to “Agent
Search” page)

14. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization
“Find an Agent” drove more people to the “Drive Safe &
Save” page than the “Drive Safe & Save” experience did!

15. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 14
Version A: Version B:
Which mailer generated the highest lift in
Life insurance sales?

16. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 15
Claims Contact Center
Claims Contact Center – “Out of Scope” Calls
Can proactive sharing of contact info reduce misdirected or “out of scope” Claims
calls?

17. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 16
Multivariate Discount Double Check Test
• Explored 7 different page features as tested ‘factors’ (text, graphics, layout,
images, colors, etc.)
• Results in 384 distinct web page variations (with a subset of 200+ valid
combinations)
• Only needed to physically test 58 combinations
• Improved quote starts by 50% and quote completes by 10%

18. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 17
• Setting the context
• State Farm examples, from simple to sophisticated
• Blueprint for getting testing started (or growing testing) at your firm
• Responding to resistance
What are we doing today?

19. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 18
My roadmap for launching testing
Be Persistent
It will take legwork and determination to find the
partner in step two
Be Ambitious Aim for moderate or greater business impact
Be Careful
Carefully execute the approach with business
partners
Be Ready Build the “success story” deck before you need it
Be A Teacher There is a significant initial education component
Be Opportunistic Follow the path of least resistance

20. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 19
Best Practices: Set Up for Success
Assemble the
right structure
Identify key roles at the outset
(business, stats/DOE, execution, structural or creative design)
Assemble the
right players
Don’t just fill seats with “good people.” If you need
an SME in a certain area, then staff that role with an SME
Assemble the
right questions
A surprising amount of time in early DOE engagements is spent
helping the business partner define the problem as a quantifiable
hypothesis
Assemble the
right scope
Too ambitious and you fail spectacularly
Not ambitious enough and you really don’t move the needle
Assemble the
right controls
Make sure to monitor execution and data collection from Day 1
and throughout the process

21. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 20
• Setting the context
• State Farm examples, from simple to sophisticated
• Blueprint for getting testing started (or growing testing) at your firm
• Responding to resistance
What are we doing today?

22. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 21
Common Arguments
Regulation
Initial resistance to statistical testing typically falls into
three categories:
Complication
Organizational
Tradition

23. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 22
Regulation
Our industry is too heavily regulated
for this
Didn’t stop Finance, Pharma, and others
from successful transition to DOE
Even if allowed, the regulators would
never understand it
It’s not “fair” to randomly assign
strategies to customers in a test
Successful testing industries proactively
educate regulators on key concepts
• Current strategy isn’t “fair” either
• Jonas Salk & polio
• “Beautiful experiment over which the
epidemiologist could become quite
ecstatic but would make the
humanitarian shudder”
Perceived Barrier Response

24. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 23
Complication
Too complicated Nonsense – no more complicated than
predictive modeling, just new to your firm
Ignoring complex relationships doesn’t
mean they aren’t there
Start small, manageable
• Design according to cost constraints
• Much cheaper than hoping we guessed
correctly (think of it as “buying data”)
This can be handled by applying
“exclusions” to the design
Complexity of analysis
Too difficult to execute
Some combinations don’t make
business sense
Too expensive
Perceived Barrier Response

25. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 24
Organizational Tradition
“We tried it once and we saw no
improvement” (i.e. learned nothing)
• You don’t hit a home run every time
• You learned that those strategies were
equivalent
• Often gain fresh, real insights that defy
historical logic or perceived wisdom
• “The greatest obstacle to discovery is not
ignorance - it is the illusion of knowledge.”
- Daniel Boorstin
“We intuitively know our business”
Perceived Barrier Response
“Our external consulting firms tell us
all we need to know about course
correcting”
Designed testing consistently outperforms
consultant-provided solutions
“I have not failed 10,000 times. I have succeeded in
proving that those 10,000 ways will not work. When I
have eliminated the ways that will not work, I will find
the way that will work.”
- Thomas Edison
WD-1 WD-39 didn’t work!
Today WD-40 can be found in 4/5 US homes

26. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 25
Questions/Comments
Andy.Pulkstenis.re6i@StateFarm.com

27. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 26
Engagement Model
Business Partner
Statistical Design
Implementation/Execution
Structural and Creative Design
TestingTeam
Ensure alignment with broader
strategic goals
Strategic Approval
“Testing Council” of SME’s
ensures alignment with
statistical best practices and
production constraints
Structural Approval
• Statistical DOE
• Production EnvironmentDocument key hypotheses,
findings, participants, and
actions taken
Test Results Repository
Execute/Monitor/Analyze/Take Action
Ensure alignment with broader
strategic goals
Strategic Approval
Execute/Monitor/Analyze/Take Action
“Testing Council” of SME’s
ensures alignment with
statistical best practices and
production constraints
Structural Approval

28. Contains CONFIDENTIAL information which may not be disclosed without
express written authorization 27
What do I mean by “test-and-learn culture?”
• Designed experiments inform (and determine) strategy
development
• Understanding of when to settle for simple “A/B” testing and
when to use more sophisticated multifactor tests
• Internal standards/best practices for test design
• Formal internal governance
A true testing culture demonstrates the following elements:



