Opticon 2017 - Building a Culture of Experimentation at IBM

1. opticon2017
Building a Culture
of
Experimentation
at IBM
Ari Sheinkin
VP of Marketing Analytics, IBM

2. opticon2017
Failure:
A Success
Story
Intro
Chapter 1: Ned
Chapter 2: The Blue Period
Chapter 3: Golden Cow

3. opticon2017
Intro:
Expectation vs Reality
The vision: data-driven marketing is not a marketer with data.
It is a marketer who uses data to form hypotheses and test and learn.
The number of marketers: 5,000+
The resources: 3 people supporting all testing at IBM.

4. opticon2017opticon2017
Chapter 1:
NED

5. Ned,
my nemesis
Hey there, Ari!
How are
you doing?

6. opticon2017
The current state of testing
adoption
Operational:
Rolled out the platform and training
• 23 Business Units
• 170 Countries
2015 2017
Testers testing 1 1,531
Active tests 5 1,238
Changes it led to:

7. opticon2017
Lessons learned
We don’t need an
experimentation
hero, we need
testing
democracy.
We need clear
job roles that
hold individuals
accountable
for testing.
We need clear
metrics to define
the success of
a good testing
program.
We need a Center
of Excellence to
scale support for
5,000 marketers
around the world.

8. opticon2017opticon2017
Chapter 2:
THE BLUE PERIOD

9. opticon2017
Classic IBM
CTA Copy CTA Styling

10. opticon2017
Beyond the blue
Personalization
1st + 3rd party data
2017 Q3
A/B Testing
Button colors + CTAs
2015
+1654%

11. opticon2017
Running
a test
• One-off experiments
on low value pages
• No real action taken
on results
• Experiments not
iterated upon
vs Building a
testing plan
• Make testing a part of
the campaign planning
process and lifecycle
• Iterate on
experiments, always
learning from previous
results
• Taking chances on
bolder ideas
opticon2017

13. opticon2017opticon2017
Chapter 3:
GOLDEN COW

14. opticon2017
In the past
We feared experimentation because we feared failure.

15. opticon2017
Current state of culture
• Quarterly contests
• Office hours
• Testing Slack
channel
• Enablement
• Newsletters
• Internal testing
web page
• Testing Blitz: 30
tests in 30 days
Rituals Repetition Recognition
The culture we’re dreaming of?
Where the courage to fail is rewarded.
opticon2017

16. opticon2017
Lessons learned
We can build a testing culture
by winning
Hearts and minds
One person at a time.

17. opticon2017

18. opticon2017
To conclude, we’ve failed.
But we’ve also learned a lot. And therefore have
had a lot of successes. And we’ve iterated.
Isn’t that the whole point of experimentation?
“ “

19. opticon2017
Q&A

20. opticon2017
THANK YOU

21. opticon2017

22. opticon2017