1.
Bart.Schutz@onlinedialogue.com
#
Marshmallows, "
& the Psychology"
behind Evidence-Based Growth
Bart.Schutz@onlinedialogue.com
#

2.
Bart.Schutz@onlinedialogue.com
#
The Marshmallow Test

3.
Bart.Schutz@onlinedialogue.com
#
The Marshmallow Test
Self-control:
health, wealth & even happiness

4.
Bart.Schutz@onlinedialogue.com
#
Homo Sapiens often is better off when
excerting self-control

5.
Bart.Schutz@onlinedialogue.com
#
Part 1) "
Psychology

6.
Bart.Schutz@onlinedialogue.com
#
We are a Dual-Processing Brain
Fast & automated
Intuitive & associative
Effortless
Large capacity
Reasons & rules
Requires attention
Effortful
Small capacity
Subconscious,
emotional
processes
Conscious,
rational
processes
(always on)
(lazy controller)
System 1
System 2

7.
Bart.Schutz@onlinedialogue.com
#
Feel your own 2 systems

8.
Bart.Schutz@onlinedialogue.com
#
A user = "
2 processing systems

9.
Bart.Schutz@onlinedialogue.com
#
The inconvenient truth:
The systems create ‘illusions of control’

10.
Bart.Schutz@onlinedialogue.com
#
Do we have Conscious Control?
A couple has 2 kids. "
One of them is a boy.
How big is the chance that
the other kid is a boy too?
33%
?

11.
Bart.Schutz@onlinedialogue.com
#
Do we have Conscious Control?
A couple has 2 kids. "
One of them is a boy.
How big is the chance that
the other kid is a boy too?
33%
?
The inconvenient truth:
‘Conscious control is often an illusion‘

12.
Bart.Schutz@onlinedialogue.com
#
Do we have Reasons?

13.
Bart.Schutz@onlinedialogue.com
#
Do we have Reasons?
The inconvenient truth:
‘Reasons are often post-decision alibi’s‘

14.
Bart.Schutz@onlinedialogue.com
#
Do we have Free Will?
I Know! Calculations are hard work.
But it’s nearly over.
Come on, one more…
123 + 5
QUICK!
THINK ABOUT A
COLOR AND A
TOOL!
At the end of this test,
I ask a question.
Don’t stop and think about it
Answer is immediately!
How much is:
15 + 6
3 + 56
89 + 2
12 + 53
75 + 26
25 + 52
63 + 32

15.
Bart.Schutz@onlinedialogue.com
#
Do we have Free Will?

16.
Bart.Schutz@onlinedialogue.com
#
Do we have Free Will?
The inconvenient truth:
‘Free-will is often an illusion‘

17.
Bart.Schutz@onlinedialogue.com
#
Do we have a reliable Memory?

18.
Bart.Schutz@onlinedialogue.com
#
Do we have a reliable Memory?
The inconvenient truth:
‘Experience & memory are quite distinct’

19.
Bart.Schutz@onlinedialogue.com
#
We are a Dual-Processing Brain
Fast & automated
Intuitive & associative
Effortless
Large capacity
Reasons & rules
Requires attention
Effortful
Small capacity
Subconscious,
emotional
processes
Conscious,
rational
processes
(always on)
(lazy controller)
System 1
System 2

20.
Bart.Schutz@onlinedialogue.com
#
You and your users usually "
CANNOT help you with optimizing!

21.
Bart.Schutz@onlinedialogue.com
#
Usability Test insights?
B
A
Winner

22.
Bart.Schutz@onlinedialogue.com
#
Usability Test insights?
B
A
Winner
User Testing is often
Useless Testing

23.
Bart.Schutz@onlinedialogue.com
#
What if users had designed Booking.com?

24.
Bart.Schutz@onlinedialogue.com
#
CRO = User-centric
Psychology

25.
Bart.Schutz@onlinedialogue.com
#
Part 2) "
Data & Statistics

26.
Bart.Schutz@onlinedialogue.com
#
Behavior has become data

27.
Bart.Schutz@onlinedialogue.com
#
Behavioral data analysis

28.
Bart.Schutz@onlinedialogue.com
#
Behavioral data analysis
The convenient truth:
1st party data!

29.
Bart.Schutz@onlinedialogue.com
#
Behavior has become testable

30.
Bart.Schutz@onlinedialogue.com
#
Behavior has become testable
The convenient truth:
!!! 1st party evidence !!!

31.
Bart.Schutz@onlinedialogue.com
#
Evidence has become FREE
Free A/B-testing: GTMtesting.com

32.
Bart.Schutz@onlinedialogue.com
#
Tools easy: lie!
Evidence has become Easy

33.
Bart.Schutz@onlinedialogue.com
#
You must have the basics of "
Statistics & Data Quality right!

34.
Bart.Schutz@onlinedialogue.com
#
The inconvenient truth:
CRO = ‘The illusion of Growth‘

35.
Bart.Schutz@onlinedialogue.com
#
POWERRRR!!!! (over significance)
Sample Size & Uplift calculator: abtestguide.com/abtestsize/

36.
Bart.Schutz@onlinedialogue.com
#
POWERRRR!!!! (over significance)
Sample Size & Uplift calculator: abtestguide.com/abtestsize/
Lack of power:
Big issue in CRO!

37.
Bart.Schutz@onlinedialogue.com
#
POWERRRR!!!! (over significance)
Sample Size & Uplift calculator: abtestguide.com/abtestsize/
Calculate
Minimum Required Uplift

38.
Bart.Schutz@onlinedialogue.com
#
Frequentistic (t-test): abtestguide.com/calc/
Which statistics for analyzing?
Bayesian: abtestguide.com/bayesian

39.
Bart.Schutz@onlinedialogue.com
#
Frequentistic (t-test): abtestguide.com/calc/
Which statistics for analyzing?
Bayesian: abtestguide.com/bayesian
Use Bayesian to grow
Use Frequentist to learn

40.
Bart.Schutz@onlinedialogue.com
#
But: Do you meet the assumptions of the statistics?
effect

41.
Bart.Schutz@onlinedialogue.com
#
But: Do you meet the assumptions of the statistics?
effect
Track users 100%, OR "
incorporate the pollution

42.
Bart.Schutz@onlinedialogue.com
#
But: Do you meet the assumptions of the statistics?
‘Ceteris Paribus’

43.
Bart.Schutz@onlinedialogue.com
#
But: Do you meet the assumptions of the statistics?
Check for
Skewed Samples
‘Ceteris Paribus’

44.
Bart.Schutz@onlinedialogue.com
#
OMG,
All my A/B-tests
were useless
!?

45.
Bart.Schutz@onlinedialogue.com
#
Tools easy: lie!
Evidence has become Easy

46.
Bart.Schutz@onlinedialogue.com
#
Part 3) "
Evidence-Based Growth

47.
Bart.Schutz@onlinedialogue.com
#
&
Evidence Based Growth
√ Data√ Psychology

48.
Bart.Schutz@onlinedialogue.com
#
CRO => "
Evidence-Based Growth (EBG)

49.
Bart.Schutz@onlinedialogue.com
#
3 levels of EBG!

50.
Bart.Schutz@onlinedialogue.com
#
EARN
adding direct value
LEARN
user behavior
TURN
Innovate
3 levels of CRO EBG

51.
Bart.Schutz@onlinedialogue.com
#
Level 1: Transactional"

52.
Bart.Schutz@onlinedialogue.com
#
Level 1: Transactional"
Earn – Earn - Earn

53.
Bart.Schutz@onlinedialogue.com
#
1st Bandwidth: How many experiments can you run?

54.
Bart.Schutz@onlinedialogue.com
#
Type 3: Loss
preven.on
Type 2: Earn Type 1: Learn
Prob of upli+ > 0% = 75% Prob of upli+ > 3% = 75%
(CO1/2/3 > 2%)
Frequen>st
(1-sided, 95%)
Bandwidth 20 per week 8 per week 5 per week
1st Bandwidth: How many experiments can you run?

55.
Bart.Schutz@onlinedialogue.com
#
“Done,
absolutely everything
Is tested
REALLY
Booking.com
Evidence Everything

56.
Bart.Schutz@onlinedialogue.com
#
Evidence Everything

57.
Bart.Schutz@onlinedialogue.com
#
“Our success at Amazon
is a function of
how many experiments
we do per year, per month, per
week, per day…”
Jeff Bezos, CEO Amazon
Focus on Regret & Velocity

58.
Bart.Schutz@onlinedialogue.com
#
“Move fast and break things.
Unless you are breaking stuff, you
are not moving fast enough.
…”
Mark Zuckerberg, FaceBook
Done is better than Perfect

59.
Bart.Schutz@onlinedialogue.com
#
Earn = Transactional

60.
Bart.Schutz@onlinedialogue.com
#
Quantity over Quality

61.
Bart.Schutz@onlinedialogue.com
#
Quantity over Quality
The BIG fish will not eat the SMALL, "
The FAST fish will eat the SLOW

62.
Bart.Schutz@onlinedialogue.com
#
Earn = Transactional

63.
Bart.Schutz@onlinedialogue.com
#
Data Driven
Transactional
C!
CREATE
A!
ANALYZE
T!
TEST

64.
Bart.Schutz@onlinedialogue.com
#
EARN => CTA–CTA –CTA!

65.
Bart.Schutz@onlinedialogue.com
#
Level 2: Intellectual"

66.
Bart.Schutz@onlinedialogue.com
#
Level 2: Intellectual"
Learn – Learn – Learn

67.
Bart.Schutz@onlinedialogue.com
#
Evidence Everything
Customer Intelligence
A!
ANALYZE
C!
CONCLUDE
C!
CREATE
A!
ANALYZE
T!
TEST

68.
Bart.Schutz@onlinedialogue.com
#
Heatmaps
Recordings
Web Analytics
Contextual Data
Previous tests
Algorithm outcomes
Mission
Vision
Strategy
Positioning
Goals
Scientific research
Competitors
View
(1st party)
Value
Customer Service
Surveys
Feedback tools
User & Market research
Social Media
Behavior & Experience Determinant Study
VoiceVerified
(1st party)
Verified

69.
Bart.Schutz@onlinedialogue.com
#
Behavior & Experience Determinant Study
get more insights from science:

70.
Bart.Schutz@onlinedialogue.com
#
Behavior & Experience Determinant Study

71.
Bart.Schutz@onlinedialogue.com
#
Behavior & Experience Insights Grid

72.
Bart.Schutz@onlinedialogue.com
#
Behavior & Experience Insights Grid

73.
Bart.Schutz@onlinedialogue.com
#
Behavior & Experience Insights Grid

74.
Bart.Schutz@onlinedialogue.com
#
Case: Hostelworld

75.
Bart.Schutz@onlinedialogue.com
#
A
B
A
B
Winner
Loser
Different brain systems per device

76.
Bart.Schutz@onlinedialogue.com
#
System 2
||
/
controling
System 1
System 1
/
||
Little control of
System 2
Learn = Customer Intelligence

77.
Bart.Schutz@onlinedialogue.com
#
1st order learning: Local maximum

78.
Bart.Schutz@onlinedialogue.com
#
LEARN => FACT – FACT – FACT!

79.
Bart.Schutz@onlinedialogue.com
#
Level 3: Transformational"
Innovate & Disrupt

80.
Bart.Schutz@onlinedialogue.com
#
Evidence Everything
Customer Intelligence
A!
ANALYZE
C!
CONCLUDE
C!
CREATE
A!
ANALYZE
T!
TEST

81.
Bart.Schutz@onlinedialogue.com
#
F!
FIND
T!
TELL
A!
ANALYZE
C!
COMBINE
C!
CREATE
A!
ANALYZE
T!
TEST
F!
FIND
T!
TRANSFORM
A!
ANALYZE
C!
CONCLUDE
C!
CREATE
A!
ANALYZE
T!
TEST
Evidence Everything
Transformational

82.
Bart.Schutz@onlinedialogue.com
#
B
A
Winner
Implicit safety

83.
Bart.Schutz@onlinedialogue.com
#
Case: Hostelworld On–The–Go?

84.
Bart.Schutz@onlinedialogue.com
#
The future of CRO = EBG"
A transformational understanding "
of what drives homo sapiens

85.
Bart.Schutz@onlinedialogue.com
#
Let’s rewind

86.
Bart.Schutz@onlinedialogue.com
#
The future of CRO = EBG"
A transformational understanding "
of what drives homo sapiens

87.
Bart.Schutz@onlinedialogue.com
#
LEARN => FACT – FACT – FACT!

88.
Bart.Schutz@onlinedialogue.com
#
EARN => ACT – ACT – ACT!

89.
Bart.Schutz@onlinedialogue.com
#
CRO => "
Evidence-Based Growth (EBG)

90.
Bart.Schutz@onlinedialogue.com
#
You got to have your
Data & Statistics right

91.
Bart.Schutz@onlinedialogue.com
#
CRO = User-centric
Psychology

92.
Bart.Schutz@onlinedialogue.com
#
You and your users usually "
CANNOT help you with optimizing!

93.
Bart.Schutz@onlinedialogue.com
#
A user = "
2 processing systems

94.
Bart.Schutz@onlinedialogue.com
#
Homo Sapiens often is better off when
excerting self-control

95.
Bart.Schutz@onlinedialogue.com
#
Internet = The
Biggest Psychological experiment ever

96.
Bart.Schutz@onlinedialogue.com
#
Internet = Data & Experiments
>100 billion ‘people’ are
in a test every month

97.
Bart.Schutz@onlinedialogue.com
#
An abundance of data is tracked

98.
Bart.Schutz@onlinedialogue.com
#
Knowledge
Knowledge
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
I am in control
No, you’re not, "
I know you better than
you know yourself
Internet = Data & Experiments
Bart.Schutz@onlinedialogue.com

99.
Bart.Schutz@onlinedialogue.com
#
Knowledge
Knowledge
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
a/b-test a/b-test a/b-test a/b-test
Internet = Data & Experiments
I know how to inject my
goal into your mind,
separating you from your
Money & Time & Free-will
Bart.Schutz@onlinedialogue.com

100.
Bart.Schutz@onlinedialogue.com
#
With great intelligence and influence,
comes great responsibility

101.
• Bart@OnlineDialogue.com
• nl.linkedin.com/in/bartschutz/
• @BartS
STATISTICS
Use =>
Abtestguide.com/calc
CONSUMER PSYCHOLOGY
Learn =>
wheelofpersuasion.com/course
Download these slides => http://ondi.me/cboostdk
EVIDENCE BASED GROWTH
Incorporate =>
info@OnlineDialogue.com
Co-founder
Advisory Board Dutch
Institute for Psychologists
EU Federation of Psychology Associations
Chair Task Force ‘Internet & Psychology’
Tak!
Bart Schutz