Driving your marketing automation with multi-armed bandits in real time

Copyright © WLOG Solutions & TouKCopyright © WLOG Solutions & TouK
Driving your marketing
automation with multi-armed
bandits in real time

Agenda
● Dot.Com business case
● The modern A/B testing
● The architecture
● Numbers (!)
● Key takeaways

Dot.Com business case

Dot.Com is a unicorn that
wants to start making a
revenue from real customers
not the investors.

Copyright © WLOG Solutions & TouK
They pivoted from dating portal and
created a NoHotel service.

They pivoted from dating portal and
created a NoHotel service.
Uniqueness was that there was no
hotel but you could still rent a room
and pay with bitcoin.

Which type of room do you want to book?
Villager Parisian Poles on vacation
Book now Book now Book now
only 100Ƀ day only 150Ƀ day only 50Ƀ day

After first year of investing rounds...

After first year of investing rounds...
Real revenue has reached
astonishing value of 0Ƀ.

They needed to pump up an
income before the next
investment round.

They needed to pump up an
income before the next
investment round.
They had a week to do it.

They hired consultants from
A/B consulting company.
Consultants after an audit
recommended to use A/B testing to
pump up the revenue.

http://www.dilbert.com/strips/comic/2014-03-15/

Created three different types of questions:
- Which type of room do you want to book?
- With whom do you want to spend a night
today?
- What extent does the African swallow
reach?

Generated 3! (eng. six) versions of the page with
shuffled types of rooms.

Villager
Book now Book now
Parisian Poles on vacation
Book now
Generated 3! (6) versions of the pricing boxes.

In total NoHotel webpage had 108
versions (=3*6*6).

They had to choose...
How in a week generate max income from
those versions?
Which version is the most profitable (best)?

The modern A/B testing

Classical A/B testing
1. Split stream of customers landing on a webpage onto
all alternatives randomly.
2. Gather results after testing N customers for each
alternative.
3. Select the best one (?)
4. Use the best one to earn money.

A 1
A 2
A 108
....
100/N
115/N
110/N
....
A 1
A 2
A 108
....
100/N
115/N
110/N
....
Explore in a random way. Exploit the best

Positive positives
- easy to implement
- unbiased
Negative positives
- technical parameters
- burned time for
selecting the best
parameter
- problem agnostic
- The best is not the most
profitable to exploit.

Modern A/B testing (bandits)
- Epsilon greedy
- …
- and variants

A 1
A 2
A 108
....
100/N
115/N
110/N
....
A 1
A 2
A 108
....
100/N
115/N
110/N
....
Explore
Start with all alternatives equal and
learn in an online mode.
Exploit the best with some
randomisation.

Context?
(context, arms)
You do not control this.
(e.g. visit hour, season, ...)
You do control this.
(e.g. price, order, img, ...)

Positive positives
- easy to implement
- more effective
exploration
Negative positives
- technical parameters
- burned time for
selecting the best
parameter
- problem agnostic
- The best is not the most
profitable to exploit.

Optimal learning - a unified theory
- Offline
- We want to optimise probability to find the best option.
- On-line
- We want to optimise sum of revenues of our actions.
- Some keywords
- knowledge gradient, Gittins indices, Bayes,
exploration-exploitation
Sophisticated but a must if you perform a large
number of experiments.

What is the secret “sauce”?
time
Ƀ
Event Payoff
Delayed payoff
What to do when waiting for
the payoff?
Exploiting only the best is the
inferior policy.

Architecture

Just code it?
- Bandit algorithms are simple to code
- But how to:
- Understand
- Tune
- Control
- Monitor

Model in R - dev and deployment
DevProduction
.zip
Continuous
integration
Version
control
Local CRAN
MRAN
REST API
Plumber REST API

Streaming with Flink
- Scaling
- Efficient state processing
- Windows and aggregates

Flink - state for bandits

Business rules
- Quick promotions
- Dynamic thresholds
- Expert rules
- All by analysts/business?
?

Business rules with Nussknacker

80% with UI
20% with code

Recommendation
state
Stats
preaggregation

Further scaling?
What if invoking R
via REST interface is
not fast enough?

Further scaling?
Multiplicate R
with
Kubernetes

Further Further scaling?
Portable
Format
for
Analytics?
FastR
on
GraalVM?

Results aka numbers

Results
● Expected conversion ratio
○ e-greedy: 750
○ Optimal learning: 883
○ Improvement: 15%
● Probability to identify best
alternative:
○ e-greedy: 81%
○ Optimal learning: 91%
Advanced Maths added value
Scenario for A/B tests
● 6 alternatives with the
following conversion ratios: 5%,
6%, 7%, 8%, 9% i 10%
● Sample size: 10000 users

Results
● Expected conversion ratio
○ e-greedy: 750
○ Optimal learning: 883
○ Improvement: 15%
● Probability to identify best
alternative:
○ e-greedy: 81%
○ Optimal learning: 91%
Advanced Maths added value
Scenario for A/B tests
● 6 alternatives with the
following conversion ratios: 5%,
6%, 7%, 8%, 9% i 10%
● Sample size: 10000 users
With mathematics you can make algorithm
parameterless and more effective!

Reacting to change

Reacting to change
Could converge even faster with drift detection or
memory weighting.

~20k/s on node for
experiment
Performance?

Key takeaways

● Business parameters are good, technical are bad.
● It pays off to use mathematics for A/B testing.
● Automated deployment is crucial when you go on
production.
● Flink and Nussnacker give performance and configuration
capabilities.
● R Suite makes R ready for production.

Wit Jakuczun
CEO
wit.Jakuczun@wlogsolutions.com
+48 601820620
http://www.wlogsolutions.com
Maciek Próchniak
mpr@touk.pl
@mpproch
http://touk.pl
52

Driving your marketing automation with multi-armed bandits in real time

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Driving your marketing automation with multi-armed bandits in real time

Similar to Driving your marketing automation with multi-armed bandits in real time (20)

More from Wit Jakuczun

More from Wit Jakuczun (12)

Recently uploaded

Recently uploaded (20)

Driving your marketing automation with multi-armed bandits in real time