Machine Learning driven Quantum Optimization for Marketing

www.deeplearningitalia.com 1
DAL DEEP LEARNING AL QUANTUM COMPUTING
Le nuove frontiere tecnologiche che stanno cambiando il nostro presente e il futuro
In collaborazione con: Fondazione Bruno Kessler

Machine Learning driven Quantum
Optimization for Marketing
Calogero Zarbo
Senior AI Research Scientist @Satispay
www.deeplearningitalia.com

3
The Problem
Our main Consumer acquisition channel is the MemberGetMember (MGM) campaign,
counting for more than 40% of the total consumer acquisition.
MGM campaigns are a substantial % of the overall marketing budget: about 25M Euros in
2022.
Even slight optimizations in this area may bring in huge results in terms of savings

4
The Solution
Personalized MGM campaigns
The assumptions are:
● Each user has unique pattern of being engaged with the app
● These patterns can be proxied by how the app is being used
Why you need AI:
● Not all customers sees every possible incentives a reasonable number of times
● Fill the gap with real measured data is costly

5
Data Pipeline & Strategy
Interaction with the BI team is crucial to make all of this happens
● Extract key information from internal datalake about registration, activation,
transactions and MGM activity
● Ground preparation for A/B test (Power Analysis)
BI not only provide data, but strategize the whole testing process:
1. Define the number of consumer we can work with
2. Stratified sampling at district level for both test and control group
3. Handle the communication with marketing for the chosen incentive allocation
4. Results harvesting and KPI evaluation

6
The Target
We experiment various type of target, but we decided that the most important ones
would be:
● Mean of the #Matches @ incentive level (appreciation)
● Presence of the Match @ incentive level (activation)
They express two slightly different perspectives in the user appreciation of a particular
campaign.

7
Data Analysis Plan (DAP)TM
TR
Set
VAL
Set
TS
Set
Normalization Training
Tuning
&
Model Selection
Training
Testing
&
Report
Normalization
©MPBA all right reserved

8
The Models
We tried different models:
● LightGBM
● XGBoost
● Transformers
After setting up a proper Data Analysis Plan (DAP), we discarded the first two options due
to their weakness towards unbalanced datasets.
Matthew Correlation Coefficient (MCC) < 0.4

9
The FT-Transformer
https://paperswithcode.com/method/ft-transformer

10
Wide&Deep Framework
https://arxiv.org/pdf/1606.07792v1.pdf

11
The Appreciation Curve
Here is shown the average appreciation curve of MGM at different levels of incentive.

12
The personalized curve

13
The Setup so far..
We choose a time span of 6 months history
For each Incentive we trained:
● A transformer for the Activation flag in the last 6 months (classification)
● A transformer for the Mean #invitation in the last 6 months (regression)
Phase 1 - The Backtesting
● Activation flag: MCC: 0.91 overall → Good to go
● Mean #invitation: R2 Score: 0.31 overall → Good to go
Phase 2 - The A/B Test
● Decision making process missing

14
The whole system
BI
Appreciation
Activation
Decision
Making
Machine
Marketing

15
The decision maker
Now that I have a level of appreciation for each user, for each incentive level, and a
probability of activation for each user, for each incentive level how to decide who gets
what incentive?

16
The decision maker
what incentive?
All possible combinations of (User,Incentives)?

17
The decision maker
what incentive?
All possible combinations of (User,Incentives)?
This seems like an NP problem: which allocation would maximize the appreciation keeping
the cost under a certain target?

18
Mathematical formalization
pij
= predicted appreciation of ith
user on jth
incentive
xij
= binary variable 0/1 whether or not assign that particular ith
user the jth
incentive
cj
= the cost of the jth
incentive
aij
= predicted activation of ith
user on jth
incentive
max sumij
(xij
* aij
* pij
)
s.t.
sumi
(xij
) = 1 for each jth
user
sumij
( (xij
* aij
* pij
* cj
) - CAC target) = 0
sumij
(xij
* aij
* pij
) >= Appreciation target

19
Trying all possible solutions?

20
D-Wave Quantum Annealer
Entanglement:
Two QuBits interact with each other without being
directly stimulated.

21
Entanglement:
Tunnelling:
A QuBit can go in any better place in the solution space
without paying the price.

22
Entanglement:
Tunnelling:
A QuBit can go in any better place in the solution space
without paying the price.
Superposition:
A QuBit can be 0 and 1 at the same time

23
Example of allocations

24
The KPI framework
Why do I
even exist?
We’re OK
No real gains

25
The KPI framework
Why do I
even exist?
I might have to update
linkedin.
We’re OK
No real gains

26
The KPI framework
Why do I
even exist?
Target missed
linkedin.
We’re OK
No real gains

27
The KPI framework
Real gains
Why do I
even exist?
Target missed
linkedin.
We’re OK
No real gains

28
The first A/B Test
Number of People: 16k for each group
Incentives: 20€, 30€, 40€, 50€
Geographic area: all north-west of italy
Sampling method: 1% of each district
Can the system trigger more invitations
keeping the cost close to the group A?

29
The first A/B Test
TARGET
MISSED
Incentives: 20€, 30€, 40€, 50€
Sampling method: 1% of each district
- No
Cost - p-value: 0.0188
Match - p-value: 0.071

30
What was wrong?
● Is the whole system not suitable for the task?
● Were the training data not representative?
● Did we introduce any confounding factor?

31
What was wrong?
● Is the whole system not suitable for the task?
● Were the training data not representative?
● Did we introduce any confounding factor?
Our speculations were:
1. The appreciation variables already include the activation information. Provide both of
them might be misleading. Let’s remove it.
2. The mean of # of matches computed on 6 months might not be representative
enough. Let’s try 3 months.

32
The whole system -V2
BI
Appreciation
Activation
Decision
Making
Machine
Marketing

33
Mathematical formalizationV2
pij
user on jth
incentive
xij
user the jth
incentive
cj
incentive
aij
= predicted activation of ith
user on jth
incentive
max sumij
(xij
* aij
* pij
)
s.t.
sumi
(xij
) = 1 for each jth
user
sumij
( (xij
* aij
* pij
* cj
) - CAC target) = 0
sumij
(xij
* aij
* pij

34
Mathematical formalizationV2
pij
user on jth
incentive
xij
user the jth
incentive
cj
incentive
max sumij
(xij
* pij
)
s.t.
sumi
(xij
) = 1 for each jth
user
sumij
( (xij
* pij
* cj
) - CAC target) = 0
sumij
(xij
* pij

35
The second A/B Test
Incentives: 20€, 30€, 40€, 50€
Sampling method: 1.5% of each district

36
The second A/B Test
Incentives: 20€, 30€, 40€, 50€
- It seems so
TARGET
HIT
Match - p-value: 1.56e-9

37
The third A/B Test
Incentives: 20€, 30€, 40€, 50€
decreasing also the cost?

38
The third A/B Test
Incentives: 20€, 30€, 40€, 50€
decreasing also the cost?
- We can’t say
TARGET
UNK
Match - p-value: 0.0573

39
Cost Analysis - The Campaign
To maximize the campaign returns we need to run simulations at different CAC target as
follows:
● Start: 30€, End: 50€, Step: 0.5€
● Total of: 40 simulations per campaign
● For this exercise let’s optimize a campaign of about 50 000 users
● Total number of variables per campaign: #Users x #Incentives (20€, 30€, 40€, 50€) =
200 000 variables
● Campaign frequency: roughly one campaign every 2 weeks

40
Cost Analysis - HiGHS
● Time per simulation: ~37 hours
● Machine type: c7g.8xlarge
● Machine cost: 1.15$/h - 20% discount: 0.92 $/h
● Cost per simulation: ~34$
● Cost per campaign: ~1361$ -> 62 computing days -> unfeasible
NB. Reducing the number of simulations, reduce dramatically the quality of the tuned
campaign.

41
D-Wave Quantum Solver
● Time per simulation: ~5 minutes
● Machine type:Advantage 6.2
● Machine cost: 125$/h - no discount - we will negotiate it eventually
● Cost per simulation: 11$
● Cost per campaign: 440$ -> 3.3 computing hours -> feasible
NB.The cost is computed assuming the solver is used only for MGM campaigns, which is
unlikely since we will have other optimization projects that would leverage such
technology and therefore lower the offset price. (i.e. optimal top-up for fraud exposure
minimization)

42
What’s in production?
Marketing
Console
ML & QC
Microservice
Districts
Target CAC
Risk Appetite
Simulations
Go
Live!
Simulation Pick

43
Final* Remark
AI do not market your product.AI do not create engagement.AI do not make satispay
more appealing.
AI use data to point out where the great work of marketing is having a tangible positive
impact.
Without the talent of our colleagues, in all departments, there is no AI that can make
Satispay what is now and what will be in the future.

44
Thank you!
GiuseppeValetto
AI Product Owner
Dario Brignone
CTO
Marta Jovanovic
Data Analyst
Giulia Guzzetti
Marketing Manager
Giulia Schiratti
Marketing Specialist
Angelo La Malva
Data Analyst

45
45
SPAZIO ALLE DOMANDE!
In collaborazione con: Fondazione Bruno Kessler
DAL DEEP LEARNING AL QUANTUM COMPUTING
Le nuove frontiere tecnologiche che stanno cambiando il nostro presente e il futuro

Machine Learning driven Quantum Optimization for Marketing

Recommended

Recommended

More Related Content

Similar to Machine Learning driven Quantum Optimization for Marketing

Similar to Machine Learning driven Quantum Optimization for Marketing (20)

More from Deep Learning Italia

More from Deep Learning Italia (20)

Recently uploaded

Recently uploaded (20)

Machine Learning driven Quantum Optimization for Marketing