Data-Driven Recommender Systems

Data-Driven Recommender Systems
Sequences of Recommendations
J´er´emie Mary
SequeL Team
Lille Nord-Europe

Recommendation
- SequeL - jeremie.mary@inria.fr 2
Set of something -Ad, items, movies or
music to recommend, . . .
We want to answer questions not clearly
stated by the user in a such way we
maximize the click/buying probability

Netflix Challenge (2006)
From xkcd.com
One million dollar challenge
Offline recommendation,
17 770 films,
480 189 users,
100 480 507 rates (with a timestamp),
Evaluation by RMSE using 1 408 342
”secret” rates. 0.9525 for the initial
model, million for the first 0.8572
Won in 2009 by the Belkor team.

Matrix formulation - Collaborative ﬁltering
Item1 Item2 Item3 . . . Itemm
User1 5 4 ? . . . ? ?
User2 5 4 4 . . . 1 5
... ? ? 1 . . . ? 5
Usern ? ? 2 . . . 4 5
Same formulation for many applications: image processing, latent
semantic analysis, factor analysis, clustering. . .
Goal: Minimize the Root Mean Square Error.
i,j
(mi,j − ˆmi,j)2

Low Rank formulation
Using the minimization formulation of the SVD we work on:
i,j mij known
(mij − ui ·t
vj − biasi,j)2
+ Ω(U, V)
With ui (resp vj) the ith row of U (resp jth column of V ).
Minimization is done by
stochastic gradient or
Alternating Least Squares.

Nice Theoretical Problem 1/2
Minimizing Ut
V ∗
where M ∗ = tr(MM∗)1/2 is the trace norm and with no
modiﬁcation of known values and with no noise.
We have a convex relaxation which is going to promote
sparsity on U and V.
[Cand`es and Tao, 2010] If the rank of M of dimension n × m
is k and has a strong incoherence property, with high
probability we recover from a random uniform sample
provided that the number of known rating is νk logO(1)
(ν),
where ν = max(n, m)

Nice Theoretical Problem 2/2
With Ω(U, V) = 0, zero padding and iterative thresholding of
singular values [Chatterjee, 2012] get a guarantee in
||M∗||1
m
√
np + 1
np with p is the fraction of -iid- observed values.
With Ω(U, V) = λ UtV 1, in the noisy setting and relaxed
conditions on the sampling [Klopp, 2014] achieve a bound on
the MSE in O(k
n log(m + n) max(n, m))
Can also be cast in the bayesian framework. This is known as
variants of Probabilistic Matrix Factorization. Allows insertion
of prior knowledge.

Nice Practical problem 1/2
Alternate Least Square strategy can be eﬃciently distributed
thought the Map Reduce paradigm and even better nodes can
cache a part of the data to minimize network cost. This is a
part of MLlib with Spark and of Vowpal Wabbit.
It is possible to obtain eﬃcient approximations of the
decomposition using randomly sampled subsets of given
matrices. The complexity can even be independent of the size
of the matrix using the length squared distribution of the
columns! [Frieze et al., 2004]

Nice Practical problem 2/2
Possible to play with regularizations and bias.
[Zhou et al., 2008] regularizes using:
Ω(U, V)
def
= λ


i
#J (i)||ui ||2
+
j
#I(j)||vj||2


J (i) (resp I(j)) is the number of rates given by user i (resp
given to item j). 0.89 RMSE.
Possible to play with the content of the matrix
[Hu et al., 2008].
Adaptations as generalized linear factorization -including
logistic one and to ranks [Guillou et al., 2014] and subsumed
by Factorization Machines [Rendle, 2010]

But
(Identically) Independently distributed observations
assumption
Guarantees holds for the estimated reconstructed matrix, not
for for the learnt factorization - U and Vt.
Netﬂix was won by a -never used in production- ensemble of
more than 100 models with an individual score around 0.91
The actual problem is not about minimizing a RMSE. We
want to use our decomposition online to make some
recommendations.

But
Existence of Hubs - very problematic for music recommender
systems.
Cold start problem both for new users and new items,
How to mix history of navigation, clicks, sales ?
How to handle additional constraints as lifetime and budgets ?
We face a sequential problem with some probabilities to estimate
and we need a policy.

Bandits
Items are the arms, group your users, and among each group play
the arm with the highest upper bound on the CTR estimate.
Example with UCB: ˆµi + α · log(T)
ni

Cumulative Regret
Cumulative regret after T step is
RT = Tµ∗
−
T
i=1
E(µπ
)
With bad policies you can face a linear growing of the regret.
There is an asymptotic lower bound:
lim inf
T→∞
E(RT )
log(T)
≥
k
i=1
∆i
KL(Pi ||P∗)
Many algorithms : εn-greedy, EXP3, UCB, UCB-v
KL-UCB,Thompson Sampling, Gitting indexes,. . . with a
matching upper bound for T large enough.
Exploration has a logarithmic cost in T.

LinUCB / Kernel UCB
O
Rk
û
confidence ellipsoid
v2
v1
˜u
(1)
n+1
argmax
j
û.vj(t)T
+α vj(t)A−1vj(t)T ,
α paramètre d’exploration
A = t−1
t =1 vjt
(t ).vjt
(t )T + Id.
Analysis in
[Abbasi-yadkori et al., 2011].
Extended to kernels
[Valko et al., 2013] and
generalized model
[Filippi et al., 2010].

Low rank bandits
At timestep t, user it is drawn randomly. Items are seen as bandits
arms with their description given by the learnt V matrix. Can be
reversed for new items [Mary et al., 2014a].
Close to a set of regularized set of LinUCB but the
factorization part does not allow to carry the analysis.
Contextual bandits regret bound is in O(d log2
T). Can we
take advantage of the lower dimensional space?
When some new items (or users) occurs at constant rate, the
regret growing will be linear.

The tighter conﬁdence bound the better ?
Yes! KL-UCB tends to be the best on controlled experiments.
Not really
In real world arms probabilities ﬂuctuates
LinUCB behavior is unclear for non-linear dependencies
We need to focus on evaluation on real datasets

ICML’11 Challenge
Item 1
Item 1
Item 1
Item 1
Item 1
Item 1
Item 2
Item 2
Item 3
Item 4
Item 4
Item 4
Item 4
Item 5
Item 5
Item 6
Item 6
Item 6
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1....
....
....
....
....
Batch 1
Batch 2
Batch N
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Some features
Item 1
Item 1
Item 3
Item 4
Item 5
Item 6 ?
?
?
?
?
Some features
Some features
Some features
Some features
Some features
.........
Some features ?
For each batch - sequentially -
the algorithm selects a display.
Some features Item 5 0
Only the reward of the selected
display is revealed for learning.
Goal: Maximize the sum
of revealed rewards.

ICML’12 Challenge
Yahoo! provided some data of their frontpage with random
uniform allocation of news.
Context Pool of current displayed Clic
(137 features) articles (around 30) article
x1 P1 a1 r1
...
...
...
...
xT PT aT rT

Evaluation [Li et al., 2011]
For a policy π the CTR estimate is computed using rejection
sampling [Li et al., 2011] on a dataset collected with a random
uniform policy.
h0 ← ∅ , GA ← 0, T ← 0
for all t ∈ {1..T} do
π ← A(hT )
if π(xt) = at then
hT+1 ← hT + {(xt, at, rt)}
GA ← GA + r, T ← T + 1
else
/* Do nothing, the record is completely ignored.*/
end if
end for
return ˆgA = GA/T

Remarks
Reported score is the CTR ∗ 10 000. Two rounds : only one
submission allowed for second round.
Only one data row out of K is used on average. With non
random policy we could derive similar algorithms at a cost of
an increased variance for some policies (those using action
with less samples).
The estimator is only asymptotically unbiased. It can be made
closer making use of the knowledge of the sampling
distribution.
The unbiased estimator is not admissible for MSE
[Li et al., 2015]. The diﬀerence is important only for action
with a small number of selection.

Results of ﬁrst round
q
q q q
q q q q q q
q q q q q
q
q
q q
q
q q
q q q q q q q q q q q q q q q q q q q q q
0 10 20 30 40
0200400600800
Participants
Scoreonfirstdataset
Complete list:
http://explochallenge.inria.fr/leaderboard/
Some methods where non contextual.

Overﬁtting / Results of 2nd
round
0 1 2 3 4 5 6
−250−200−150−100−50050
Number of submission (LogScale)
DifferencebetweenPhase2andPhase1
“bad” submissions
< always last
szatymaz
???
Jamh
● ●
winner
0 5 10 15 20 25 30
400500600700800900 Participants
Scoreonfinaldataset
1. The expert
Montanuniversitaet Leoben
2. José Antonio Martín (jamh)
Universidad de Madrid
3. Meng-Lun Wu (Allen)
NCU Taiwan
Winner
of phase 1
Second
of phase 1

ICML’12 Challenge - UCB-v
From [Audibert et al., 2009]
ˆµ = µ +
c · µ · (1 − µ) · log(t)
n
+ c ·
0.5 − µ
n
log(t)
with t current time step, n number of display of the news, µ
empirical mean of the CTR, c contant parameter.

Temporal eﬀects
From Bee-Chung Chen, time eﬀects on CTR for news.
Lot of news with low variance.

Bootstrapped replay on expanded data
We do not want to discard O(1/K) row of the dataset on average.
Under mild hypothesis using bootstrap we have an unbiased
estimator of the CTR distribution with a speed in O(1/L) where L
is the size of the dataset [Mary et al., 2014b].
+
++
+
+
+
+
+
+
+
++++
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+++++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
++
++
+
+
+
+ +
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+ +
++
++++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+++++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+++
+
+
+
+
++
++
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+++++
+
++++
+
+
+
+
+
++++
+
++
+
++
+
+
+
++++
+
+
+
+
+
+
++
+
+
+
+
+
++
+
++
+++
+
++++++ ++++
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+++++
+
+
+
+
+
++
+
+
+
+
+
+
+
++++++
+
+
++++
+++
+ +
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++++
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
++
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++++
+
+
+++
+
+
+
++
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
++
+
+
++
++
+
+
+
++
+
++
+
+++
+
+++
+
+
+++
+
++
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+++
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+++++
++
+
+
+
+
+
++
+
+
+
++
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+++
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+
+
++
+
++++
+
++++
+
+++
+
+ +
+
+
++
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
++
+
+
+
+
+
+
++++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+++++++++
+
++
+
+
++++++
+
+
++
+
++
+
+
+
++++++++
+
++
+
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++++++ +
+
+
+
+
+
+
+
+
+
+++++
+
+
+
++
+++
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
++++
+
+++++
+++
+
++
+
+++
+
+
+
+
+
+++
+
+
+++++++
+
++
+
+++++++++
+
+
++
+
++
+
+
+
+
+
+
+
+
+++
++
+
+
+
+
+
+
+
+
+
+ ++
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++++++++
++
+
+
+
+
+++
+
++++++++++
+
++
+
+
+
++
+
+
+
+
++++
+
+++
+
+
+
+
+
+
++
+
+
+
+
+
+++
+
+++
+
++++
+
+
+
++++++++++++ ++
+
+
++
+
+
+
++++++
+
++++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++++
+
+++++
+
+
++++++++
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+++
+
+
+
+
+
+
++++
+
+
+
++
+
+
+
+
+
++
+
+
+
++
+
+
++
+
+++++
+
++
+
+
+
+
+
+++
++
+
++
+
+
+
++
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+
+
+
++
+++
+
++++ ++
+
+
+
+
+
+++
+
++
++
+
+
+
+
+
++
+
+
++++++
++
+
+
+++
+
++ +++++
+
+
+++
++++++++++
+
+
++
+
+
+
+++ +
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
++
+
+++++++
+
+
+
+
+++++
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++++++
+
+++++++
+
++++
+
+++++
+
++
+
+
+
+++
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+++++++++
+
+++
+
+++++
+
+
+++
+
+
+
+
+
+
+
+
++++
+
++
+
+
+
+
++
+
+
+
+
+
+
+++++++
+
++
+
++++++
+++++++++++
+
++++++++ ++++
+
+++++
+
+++
+
+++++
+
+
+
+
+
+
+
++
+
+
+
++++
+
++
+
+++++++++ +++
+
++++++ ++++
+
++++
+
+
+++++
+
+
+
++++
+
+++
+
++ +++
+
++
+
++
++
+++
+
++++
+
+
++
+
++++
+
+
+
+++++++
+
+
++++++++++
++++++++
+++++++
+
++
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
++++++++
+
++
+
+
++++
+
++++
+
+++++++
+
+
+
+
+++
++
++
+
++
+
+
+
+
+
+
+
+
+++++
+++++++++
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+++
+
+++++++++
++++++++
+
+
+++
+
+++
+
+
+
++++++
+
+
++
+
+
++
+
+++++
+
+
+
+
+
++
+
++
+
+
++
+
+
++++
++
+
++++
+
+
+
++++++++++
++++++++
+
+
++
+
+++
+
+
+
+++++
+
+
+++++
+
++++++++
+++
+
+
++++ +++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
++
+
+++
+
+
+
+
+
+
+
+
+
++++++
+
+
+
++++++
+
+
++++
+
+++
+++
+
++++++
+
++
+
+
+
++++
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++++++++++
+
+
+
+
++
+
+
+
+
+
++
+
++
+
+
+
+++++++++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++++++
+
+
+
+
+
+
++
+
++
+
+++++++
+
++++++++++++
++++++++
+
++++
+
++++++
+
++++++++ ++++++++
+
+
+++
+
+
+
+
+++
++++++++++++++++++ +++++
+++
+
+
++++++++++
+
+
+
+++++++
+
+
+
+
+
+
+
+
++
++++
+
+++++
++
+
+
+
++++
+
+++
++
++
++
+
+++
+
+
+
+
+
+
+
+
+
+
+++++++
+++
+
+++++
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+++
+
+
+++++ +++++
+
++++
++++++++ ++++++
+
++
+
++++++++
+
+
+
+
+
++
+++
+++++++++
+
+++
+
+++++
+
+
+
+
+
+
+
+
+
+
+
+++++
+
+++ +
+
+
++
+
+
+
+
+
+
+
++++++++++++++++++
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
++
+
+++
+
+
+++
+
++++
+
+++
+
+
+
++
++
+
++
+
+
++++++
+
+
+
+
+
++++++++++
+
+
+++
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
++++++++
+
+++
+
+
++
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
++
+
+
+
+++
+
+++
+
+
++
+
+
+
+
++
+
++
+
++++++
+
+
++
+
+++++++++
+
+
+
+
+
+
+++++
+
++
+
+++
+++
+
++
+
+
++
++
+
++++++
+
+
++++
+
+
+
+
+
+++++++
+
++
+
++
+
+
+++
+
++++++++
+
+
+
+
+
+
+
+
+
+++
+
+
+
++++
+
++
+++++
+
++++
+
+
++++
+
++
+
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+ +
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+++++++++
+
+++++++++++++
+
++
+
+
++++
+
++
+++++++
+
++++++++++++
+
+
+++
+
+
+
++
++++++++++
++
+
+
++++
+
+
+
+
+
+
+
+
+
+
+
+
+
++++
+
++++
+
++
+
++++++
+
+
+
+
+
+
+
+
+
+
+
+
+++++++
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
++++
+++++++
+
++ +++
+
+
+
+
+
+
+
++++++++++
+
+
+
+
+
++
+
+
+
+++++
+
+++
+
+
+
+
++
+
+
++++
+
++++++++++++++++++
++++++++++
+++++++++
++++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++++++++
+
+
+
+
+
+
+
+
+
+
+
+
+
++++
+
+
+
++ +
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
++
+
++
+
++
+
++++++++++++
+
++++
+
++ +++
+
+
+
+
+
++
+
++
+
+++
+
+
+
+
+
+
+
++
+
+
++
++++
+
++
+
+
+
+
+
+++++
+
+
+
+
+
++
++
+
+
+
+++
+
+++++++ ++
+
+
+
++
+
+
+
+
+
+++++
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
++
+
+
+
+
++++
+
++++
+
++++++++++ +++
++
+
+
++
+
+
+
+
+
+++++
+
+++++++++++
+
+++++++++++++
+
+++
+++
+
++
+
+++
+++++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+++++
+
+++++++
+
+++++++++
+++
+
+
+
+++
+
+
++++++++
+++++++
+
+
++
+
++
+
++
+
+
+
++
+
+++++++
+
+++++
+
+
+
+++
+
+
+++++
+
+
+
++
++
+
+
++
+
++
+
+++++++++++++++
+++
+
+++
+
+
+
+++++++++++
+
+++++
+
++
+
++++++++
+
+
+
+
++
+
++
+
++++++
+
+++
+
+
+
+
+
++
+
+
+
+++++++++
++++
+
+
+++++++++
+
+++++++++++ +
+
++
+
++++
++++++++
+
++++++
+
++++
+
+
+
+
+
+
+++
+
+
+
+
+
+
++++
+
++
+
+
++++++ ++
+
+
+++
+
+
+
+
+
++++++
+
+
+
+
+
++
+
++
+
+
+++
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+++++
+
+
+
+
+
+++++
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+++
+
+
+++++
+
++
+
+
+
++++
+
+++
+
+
++
+
+
+
+
+
+
++
+
+++
+
++
+++++
+
++++
+
++
+
+
++
+++
+
+
++++
+
++
+
++
+
++
+
+
+
+
+
+
+
+
++
+++
+
+
+++++++++
+
+++++++++
+++
+
+++++
+
+
+
+
++
+
+++
+
+
+
+
++
+
++
+
+
+
+
+
+
+++
+
+
++
+
+++
+
+++++++++++++
+
+
+
++
++
+
+
+
++++++++
+
+
+
+++
+
+++++++++
+
+++
+
+
++
+
++
+
+
+
+
+
+
++
++
+
+
+
+
++
+
+
+
++
+
+
+
+++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++++++
+
+
+
+++
+++++++++
+
++
+
+
++++++
+
+
+
++
++
+
+
+
+
+
++++++++++
+
+
+
+
++
+
+
+++++
+
++++
+
+
+
++
+
++
+
+
+
+
+++++++
+
+
+++
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
++
++
+
+
+
+
+
++
+
+
+
+
+
+
+ +
+
+
++
++
+
+
++
+
+
+
+
+
++
+
+
+
+
++
++
++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
++++++++++
+
+
+
++
++
+
+
+
++
++
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+++++++
++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+++
+
+
++++++
+
++
+
+
+
+
+
++
+
+++++
+
+++++
+
+++
+
+++
+
++++
+
+++++++
+
+
+
+
+
+
+
+
++
+
+++
+
+
+ +
+
+
+
++
+
+++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++++++++++ +++
+
+
+
++++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+++++++++++
+
+ +
++
+
+
+
++
+
+
+
+
+
+
+
++++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++++
+
++
+
+++
+
++
+
+
+
+
+
++
+
++
+
++
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
++
+
+
+
+
+
+
+
++
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +
+++
+
++
+
++
+
+
++
+
++++
+
+
+
+
+
+
+
++++
+
+
+
+
+
++
+
+
+
+++++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
++
+
+
+
+
+
+++
+
++++++
+
++
+++
+
++++
+
+++++++
+
+
+
++++
+
++++
+
+
+
+
+
+
+
+
+
+
+
++
+
++
++
++++++
+
++++++
+
++
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++++
+
+
++
++++
+
++
+
+
+
++
+
+
+
+
+
+
+++
+
+++
+
++
+
+
++
+
+
+
+++++
+
+++++
++
+
+
+
++
+
+
+
+
+
+
++
+
+
+
++
+
+
+++
+
++++
++++++++
+
+
+
+
+
+
+
++++
+
++++++++
+
++++++
+
+++
+
+
+
+
+
+
+
+
+
+
++++++
+
++++
+
+++++++
+
+
+
+
+
+
++
+
+
+
+
++
++
+
++
+
+
++
+
++++++++
+
+
+++++
+
+
+
+
+
+
+
++
+
++ +
+
+++++
++
+
+
+
+
+
++
+
++
+
+
+++++
+
++++
+
++
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
++
+
+++++
+
+
++++
+
+
+
+
++
+
+
+
+
+
+
+++
+
+++
+
++
++
+++++++++++++++++
+
+++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+++
+
+
+
++
+
+
+
++++
+
++++
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+++++
+
++++++++++++++
+
+
+
+
+
+
+
+
+
+
++++
+
+++++
+
+
+
+
+
++
+
+
+
+
++
+
++
+
+
+
+
++++++++++ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++++
+
+++++++++ +
+
+++++
+
++
+
++++++
+
+
+
+
+
++
+
++++++++++++++++++++++++++++++++++++++
+
+++++++++++++
+++
+
+
+
+
+
++++++++++++
++
+
+
+
+
+
+++
++++++++++
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
++++
+
++
+
+
+
+
++
+
+
+
+
+
++
+++
+
++
+
+++
+
+++
+++
+
+++++++
+
+++++
+
+
+
+
+++++++++
++
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
++
+
++
+
+
+
+
+++++
+
++++
+
++++++++++++
+
+
+
++
+
+++
+
+++++++ +
+
+
+
+
+
++++
+
+
++
+
+
+
+
++
++++++
+
+++
+
+
++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++++++++++++
+
+
+
+
++
+
++
+
++
+
+
+
++
+
+
+
+
+
+
+
++++++++++++
+
+
+
+
+
++++++
+
+
++++++++++++
+
+++++++++
+
+
+
+++++++
+
+++
+
+
++++++++
+
+++
+
+ +
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+++++++++++++++++
+
+
+
++
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+++
+
+
+
++
+
+
+
+
+
+++
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
++
+
+++
+
++++++++++++
+
+
++
+
+
++
+
+
+
+++
+
+
++
+
+
++
++
+
+
+
+
++++++
+
+++++++ +++
+
+
+
+
+
++
+++++++
+
+
+
+
+
+
+
+
+
++++
++
+
++
+
++++ +
+
+
+
+++
+
+
+
+
+
+++
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
++
+
+
++
+
+++
+
++++++++++++ +
+
+
+
+
+
+++
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+++
+
+
+
+++++
+
++
+
+
++
+
++++++++++++++
++
+
++
+
++++
+++++++++
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+++
+
+
+
+
+
++++++
+
++
+
+
++
+
+++++++ +
+
++
+
+
++
+
+
++
+
+++
+
+++
++
++
+
++
+
+
+
+
+
++
+
+
++
+
+ +
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+++
+
+
+
++++++++++++
++++
+
+
+
+
+
+
++
+
+
++
+
+
+
+++
+
+
+
+
+
+
+
+
+++++
+
+
+
+
+
+
++
+
++++++
+
++
+
++++++
+
+
+++++
+
++
+
+++++++
+
+
+
++
++
+
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+++
+
++
++++++
+
+
+
+
++
+
+++++++
+
+
+
+
+
++
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++++
+
+++
+
+
+
+
++
+
+++
++
+
+++++
+
+
++
+
+++++++++++++++++
+
+
+
+
+++
+
++
+
+
+
+
++
+
++
+
+
++
+
+
+
+
+
+
+
++++++
+
++
+
+
+
+
+
+++
+
+
+
+++
+
+
++
++
+
+
+
+
+
+
+
+
+++
+++
+
++++++ +
+
+
+
++
+
++
+
+
++++++++++
+
++++
+
++++
+
+
+
++
+
++
+
+
+
+++
+
++++
+
+
+
+
+
+
+
+
++
++
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+++++++++++
+
+
+
+++
+
+
+
++
+
++
+
+
+++ +
+
+
+
+
+++++
+
++++++
++
+
+++
+
+
+
+
++++++
+
+
+
+
+
+
+ +
+
+
+
+
++
+
+
+
+
++
+
+
+
+
++
+
+
+
+
++
+
+
++
+
+
++++++++
+
+
+++
+
+
+
++
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
++
+
+
++++
+
++
+
+ +
+
+
++
+
++
+
+++++
+
+
+
+
+
+++++++++
+
++++++
+
+++++++++++++++++++++++ +
+
+++++
+
+
+
+
+
+
+
+
+
+
+
++
+
+++++
+
+
+
+
+
+
+
+
+
+
++
+
+
+++
+
++
+
+
+
+
+
++
+
+
+
+++++
+
+
++
+
+
+
+
+
+
+
++++
+
++++++
+
+++
+
++
+++++++
+
++
+++
+
+
+
++++
+++++++++++++++++++++
+
+
++
+
+
+
+ ++
+
++++++++++++
+
++++
+
+
+++
+
+
+++
+
+++++++
+
+ ++
++
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++++++++
+
+
+
++++++++++++
+
+
++
+
+
+++
+
+
++
+
+
+
++++++++++
+
+
+
+
+
+++++
+++
+
++++++
++++++++++ +
+
+++
+
+
++
+
+++++++
+
+
+
++
+
+++++++++++++++++ ++
+
+
+
+++++++++++++
+
+
+
+++
+
+
++
+
+++++++++++ ++
+
+++
+
++
+
+
+
+++++++
+
+
+
+
++
+
+
+
+
+++++++++
++++++++++
+
+
++++++++
+
+
++++
+
+
+
+ +
+
+
+
+++++
+
++++++++++ +
+
++++
+
+
++
+
++
+
+++++++
+
+++++++
+++++++++
++++++++++ +
+
+
+++
+
+++
+
+
+
+
+
+
+++
+ +
+
+
+
+
+++
+
+
+
+
+
+
+
++
+
++ +
+
+
+
+
++
+
+
+
+++++++++
+
++++
+
+
+
++
+
+
++++++++
++
+
++++
+
+
+
+++
+
+
+
+++++
+
++
+
+
+
+++ +++
+
+++
+
++++
+
++++++++
+
++++++++
+++++++
+
++
+
+
+
+
++
+
+
+
+ +
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
++
+++
+
++++++
++++++++
+
+
+
+
++
+
+
+
+
+
+
+++
+
+
+++++
+
+
+
+
+++
+
+
+
+
+
+
+++
+
+++
++
+
++
+
++
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+++++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+++++++
+
+
+
+
+
+
+
++
+
++
+
+++++
+++++
+
++ +
+
++
+
+
+
+
++
+
++
+
+++
+
+ +++
+
++++++
+
++
++
+
+
+++
+
+
+
++
+
+
+
+
+
++
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+++
+
++++
+
+
+++
+
+
+
++++++++++
+
++ +
+
+
+
+
+
+
++
++
+
+++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++++++
+
+
+
++
+
+++++
+
+
+
+
+
+
+
+
+
+ +
+
+
+
+++++
+
+
+++++++
+
+
++++
+
+++
+
+
+
+++++++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++++++
+++++
+
+++++
+
+++++++
++++++++++++++++++++++++++
+++++++++++++++++
+
+++++
+
+
+
+
+
+++
+
+++++
+
+
+
+
+
+
+
+
++++++++++
+
++
+
++++
+
+
+
+++++++
+
+ +
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
++
++
+
+
+
+
+
+
++
+
+
+
+ +
+
+
+
+
+
+
+
++
++++++++
++
+++
+
+
+
++
+
+
+
+
+
+
+
+++
+
+
+++++
+
+
+++
+
+
+
+
+
+
++++ +++
+
+
+
+
++
+
++++++++++
+
++
++
+
+
++
+
+
+++++++++ +++
+
++++++++++
+
++++++++
+
++++
+
+++++++
+
+
+
++
+
+
+
+
+
++
+
++
+
+++
+
+
+
++
+
+++
+
+
+
+
+
++
+
+
+
+
+
++
++++++++++
+
+
+
+
+
+
++++
++
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
++
+
++++
+
++++++++++
+
+++++
+
++++
++
+
+
+++++
+
+
+
+
+
+
+
+
+
+
+
+++++++++
++++++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+++
+
+
+
++
+++++++++
+
+
++++
++
++
++
++
+
+
+
+
+
+
+
+++
+
+++++++++++++++
++++++++++
++++++++++
+
+
+
++
+
+
+++
+
++++
++++++++
+
+
+
+
+
+
+
+
++
+++++
+
+
+
++++++++
+
+++++++
+
+
+
+++
+
++
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++ +
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++++
+
+
+
++
+
+
+
+
+
++++
+
+++++ +
+
+
++
++
+
+
+
+
+
+
++++
+
++
++
+
++
+++
+
+
+
+
+
+
+
+++
++
+
+
+
+
+
++
+
++
+
++
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+++
+
+
++
+
+
+
+
+
+
++
+
+
+
++++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
++
++
+
+++
+
+
++
+
++++
+
++
+
+
++
+
++
+
++
+
+
++++
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++++++
+
++
+
+
+
+
+
+
+
+
+++
+
++
++
+
+
+
+
+
+
+
++++
+
+
+
+
++
+
+++
+
+
++
+++++++
+
++
+
+
+
++++++
+
+
+
++++
+
+
+
+
++
+
+++
+
+
+
+ ++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++++++++
+
+
+
++
+
+
++++
++
+
+
+
++++
+
+
++
+
+
+
+++
+
+
++
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+++++++
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
++
+
+
0 50000 100000 150000 200000 250000 300000
−0.050.000.050.100.150.20
Dataset size
EstimatedCTR−ActualCTR
+++
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
++
+
++
+
+
+
++
++++
++
+
++
++
++
+
+
++
+++
+
++
+
+
+
+
+
+++++
+
+
+
+
+
+++++
+
++
+++
+
+
+
++++
+
+ ++++++
++++
+
+
+
+
+++
+++ ++++++
++++
++++
+
+
+
+++ +++
+++
++
+
+
+++
+
+
++
+
++
+
+
+
+
++
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+++
+
+++++++
++
+
+
++
+++
+
+++
++
+++
+++
+
+
+++
+
+
+
+
+++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
++
+
+
++
++
+
+
++
+
+ +
+
+
+
++
+++
+
+
+
+
+
+
+
++
+
+ ++
+
+
+
+
+
++
+
+++
++
+
++++
++
+
+
+++
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
++
++
+
+
+
++
++
+
++++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+++
+
++
+
++
+
+
+
+ +
+
+
++
+
++
++
++
+
+++
+
+
++
+
++++
+
+++++
+
+
+
++++
+
+ ++++
+++
+
++ +
+
+
+
+
+
++
++
+
++
++
+++
+
+
+++++
+
+
+
+
+
+
++
+
+
++
++
+
+
++++++
+
+++
+
++
+
+
+
+
+
+
+++
+
+
+++
+
+
++
++++++
+
+
+++
++
+
+
++
+
++
+
++
+
+
++
+
++++++++++
+
++++
+
+++
+++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +
+++
+
+
+
++
+
++
+
+
+
+
++
+
+
+
++
+
+
++
+
+
+
+++
+++++
+
+
+
++
++
++
+
+
+
++
+
++
+++
++
+
+
++
++
++
++
++
++
++
+
+++ ++
+
++++
+++
+
+
+
+
+++
+
+
+
+
+
+
++
+
+
+
++ ++
+
+++++
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +
++
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+++++
+
++
+
++
+
+++
+++++
+
++
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+++
+
++
+
+
+
+
+
+
+
++
+
+++
+
+
+
+
++
+
+
+
+++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+++
+
+
++
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++++
+
++
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+
+
+
+
++
+
++
+
+
+
+
+
+
++
+
++++
+
+
+
+
+++++
+
+
++++++++
+
+
+
+
+
+
+
+
+
+
++
+
+
+++
+
+
+
+
+
+
++
++
+++
+
++++++
+
+
+
+
++
+++
+
+
+
+ +
+
++
+
++
+
+++++
+
+
+
+
+++
+
+
++
+
+
+
++
++
+
++
+
+
+
+
+
+
+
+++
+
++
+++
+
+
+
+
+
+
++++ +
+
+
++
++++
+
+++
+++++
++
+
+
+
+
+
+
++
+
++
+
+
+
+
+
+
+
++
+
+++
+
+
++
+
++
+
++
+
+
+
+
+
++
+
+
+
+
+
+
++
+ ++++++++
+
+
+
+++
+
++
+
+
+
+
+
+
+
+
+
+++
+
++
+
+
++
+++
+
+
+++
+
+
+++
++++
+
+
+++++
+
++++
+
++
+
+++
++
++
+
+
+
+
+
+
+
++
+
+
+
++ ++++
++
+
++
+
++
+
+
+
+
+
++++
+
+++
+
+
+
++
+
+
+
+
+
+
++++
+
+
+
++
++
++
+
+++++
++
+
+
+
+
+++
+
+
++
+
+
+
+
+++++
+
++ +
+++
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
++
+
+++
+++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+ +
+
+++
+
+
+
+++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+++
+
+
+
+
+
+
+
+
+++
+
+
+++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+ +
++
++
+
++
+
+
+
+
++
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+++
+++
+
++
+++
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+++
+
+
+
+
+++
+
+
++
+
+
+++
+
+
+
+
++++++++ +
++
+
+
+
+
+++ +
+++++++++
+
++
+
+++++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+++
+
+
++
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+ +
++
+
+
++++
+++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+++
+
+
+
+
+
+
+
++
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
++
+
+
++
++
+
+
++
+
++
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
++++
+
+
+
+
+ ++
+
+
+
+
++
+
+++
+
+
+
++
+
+
+
+
+++
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
++
+
+++
+
+
+
+
+
+
+
+
+
+
++++
+
+
+
+
+
+ +
+
+
+++
++
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+ +++
+
+
+++
++
++
+++
+
+
+
+
+
+++
++
++
+
+
+
+
+
+
+++
++++
++
+
+
+
++
+
++
+
+
+
++
+
+
+
+
+
++
+
+
++
+
++
+
+
+
+++++
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
++
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+++
+
+
+
+ ++
++
+
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+++
+
+
++
+++
+
++
+
+
++
++++
++
+++++++
+
++
+
+
++++
+
+
+++
+
+
+
+
+ +
+
+
+
+
++
++
+
+
+
++
+
+
++
+
+
+
++++
+
+
+
+
+
+
+
+
+
+
++++++++
+
+++
+
++
+
+
+
++
+
+
+
+++
++++++++++
+
+++
+
+
++
+
+
++
+
+
+
+
+
+ +
++++
+
+
+
+
+
+
+++
+
+
+
++
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+++
+
+
+
++
+
+
+
+
++
+
++
+
++
+
+
+
++
+
+
+
+++
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ ++++++
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
++
+
+
+
++
+
+
+
+
+
++
++
++++
+
+
+++
++++
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+++
+
+
+++
+
+
+
++
+
+
+
+
+
+
+++
++++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++++
+
+
+
++
+
+
+
++
++
++
+
+
+
+
+
+
+
+++
+
++++
+++
+
+
++++
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
++
+
+
+
+
+
+
++
++
+
+
+
++
+
+++
+
+
+++
++
+
+
+
+++
+
+
+
++
+
+
++
+
++++
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
++
++
++
+
+++
+++
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+ +
+
+
+
+
+
+
++
++
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+ +
++
+
+
++
+
+
++
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ ++
+
+
+
+
+
++
+
+
++
+
+
+
+
++
+
+
+
++
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
++
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+ +
++
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
++
++
+
+
+
++
+
+
+
+
++
+
+
+
++
+
++
+
+
+
+
+
+
+ +
+++
+
+
+
+
+
+
+
+
+
+
++
+++
+
+++
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+
+
++
+++
+
+
+
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+++
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
++ ++
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+++
+
+
++
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+++
++++
++
+
+
+
+
+
+
+
+
+
+ ++
++
+
+++
+
+
+
++
+
+
+
+
+
+
+ ++++
++
+
+++
+
+++
+
+
+++
+
+
+
++
++
+
+
+
+
++
+++
++
+
+
+ +++++++
++
+
+
+
+
++
+
+
+
+
+
+
+
+++
+
+
++++
+
+
+
+
+
+
+
+
+
+
+++
+
++
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
++
+
+
+
+
++
+
++
+
+
+
+
+
+
++++++
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
++
+
+
+
++
+++
+
+
+
+
+
+
+
+
+
++
+
++
++
+++
+
+
+++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
++++
++
+++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
++
+
+
++
++
++
+
++++
++
++
+
+
+
++
+
+
++
+
+
+
+++++
+
+
++
+
+
+++
+
++
++++
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
++
+
+
+
+
++
+
+
+
++
+++
+
++
+
+
++++
+
+
+
+
+
+
+
+
+
+
++
+
++
+
++
+
+
++
+
+
+
+
+++
+++
+
+
+
+
+
++
+
+
++
+
+
+
+++
+
++
+
++
++
+
+
+
+
+
++
++
+
+
+
+
++
+
+++++
++++++++
+
+
+
+++
+
+
+
++
+
+
+
+
++
+
+
+
++
+
+
+
+
+
++
+
+
++
+
+
+
+ ++
+++++
++
+
+
+
+
+++
+
+++
+++
+
+
+
+
+
++
+
+
++
+
+
+++++
+++
+
+
+
+
+
+
+
+
+++
++
+++
+
++
+
+
+
+
+
+++++
++
+
+
+
+
+ ++++++++
++
+
+
+
+
+
++++
++
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
++
+
+
+++
+
+
+
+
+
+
+
+
+
+++ ++
++
+
+++
++
+
+
+
+
+
+
+
+
++ +
+
+
+++
++
+
+
+
++
++
+++
+
+
++
++++++
++
+
+
+++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+++
+
+
+
+
+++++++
+ +
+
+
+
+
+
+
+
+
+++
+
+
+
+
++
+
+
++
+
+
++
+
+
+
+ +
+
+
+
+
++
+
+
+ ++
+
+
+
++
+++
+
+++++++
++++
+
+
+
++
++
+
+
+
+
+
+
++
+
+
+
++++
+++
+
++++
+
++
+
+
+
++
++
+
+++
+
+
+
+
++++
++
+
+
+
+
+
+
+++
+
+++
+
+
+
+
++
+
+
++
+
++
+
+++
+
+
+
+ +++++++
+++
+
++
+
+
+
+
+
+
+
+++++
+
+++++++
+
+
+
+
++
+++
+
+
++
+
+++
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
++++
+
+
+
+
+
+
++
+
+
++++++++
+
+
++
+
++
+
+
+
+
+++++++
+
+
+
+
+
++
+++
+
+
+
+
+
+
+
++
+
++
+++++++
++
+
+
+
++
++
++
+
++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+++
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
++
++ ++
+
+
+++
+
++
+++
+
+
++
+
+
+
+
+
+
+
++
+
+
+
++
+
++
++
+
+++
+
+
+
+
++
+
++
+
+
+
+
+
++
++
++
+
++
+
+
+
+++
+
+
+
++
+
+
+
+
+
+
+
+
+
++++
+
+
+ +
+
+
+
+++
++
+
+++
++
+
+
++
+
+
+
+
++
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++
+
+
+
+
+
+
+
++
+++
+
++
++
+
+
+
+
++
+
++ +
+++
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+++
+ +
++
+++++++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++ +
+++
+
+
+
+++
+
+
+
++
+
++
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
++
+
+ +
+
+++
+
++
+
+
+
+
+
+
+
+
+
++
+++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+++
+
+
+
+
+
++
+
+ ++
++
++
+++
+
+
+
+
+++++++
++++++
+
++
+
+
++
+
+
+
+++
+ ++++
+
+++
++++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+ ++
+
+
++
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
++
++
+
++++
+
+++++
+
++
+
+
+
++
+
+
+
+
+
+
+
+
++
++
++++
+++
+
++
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
++
+
++
+
+++
++
+++
+
++
+
++
+++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+++ ++
++
++
+
++
+
++
+
+
++
+
++
+
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++++
++
+++++
+
+
+
++
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+++
++
+
+++
+
+
+
+
+
+++
+++
++
+
+
+++
+
+
+
++
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++++++
++
+
++
+
+
+++++
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+ ++
+
+
+
++
+
+++
++
+
+
+
+
+
+
+
+
+++
+
+
+
++
++
+
+
+
+
++
+
+
+
+++
+
+
+
+
+
+
+
+
+
++++
++
+
++
+
++
++
+
+
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
++++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
++ +
+
+
++
+++
+
+
++
+
+
++++++
++
+
++++
+
+
++
+++++
+++
+
+
+
++
+
+
+
+
++
+
+
+
+++++
+
+
+
+
+
+
+
+
++
+
+
+++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++
++
+
+
+
+++ +++++++
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++
++
++
+++
+
++++
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+++++
+
++
+
+++
+
+
+
+
+
+
+ +++
+
+
+
++
+++
+
+
+
+
+
+
+
+
+
+
++
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+++
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
++
+
+
+ +
+
+
++
++
+
++
+
+
+
+
++++
+
+
+
+
++
+
+++
+
+
+
+
+++++
+
++
+
++
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
++
+
+++
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
++
+
+
+
+
+
++
+
+
+
+
+
++
+
++
+
+
+
+
++
++
+
+
+
+++++
+
+
+
+
++
+
+
+
++
+
+++
+
++
+
+
++
+
+
++
+
+
+
+
++
+
++
+
+
+++
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+ +
+
+
++++
+
++
+
+
++
+
+
++
+
+
+
+
+
++
+
++
+
+
+
+
++
+++
+
+
+ +
+
+
+
++
++
++
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+++
+
+
+
+
+++
+
+
+
+
+
+
++
++
+
+
+
++
+
+
+++
+
+
+
+
+
+
+
+
++
+++
+
+
+
+
+
+
+
+
+++
+
+
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+++
+
+
+
+
+
+
+
+
++
++
++
+
+
+
+
+
+++
+
+
+
+
+++
+
+
+
++
+
+
+
+ ++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
++
+++
+
+
++
+
+
+
++
+
+
+
++
+
+
+
++
+
++
+
+
+
++
+
+
++
++
+
+
++
+
+
++++
+
++
+
+
+
+
+
+
+
+++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++
+
+
++
++
+
++
++
++++
+
+
++
+++
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+++
+
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
++
+
+
+
+
+ +
+
++
+
+
++
+++
+
+
+
++
+
++
+ +
++
++
++++
+
+
+
++
+
+
+
++
++
+
++
++
+
+
+ +
+
+++
++
++
+ +
++
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+++
+
+
+
+
+
+
++++
++
+
+
+
++
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++
+
+
+
+
+
+
+++
+++
+
+
++++
+
+++++
+
+
++
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+++
++
+
++++++
+
+
+
++
+
++
+
+
+
++
+
+
+
+
+
++
+++
+
+
+
+
+
+++
+
++++++
+
++
+
+
++
+
+
+
+
+
+++
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+++
+
+
+
+
++
+
+
+++
+
+
+
++
+
+
++
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+++
+
+
+
+
+
++
+++
+
++
++
++
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
++++
+
+
+++++++
+++
+
+
+
+
+
++
+
+
++
+++
+
++
++
+
+
++++
+
+
+
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+++
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
++
++++
+
+
+
+
+
+
+
++
+
+
++
+
+
+
++
+
+
+++
+
++
+
+
+++
+++++++
+
+
+
+++++
+
+
+
+
+
++
+
++
+
+++++
+
+++++
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++
++
+
+++
+
+
+
++
+
++
++
++
+
+
+
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+++
+
+
+
+
+
+
+
+
+++
+
++
+
++++
+
++
++
+
+
+
+++
+
++
+
+
++++
++
+
++
+
+
+
++++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+++
+
+
+
+
+
+
+
++
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+ +++
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+++
+++
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
++
+
+
+
+
+
+
+
+
++++
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+++
++
+
+++
++
+
+
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
+
+
++
+
+
+
+
+
++++
+
+++
+
++
+
+
+
+
++
+++
+
+
+
+
+
+
+
+ ++
+
+
+
+++++
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
++++
+
+
++
++
+
+
+
+
+++++
+
+
+
++++++
++
+
++++++++
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
++++++
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+ ++
++
+
+++
+
+ ++++
+++
+
++
+
+
+
+
+
+
+
+
++
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
++++
+
+++++
+
+
+
+++
+
+
+
+
+
+
+
+ +++++
++
+
+
++
+
+
+
+
+
++
+
+
+++
+
++
+
+
+
+
+++
++
+
++
+
+
+
+
++
+
+
+
+
+
+ ++
+
++
+
++
+++
+
+++
+
++
++
+++
+
+
++++++
+++
+++
+
++
+
+
+
+
+
+
+
+
+
+ ++++
+
+
++
++ +
+
++++++++
+
+
+
+
+++
+
+
+
+++
+++
+
+
++
+
+++++++
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
++
+
++
+
+
+
+
+
++
+
+
+
+++
+ +
+++++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
++
+
++
+
++
+
+
+
+
+++
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
++
+
+++
+
+
+
++
++
++
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++++
+
+++
+
+
+
+
++
+
+++
+
+
+
+
++
++++++
+
++
+
++
+++
+
+
Replay
BRED
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0 5 10 15 20
Bandwidth: the amount of Jittering (x 1/√L)
replay or
S-BRED, any B, no Jitter
MAE
Absolute bias

Limited Stock/Budget Impact
Comparison :
Greedy allocation vs Stock aware allocation.
Stock 100 100
Item 1 Item 2
100 → Profile 1 0.8 0.1
100 → Profile 2 0.8 0.5
Stock 100 100
Item 1 Item 2
100 → Profile 1 50 - 62.5 3.75 - 37.5
100 → Profile 2 50 - 62.5 18.75 - 37.5
Budget 100 100
Item 1 Item 2
100 → Profile 1 80 - 100 0 - 0
100 → Profile 2 20 - 25 37.5 - 75
Independent allocation.
Result : 122.5 sales
Optimal allocation.
Profile 1 : 100% for Item 1, 0% on Item 2
Profile 2 : 20% for Item 1, 75% on Item 2
Result : 137.5 sales

Adaptive Linear Programing
With known click probabilities and visitor’s distribution we
could use linear programming under constraints to solve the
problem.
With unknown probabilities we can adapt the optimistic
strategy to explore/exploit with the allocation policy
[Gabillon et al., 2010] including with some conﬁdence
guarantees [Girgin et al., 2011].
advertising campaigns – an approximate planning approach 5
e highest
employ a
bability of
expected
hastic ex-
ith possi-
with lower
mes of the
heir click
nces even
programming solution mentioned before.
Observing Fig. 3, it is easy to see that the interactions be-
tween the advertising campaigns materialize as overlapping
Fig. 4 The timeline divided into intervals and parts. Ij denotes the jth in-
terval [t , t ] and a denotes the allocation for advertising campaign Ad

Life Long Learning
Current policy  Randomization to maximize
the evaluation of promising.
Convex optimization
Randomized policy rand
New dataset
Go live with rand.
Promising
areas
Research of new policies:
Within the computed confidence interval (CI), a policy
does not exceed the maximum acceptable risk.
new



new

rand

promising

Analytic CI
Bootstrap CI
Learning
process
Can be automated!
[Bottou et al., 2012] proposed a step in this direction with an
explicit model of dependancies.

Less Supervision Required
Multitask learning / Tensor
formulations, better
aggregation
Back to Reinforcement
maybe with several actions
choices. Use of the
estimates is more important
than their quality - 55,
Nuukik.
Deep learning is coming.
Already able to win the
Netﬂix challenge using
User/Items sparse
autoencoders.
I1
I2
I3
I1
I2
I2I3
X1
I1
I2
I3
I2
I1
I1I3
X2
I1
I2
I3
I3
I2
I1I2
X3

Conclusion on Recommender Systems
Theory provides a lot of different attack angles,
Recommendation is a sequential problem,
Evaluation is the fundamental difficulty,
We can expect more from the flow of data,
Future is the open loop and requires collaboration with
compagnies.

Bibliographie I
Abbasi-yadkori, Y., Pal, D., and Szepesvari, C. (2011).
Improved algorithms for linear stochastic bandits.
In Advances in Neural Information Processing Systems 24th
(NIPS), pages 2312–2320.
Audibert, J.-Y., Munos, R., and Szepesvári, C. (2009).
Exploration-exploitation tradeoff using variance estimates in multi-armed bandits.
Theor. Comput. Sci., 410(19):1876–1902.
Bendersky, M., Pueyo, L. G., Josifovski, V., and Harmsen, J. J. (2014).
Up next: Retrieval methods for large scale related video suggestion.
In Proceedings of KDD 2014.
Bottou, L., Peters, J., Quiñonero Candela, J., Charles, D. X., Chickering, D. M., Portugualy, E., Ray, D.,
Simard, P., and Snelson, E. (2012).
Counterfactual reasoning and learning systems.
Technical report, arXiv:1209.2355.
Candès, E. J. and Tao, T. (2010).
The power of convex relaxation: Near-optimal matrix completion.
IEEE Trans. Inf. Theor., 56(5):2053–2080.

Bibliographie II
Chatterjee, S. (2012).
Matrix estimation by universal singular value thresholding.
pre-print.
Filippi, S., Cappe, O., Garivier, A., and Szepesvári, C. (2010).
Parametric bandits: The generalized linear case.
In Lafferty, J., Williams, C., Shawe-Taylor, J., Zemel, R., and Culotta, A., editors, Advances in Neural
Information Processing Systems 23, pages 586–594. Curran Associates, Inc.
Frieze, A., Kannan, R., and Vempala, S. (2004).
Fast monte-carlo algorithms for finding low-rank approximations.
J. ACM, 51(6):1025–1041.
Gabillon, V., Mary, J., and Preux, P. (2010).
Affichage de publicitur des portails web. (best paper award).
In 10e Extraction, Gestion des Connaissance (EGC).
Girgin, S., Mary, J., Preux, P., and Nicol, O. (2011).
Managing advertising campaigns - an approximate planning approach.
In Frontiers of Computer Science.
Guillou, F., Gaudel, R., Mary, J., and Preux, P. (2014).
User Engagement as Evaluation: a Ranking or a Regression Problem?

Bibliographie III
Hu, Y., Koren, Y., and Volinsky, C. (2008).
Collaborative filtering for implicit feedback datasets.
In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, ICDM ’08, pages
263–272, Washington, DC, USA. IEEE Computer Society.
Klopp, O. (2014).
Noisy low-rank matrix completion with general sampling distribution.
Bernoulli, 20(1):282–303.
Li, L., Chu, W., Langford, J., and Wang, X. (2011).
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms.
In Proc. Web Search and Data Mining (WSDM), pages 297–306. ACM.
Li, L., Munos, R., and Szepesvári, C. (2015).
Toward minimax off-policy value estimation.
In Lebanon, G. and Vishwanathan, S. V. N., editors, Proceedings of the Eighteenth International Conference
on Artificial Intelligence and Statistics, AISTATS 2015, San Diego, California, USA, May 9-12, 2015,
volume 38 of JMLR Proceedings. JMLR.org.
Mary, J., Gaudel, R., and Preux, P. (2014a).
Bandits Warm-up Cold Recommender Systems.
Research Report RR-8563, INRIA Lille ; INRIA.

Bibliographie IV
Mary, J., Nicol, O., and Preux, P. (2014b).
Improving offline evaluation of contextual bandit algorithms via bootstrapping techniques.
In Proc. ICML, JMLR WCP, Beijing - China.
Rendle, S. (2010).
Factorization machines.
In ICDM 2010, The 10th IEEE International Conference on Data Mining, Sydney, Australia, 14-17
December 2010, pages 995–1000.
Valko, M., Korda, N., Munos, R., Flaounas, I., and Cristianini, N. (2013).
Finite-time analysis of kernelised contextual bandits.
In Conference on Uncertainty in Artificial Intelligence.
Weinberger, K., Dasgupta, A., Langford, J., Smola, A., and Attenberg, J. (2009).
Feature hashing for large scale multitask learning.
In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pages
1113–1120, New York, NY, USA. ACM.
Zhou, Y., Wilkinson, D., Schreiber, R., and Pan, R. (2008).
Large-scale parallel collaborative filtering for the netflix prize.
In Proceedings of the 4th international conference on Algorithmic Aspects in Information and Management
(AAIM), pages 337–348, Berlin, Heidelberg. Springer-Verlag.

Spotlight on fast low rank approximation (1/3)
Let p1, p2, . . . , pn be nonnegative reals adding up to 1. Pick
j ∈ {1 . . . n} with probability pj.
For any vector v, consider the vector valued random variable
X =
M(j)vj
pj
Then E(X) = Mv. So X is an unbiaised estimator of Mv and
Var(X) =
n
j=1
M(j) 2v2
j
pj
− Mv 2
- jeremie.mary@inria.fr 35

So if pj =
M(j) 2
2
M F
, after s samples if we note Y the random
variable which avegages the realizations, then E(Y ) is an estimator
of Mv with
Var(Y ) =
1
s
M 2
F v 2
→ fast stochastic matrix vector multiplication.

Let j1, . . . , js be s i.i.d. random variables taking values in {1 . . . n}
such as the probability that j1 is equal to i ∈ [n] is proportional to
M(i) 2
2.
Let B a matrix such that its ith column is 1√
spji
Mji . Let u1, . . . , uk
be the k top left singular vectors of B. A low rank approximation
to M is
˜M =
k
i=1
ui ui M.
One can prove that for s = 4k/ε2 this approximation satisﬁes:
E M − ˜M 2
F ≤ M − M∗ 2
F + ε M 2
F
where M∗ is the best rank k approximation to M.

Factorization Machines
Vowpal Wabbit in [Weinberger et al., 2009] adds the hashing trick
which is a kind of regularization.
Won the Criteo challenge on Kaggle (0.44 for logistic loss while
best logistic regression where at 0.46).

Deezer
Millions of songs, only 20% signiﬁcantly listened (popularity
matters here).
Hard problems with negative feedbacks and new songs
Hybrid system (some tags are available).
Similar problem for Youtube recommendations
[Bendersky et al., 2014]
Demo – requires a logged deezer account
http://www.inria.fr/~mary/projets/deezer/

Clustering of time series
A pair of times series should belong to the same cluster if and
only if the distribution that generates them is the same.
It is possible to build a distance for stationary ergodic
processes ρ1 and ρ2 :
d(ρ1, ρ2) =
∞
m,l=1
wm,l
B∈Bm,l
|ρ1(B) − ρ2(B)|
Where wm,l = wm · wl and wi = 2−i , i ∈ N and the sets
Bm,l , m, l ∈ N are obtained via the partitioning of R into
cubes of dimension m and volume 2−ml .

Telescope distance (1/2)
From samples X = (X1 . . . Xn) and Y = (Y1 . . . Ym)
run a classifier on the two samples, considering each
Xi , i = 1, . . . , n as a class-0 example and each Yi , i = 1, . . . , m
as class-1 example. T1
x is the number of correctly classified
examples for class-0.
T1
y is the number of misclassified sample from class-1.
d1 = |T1
x /n − T1
y /m|

Telescope distance (2/2)
Do the same thing with k − tuples to
obtain
dk = |Tk
x /(n − k + 1) − Tk
y /(m − k + 1)|
d =
√
n
k=1
wkdk
where wk = 1/k2 but can be any
summable sequence.
5 10 15 20
5101520
Sequence number
Sequencenumber

Data-Driven Recommender Systems

More Related Content

What's hot

Viewers also liked

Similar to Data-Driven Recommender Systems

More from recsysfr

Data-Driven Recommender Systems