Slides of my presentation at CIKM2018 about version 2 of the GRU4Rec algorithm, a recurrent neural network based algorithm for the session-based recommendation task. We discuss sampling strategies and introduce additional sampling to the algorithm. We also redesign the loss function to cope with additional sampling. The resulting BPR-max loss function is able to efficiently handle many negative samples without encountering the vanishing gradient problem. We also introduce constrained embeddings which speeds up the conversion of item representations and reduces memory usage by a factor of 4. These improvements increase offline measures up to 52%. In the talk we also discuss online A/B test and the implications of long time observations. Most of these observations are exclusive to this talk and are not in the paper. You can access the preprint version of the paper on arXiv: https://arxiv.org/abs/1706.03847 The code is available on GitHub: https://github.com/hidasib/GRU4Rec

1. GRU4Rec v2
Recurrent neural networks with top-k gains for
session-based recommendations
Balázs Hidasi
@balazshidasi

2. At a glance
• Improvements on GRU4Rec [Hidasi et. al, 2015]
 Session-based recommendations with RNNs
• General
 Negative sampling strategies
 Loss function design
• Specific
 Constrained embeddings
 Implementation details
• Offline tests: up to 35% improvement
• Online tests & observations

3. GRU4Rec overview

4. Context: session-based recommendations
• User identification
 Feasibility
 Privacy
 Regulations
• User intent
 Disjoint sessions
o Need
o Situation (context)
o „Irregularities”
• Session-based recommendations
• Permanent user cold-start

5. Preliminaries
• Next click prediction
• Top-N recommendation (ranking)
• Implicit feedback

6. Recurrent Neural Networks
• Basics
 Input: sequential information ( 𝑥 𝑡 𝑡=1
𝑇
)
 Hidden state (ℎ 𝑡):
o representation of the sequence so far
o influenced by every element of the sequence up to t
 ℎ 𝑡 = 𝑓 𝑊𝑥 𝑡 + 𝑈ℎ 𝑡−1 + 𝑏
• Gated RNNs (GRU, LSTM & others)
 Basic RNN is subject to the exploding/vanishing gradient problem
 Use ℎ 𝑡 = ℎ 𝑡−1 + Δ 𝑡 instead of rewriting the hidden state
 Information flow is controlled by gates
• Gated Recurrent Unit (GRU)
 Update gate (z)
 Reset gate (r)
 𝑧𝑡 = 𝜎 𝑊𝑧 𝑥 𝑡 + 𝑈𝑧ℎ 𝑡−1 + 𝑏 𝑧
 𝑟𝑡 = 𝜎 𝑊𝑟 𝑥 𝑡 + 𝑈𝑟ℎ 𝑡−1 + 𝑏 𝑟
 ℎ 𝑡 = tanh 𝑊𝑥 𝑡 + 𝑟𝑡 ∘ 𝑈ℎ 𝑡−1 + 𝑏
 ℎ 𝑡 = 𝑧𝑡 ∘ ℎ 𝑡−1 + 1 − 𝑧𝑡 ∘ ℎ 𝑡
ℎ
r
IN
OUT
z
+

7. GRU4Rec
• GRU trained on session data, adapted to the
recommendation task
 Input: current item ID
 Hidden state: session representation
 Output: likelihood of being the next item
• Session-parallel mini-batches
 Mini-batch is defined over sessions
 Update with one step BPTT
o Lots of sessions are very short
o 2D mini-batching, updating on longer sequences (with
or without padding) didn’t improve accuracy
• Output sampling
 Computing scores for all items (100K – 1M) in every
step is slow
 One positive item (target) + several samples
 Fast solution: scores on mini-batch targets
o Items of the other mini-batch are negative samples for
the current mini-batch
• Loss functions: cross-entropy, BPR, TOP1
GRU layer
One-hot vector
Weighted output
Scores on items
f()
One-hot vector
ItemID (next)
ItemID

8. Negative sampling & loss
function design

9. Negative sampling
• Training step
 Score all items
 Push target items forward (modify model parameters)
• Many training steps & many items
 Not scalable
 𝑂 𝑆𝐼 𝑁+
• Sample negative examples instead  𝑂 𝐾𝑁+
• Sampling probability
 Must be quick to calculate
 Two popular choices
o Uniform  many unnecessary steps
o Proportional to support  better in practice, fast start, some
relations are not examined enough
 Optimal choice
o Data dependent
o Changing during training might help

10. Mini-batch based negative sampling
• Target items of other examples
from the mini-batch  as
negative samples
• Pros
 Efficient & simple implementation
on GPU
 Sampling probability proportional
to support
• Cons
 Number of samples is tied to the
batch size
o Mini-batch training: smaller
batches
o Negative sampling: larger
batches
 Sampling probability is always the
same
𝑖1,1 𝑖1,2 𝑖1,3 𝑖1,4
𝑖2,1 𝑖2,2 𝑖2,3
𝑖3,1 𝑖3,2 𝑖3,3 𝑖3,4 𝑖3,5 𝑖3,6
𝑖4,1 𝑖4,2
𝑖5,1 𝑖5,2 𝑖5,3
Session1
Session2
Session3
Session4
Session5
𝑖1,1 𝑖1,2 𝑖1,3
𝑖2,1 𝑖2,2
𝑖3,1 𝑖3,2 𝑖3,3 𝑖3,4 𝑖3,5
𝑖4,1
𝑖5,1 𝑖5,2
Input
Target
…
𝑖1,2 𝑖1,3 𝑖1,4
𝑖2,2 𝑖2,3
𝑖3,2 𝑖3,3 𝑖3,4 𝑖3,5 𝑖3,6
𝑖4,2
𝑖5,2 𝑖5,3
…
𝑖1 𝑖5 𝑖8
𝑦1
1
𝑦2
1
𝑦3
1 𝑦4
1
𝑦5
1
𝑦6
1
𝑦7
1
𝑦8
1
𝑦1
3
𝑦2
3
𝑦3
3
𝑦4
3
𝑦5
3
𝑦6
3
𝑦7
3
𝑦8
3
𝑦1
2
𝑦2
2
𝑦3
2 𝑦4
2
𝑦5
2
𝑦6
2
𝑦7
2
𝑦8
2
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 1
0 0 0 0 1 0 0 0

11. Solution: add more samples!
• Add extra samples
• Shared between mini-batches
• Sampling probability: 𝑝𝑖 ∼ supp 𝛼
 𝛼 = 0  uniform
 𝛼 = 1  popularity based
• Implementation trick
 Sampling interrupts GPU computations
 More efficient in parallel
 Sample store (cache)
o Precompute 10-100M samples
o Resample is we used them all

12. Another look the loss functions
• Listwise losses on the target+negative samples
 Cross-entropy + softmax
o Cross-entropy in itself is pointwise
o Target should have the maximal score
o 𝑋𝐸 = − log 𝑠𝑖 , 𝑠𝑖 =
𝑒 𝑟 𝑖
𝑗=1
𝐾 𝑒
𝑟 𝑗
o Unstable in previous implementation
– Rounding errors
– Fixed
 Average BPR-score
o BPR in itself is pairwise
o Target should have higher score than all negative samples
o 𝐵𝑃𝑅 = −
1
𝐾 𝑗=1
𝐾
log 𝜎 𝑟𝑖 − 𝑟𝑗
 TOP1 score
o Heuristic loss, idea is similar to the average BPR
o Score regularization part
o 𝑇𝑂𝑃1 =
1
𝐾 𝑗=1
𝐾
𝜎 𝑟𝑗 − 𝑟𝑖 + 𝜎 𝑟𝑗
2
• Earlier results
 Similar performance
 TOP1 is slightly better

13. Unexpected behaviour of pairwise losses
• Unexpected behaviour
 Several negative samples  good results
 Many negative samples  bad results
• Gradient (BPR wrt. target score)

𝜕𝐿 𝑖
𝜕𝑟 𝑖
=
1
𝐾 𝑗=1
𝐾
1 − 𝜎 𝑟𝑖 − 𝑟𝑗
• Irrelevant negative sample
 Whose score is already lower than 𝑟𝑖
o Changes during training
 Doesn’t contribute to optimization: 1 − 𝜎 𝑟𝑖 − 𝑟𝑗 ∼ 0
 Number of irrelevant samples increases as training progresses
• Averaging with many negative samples  gradient vanishes
 Target will be pushed up for a while
 Slows down as approaches the top
 More samples: slows down earlier

14. Pairwise-max loss functions
• The target score should be higher than the maximum
score amongst the negative samples
• BPR-max
 𝑃 𝑟𝑖 > 𝑟MAX|𝜃 = 𝑗=1
𝐾
𝑃 𝑟𝑖 > 𝑟𝑗 𝑟𝑗 = 𝑟MAX, 𝜃 𝑃 𝑟𝑗 = 𝑟MAX|𝜃
 Use continuous approximations
o 𝑃 𝑟𝑖 > 𝑟𝑗 𝑟𝑗 = 𝑟MAX, 𝜃 = 𝜎 𝑟𝑖 − 𝑟𝑗
o 𝑃 𝑟𝑗 = 𝑟MAX|𝜃 = softmax 𝑟𝑘 𝑘=1
𝐾
=
𝑒
𝑟 𝑗
𝑘=1
𝐾 𝑒 𝑟 𝑘
= 𝑠𝑗
– Softmax over negative samples only
 Minimize negative log probability
 Add ℓ2 score regularization
• 𝐿𝑖 = − log 𝑗=1
𝐾
𝑠𝑗 𝜎 𝑟𝑖 − 𝑟𝑗 + 𝜆 𝑗=1
𝐾
𝑟𝑗
2

15. Gradient of pairwise max losses
• BPR-max (wrt. 𝑟𝑖)

𝜕𝐿 𝑖
𝜕𝑟 𝑖
=
𝑗=1
𝐾
𝑠 𝑗 𝜎 𝑟 𝑖−𝑟 𝑗 1−𝜎 𝑟 𝑖−𝑟 𝑗
𝑗=1
𝐾 𝑠 𝑗 𝜎 𝑟 𝑖−𝑟 𝑗
 Weighted average of BPR gradients
o Relative importance of samples:
𝑠 𝑗 𝜎 𝑟 𝑖−𝑟 𝑗
𝑠 𝑘 𝜎 𝑟 𝑖−𝑟 𝑘
=
𝑒
𝑟 𝑗+𝑒
𝑟 𝑗+𝑟 𝑘−𝑟 𝑖
𝑒 𝑟 𝑘+𝑒
𝑟 𝑗+𝑟 𝑘−𝑟 𝑖
o Smoothed softmax
– If 𝑟𝑖 ≫ max 𝑟𝑗, 𝑟𝑘  behaves like softmax
– Stronger smoothing otherwise
– Uniform  softmax as 𝑟𝑖 is pushed to the top

16. Number of samples: training times &
performance
• Training times on GPU
don’t increase until the
parallelization limit is
reached
• Around the same place
• Significant improvements
up to a certain point
• Diminishing returns after

17. The effect of the α parameter
• Data & loss dependent
 Cross-entropy: favours lower values
 BPR-max: 0.5 is usually a good choice (data dependent)
 There is always a popularity based part of the samples
o Original mini-batch examples
o Removing these will result in higher optimal 𝛼
• CLASS dataset (cross-entropy, BPR-max)

18. Constrained embeddings &
offline tests

19. Unified item representations in GRU4Rec (1/2)
• Hidden state x 𝑊𝑦  scores
 One vector for each item  „item feature matrix”
• Embedding
 A vector for each item  another „item feature matrix”
 Can be used as the input of the GRU layer instead of the one-hot
vector
 Slightly decreases offline metrics
• Constrained embeddings (unified item representations)
 Use the same matrix for both input and output embedding
 Unified representations  faster convergence
 Embedding size tied to hidden the size of the last hidden state

20. Unified item representations in GRU4Rec (2/2)
• Model size: largest components scale with the items
 One-hot input: 𝑋
o 𝑊𝑥
0
, 𝑊𝑟
0
, 𝑊𝑧
0
, 𝑊𝑦  𝑆𝐼 × 𝑆 𝐻
o 𝑈ℎ
0
, 𝑈𝑟
0
, 𝑈𝑧
0
 𝑆 𝐻 × 𝑆 𝐻
 Embedding input: 𝑋/2
o 𝐸, 𝑊𝑦  𝑆𝐼 × 𝑆 𝐻
o 𝑊𝑥
0
, 𝑊𝑟
0
, 𝑊𝑧
0
 𝑆 𝐸 × 𝑆 𝐻
o 𝑈ℎ
0
, 𝑈𝑟
0
, 𝑈𝑧
0
 𝑆 𝐻 × 𝑆 𝐻
 Constrained embedding: 𝑋/4
o 𝑊𝑦  𝑆𝐼 × 𝑆 𝐻
o 𝑊𝑥
0
, 𝑊𝑟
0
, 𝑊𝑧
0
, 𝑈ℎ
0
, 𝑈𝑟
0
, 𝑈𝑧
0
 𝑆 𝐻 × 𝑆 𝐻
GRU
layer
H
ItemID
dot(𝐻,𝑊𝑦)
scores
GRU
layer
H
ItemIDs
dot(𝐻,𝑊𝑦)
scores
E[ItemIDs
]
embedding
GRU
layer
H
ItemIDs
dot(𝐻,𝑊𝑦)
scores
𝑊𝑦[ItemIDs]
embedding

21. Offline results
• Over item-kNN
 +25-52% in recall@20
 +35-55% in MRR@20
• Over the original GRU4Rec
 +18-35% in recall@20
 +27-37% in MRR@20
• BPR-max vs. (fixed) cross-entropy
 +2-6% improvement in 2 of 4 cases
 No statistically significant difference in the other 2 case
• Constrained embedding
 Most cases: slightly worse MRR & better recall
 Huge improvements on the CLASS datasset (+18.74% in recall, +29.44% in MRR)

22. Online A/B tests

23. A/B test - video service (1/3)
• Setup
 Video page
 Recommended videos on a strip
 Autoplay functionality
 Recommendations are NOT recomputed if the user clicks on any of
the recommended videos or autoplay loads a new video
 User based A/B split
• Algorithms
 Original algorithm: previous recommendation logic
 GRU4Rec next best: N guesses for the next item
 GRU4Rec sequence: sequence of length N as the continuation of
the current session
o Greedy generation

24. A/B test - video service (2/3)
• Technical details
 User based A/B split
 GRU4Rec serving from a single GPU using a single thread
o Score computations for ~500K items in 1-2ms (next best)
 Constant retraining
o GRU4Rec: ~1.5 hours on ~30M events (including data collection and
preprocessing)
o Original logic: different parts with different frequency
 GRU4Rec falls back to the original logic if it can’t recommend or times
out
• KPIs (relative to the number of recommendation requests)
 Watch time
 Videos played
 Recommendations clicked
• Bots and power users are excluded from the KPI computations

25. A/B test - video service (3/3)

26. A/B test – long term effects (1/3)
• Watch time
0.96
0.98
1
1.02
1.04
1.06
Rel.improvement
(watchtime/rec)
Next best
Sequence
Watchtime/rec
Baseline
Next best
Sequence

27. A/B test – long term effects (2/3)
• GRU4Rec: strong generalization capabilities
 Finds hidden gems
 Unlike counting based approaches
o Not obvious in offline only testing
• Feedback loop
 Baseline trains also sees the feedback generated for recommendations of other groups
 Learns how to recommend hidden gems
• GRU4Rec maintains some lead
 New items are constantly uploaded
• Comparison of different countries
Watchtime/rec
(increase)

28. A/B test – long term effects (3/3)
• Videos played
 Next best and sequence switched places
 Sequence mode: great for episodic content, can suffer
otherwise
 Next best mode: more diverse, better for non-episodic
content
 Feedback loop: next best learns some of the episodic
recommendations
1
1.01
1.02
1.03
1.04
1.05
1.06
Relativeimprovement
(videosplayed/rec)
Next best
Sequence

29. A/B test - online marketplace
• Differences in setup
 On home page
 Next best mode only
 KPI: CTR
• Items have limited lifespan
 Will GRU4Rec keep its 19-20% lead?
CTR
Baseline
GRU4Rec

30. Thank you!
Q&A
Check out the code (free for research):
https://github.com/hidasib/GRU4Rec
Read the preprint: https://arxiv.org/abs/1706.03847