Many of today’s most engaging—and commercially important—applications provide personalised experiences to users. Collaborative filtering algorithms capture the commonality between users and enable applications to make personalised recommendations quickly and efficiently. The Alternating Least Squares (ALS) algorithm is still deemed the industry standard in collaborative filtering. In this talk Sophie will show you how to implement ALS using Apache Spark to build your own recommendation engine. Sophie will show that, by splitting the recommendation engine into multiple cooperating services, it is possible to reduce the system’s complexity and produce a robust collaborative filtering platform with support for continuous model training. In this presentation you will learn how to build a recommendation system for the case where recorded data is explicitly given as a rating, as well as for the case where the data is less succinct. You will walk away from this talk with the knowledge and tools needed to implement your own recommendation system using collaborative filtering and microservices.

1. BUILDING STREAMING
RECOMMENDATION
ENGINES ON APACHE SPARK
Rui Vieira
Software Engineer

2. Overview
Collaborative Filtering
Batch Alternating Least Squares (ALS)
Streaming ALS
Apache Spark
Distributed Streaming ALS

3. Collaborative Filtering

4. Collaborative Filtering
Users, products and ratings
(user, product) ⟼ rating
Collaborative
“Filtering”

5. Collaborative Filtering

6. Collaborative Filtering
A
?
B
?

7. Collaborative Filtering
A B
? ?

8. Alternating Least Squares
R =
user 1 user 2 user 3 · · · user N
⎡
⎢
⎢
⎢
⎢
⎣
⎤
⎥
⎥
⎥
⎥
⎦
1 4.5 ? · · · 3 product 1
? 3 3 · · · 4 product 2
5 3 ? · · · ? product 3
...
...
...
...
...
...
2 4 1 · · · ? product M

9. Alternating Least Squares
=
̂𝖱x,y = 𝖴x 𝖯T
y
𝖯T
y
𝖴x
̂𝖱

10. Batch ALS
loss =
∑
x,y
(ϵx,y)
2
+ λx + λy
ϵx,y = rx,y − ̂rx,y

11. Alternating Least Squares
=
̂𝖱 𝖴
𝖯T

12. Alternating Least Squares
=
̂𝖱 𝖴
𝖯T

13. Alternating Least Squares
R =
user 1 user 2 user 3 · · · user N
⎡
⎢
⎢
⎢
⎢
⎣
⎤
⎥
⎥
⎥
⎥
⎦
1 4.5 3.8 · · · 3 product 1
3.2 3 3 · · · 4 product 2
5 3 3.4 · · · 3.1 product 3
...
...
...
...
...
...
2 4 1 · · · 2.7 product M

14. Batch ALS
1
2
3
4
300
1 2 3 4 300
70 82 60 54 65
70 86 68 67 72
96 103 82 82 77
90 87 68 93 82
38 48 44 51 35
…
…

15. Batch ALS
1
2
3
4
300
1 2 3 4 300
70 82 60 54 65
70 86 68 67 72
96 103 82 82 77
90 87 68 93 82
38 48 44 51 35
…
…

16. Batch ALS

17. Batch ALS

18. Batch ALS

19. Batch ALS

20. Streaming ALS
Can we update the model with a data stream?
Stochastic Gradient Descent (SGD)
Bias SGD (B-SGD)

21. Streaming ALS
bx,y = μ + bx + by
̂r⋆
x,y = bx,y + ̂rx,y

22. Streaming ALS
ˆrx,y = µ + bx + by + Ux · PT
y
loss =
x,y
⎛
⎜
⎝rx,y − ˆrx,y
ϵx,y
⎞
⎟
⎠
2
+ · · ·

23. Streaming ALS
bx bx + (✏x,y xbx)
by by + (✏x,y yby)
bias
factors
Ux Ux + (✏x,yPy
0
xUx)
Py Py + ✏x,yUx
0
yPy

24. Streaming ALS

25. Streaming ALS

26. Streaming ALS

27. Streaming ALS
Batch
B-SGD
Predictions

28. Streaming ALS

29. Streaming ALS

30. APACHE SPARK

31. Apache Spark
Array[T]
RDD[T]
partition 1
partition 2
partition 3
RDD[T]
partition 1
partition 2
partition 3
.map

32. MLlib ALS
val model = ALS.train(ratings, rank, iterations, lambda)

33. MLlib ALS
val model = ALS.train(ratings, rank, iterations, lambda)
case class Rating(int user, int product, double rating)
val ratings: RDD[Rating]

34. MLlib ALS
val model = ALS.train(ratings, rank, iterations, lambda)
case class Rating(int user, int product, double rating)
val ratings: RDD[Rating]
val rank: int

35. MLlib ALS
val model = ALS.train(ratings, rank, iterations, lambda)
case class Rating(int user, int product, double rating)
val ratings: RDD[Rating]
val rank: int
val iterations: int

36. MLlib ALS
val model = ALS.train(ratings, rank, iterations, lambda)
case class Rating(int user, int product, double rating)
val ratings: RDD[Rating]
val rank: int
val iterations: int
val lambda: Double

37. MLlib ALS
> val model = ALS.train(ratings, rank, iterations, lambda)
model: MatrixFactorizationModel
class MatrixFactorizationModel {
val userFeatures: RDD[(Int, Array[Double])]
val productFeatures: RDD[(Int, Array[Double])]
}

38. Spark Streaming ALS
RDD[Rating]
DStream[Rating]
Window 1 Window 2 Window 3 Window 4
RDD[Rating] RDD[Rating] RDD[Rating] RDD[Rating]

39. Spark Streaming ALS
DStream[Rating]

40. Spark Streaming ALS
DStream[Rating]
Window1
RDD[Rating]
model = StreamingALS.train(rdd1, params)
RDD[Rating]
model = model.train(rdd2)
Window2
RDD[Rating]
model = model.train(rdd3)
Window3

41. What do we need?
user latent factors
product latent factors
calculate the global bias
calculate user specific bias
calculate product specific bias

42. user latent factors
product latent factors
calculate the global bias
calculate user specific bias
calculate product specific bias
What do we need?

43. Spark Streaming ALS
Window1
RDD[Rating]

44. Spark Streaming ALS
Window1
RDD[Rating]
(👨, 🏀, 3.5)
(👩, 🎾, 3.0)
(%, ⚽, 1.0)
(', 🏈, 4.0)
(), ⚽, 5.0)
RDD[Rating]

45. Spark Streaming ALS
Window1
RDD[Rating]
(👨, 🏀, 3.5)
(👩, 🎾, 3.0)
(%, ⚽, 1.0)
(', 🏈, 4.0)
(), ⚽, 5.0)
RDD[Rating]
(🏀,(👨, 3.5))
(🎾,(👩, 3.0))
(⚽,(%, 1.0))
(🏈,(', 4.0))
(⚽,(), 5.0))
RDD[(Int, Rating)]
.m

46. Spark Streaming ALS
Window1
RDD[Rating]
(👨, 🏀, 3.5)
(👩, 🎾, 3.0)
(%, ⚽, 1.0)
(', 🏈, 4.0)
(), ⚽, 5.0)
RDD[Rating]
(🏀,(👨, 3.5))
(🎾,(👩, 3.0))
(⚽,(%, 1.0))
(🏈,(', 4.0))
(⚽,(), 5.0))
RDD[(Int, Rating)]
.m
(👨,(🏀, 3.5))
(👩,(🎾, 3.0))
(%,(⚽, 1.0))
(',(🏈, 4.0))
(),(⚽, 5.0))
RDD[(Int, Rating)]
.map

47. Spark Streaming ALS

48. Spark Streaming ALS
(👨, (🏀, 3.5))
(👩, (🎾, 3.0))
(%, (⚽, 1.0))
(', (🏈, 4.0))
(), (⚽, 5.0))
RDD[(Int, Rating)]

49. Spark Streaming ALS
(👨, (🏀, 3.5))
(👩, (🎾, 3.0))
(%, (⚽, 1.0))
(', (🏈, 4.0))
(), (⚽, 5.0))
RDD[(Int, Rating)]
(👨, [0.123, -0.234, …]))
(👩, [0.934, 0.526, …])
(%, [0.421, -0.594, …])
(', [0.034, 0.661, …])
(), [0.713, -0.335, …])
RDD[(Int, Factor)]
.map

50. Spark Streaming ALS

51. Spark Streaming ALS
(👨,(🏀, 3.5))
(👩, (🎾, 3.0))
(%,(⚽, 1.0))
(',(🏈, 4.0))
(),(⚽, 5.0))
RDD[(Int, Rating)]

52. Spark Streaming ALS
(👨,(🏀, 3.5))
(👩, (🎾, 3.0))
(%,(⚽, 1.0))
(',(🏈, 4.0))
(),(⚽, 5.0))
RDD[(Int, Rating)]
(👨, [0.123, -0.234, …])
(👩, [0.934, 0.526, …])
(%, [0.421, -0.594, …])
(', [0.034, 0.661, …])
(), [0.713, -0.335, …])
RDD[(Int, Factor)]

53. Spark Streaming ALS
(👨,(🏀, 3.5))
(👩, (🎾, 3.0))
(%,(⚽, 1.0))
(',(🏈, 4.0))
(),(⚽, 5.0))
RDD[(Int, Rating)]
(👨, [0.123, -0.234, …])
(👩, [0.934, 0.526, …])
(%, [0.421, -0.594, …])
(', [0.034, 0.661, …])
(), [0.713, -0.335, …])
RDD[(Int, Factor)]
.join

54. Spark Streaming ALS
.join
(🏀, (👨, 3.5, [0.123, -0.234, …]))
(🎾, (👩, 3.0, [0.934, 0.526, …]))
(⚽, (%, 1.0, [0.421, -0.594, …]))
(🏈, (', 4.0, [0.034, 0.661, …]))
(⚽, (), 5.0, [0.713, -0.335, …]))
RDD[(Int, (Int, Double, Factor))]
user latent factors

55. Spark Streaming ALS

56. Spark Streaming ALS
(🏀, (👨, 3.5, [0.123, -0.234, …]))
(🎾, (👩, 3.0, [0.934, 0.526, …]))
(⚽, (%, 1.0, [0.421, -0.594, …]))
(🏈, (', 4.0, [0.034, 0.661, …]))
(⚽, (), 5.0, [0.713, -0.335, …]))
RDD[(Int, (Int, Double, Factor))]

57. Spark Streaming ALS
(🏀, (👨, 3.5, [0.123, -0.234, …]))
(🎾, (👩, 3.0, [0.934, 0.526, …]))
(⚽, (%, 1.0, [0.421, -0.594, …]))
(🏈, (', 4.0, [0.034, 0.661, …]))
(⚽, (), 5.0, [0.713, -0.335, …]))
RDD[(Int, (Int, Double, Factor))]
(🏀, [0.764, 0.254, …]))
(⚽, [0.136, 0.933, …])
(🎾, [0.663, -0.134, …])
(🎾, [0.811, 0.535, …])
(🏈, [0.234, -0.579, …])
RDD[(Int, Factor)]
.join

58. Spark Streaming ALS
(🏀, (👨, 3.5, [0.123, …], [0.764, …] ))
(🎾, (👩, 3.0, [0.934, 0.526, …], [0.933, …]))
(⚽, (%, 1.0, [0.421, -0.594, …], [0.663, …]))
(🏈, (', 4.0, [0.034, 0.661, …], [0.811, …]))
(⚽, (), 5.0, [0.713, -0.335, …], [0.234, …]))
RDD[(Int, (Int, Double, Factor, Factor))]

59. user latent factors
product latent factors
calculate the global bias
calculate user specific bias
calculate product specific bias
What do we need?

60. Spark Streaming ALS
(🏀, (👨, 3.5, [0.123, …], [0.764, …] ))
(🎾, (👩, 3.0, [0.934, 0.526, …], [0.933, …]))
(⚽, (%, 1.0, [0.421, -0.594, …], [0.663, …]))
(🏈, (', 4.0, [0.034, 0.661, …], [0.811, …]))
(⚽, (), 5.0, [0.713, -0.335, …], [0.234, …]))
RDD[(Int, (Int, Double,

61. Spark Streaming ALS
(🏀, (👨, 3.5, [0.123, …], [0.764, …] ))
(🎾, (👩, 3.0, [0.934, 0.526, …], [0.933, …]))
(⚽, (%, 1.0, [0.421, -0.594, …], [0.663, …]))
(🏈, (', 4.0, [0.034, 0.661, …], [0.811, …]))
(⚽, (), 5.0, [0.713, -0.335, …], [0.234, …]))
RDD[(Int, (Int, Double,
(🏀, (👨, 3.5, [0.123, …], [0.764, …] ))
(🎾, (👩, 3.0, [0.934, 0.526, …], [0.933, …]))
(⚽, (%, 1.0, [0.421, -0.594, …], [0.663, …]))
(🏈, (', 4.0, [0.034, 0.661, …], [0.811, …]))
(⚽, (), 5.0, [0.713, -0.335, …], [0.234, …]))
RDD[(Int, (Int, Double, Factor, Factor))]

62. user latent factors
product latent factors
calculate the global bias
calculate user specific bias
calculate product specific bias
What do we need?

63. Spark Streaming ALS
✏x,y = rx,y ˆrx,y
ˆrx,y = µ + bx + by + Ux · PT
y
bx bx + (✏x,y xbx)

64. Spark Streaming ALS
✏x,y = rx,y ˆrx,y
ˆrx,y = µ + bx + by + Ux · PT
y
bx bx + (✏x,y xbx)

65. Spark Streaming ALS
ˆrx,y = µ + bx + by + Ux · PT
y

66. Spark Streaming ALS
ˆrx,y = µ + bx + by + Ux · PT
y
(⚽, (%, 1.0, [0.421, -0.594, …], [0.663, …]))
RDD[(Int, (Int, Double, Factor, Factor))]
predicted(⚽, %) = µ + bx + by + [0.421, -0.594, …] x [0.663, …]T = 2.3

67. Spark Streaming ALS
✏x,y = rx,y ˆrx,y
bx bx + (✏x,y xbx)

68. Spark Streaming ALS
✏x,y = rx,y ˆrx,y

69. Spark Streaming ALS
✏x,y = rx,y ˆrx,y
(⚽, (%, 1.0, [0.421, -0.594, …], [0.663, …]))
RDD[(Int, (Int, Double, Factor))]
predicted(%,⚽) = 2.3
rating(%,⚽) = 1.0
error(%,⚽) = rating(%,⚽) - predicted(%,⚽) = -1.3

70. Spark Streaming ALS
bx bx + (✏x,y xbx)
by by + (✏x,y yby)
gradients

71. Spark Streaming ALS
bx bx + (✏x,y xbx)
by by + (✏x,y yby)
gradients
(👨, 0.932, [0.123, -0.140, …])
(👩, 0.101, [0.334, 0.273, …])
(%, 0.128, [0.957, -0.247, …])
(', 0.242, [0.038, 0.883, …])
(), 0.245, [0.283, -0.953, …])
RDD[(Int, Double, Factor)]

72. Spark Streaming ALS
bx bx + (✏x,y xbx)
by by + (✏x,y yby)
gradients
(👨, 0.932, [0.123, -0.140, …])
(👩, 0.101, [0.334, 0.273, …])
(%, 0.128, [0.957, -0.247, …])
(', 0.242, [0.038, 0.883, …])
(), 0.245, [0.283, -0.953, …])
RDD[(Int, Double, Factor)]
(🏀, 0.274, [0.445, -0.233, …])
(🎾, 0.483, [0.843, 0.023, …])
(⚽, 0.595, [0.284, -0.987, …])
(🏈, 0.103, [0.340, 0.328, …])
(⚽, 0.253, [0.472, -0.274, …])
RDD[(Int, Double, Factor)]

73. Spark Streaming ALS

74. Spark Streaming ALS
(👨, 0.932, [0.123, -0.140, …])
(👩, 0.101, [0.334, 0.273, …])
RDD[(Int, Double, Factor)]

75. Spark Streaming ALS
(👨, 0.932, [0.123, -0.140, …])
(👩, 0.101, [0.334, 0.273, …])
RDD[(Int, Double, Factor)]
(👨, 0.932)
(👩, 0.101)
RDD[(Int, Double)]
∇bx

76. Spark Streaming ALS
(👨, 0.932, [0.123, -0.140, …])
(👩, 0.101, [0.334, 0.273, …])
RDD[(Int, Double, Factor)]
(👨, 0.932)
(👩, 0.101)
RDD[(Int, Double)]
(👨, [0.123, …])
(👩, [0.334, …])
RDD[(Int, Factor)]
∇bx ∇Ux

77. Spark Streaming ALS
(👨, 0.932, [0.123, -0.140, …])
(👩, 0.101, [0.334, 0.273, …])
RDD[(Int, Double, Factor)]
(🏀, 0.274, [0.445, -0.233, …])
(🎾, 0.483, [0.843, 0.023, …])
RDD[(Int, Double, Factor)]
(👨, 0.932)
(👩, 0.101)
RDD[(Int, Double)]
(👨, [0.123, …])
(👩, [0.334, …])
RDD[(Int, Factor)]
∇bx ∇Ux

78. Spark Streaming ALS
(👨, 0.932, [0.123, -0.140, …])
(👩, 0.101, [0.334, 0.273, …])
RDD[(Int, Double, Factor)]
(🏀, 0.274, [0.445, -0.233, …])
(🎾, 0.483, [0.843, 0.023, …])
RDD[(Int, Double, Factor)]
(👨, 0.932)
(👩, 0.101)
RDD[(Int, Double)]
(👨, [0.123, …])
(👩, [0.334, …])
RDD[(Int, Factor)]
(🏀, 0.274)
(🎾, 0.483)
RDD[(Int, Double)]
∇bx ∇Ux ∇by

79. Spark Streaming ALS
(👨, 0.932, [0.123, -0.140, …])
(👩, 0.101, [0.334, 0.273, …])
RDD[(Int, Double, Factor)]
(🏀, 0.274, [0.445, -0.233, …])
(🎾, 0.483, [0.843, 0.023, …])
RDD[(Int, Double, Factor)]
(👨, 0.932)
(👩, 0.101)
RDD[(Int, Double)]
(👨, [0.123, …])
(👩, [0.334, …])
RDD[(Int, Factor)]
(🏀, 0.274)
(🎾, 0.483)
RDD[(Int, Double)]
(🏀, [0.445, …])
(🎾, [0.843, …])
RDD[(Int, Factor)]
∇bx ∇Ux ∇by ∇Py

80. Spark Streaming ALS

81. Spark Streaming ALS
(👨, 0.932)
(👨, 0.101)
RDD[(Int, Double)]
b(👨) += Σ∇b(👨)

82. Spark Streaming ALS
(👨, 0.932)
(👨, 0.101)
RDD[(Int, Double)]
(👨, [0.123, …])
(👨, [0.334, …])
RDD[(Int, Factor)]
b(👨) += Σ∇b(👨) U(👨) += Σ∇U(👨)

83. Spark Streaming ALS
(👨, 0.932)
(👨, 0.101)
RDD[(Int, Double)]
(👨, [0.123, …])
(👨, [0.334, …])
RDD[(Int, Factor)]
(⚽, 0.274)
(⚽, 0.483)
RDD[(Int, Double)]
b(👨) += Σ∇b(👨) U(👨) += Σ∇U(👨) b(⚽) += Σ∇b(⚽)

84. Spark Streaming ALS
(👨, 0.932)
(👨, 0.101)
RDD[(Int, Double)]
(👨, [0.123, …])
(👨, [0.334, …])
RDD[(Int, Factor)]
(⚽, 0.274)
(⚽, 0.483)
RDD[(Int, Double)]
(⚽, [0.445, …])
(⚽, [0.843, …])
RDD[(Int, Factor)]
b(👨) += Σ∇b(👨) U(👨) += Σ∇U(👨) b(⚽) += Σ∇b(⚽) U(⚽) += Σ∇U(⚽)

85. Spark Streaming ALS
DStream[Rating]
Window 1 Window 2 Window 3 Window 4
RDD[Rating] RDD[Rating] RDD[Rating] RDD[Rating]
StreamALS
RDD[Int, Factor] RDD[Int, Factor]
StreamALS

86. Spark Streaming ALS
RDD[(Int, (Int, Double)]
(🧔, (🎱, 3.5)) (,, (⚾, 1.0)) (👨, (🏀, 4.5)) …
RDD[(Int, (Int, Double)]
(🎱, (🧔, 3.5)) (🏈, (', 3.0)) (⚽, (%, 1.5)) …
Windowt
RDD[Rating]
(👨, 🏀, 4.5)
(👩, 🎾, 4.5)
(%, ⚽, 1.5)
(', 🏈, 3.0)
(), ⚽, 2.0)
(🧔, 🎱, 3.5)
(,, ⚾, 1.0)
(., ⚾, 2.5)
.map

87. Spark Streaming ALS
RDD[(Int, (Int, Double), Int, Factor)]
(👨, (🏀, 4.5), 👨, (0.12, [0.9,…]))
(👩, (🎾, 4.5), (👩, (-0.1, [1.37,…])))
(🧔, (🎱, 1.5), None)
…
RDD[(Int, (Int, Factor)]
(👨, (0.12, [0.9,…]))
(👩, (-0.1, [1.37,…]))
(%, (1, [0.123,…]))
…
RDD[(Int, (Int, Double)]
(👨, (🏀, 4.5)) (👩, (🎾, 4.5)) (🧔, (🎱, 1.5)) …
userRatings.fullOuterJoin(userFactors).map {
case (userId, (_, userFactors)) =>
(userId, userFactors(featureGenerator.nextValue()))
}
.fullOuterJoin

88. Data
MovieLens
Widely used in recommendation engine research
Variants
Small - 100,000 ratings / 9,000 movies / 700 users
Full - 26 million ratings / 45,000 movies / 270,000

89. Training batch ALS
val split: Array[RDD[Rating]] = ratings.randomSplit(0.8, 0.2)
val model = ALS.train(split(0), rank, iter, lambda)

90. Training batch ALS
val split: Array[RDD[Rating]] = ratings.randomSplit(0.8, 0.2)
val model = ALS.train(split(0), rank, iter, lambda)
val predictions: RDD[Rating] = model.predict(split(1).map { x =>
(x.user, x.product))
}
val pairs = predictions.map(x => ((x.user, x.product), x.rating))
.join(split(1).map(x => ((x.user, x.product), x.rating))
.values
Val RMSE = math.sqrt(pairs.map(x => math.pow(x._1 - x._2, 2)).mean())

91. Training streaming ALS
spark
master
spark
worker
spark
worker
spark
worker
spark
worker
kafka

92. Training streaming ALS
val model = StreamingALS(rank, iterations, lambda, gamma)
trainingStreamSet.foreachRDD { rdd =>
model.train(rdd)
val RMSE = calculateRMSE(model, validation)
}
RDD[Rating] RDD[Rating] RDD[Rating]
kafka

93. Comparison
1.1
1.2
1.3
0 20 40 60 80
t
RMSE

94. TO CONSIDER…

95. To consider
“Cold start”
Same as batch ALS
Too few observations = meaningless
Train offline

96. To consider
Hyper-parameter estimation
Data
Parameters A
Model
Data
Parameters B
Data
Parameters C
Data
Parameters D

97. To consider
Hyper-parameter estimation
Data
Parameters A
Model
Data
Parameters B
Data
Parameters C
Data
Parameters D
Model
Parameters A
Model
Parameters B
Parameters C
Parameters D
Model

98. To consider
Hyper-parameter estimation
Data
Model A
Data
Model B
Data
Model C
Data
Model D
Model B
Model C
Model D
Model B
Model D
Model B

99. To consider
Partition A Partition BJoin
Partitioning

100. To consider
RDD random access?
val u = userFeatures.lookup(userId)
val p = productFeatures.lookup(productId)
val predicted = model.predict(userId, productId, u, p, globalBias)
R
=
U
PT
x
y

101. Links
Blog post:
https://ruivieira.github.io/a-streaming-als-
implementation.html
radanalytics.io

102. THANK YOU