Many of today’s most engaging – and commercially important – applications provide personalised experiences to users. Collaborative filtering algorithms capture the commonality between users and enable applications to make personalised recommendations quickly and efficiently. The Alternating Least Squares (ALS) algorithm is still deemed the industry standard in collaborative filtering. In this talk Sophie will show you how to implement ALS using Apache Spark to build your own recommendation engine. Sophie will show that, by splitting the recommendation engine into multiple cooperating services, it is possible to reduce the system’s complexity and produce a robust collaborative filtering platform with support for continuous model training. In this presentation you will learn how to build a recommendation system for the case where recorded data is explicitly given as a rating, as well as for the case where the data is less succinct. You will walk away from this talk with the knowledge and tools needed to implement your own recommendation system using collaborative filtering and microservices.

1. Building an Implicit
Recommendation Engine
Sophie Watson @sophwats sophie@redhat.com
#SAISDS12

2. Building an Implicit
Recommendation Engine
Sophie Watson @sophwats sophie@redhat.com
#SAISDS12

3. #SAISDS12
}
}
movies
books
food
people to follow
music
hotels

4. #SAISDS12
Outline
Collaborative Filtering
Alternating Least Squares

5. #SAISDS12
Collaborative Filtering

6. #SAISDS12
Collaborative Filtering

7. #SAISDS12
Collaborative Filtering

8. #SAISDS12
Collaborative Filtering
Product A
Product B
Product C
1 2 3
5
3
5
5
3
3
5
3
?

9. #SAISDS12
Collaborative Filtering
Product A
Product B
Product C
1 2 3
5
3
5
5
3
3
5
3
4

10. #SAISDS12
R =
user 1 user 2 user 3 user N
product 1
product 2
product 3
product M
· · ·
· · ·
· · ·
· · ·
· · ·
···
···
···
···
···
···
1
5
2
?
?
?
?
?3
3
4.5
4 1
3 4
3
Alternating Least Squares

11. #SAISDS12
Rkj
k
j

12. #SAISDS12
Alternating Least Squares
R =
user 1 user 2 user 3 user N
product 1
product 2
product 3
product M
· · ·
· · ·
· · ·
· · ·
· · ·
···
···
···
···
···
···
1
5
2
3.8
3
3
4.5
4 1
3 4
3
3.4
3.2
3.1
2.7

13. #SAISDS12
Implicit Data
Song A
1
1 play

14. #SAISDS12
Implicit Data
Song A
1
1 play
Song B 0 plays

15. #SAISDS12
Implicit Data
Song A
1
1 play
Song B 0 plays
Song C 100 plays

16. #SAISDS12
Implicit Alternating Least Squares

17. #SAISDS12
Implicit Alternating Least Squares
preference
The aim:

18. #SAISDS12
Implicit Alternating Least Squares
The recorded data:
preference
recording

19. #SAISDS12
Implicit Alternating Least Squares
The recorded data:
preference
recording

20. #SAISDS12
Implicit Alternating Least Squares
Confidence:
preference
recording

21. #SAISDS12
Implicit Alternating Least Squares
Confidence:
preference
recording
tuning parameter

22. #SAISDS12
Implicit Alternating Least Squares
Minimisation:
preference
recording
user vector Item vector

23. #SAISDS12
What does Spark offer?

24. #SAISDS12
Data
Lastfm dataset
17 million recordings / 360k users / 200k artists
user id
artist id
}artist name
number of plays

25. #SAISDS12
Building the model
(user id, product id, recording)
}tuning parameters

26. #SAISDS12
Tuning Parameters
R
Training
Validation
60%
40%

27. #SAISDS12
Rank
}
}

28. #SAISDS12
Rank
Rank
MeanSquaredError

29. #SAISDS12
Lambda
Minimisation:
+ λ ( ll U ll + ll X ll )
Meansquarederror
Lambda

30. #SAISDS12
Alpha
recording

31. #SAISDS12
Alpha
recording
relates to scale of recording

32. #SAISDS12
Iterations
Iterations
MeanSquaredError

33. #SAISDS12
Making Predictions
(user id, item id)

34. #SAISDS12

35. #SAISDS12

36. #SAISDS12

37. #SAISDS12

38. #SAISDS12

39. #SAISDS12
Microservices

40. #SAISDS12
Microservices

41. #SAISDS12
Microservices

42. #SAISDS12
radanalytics.io

43. #SAISDS12
Collaborative Filtering
Alternating Least Squares@sophwats
sophie@redhat.com