Building your first recommender - Jettro Coenradie - Codemotion Amsterdam 2018

1.
Building your firstrecommender Jettro Coenradie AMSTERDAM | MAY 8-9, 2018

2.
Jettro Coenradie Fellow at Luminis Amsterdam specialised in (Elastic) search experimenting with Machine Learning @jettroCoenradie https://www.linkedin.com/in/jettro/ https://github.com/jettro

3.
https://nl.wikipedia.org/wiki/Free_Record_Shop

4.
The Goal

5.
The Goal

6.
The Goal

7.
To increase salesand happiness

8.
To increase salesand happiness Advertorial

9.
To increase salesand happiness Advertorial Recommendation

12.
Advertorial

13.
Not based on knowledge we have about you, no context. Advertorial

14.
Not based on knowledge we have about you, no context. Based on what we know about you and about the items we have. Advertorial Recommendation

15.
Having posters onthe wall

16.
Having posters onthe wall Advertorial

17.
Regular customer wantsto buy something new

18.
Regular customer wantsto buy something new Recommendation

19.
Play music inthe store

20.
Play music inthe storeAdvertorial

21.
Play music inthe store Saturday morning Advertorial

22.
Play music inthe store Saturday morning Advertorial

23.
Play music inthe store Saturday morning Advertorial Recommendation

24.
Risk of Recommendations

25.
Risk of Recommendations •Would you recommend something that is out of stock?

26.
Risk of Recommendations •Would you recommend something that is out of stock? • Would you advise different items during the seasons?

27.
Risk of Recommendations •Would you recommend something that is out of stock? • Would you advise different items during the seasons? • Does personal taste influence your recommendations?

28.
Who buys musicthese days?

30.
Advertorial

31.
Advertorial Recommendation

32.
Goal: Have users listento many different songs Make users stick around

33.
Analytics Dashboard tosupport our Goals • Track returning users • Track amount of songs a user listens to • Track amount of recommended songs are listened to • ????

34.
Assumption Recommendations empower our application

35.
Type of recommendations •Collaborative filtering • Content / Knowledge based filtering

36.
Collaborative filtering Find similar users using behaviour of the users

37.
Collaborative filtering Find similar users using behaviour of the users Explicit Ratings • Rates song with amount of stars

38.
Collaborative filtering Find similar users using behaviour of the users Explicit Ratings • Rates song with amount of stars Implicit Ratings •Users click on a song • Users listens to complete song (or stops after half a minute)

39.
Content/Knowledge based filtering Find similar music using properties of the music •The genre of the music • The artist • Producer

40.
Create recommender using Collaborativefiltering

41.
Create recommender using Collaborativefiltering Obtain all rating data

42.
Create recommender using Collaborativefiltering Obtain all rating data Find similar users

43.
Create recommender using Collaborativefiltering Obtain all rating data Find similar users Predict missing ratings

44.
Create recommender using Collaborativefiltering Obtain all rating data Find similar users Predict missing ratings Present recommendations

45.
Item album 1 album 2 album 3album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User User-item matrix Obtain all rating data

46.

47.

48.

49.
user i user jsimilar item m item n similar ratedrated

50.
user i user jsimilar item m item n similar ratedrated user based

51.
user i user jsimilar item m item n similar ratedrated item based

52.
user i user jsimilar item m item n similar ratedrated item based user based

53.
Find similar users Item album 1album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User

54.
Find similar users Item album 1album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User

55.
Find similar items Item album 1album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User

56.
Find similar items Item album 1album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User

57.
Similarity How to determine if users or items are similar == == ? ?

58.
Definition of similarity sim(itemi,itemi ) = 1 sim(itemi ,itemj ) = 0

59.
Similarity based ondistance similarity(itemi ,itemj ) = 1− distance maximum_distance

60.
User similarity examples •Jaccard Distance - Compare items liked between users • L1-Norm - Compares ratings given by users the Manhatten way • L2-Norm - Compares ratings the Pythagoras way • Cosine - Compares ratings the angle between vectors way

61.
Jaccard Distance album 1 album 2album 3 album 4 user 1 user 2 4 4

62.
Jaccard Distance album 1 album 2album 3 album 4 user 1 user 2 4 4 distance(user1,user2 ) = #albums_liked _by_both #albums_liked _by_any = 1 3

63.
Jaccard Distance album 1 album 2album 3 album 4 user 1 user 2

64.
Jaccard Distance album 1 album 2album 3 album 4 user 1 user 2 distance(user1,user2 ) = #albums_ same_opinion #albums_with_opinion = 2 3

65.
album 1 album 2 album 3album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 Simplify to 2D space

66.
album 1 album 2 album 3album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 album 1 album 2 user 1 4 1 user 2 4 3 user 3 1 5 Simplify to 2D space

67.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 L1 Norm Mean Absolute Error

68.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 L1 Norm D1 D2 Mean Absolute Error

69.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 L1 Norm D1 D2 Mean Absolute Error = 1 2 4 −1 + 3− 5( )= 2.5 distance(user2,user3 ) = 1 n ratinguser2 ,albumi − ratinguser3,albumi i=1 n ∑

70.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 L2 Norm Root Mean Squared Error

71.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 D1 L2 Norm Root Mean Squared Error

72.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 D1 L2 Norm = 1 2 4 −1 2 + 3− 5 2 = 1 2 13 = 1.8 distance(user2,user3 ) = 1 n ratinguser2 ,albumi − ratinguser3,albumi 2 i=1 n ∑ Root Mean Squared Error

73.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 Cosine

74.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 Cosine ∠23 ∠12

75.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 Cosine ∠23 ∠12 sim(user2,user3 ) = ratinguser2 ,albumi i ratinguser3,albumi i=1 n ∑ ratinguser2 ,albumi 2 i=1 n ∑ i ratinguser3,albumi 2 i=1 n ∑

76.
album 1 album 2 user 1 41 user 2 4 3 user 3 1 5 L1 norm L2 norm Cosine user 1 user 2 0.75 0.65 0.92 user 1 user 3 0.13 0.12 0.43 user 2 user3 0.38 0.36 0.75

77.
Normalising • Use average of ratings for each album • Baseline ratings using the average rating

78.
Normalising • Use average of ratings for each album • Baseline ratings using the average rating 1 2 3 4 5 u1u2 u3 u4 u5

79.
Normalising • Use average of ratings for each album • Baseline ratings using the average rating 1 2 3 4 5 u1u2 u3 u4 u5 2.6

80.
Normalising • Use average of ratings for each album • Baseline ratings using the average rating 1 2 3 4 5 u1u2 u3 u4 u5 2.6 Normalised rating: u3 = 2 - 2.6

81.
Normalising • Use average of ratings for each album • Baseline ratings using the average rating 1 2 3 4 5 u1u2 u3 u4 u5 2.6 Normalised rating: u3 = 2 - 2.6 ratinguser2 ,albumi − ratinguser2 ( )i ratinguser3,albumi − ratinguser3 ( )i=1 n ∑ ratinguser2 ,albumi − ratinguser2 ( ) 2 i=1 n ∑ i ratinguser3,albumi − ratinguser3 ( ) 2 i=1 n ∑

82.
Find Neighbourhood • Find most similar users •Users that have rated at least one item I have rated • Score indicating how similar the user is to me

83.
Predicting missing ratings Predict the ratings a user would give

84.
Predictions - Classificationway Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User

85.
Predictions - Classificationway Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User Predicted rating for user 1 and album 3

86.
Predictions - Classificationway Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User Predicted rating for user 1 and album 3 1 x 4

87.
Predictions - Classificationway Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User Predicted rating for user 1 and album 3 1 x 4 2 x 1

88.
Predictions - Classificationway Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User Predicted rating for user 1 and album 3 1 x 4 2 x 1 so prediction becomes 1 1

89.
Predictions - Regressionway Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User Predicted rating for user 1 and album 3

90.
Predictions - Regressionway Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User Predicted rating for user 1 and album 3 = 4 + 1 + 1 3

91.
Predictions - Regressionway Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User Predicted rating for user 1 and album 3 so prediction becomes 2 2 = 4 + 1 + 1 3

92.
Predictions • Normalise ratings by users • Give more similar users higher impact •Only use ratings when more than ‘x’ persons have rated

93.
The story sofar

94.
The story sofar

95.
The story sofar

96.
The story sofar

97.
The story sofar Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User

98.
The story sofar Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User == == ? ?

99.
The story sofar Item album 1 album 2 album 3 album 4 user 1 4 1 3 user 2 4 3 4 2 user 3 1 5 1 4 user 4 1 1 4 User == == ? ? Predict

100.
Demo https://rolling500.luminis.amsterdam

101.
Logs Queries Clicks Rolling 500Recommendation Recommendations Ratings

102.

103.

104.

105.

106.

107.

108.

109.

110.

111.
{ "album": "War", "image": "rs-137178-883f5fe955b2745cd539.jpg", "information":"<p>U2 were on the cusp of becoming one of the Eighties' most important groups when their third album came out. It's the band's most overtly political album, with songs about Poland's Solidarity movement ("New Year's Day") and Irish unrest ("Sunday Bloody Sunday") charged with explosive, passionate guitar rock.</p>", "sequence": 223, "order": 279, "label": "Island", "artist": "U2", "year": "1983", "id": 350640 } { “user_id": “39cf6ada-6a79-4d10-9537-0fe65179b7d2", "ratings": [0,0,0,4,0,0,0,1,0,0,0,0,5, … ,0,0,0,3,5,1,0,0] } The Data

112.
Lenskit - http://lenskit.org Open-Source Tools for Recommender Systems

113.
ItemRecommender RatingPredictor TopNItemRecommender SimpleRatingPredictor Application implementsimplements uses

114.
ItemRecommender BaselineScorer UserVectorSimilarity EventDao UserMeanItemScorer CosineVectorSimilarity EventCollectionDao TopNItemRecommender ItemScorer UserUserItemScorer implements implements uses uses implements implements implements

115.
private LenskitConfiguration configureUserSimilarity( List<Rating>foundRatings, Map<String, Long> userIdMapping) { LenskitConfiguration config = new LenskitConfiguration(); config.bind(ItemScorer.class).to(UserUserItemScorer.class); config.bind(BaselineScorer.class, ItemScorer.class).to(UserMeanItemScorer.class); config.bind(UserSimilarityThreshold.class, Threshold.class).to(NoThreshold.class); config.bind(UserMeanBaseline.class, ItemScorer.class).to(ItemMeanRatingItemScorer.class); config.bind(UserVectorNormalizer.class).to(BaselineSubtractingUserVectorNormalizer.class); config.within(UserVectorSimilarity.class).bind(VectorSimilarity.class).to(CosineVectorSimilarity.class); List<MutableRating> ratings = convertRatings(foundRatings, userIdMapping); config.bind(EventDAO.class).to(EventCollectionDAO.create(ratings)); return config; }

116.

117.

118.

119.

120.

121.
public class Rating{ private String userId; private int[] ratings; } public class MutableRating implements Rating { private long uid; private long iid; private double value; }

122.
private List<MutableRating> convertRatings(List<Rating>foundRatings, Map<String, Long> userIdMapping) { List<MutableRating> ratings = new ArrayList<>(); CurrentUserId userId = new CurrentUserId(); foundRatings.forEach(rating -> { String userIdString = rating.getUserId(); long curUser = (userIdMapping.containsKey(userIdString)) ? userIdMapping.get(userIdString) : userId.getUserId(); userIdMapping.put(userIdString, curUser); for (int i = 0; i < rating.getRatings().length; i++) { if (rating.getRatings()[i] > 0) { ratings.add(createRating(i, rating.getRatings()[i], curUser)); } } }); return ratings; } private MutableRating createRating(long item, long rating, long user) { MutableRating mutableRating = new MutableRating(); mutableRating.setItemId(item); mutableRating.setRating(rating); mutableRating.setUserId(user); return mutableRating; }

123.

124.

125.

126.

127.
Implementing the story

128.
Implementing the story Rolling500

129.
Implementing the story Rolling500

130.
Next chapters ofthe story

131.
Segmentation - Selectingthe Neighbourhood Use clustering technologies to pre-select a group

132.
K-means clustering

133.
K-means clustering

134.
K-means clustering

135.
K-means clustering

136.
K-means clustering

137.
Cold cases Users or items without ratings

138.
Cold users They do not have ratings yet, so how to find similar users? • Can we use information we have from them? •Ask them to rate a few items first • Provide them with the top selling items

139.
Cold items Nobody rated them yet • Ask users to give them a rating •Add a new items list to the screen. • Use Content/knowledge based recommendations

140.
Content based Filtering •Use properties of products like: artist, label, year, singer, guitarist, etc • Tags or categories: • Contains a lengthy guitar solo • Singer / songwriter • Contains violins • Female singer

145.
Find “more likethis” items • Simple approach: use a query in elasticsearch with TF/IDF or BM25 • Good to use things like stop words, synonyms, phrase matches, stemming, etc • Use Entity Extraction to find Names, Places, dates, etc

146.
Nirvana Nevermind The overnight-success story of the 1990s, Nirvana's second album and its totemic first single, "Smells Like Teen Spirit," shot up from the nascent grunge scene in Seattle to kick Michael Jackson off the top of the Billboard album chart and blow hair metal off the map. No album in recent history had such an overpowering impact on a generation – a nation of teens suddenly turned punk – and such a catastrophic effect on its main creator. The weight of fame led already troubled singer-guitarist Kurt Cobain to take his own life in 1994. But his slashing riffs, corrosive singing and deviously oblique writing, rammed home by the Pixies-via- Zeppelin might of bassist Krist Novoselic and drummer Dave Grohl, put the warrior purity back in rock & roll. Lyrically, Cobain raged in code – shorthand grenades of inner tumult and self-loathing. His genius, though, in songs like "Lithium," "Breed" and "Teen Spirit" was the soft-loud tension he created between verse and chorus, restraint and assault. Cobain was a pop lover at heart – and a Beatlemaniac: Nevermind producer Butch Vig remembers hearing Cobain play John Lennon's "Julia" at sessions. Cobain also fought to maintain his underground honor. Ultimately, it was a losing battle, but it is part of this album's enduring power. Vig recalls when Cobain was forced to overdub the guitar intro to "Teen Spirit" because he couldn't nail it live with the band: "That pissed him off. He wanted to play [the song] live all the way through."

147.
@jettroCoenradie https://www.linkedin.com/in/jettro/ https://github.com/jettro https://rolling500.luminis.amsterdam https://github.com/jettro/rolling500 http://lenskit.org http://elasticsearch-learning-to-rank.readthedocs.io

Building your first recommender - Jettro Coenradie - Codemotion Amsterdam 2018

More Related Content

What's hot

Similar to Building your first recommender - Jettro Coenradie - Codemotion Amsterdam 2018

More from Codemotion

Recently uploaded

Building your first recommender - Jettro Coenradie - Codemotion Amsterdam 2018