This document discusses techniques for recommender systems including multi-armed bandit (MAB), Thompson sampling, user clustering, and using item features. It provides examples of how MAB works using the ε-greedy approach and explores the tradeoff between exploration and exploitation. User clustering is presented as a way to group users based on click-through rate to improve targeting. Finally, it suggests using different item features like images, text, and collaborative filtering data as inputs to recommendation models.

1. PyCon Korea 2019
, .
|
1

2. 2
/
+

3. ?

5. ?
5

6. ?
6

7. ?
7

8. ?
8

9. ?

10. ?

13. .

14. 14
( )
?
?

15. 15
10
20
20

16. 16

17. .
17

18. ?

19. .
- -

20. .
- -

21. ?

22. 22
( )
?
?

23. :
23
( )
?
?
( ) .

24. 24
10
20
20

25. : .
25
10
20
20
20

26. 26

27. : .
27

28. .

29. ?

30. .
30
/
TP.

31. .

32. .
!32

33. 33
…

34. 34
- - ViewerEnd

35. • :
CTR(%)
•
• MAB(Multi Armed Bandit)
• User Clustering
-
!35

36. MAB(Multi Armed Bandit)
• MAB = Exploration( ) and Exploitation( ) Trade-off
• 10%( ) Feedback (impression,
click)
* ε-greedy MAB .
!36

37. • Feedback CTR(%) .
• CTR(%) = # of clicks / # of impressions
Exploration
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1% 6.3%
3.6% 6.7% 8.0% 3.1% 3.6% 2.0% 4.4% 3.1% 7.3% 8.2% 2.7%
4.4% 8.1% 0.6% 5.9% 9.2% 7.3% 8.3% 8.6% 4.2% 9.9% 6.9%
* ε-greedy MAB .
!37
MAB(Multi Armed Bandit)

38. • CTR 90% ( )
• CTR
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1% 6.3%
3.6% 6.7% 8.0% 3.1% 3.6% 2.0% 4.4% 3.1% 7.3% 8.2% 2.7%
4.4% 8.1% 0.6% 5.9% 9.2% 7.3% 8.3% 8.6% 4.2% 9.9% 6.9%
Exploitation8.0% 8.2%
* ε-greedy MAB .
!38
MAB(Multi Armed Bandit)

39. 0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1% 6.3%
3.6% 6.7% 8.0% 3.1% 3.6% 2.0% 4.4% 3.1% 7.3% 8.2% 2.7%
4.4% 8.1% 0.6% 5.9% 9.2% 7.3% 8.3% 8.6% 4.2% 9.9% 6.9%
Exploitation
(10%) (90%)
&
?
:
?
: .
!39
MAB(Multi Armed Bandit)

40. 0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1% 6.3%
3.6% 6.7% 8.0% 3.1% 3.6% 2.0% 4.4% 3.1% 7.3% 8.2% 2.7%
4.4% 8.1% 0.6% 5.9% 9.2% 7.3% 8.3% 8.6% 4.2% 9.9% 6.9%
Exploitation
Exploration(10%) Exploitation(90%)
• MAB ?
• TS-MAB
ε-greedy
UCB(Upper Confidence Bound)
Lin-UCB
Thompson Sampling
NeuralBandit
LinRel (Linear Associative Reinforcement Learning)
!40
MAB(Multi Armed Bandit)

41. ε-Greedy MAB ε=0.10
41
10M Impressions
10%(ε)
1M Impressions( )
1 2 3 4 4 5
6 7 8 9 … 100

42. ε-Greedy MAB ε=0.10
42
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 0.1% 0.2% 1.0% … 2.2%
(100 ) 10k Impression

43. ε-Greedy MAB ε=0.10
43
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 0.1% 0.2% 1.0% … 2.2%
(100 ) 10k Impression
CTR = 1.5%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
50
3.0%

44. ε-Greedy MAB ε=0.10
44
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 0.1% 0.2% 1.0% … 2.2%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
50
3.0%
90%(1-ε)
9M Impressions
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%

45. ε-Greedy MAB ε=0.10
45
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 1.5% 0.2% 1.0% … 2.2%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
90%(1-ε)
9M Impressions
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%
10k Impression
CTR
Impressions CTR (3σ)

46. ε-Greedy MAB ε=0.10
46
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 1.5% 0.2%
10 Impressions
CTR
Impressions CTR

47. ε-Greedy MAB ε=0.10
47
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 1.5% 0.2% 1.0% … 2.2%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
90%(1-ε)
9M Impressions
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%
CTR
Impressions 99.7%(3σ)

48. ε-Greedy MAB ε=0.10
48
10M Impressions
10%(ε)
1M Impressions( )
1.1% 2.0% 8.2% 0.01% 4.6% 1.2%
5.2% 1.5% 0.2% 1.0% … 2.2%
Best
arm
( )
3
8.2%
7
5.2%
4
4.6%
50
3.0%
90%(1-ε)
9M Impressions
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%
CTR
Impressions 99.7%(3σ)
CTR 3.0%
3.0%
3.0% 3.0% 3.0%
3.0%3.0%

49. ε-Greedy MAB ε=0.10
49
10M Impressions
10%(ε)
1.1% 2.0%
5.2% 1.5% 0.2% 1.0% … 2.2%
Best
arm
( )
90%(1-ε)
(100 ) 10k Impression
CTR = 1.5%
CTR = 5.1%
CTR 4.74%
CTR
Impressions 99.7%(3σ)
3.0% 3.0%3.0%
Optimal Arm
Impressions (regret )

50. Thompson Sampling MAB ?

51. Thompson Sampling MAB
• (arm) CTR Beta(a,b) . ( a=click, b=unclick )
51
1
10%
Impressions : 10 50 100 200 1k 10k
2
25%
Impressions : 10 50 100 200 1k 10k

52. Thompson Sampling MAB
• (arm) CTR Beta(a,b) ( a=click, b=unclick )
52
1
10%
( ) CTR 15%
1 (10%<15%) 100 Impressions trial
.
Impression ->
Impressions : 10 50 100 200 1k 10k

53. Thompson Sampling MAB
• (arm) CTR Beta(a,b) ( a=click, b=unclick )
53
2
25%
2 CTR 25%>15%
( )
.
Impressions : 10 50 100 200 1k 10k

54. • 1K Impressions
54
CTR

55. • 10K Impressions
55
CTR

56. • 50K Impressions
56
CTR

57. • 100K Impressions
57
CTR

58. • 500K Impressions
58
CTR

59. • 1M Impressions
59
CTR

60. • 1M Impressions
60
CTR
CTR (Arm)

61. • 1M Impressions
61
CTR
CTR
( )
.
.

62. • 1M Impressions
62
CTR
CTR
( )
.
.
TS-MAB
& Trade-off
Regret( ) .

63. User Clustering
• CTR .
CTR : 7.6%
A : 25% (30 )
B : 2.1% ( )
C : 7.1% ( )
CTR
!63

64. User Clustering
• X CTR
• CTR
200 8,000

65. User Clustering
•
• 8
8
8,000 64,000
CTR

66. Clustering ?
CB(image,Text)
Feature User Feature
[0.628, 0.88, 0.376, 0.065, 0.849]
[0.508, 0.268, 0.193, 0.125, 0.425]
[0.431, 0.077, 0.012, 0.07, 0.037]
[0.915, 0.294, 0.713, 0.851, 0.423]
[0.508, 0.268, 0.193, 0.125, 0.425]
[0.607, 0.639, 0.554, 0.092, 0.297]
[0.587, 0.319, 0.094, 0.173, 0.177]
[0.409, 0.458, 0.48, 0.319, 0.783]
[0.479, 0.434, 0.618, 0.297, 0.752]
[0.467, 0.206, 0.905, 0.7, 0.568]
, , ,
, ,
, , ,
!66
1
2
3
4
5
6

67. Clustering ?
14 CB(image,Text)
Feature User Feature
[0.628, 0.88, 0.376, 0.065, 0.849]
[0.508, 0.268, 0.193, 0.125, 0.425]
[0.431, 0.077, 0.012, 0.07, 0.037]
[0.915, 0.294, 0.713, 0.851, 0.423]
[0.508, 0.268, 0.193, 0.125, 0.425]
[0.607, 0.639, 0.554, 0.092, 0.297]
[0.587, 0.319, 0.094, 0.173, 0.177]
[0.409, 0.458, 0.48, 0.319, 0.783]
[0.479, 0.434, 0.618, 0.297, 0.752]
[0.467, 0.206, 0.905, 0.7, 0.568]
, , ,
, ,
, , ,
8 (#0~#7)
?

68. -
•
• #1, #5, #6, #7
• #0, #2 #3

69. -
•
• #1, #5, #6, #7
• #0, #2 #3
/

71. #1
, ,

72. #3
, ,

73. /
#3
, ,

74. Tag

75. Tag

76. /
.

77. ?
77
#2
#1
#3
User Clustering
Targeting
CTR
1 : CTR 9.1%
2 : CTR 8.8%
3 : CTR 8.0%
4 : CTR 7.8%
5 : CTR 7.1%
6 : CTR 6.8%
7 : CTR 6.7%
…
MAB
Ranking

78. ?
78
#2
#3
User Clustering
Targeting
Ranker
MAB
Ranking
Targeter

79. + MAB

80. -ViewerEnd
• :
CTR(%)
•
• (Item Feature)
• MAB(Multi Armed Bandit)
!80

81. ?
81
#2
#3
User Clustering
Targeting
Ranker
MAB
Ranking
Targeter

82. ?
82
(Item)
Targeter
Feature
Targeting
Ranker
MAB
Ranking

83. -ViewerEnd
(CF)
(Text)
(Image)
!83
1 2 3

84. - Item Features
/ Image
(1) Image Feature
, Text
(2) Text Feature
Feedback
(3) CF-Feature
!84

85. ?
!85

86. ?
!86

87. ?
!87

88. 1 3 41 1 92 1
1 3 41 1 92 1
1 3 41 1 92 1
Image
Text( )
CF( )
!88
1 3 41 1 92 1Image Style

89. Image
Text( )
CF( )
Image Style
Style transfer network
!89
1 3 41 1 92 1

90. Image Style
Text( )
CF( )
Image Object Detection Task
Image Object
Pre-trained VGG19 Model
!90
1 3 41 1 92 1

91. Image Style
Image
CF( )
,
Keyword
Word2Vec
!91
1 3 41 1 92 1

92. Image Style
Image
Text( )
CF( )
Matrix Factorization(ALS)
with implicit feedback
(Feedback) Item-User
!92
1 3 41 1 92 1

93. !93
1 3 41 1 92 1
1 3 41 1 92 1
1 3 41 1 92 1
1 3 41 1 92 1

94. “ ”
(%)
!94

95. 0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1%
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1%
0.4% 4.4% 2.9% 7.3% 2.3% 8.7% 0.2% 1.0% 1.9% 8.1%
0.4% 6.0% 2.9% 7.3% 2.7% 5.6% 6.7% 1.0% 1.9% 8.1%
MAB
(%)
!95

96. 0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1%
0.4% 4.0% 2.9% 7.3% 2.7% 8.7% 6.7% 1.0% 1.9% 8.1%
0.4% 4.4% 2.9% 7.3% 2.3% 8.7% 0.2% 1.0% 1.9% 8.1%
0.4% 6.0% 2.9% 7.3% 2.7% 5.6% 6.7% 1.0% 1.9% 8.1%
!96

97. ?
97
(Item)
Targeting
CTR
1 : CTR 9.1%
2 : CTR 8.8%
3 : CTR 8.0%
4 : CTR 7.8%
5 : CTR 7.1%
6 : CTR 6.8%
7 : CTR 6.7%
…
MAB
Ranking

98. + MAB
!98

99. !99

100. 100
1 2 3 4 5 6
…
89 90
(Clicks)
Impression

101. 101
1 2 3 4 5 6
…
89 90
(Use Coin)
Impression

102. 102
User Cluster +
+
MAB
MAB
RankerTargeter
=
=

103. 103
+
MAB
Conditional Bandit
Exponential Smoothing
Seen Decay
Soft User Clustering
Retention Model
Unbiased Most Popular
Feature Matching
Targeter
=
=
Ranker

104. 104
• MAB
- Bandit Algorithm = Thompson Sampling(
- Reward = Click (with Unclick )
- Play Arms = Cluster Most Popular
- None Stationary = Exponential Decaying
• 2
- = # of clicks / # of impressions
- = # of use_coins / # of impressions
1. MAB

105. 105
• MAB
- Bandit Algorithm = Thompson Sampling
- Reward = Click (with Unclick )
- Play Arms = Cluster Most Popular
- None Stationary = Exponential Decaying
• 2
- = # of clicks / # of impressions
- = # of use_coins / # of impressions
1. MAB
Use Coin( )
MAB Reward Use Coin, Click + User Coin
by @brandon.lim

106. 106
1. MAB ?
(Beta) (Alpha) -20% —>
? ?
MAB ?
-20%

107. 2. Conditional Bandit
107
1 2 3 4 5 6
…
89 90
by @troye.kwon

108. 2. Conditional Bandit
108
1 2 3 4 5 6
…
89 90
Impressions
Reward=Click( )
α=click, β=unclick
MAB
by @troye.kwon

109. 2. Conditional Bandit
109
1 2 3 4 5 6
…
89
Impressions
Reward=Click( )
α=click, β=unclick
MAB
Reward=Use Coin( )
α=use-coin, β=click
MAB
by @troye.kwon

110. 2. Conditional Bandit ?
110
(Beta) (alpha)
? ?
- MAB .

111. 3. Retention Model
• : .
,
“ ” .
•
• MAB
111
by @jinny.k
+
MAB
Targeter Ranker

112. 3. Retention Model ?
112
by @jinny.k
(CTR) (CVR)
? (CTR) ?

113. 4. Seen decay
• : Negative Feedback
• click impression Ranker
• : (alpha) (Beta)
113

114. -> CTR
114
•
• Hard Clustering(k-Means) —> Soft Clustering (pLSI)
• Feature Matching
• Targetting Genre/Tag Matching
• MAB non-stationary Exponential Smoothing
• Targeting Unbiased Most Popular
• MAB Hyper parameter Turning

115. (%)
!115
Soft Clustering (pLSI) Feature Matching
Conditional Bandit Retention Model
Exponential Smoothing
MABUnbiased Most Popular

116. ?
?
!116

117. !117

118. ?
118

119. ?
119

120. ?
120

121. ?
121

122. ?
122

123. 0.1%
123
= 3.96%
= 6.70%
= 2.63%
= 0.07%

124. -> 4.56
124
= 3.96%
= 6.70%
= 2.63%
= 0.07%
4.56

125. -> 4.56
125
= 3.96%
= 6.70%
= 2.63%
= 0.07%
4.56
AB
(<0.001)
Feedback (>4.56days)

126. /
,
?

127. .
127
Base : Editor’s ( X) 1.9%
Alpha : 1 4.8%
Beta : 2 5.5%
Gamma : 3 6.5%
CTR
1 2 3 4
…

128. .
128
Base : Editor’s ( X) 1.9%
Alpha : 1 4.8%
Beta : 2 5.5%
Gamma : 3 6.5%
CTR
+ 242% + 42%
1 2 3 4
…
CTR

130. .
?

131. / ?
?

132. / ? ->
? ->

133. 133
YOU
?

134. .
!134
|