Background statement Explain what we mean by multiple objectives in this context Give two examples Motivated from the user point of view Motivated from the system point of view User point of view would improve the user experience System point of view would take into account other external factors
Background statement Explain what we mean by multiple objectives in this context Give two examples Motivated from the user point of view Motivated from the system point of view User point of view would improve the user experience System point of view would take into account other external factors
Improving the accuracy of the system does not necessarily equal to improving user experience Defined the recommender algorithm how good it performs on a metrics and how fast it can do its job But we need additional factors that would define the whole system For example if we take a VOD service User interface affects how user reads, understands find information including the recommendation How interesting the movies are that the user gets
System related factors include The underlying hardware of the system How much, how fast an item can be delivered
External factors include How much the company earns on an item, for example some items have a higher profit margin Personalised advertisement, recommending preferred items
User related and system related factors can directly improve user experience And we suggest that the combination of accuracy and other factors might improve user experience And the rest can help to maximize profit
Consider accuracy as the main objective
Combine these into a single optimization framework. Take the prediction values of any baseline algorithm - r-hat. And add additional constraints that are equally important. Baseline algorithm to minimize errors. Higher weights represent higher importance Without constraints it returns the same as the original order
Recommendation is considered to be a ranking problem Top-N list It is a linear optimization problem, so global solution can be found. Biasing the original order by using constraints
Introducing two case studies to illustrate the use of the framework The first one is aim to promote items from the long tail. Assumption Current systems are biased towards popular items Picked the first 100 items that have received the highest ratings Where SVD is likely to place them Plot the distribution Figure show the probability that some popular are placed higher on the recommendation list by all the widely used recommender algorithms
Does that reflect user needs? We assume that discovering unknown items are more valuable We aim to identify users who would like alternative choices And recommend from the long tail for them Keep recommending popular items if the user has a more mainstream taste
We add this as an inequality constraint to the framework m is a vector that contains the mainstreamness value of each items in the recommended list m_u measures the mainstreamness of the user How these values are calculated are in the paper if you are interested
We added another extra bit to the framework to diversify items
Experimenting with diversification Promote item from the long tail that differ from each other Higher covariance That would reduce the risk of such an extension
Since we approach recommendation as a ranking problem We used the following IR measures
The probability that popular items ranked higher is significantly reduced Only 32% of the users have popular items in the top position The baseline is 45% We get reduction until position four Then it is slightly worse than the baseline This is the case when user studies would provide a better way to evaluate performance
Long tail constraint alone Long tail constraint with diversification Slight performance loss for all measures, except one.
The other case – system case Adding other non-user related factors to the system the availably of certain items The aim is to rank items lower if we are about to run out of stock But also minimize performance loss
The second scenario was evaluated by simulating the stock level Of an imaginary company for 50 days We presented a recommendation list to a random number of users each day The probability that a user took an item depended on the rank The cumulative probability up to the present point was based on a, How many times an item was show in the past B, And at which ranking position To evaluate the system we monitored the waiting list size
We introduce a cut-off point from which the system will start to reorganize the ranking list.
Adding it as a constraint to the system, in a same fashion s is a vector that represents the probability that the item is available at the given time For each item in the list s_u controls how cautious the system should be with stock level. e.g if s_u is higher the system starts penalizing items later As they are getting less available
The experiment is designed to run out of stock This graph shows the waiting list size for the first 20 days With respect to parameter c It controls when the system should start penalizing items as they are running out of stock e.g if c is set to 0.8 then after it is 80% likely that an item is taken the system starts penalizing
This table show the average and the maximum performance loss per day As you can see from c=0.8 and above there is only a slight performance loss of the system.
Two papers Optimizing algorithm from the user point of view One way is to identify different errors A general framework to handle multiple goals Two scenarios to illustrate that
Improving user experience might be validated using user studies
Transcript
1.
Multiple Objectives in Collaborative Filtering
Tamas Jambor and Jun Wang
University College London
2.
Structure of the talk
• Motivation
• Multiple objectives
• User perspective
– Promoting less popular items
• System perspective
– Stock management
3.
Motivation
• In the RecSys community, many research efforts
are focused on recommendation accuracy
• And yet accuracy is not a only concern
• Practical recommender systems might have
multiple goals
4.
Improved Accuracy != Improved User experience
Algorithm
Additional
factors
Available resources
Cost of delivery
User interface
Diverse choices
Profitability per item
Advertisement
5.
Improved user experience
Available resources
Cost of delivery
User interface
Diverse choices
Profitability per item
Advertisement
Additional
factors
Accuracy
Improved
user
experience
6.
Handling Multiple objectives
• Accuracy is the main objective
– Defined in the baseline algorithm
• User perspective
– Define and consider user satisfaction as priority
• System perspective
– Consider additional system related objectives
• Objectives of the system might contradict
7.
Where to optimize?
• In the objective function or as a post-filter?
• Post-filters have the advantage to
– Add to any baseline algorithm
– Extend easily
– Add multiple goals
8.
The proposed optimization framework
(for each user)
• Add additional constraints of w
0
11:tosubject
ˆmax
w
w
rw
T
T
w
9.
Properties of the framework
• Linear optimization problem
• Recommendation as a ranking problem
• Constraints provide the means of biasing the
ranking
10.
User case – Promoting the Long Tail
Current systems are biased towards popular items
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 8 9 10
Probabilityofanyof100mostpopularitembeingat
rankingposition
Ranking Position
SVD
User-based
Item-based
Random Sample
11.
Promoting the Long Tail
• Does that reflect real user needs?
• Popular items might not be interesting for the user
• Discovering unknown item could be more valuable
• The aim is to reduce recommending popular items
– if the user is likely to be an interested in alternative
choices
– keep recommending popular items otherwise
12.
Promoting the Long Tail
• Extending the optimization framework
0
11
:tosubject
ˆmax
w
w
mwm
rw
T
u
T
T
w
13.
Promoting the Long Tail and Diversification
• Diversifying the results
0
11
:tosubject
ˆmax
w
w
mwm
wwrw
T
u
T
TT
w
14.
Diversification
• Increase the covariance between recommended
items
– Reduce the risk of expanding the system
– Provide a wider range of choice
16.
Evaluation metrics
• Recommendation as a ranking problem
• IR measures
– Normalized discounted cumulative gain (NDCG)
– Precision
– Mean reciprocal rank (MRR)
• Constraint specific measures
17.
Results: Promoting the Long Tail
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 8 9 10
Probabilityofanyof100mostpopularitembeingat
rankingposition
Ranking Position
Baseline (SVD)
Long Tail Constraint
Long Tail Constraint and
Diversification (λ=6)
Random Sample
18.
Results: Promoting the Long Tail
Baseline (SVD) LTC LTC and Div (λ=6)
NDCG@10 0.8808 0.8780 (-0.3%) 0.8715 (-1.0%)
P@10 0.8204 0.8207 (+0.2%) 0.8177 (-0.3%)
MRR 0.9518 0.9453 (-0.6%) 0.9349 (-1.7%)
19.
System case – Resource Constraint
• Introducing external factors to the system
• Stock availability of recommended items
• The aim is to rank items lower, if less of them are
available
• Minimizing performance loss
20.
Simulation
• Online DVD-Rental company
– Operates a warehouse
– Only a limited number of items are available
• Recommend items that are in stock higher in the
ranking list
21.
Simulation
• User choice is based purely on recommendation
• Simulating the stock level for 50 days
– Present a list of items to a random number of users
– The probability that the item is taken depends on the
rank
– Cumulative probability depends on how many times the
item was shown and at which rank position
22.
Cut-off point
• Threshold c controls the cut-off point from which
the system starts re-ranking items
cp
cps
s ti
titi
ti
,
,,
,
if0
if
23.
Resource Constraint
• Extending the optimization framework
0
11
:tosubject
ˆmax
w
w
sws
rw
T
u
T
w
24.
Evaluation: Monitoring the waiting list size
• Waiting list
– If item is not in stock, user puts it on their waiting list
– When item returns, it goes out to the next user
• Waiting list size represents how long a user has to
wait to get their favourite items
28.
Conclusion
• Recommender systems have multiple objectives
• Multiple optimization framework
– Expand the system with minor performance loss
– It is designed to add objectives flexibly
– It can be added to any recommender system
• Two scenarios that offer practical solutions
– Long-tail items
– Stock simulation
29.
Future plan
• Personalized digital content delivery
– Reduce delivery cost
• Diversification and the long tail
– Does recommendation kill diversity?
• Evaluate improved user experience
– User studies
31.
References
• Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms.
ACM Trans. Inf. Syst. 22(1) (2004)
• Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic
framework for performing collaborative filtering. In: SIGIR '99. (1999)
• Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for
recommender systems. Computer 42(8) (2009)
• Wang, J., de Vries, A.P., Reinders, M.J.T.: Unifying user-based and item-
based collaborative filtering approaches by similarity fusion. In: SIGIR '06:
Proceedings of the 29th annual international ACM SIGIR conference on
Research and development in information retrieval, New York, NY, ACM Press
Be the first to comment