Recent work in recommender systems mainly focuses on fairness in recommendations as an important aspect of measuring recommendations quality. A fairness-aware recommender system aims to treat different user groups similarly. Relevant work on user-oriented fairness highlights the discriminative behavior of fairness-unaware recommendation algorithms towards a certain user group, defined based on users' activity level. Typical solutions include proposing a user-centered fairness re-ranking framework applied on top of a base ranking model to mitigate its unfair behavior towards a certain user group i.e., disadvantaged group. In this paper, we re-produce a user-oriented fairness study and provide extensive experiments to analyze the dependency of their proposed method on various fairness and recommendation aspects, including the recommendation domain, nature of the base ranking model, and user grouping method. Moreover, we evaluate the final recommendations provided by the re-ranking framework from both user- (e.g., NDCG, user-fairness) and item-side (e.g., novelty, item-fairness) metrics. We discover interesting trends and trade-offs between the model's performance in terms of different evaluation metrics. For instance, we see that the definition of the advantaged/disadvantaged user groups plays a crucial role in the effectiveness of the fairness algorithm and how it improves the performance of specific base ranking models. Finally, we highlight some important open challenges and future directions in this field.

1. Experiments on Generalizability of
User-Oriented Fairness in
Recommender Systems
Mohammad Aliannejadi
IRLab
University of Amsterdam
m.aliannejadi@uva.nl
Hossein A. Rahmani
WI Group
University College London
h.rahmani@ucl.ac.uk
Mohammadmehdi Naghiaei
DECIDE
University of Southern California
naghiaei@usc.edu
Mehdi Dehghan
Abin's Lab
Shahid Beheshti University
mahdi.dehghan551@gmail.com
March 2022

2. Outline
01
Motivation
02
Fairness Taxonomy
03
User-oriented Fairness
04
Reproducibility Aspects
05
Reproducibility Methodology
06
Discussion and Findings

3. collection
recommendation
learning
Recommender Engine Echo Chambers
Matthew Effect
Information Asymmetry
Biases in RecSys Potential Consequences of Unfairness
Data
User
Chen, Jiawei, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. "Bias and debias in
recommender system: A survey and future directions." arXiv preprint arXiv:2010.03240 (2020).

4. Different Perspective in
Fairness in RecSys
● User Fairness vs. Item Fairness
● Group Fairness vs. Individual Fairness
● Single-sided Fairness vs. Multi-sided Fairness
Consumer
Providers
Provided
Items
Recommendation
Engine
Recommended
Items
Side Stakeholder
Preferences
Abdollahpouri, Himan, and Robin Burke. "Multi-stakeholder recommendation and its connection to
multi-sided fairness." arXiv preprint arXiv:1907.13158 (2019).

5. Mitigating Harmful Biases: Strategies
Pre-processing strategies In-processing strategies Post-processing strategies
e.g. data rebalancing e.g. regularization e.g. re-ranking

6. User-oriented Fairness Re-ranking (UFR)
By The WISE Lab at Rutgers University (https:/
/wise.cs.rutgers.edu/)
Li, Yunqi, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. "User-oriented fairness in recommendation." In Proceedings of the Web Conference 2021, pp. 624-632. 2021.
collection
fair recommendation
learning
Recommender System
Data
User
UFR
Post-processing
User Fairness
Single-sided Fairness
Group Fairness

7. UFR Methodology Overview
Base Rec. Models Recommendations
UFR Model
(re-ranking)
Fair Recommendation
● A Re-ranking Integer Programming Method
● Objective Function:
○ Re-ranking the recommendation list of each user provided by the baseline algorithms
● Constraint:
○ Minimizing the difference in performance between the groups of users
○ Select only K items to recommend

8. Reproducibility Aspects Overview
Effect of Data Characteristics
8 different datasets, and 6 domains.
Ability to propagate bias differs from
one another
6 deep and shallow models.
Effect of the user grouping methods
on the performance
Level of activity and consumption of popular items.
Underlying trends and trade-offs
between various evaluation
metrics.
NDCG, UGF, Novelty, GAP
Domain Base Rec. Alg. User Groups Assumptions Fairness vs. Effectiveness Metrics

9. Reproducibility Methodology
● Baselines
MostPop, BPR
● Traditionals
PF, WMF
● Neurals
NeuMF, VAECF
● User Grouping
G1: Level of activity
G2: Consumption of popular items
● User-oriented Group
Fairness (UGF)
● Beyond-accuracy, e.g.
Novelty
● Delta Group Average
Popularity (ΔGAP)
● Movie (MovieLens)
● Music (Last.fm)
● Point-of-Interest
(Gowalla, Foursquare)
● Book (BookCrossing)
● eCommerce
(AmazonToy, AmazonOffice)
● Opinion (Epinion)
Feel free to ask your questions about details on reproducibility methodology in Q&A.
Datasets
Fairness Assumption
and Evaluation
Base Ranking Models

10. Reproducibility Methodology (Cont.)
● Google Colab
Jupyter Notebooks
● Repository
https:/
/github.com/rahmanidashti
/FairRecSys
● Cornac1
recommendation toolkit
● MIP2
, Groubi3
optimization toolkit
1
https:/
/cornac.preferred.ai/
2
https:/
/www.python-mip.com/
3
https:/
/www.gurobi.com/
● Having access to the
relevance judgments
● Relevance labels are
estimated based on the
training data.
Details can be found on our repository on GitHub: https://github.com/rahmanidashti/FairRecSys
Relevance Estimation
Implementation Details Code and Data

11. Discussions
and Findings

12. Domain.
● Certain domains exhibit different patterns.
● A wider range of variance in UGF improvement on
implicit feedback datasets compared to explicit
feedback datasets.
● The higher number of interactions per user and
item can lead to better performance of fairness
model.

13. Recommendation Models.
● Observations suggest that the user grouping assumption
also affects the sensitivity of a model to fairness.
● WMF demonstrates the most robust performance in
mitigating user-oriented unfairness, exhibiting the least
variance, as well as the best average improvement in
terms of UGF.
● Comparing Figures (a) and (b), we observe two
considerably different behaviors of different base ranking
models when it comes to the variance and UGF
improvement.

14. Fairness vs. Effectiveness Metrics

15. User Grouping Methods
(a) G1
(b) G2
(c) G1
(d) G2

16. Conclusion and Future Work
● Our experiment on reproducing UFR using six base recommendation algorithms shows that UFR is
algorithm agnostic. In other words, it will improve fairness regardless of the baseline recommendation
algorithm but to a different extent.
● We found that the user grouping method is one of the most important aspects of the user fairness
algorithm since it directly affects our interpretation of an algorithm’s fair behavior.
● Exploring other methodologies for mitigating unfairness that can show less dependency on the data
distribution and domain properties should be explored in this area.

17. Thank you!
Mohammadmehdi Naghiaei
University of Southern California
naghiaei@usc.edu
@naghiaei
https://github.com/rahmanidashti/FairRecSys