SlideShare a Scribd company logo
Fair Recommendation
Systems
Sharmistha Chatterjee
Senior Manager Data Sciences at
Publicis Sapient
Author | Speaker | GDE for ML
@sharmichat82
http://techairesearch.com/
Women in Data Science - Mysuru
Key Takeaways
Fairness in Machine Learning
Need for Fairness in ML based Recommendation
Systems
Fairness-aware Machine Learning Best
Practices
Multi-sided Fairness , Platforms and Metrics
Why Fairness in Recommendations
Fair Housing Act LGBT
Fairness Act
Disability
Status
Disparate
Impact
Source - https://www.nytimes.com/2019/05/07/opinion/google-sundar-pichai-privacy.html
Disability
Laws and Policies
Ethical Artificial Intelligence – Fairness Origin
Professor Klaus Schwab - Executive Chairman of WORLD ECONOMIC
FORUM
“We must address, individually
and collectively, moral and
ethical issues raised by cutting-
edge research in artificial
intelligence and biotechnology,
which will enable significant life
extension, designer babies, and
memory extraction.” —Klaus
Schwab
Source - https://www.statista.com/chart/18805/highest-penalties-in-privacy-enforcement-actions-worldwide/
Foundation of Algorithmic Justice League
Joy Buolamwini - computer scientist and digital activist based at the
MIT Media Lab
Whether AI will help us reach our aspirations or
reinforce the unjust inequalities is ultimately up to us.
If we fail to make ethical and inclusive artificial
intelligence, we risk losing gains made in civil rights
and gender equity under the guise of machine
neutrality
Source - https://www.statista.com/chart/18805/highest-penalties-in-privacy-enforcement-actions-worldwide/
AI Regulation
Sundar Pichai
The head of Google and parent company
Alphabet has called for artificial
intelligence (AI) to be regulated
● Fair Marketplace
● Legal obligation
● Social Responsibility
● Business Requirement /Model
Source - https://builtin.com/artificial-intelligence/ai-laws-regulations
Unfair Recommendation Systems from Biases
Source - Simple Demographics Often Identify People Uniquely
Differences in demographics or other user characteristics.
Differences across online and offline communities, platforms
and contexts
Lexical, syntactic, semantic, and structural differences in the contents
Connections, interactions, or activities obtained from networks and their attributes
Society norms, prejudices, economical status
Seasonal, weekly or observed at a certain time
Population
Behavioral
Content
Production
Linking
Social/
Economic
Temporal
Types of Biases
Recommendation Systems
Differences in demographics or other user characteristics.
Differences across online and offline communities, platforms
and contexts
Lexical, syntactic, semantic, and structural differences in the contents
Connections, interactions, or activities obtained from networks and their attributes
Society norms, prejudices, economical status
Seasonal, weekly or observed at a certain time
Impact of Bias
● Biased customer reviews
● Disparate impact on minority drivers
● Unjust outcomes with low wages
Recommend Fairly to All Groups of Users
Fairness
Escalate
Source -https://course.ece.cmu.edu/~ece734/lectures/lecture-2018-10-08-deanonymization.pdf
Product Goals Stakeholder Identification
Analyze
Mitigate TransparencyMonitor Performance
Best Practices for Removing Bias
Where and How
Legal
Fairness Experts
User Researchers
Google’s Responsible AI - Expert Opinion
Where and How
Social Science
Backgrounds
Gender and sexual
orientation
Diverse Identities - Race, Nationality, Religion
Google’s Responsible AI - Diversity and Inclusion
Fairness for Individual and Groups
Pre-processing , Learning Function , Post Processing
Source - https://arxiv.org/pdf/1906.08732.pdf
FAIRNESS COMMON TERMS
Source
: https://arxiv.org/pdf/1906.08732.pdf
Envy-freeness requires
that every user should
prefer their own
allocation to that of
everyone else; it ignores
users’ qualifications and
considers preferences
Individual or metric
fairness ignores
preferences and
requires that similar
users should be
treated similarly.
Multiple task
fairness, requires
that individual
fairness is satisfied
separately and
simultaneously for
all categories
Inter-category envy-
freeness, which allows
users to specify a set of
categories that they
“care” about, and
guarantees that they
receive at least that they
care about as any other
individual
Fairness Constraints with 2 latent variables:
ProtectedItemRating(i) ⇒ UnProtectedItemRating(i)
UnProtectedItemRating(i) ⇒ ProtectedItemRating(i)
Protected(u) ∧ RATING(u, i) ∧ ItemGroup(i, g)
⇒ ProtectedItemGroupRating(g)
¬ Protected(u) ∧ Rating(u, i) ∧ ItemGroup(i, g)
⇒ UnProtectedItemGroupRating(g)
Recommendation Systems
MetricsSL
No
1.
2.
3.
4.
Statistical independence between recommendation
results and the sensitive attribute
Value unfairness (estimation error across user groups)
Absolute unfairness
Underestimation unfairness
5. Overestimation unfairness
Fairness Metrics between advantaged and disadvantaged groups
6. Non-parity Fairness
Source -https://arxiv.org/pdf/1809.09030.pdf
𝑝𝑖- d-dimensional vector representing the ith user,
𝑞 𝑗 − d−dimensional vector representing the jth item,
𝑢𝑖 𝑎𝑛𝑑 𝑣𝑗. user and item respectively
X – observed ratings
Recommendation Diversity and User Fairness
Ranking algorithms
Top-l CF
recommendations
Sample K items
uniformly – K nearest
neighbors
Greedy algo – New
recommendations above
threshold
● Individual diversity - Diverse recommendations
to the users.
● Aggregate diversity - Improve item diversity by
recommending them at least once across all
users.
● Limitations – No fairness like differential
treatment of two users or two items.
● Impact – Disparity among users increases with
Aggregate Diversity
CF Output
Source - https://www.researchgate.net/publication/324640535_User_Fairness_in_Recommender_Systems
Different Types of multi-sided Fairness
● Multi-stakeholder Recommender Systems with multiple
goals (e.g. LinkedIn, Etsy)
● Reciprocal Recommendations –Bilateral and acceptable
to both parties. (e.g. job, mentor, business partner)
○ Peer-to-peer recommendation –Sharing economy,
online advertising and scientific collaboration
● Objective – Maximize system utility
Multi-sided Fairness in Recommendation Systems
Source - http://proceedings.mlr.press/v81/burke18a/burke18a.pdf
Regularization based Sparse Linear Method (SLIM) with Neighborhood balance
● Provider Fairness – Market Diversity , avoid
monopoly by recommending minority owned
businesses.
● Consumer Fairness – Personalization, Disparate
impact of recommendation on protected
classes
Multi-sided Fairness – User Based Neighborhoods
Source -http://proceedings.mlr.press/v81/burke18a/burke18a.pdf
Unbalanced Neighbors
Balanced Neighbors
FairnessFairness
● Regression coefficient <user, item> pair
● Minimize regularized loss function
Multi-sided Fairness - Item Based Neighborhoods
Source -https://medium.com/@cfpinela/recommender-systems-user-based-and-item-based-collaborative-filtering-
5d5f375a127f
● Use case - Exposure to loans
from different geographic
regions
● Items in protected group are
in neighborhoods that have
balanced membership of
items from the unprotected
group● Balance between protected and non-
protected neighbors for each user.
Balanced Neighborhood SLIM – Balance Personalization with Fairness
Two-sided Fairness Two-sided Platforms
● Fair recommendation -> Fair Allocation
○ Maximin Share (MMS) of exposure for Producers
and Envy-Free up to One Good (EF1) fairness for
every customer
● Cardinality constrained fair allocation
○ All items are grouped into disjoint categories and no
agent receives more than a pre-specified number of
items from same category
○ Exactly k items are allocated to each customer
Datasets – GL-CUSTOM, (Relevance
Scoring) , GL-FACT and LAST.FM DATASET
–(Latent Factorization)
Source - https://people.mpi-sws.org/~achakrab/papers/patro_FairRec_WWW20.pdf
Source - https://people.mpi-sws.org/~achakrab/papers/patro_FairRec_WWW20.pdf
FAIR RECOMMENDATION SYSTEMSAdvantages and Metrics
MetricsSL
No
1.
2.
3.
4.
Fraction of Satisfied Producers
Inequality of Product Exposures
Exposure Loss on Producers
Mean-Average Envy
5. Loss and Disparity in Customer Utilities
● Advantages
○ Economic
○ Social
○ Judicial and
○ Long-term sustainability
K items
Evaluate Fair Recommender Systems - Pairwise
Comparisons Original vs Pairwise Regularization
● Pairwise regularization to optimize for
inter-group pairwise fairness.
● Corresponds to ranking performance
● Pairwise Fairness - Likelihood of a clicked
item being ranked above another relevant
unclicked item is the same across both
groups (same/opposite), conditioned on
the items have safe level of engagement.
● Real world scalable product grade systems
Source - http://alexbeutel.com/papers/kdd2019_pairwise_fairness.pdf
Intra Group and Inter Group Pairwise Fairness
Compositional Fairness
Source -
https://www.researchgate.net/publication/335755357_RecSim_A_Configurable_Simulation_Platform_for_Recommender_
Systems
Each user should be envy-free with respect to
categories (inter-category)
Combination of inter-category envy-freeness and total-variation fairness into
hybrid fairness
Individual fairness within each category
End-to-end Fair Systems with High Platform Utility
Conclusion
● Handle sparse data
● Bias from Advertisers
● Formulate right personalization and key
outcome
● Techniques to mitigate harms and mis-
behaviors
● Computational enhancement for Multi-
stakeholder fairness optimization
● Network structures that define relationships
between providers and users
● Sequential notions of fairness into recommender
systems with additional time-dependent
constraints
● Equal treatment vs equal outcome
Source: https://www.researchgate.net/publication/314971193_Optimal_Performance_vs_Fairness_Tradeoff_for_Resource_Allocation_in_Wireless_Systems
Questions?
References
● https://www.ibm.com/blogs/research/2018/09/ai-fairness-360/
● https://arxiv.org/pdf/1810.01943.pdf
● https://pair.withgoogle.com/explorables/measuring-fairness/
● https://www.research.ibm.com/artificial-intelligence/trusted-ai/
● https://pair.withgoogle.com/explorables/measuring-fairness/
● https://www.researchgate.net/publication/314971193_Optimal_Performance_vs_Fairness_Tradeoff_for_Resource_Alloca
tion_in_Wireless_Systems
● https://people.mpi-sws.org/~achakrab/papers/patro_FairRec_WWW20.pdf
● http://alexbeutel.com/papers/kdd2019_pairwise_fairness.pdf
● http://proceedings.mlr.press/v81/burke18a/burke18a.pdf
● https://course.ece.cmu.edu/~ece734/lectures/lecture-2018-10-08-deanonymization.pdf
● https://www.nytimes.com/2019/05/07/opinion/google-sundar-pichai-privacy.html
● https://arxiv.org/pdf/1906.08732.pdf
● Algorithmic Fairness - https://arxiv.org/pdf/2001.09784.pdf
● Fairness Definitions Explained- https://fairware.cs.umass.edu/papers/Verma.pdf
● https://www.researchgate.net/publication/324640535_User_Fairness_in_Recommender_Systems
● https://github.com/gourabkumarpatro/FairRec_www_2020

More Related Content

What's hot

Boston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender SystemsBoston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender Systems
James Kirk
 
Recommender system introduction
Recommender system   introductionRecommender system   introduction
Recommender system introductionLiang Xiang
 
Recommender system
Recommender systemRecommender system
Recommender system
Nilotpal Pramanik
 
An introduction to Recommender Systems
An introduction to Recommender SystemsAn introduction to Recommender Systems
An introduction to Recommender Systems
David Zibriczky
 
Tutorial on Advances in Bias-aware Recommendation on the Web @ WSDM 2021
Tutorial on Advances in Bias-aware Recommendation on the Web @ WSDM 2021Tutorial on Advances in Bias-aware Recommendation on the Web @ WSDM 2021
Tutorial on Advances in Bias-aware Recommendation on the Web @ WSDM 2021
Mirko Marras
 
Recommendation system
Recommendation systemRecommendation system
Recommendation system
Rishabh Mehta
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
Xavier Amatriain
 
Overview of recommender system
Overview of recommender systemOverview of recommender system
Overview of recommender system
Stanley Wang
 
Recommender system algorithm and architecture
Recommender system algorithm and architectureRecommender system algorithm and architecture
Recommender system algorithm and architectureLiang Xiang
 
Recommendation Systems: Applying Amazon's Collaborative Filtering Methods to ...
Recommendation Systems: Applying Amazon's Collaborative Filtering Methods to ...Recommendation Systems: Applying Amazon's Collaborative Filtering Methods to ...
Recommendation Systems: Applying Amazon's Collaborative Filtering Methods to ...
Nguyen Cao
 
Introduction to Recommendation Systems
Introduction to Recommendation SystemsIntroduction to Recommendation Systems
Introduction to Recommendation Systems
Trieu Nguyen
 
Collaborative Filtering using KNN
Collaborative Filtering using KNNCollaborative Filtering using KNN
Collaborative Filtering using KNN
Şeyda Hatipoğlu
 
[Final]collaborative filtering and recommender systems
[Final]collaborative filtering and recommender systems[Final]collaborative filtering and recommender systems
[Final]collaborative filtering and recommender systems
Falitokiniaina Rabearison
 
Anatomy of an eCommerce Search Engine by Mayur Datar
Anatomy of an eCommerce Search Engine by Mayur DatarAnatomy of an eCommerce Search Engine by Mayur Datar
Anatomy of an eCommerce Search Engine by Mayur Datar
Naresh Jain
 
Recommendation System
Recommendation SystemRecommendation System
Recommendation System
Anamta Sayyed
 
Collaborative Recommender System for Music using PyTorch
Collaborative Recommender System for Music using PyTorchCollaborative Recommender System for Music using PyTorch
Collaborative Recommender System for Music using PyTorch
Valentin Nagacevschi
 
AI-driven product innovation: from Recommender Systems to COVID-19
AI-driven product innovation: from Recommender Systems to COVID-19AI-driven product innovation: from Recommender Systems to COVID-19
AI-driven product innovation: from Recommender Systems to COVID-19
Xavier Amatriain
 
Item Based Collaborative Filtering Recommendation Algorithms
Item Based Collaborative Filtering Recommendation AlgorithmsItem Based Collaborative Filtering Recommendation Algorithms
Item Based Collaborative Filtering Recommendation Algorithmsnextlib
 
Recommender Systems: Advances in Collaborative Filtering
Recommender Systems: Advances in Collaborative FilteringRecommender Systems: Advances in Collaborative Filtering
Recommender Systems: Advances in Collaborative Filtering
Changsung Moon
 
Recommender Systems from A to Z – Model Training
Recommender Systems from A to Z – Model TrainingRecommender Systems from A to Z – Model Training
Recommender Systems from A to Z – Model Training
Crossing Minds
 

What's hot (20)

Boston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender SystemsBoston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender Systems
 
Recommender system introduction
Recommender system   introductionRecommender system   introduction
Recommender system introduction
 
Recommender system
Recommender systemRecommender system
Recommender system
 
An introduction to Recommender Systems
An introduction to Recommender SystemsAn introduction to Recommender Systems
An introduction to Recommender Systems
 
Tutorial on Advances in Bias-aware Recommendation on the Web @ WSDM 2021
Tutorial on Advances in Bias-aware Recommendation on the Web @ WSDM 2021Tutorial on Advances in Bias-aware Recommendation on the Web @ WSDM 2021
Tutorial on Advances in Bias-aware Recommendation on the Web @ WSDM 2021
 
Recommendation system
Recommendation systemRecommendation system
Recommendation system
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
 
Overview of recommender system
Overview of recommender systemOverview of recommender system
Overview of recommender system
 
Recommender system algorithm and architecture
Recommender system algorithm and architectureRecommender system algorithm and architecture
Recommender system algorithm and architecture
 
Recommendation Systems: Applying Amazon's Collaborative Filtering Methods to ...
Recommendation Systems: Applying Amazon's Collaborative Filtering Methods to ...Recommendation Systems: Applying Amazon's Collaborative Filtering Methods to ...
Recommendation Systems: Applying Amazon's Collaborative Filtering Methods to ...
 
Introduction to Recommendation Systems
Introduction to Recommendation SystemsIntroduction to Recommendation Systems
Introduction to Recommendation Systems
 
Collaborative Filtering using KNN
Collaborative Filtering using KNNCollaborative Filtering using KNN
Collaborative Filtering using KNN
 
[Final]collaborative filtering and recommender systems
[Final]collaborative filtering and recommender systems[Final]collaborative filtering and recommender systems
[Final]collaborative filtering and recommender systems
 
Anatomy of an eCommerce Search Engine by Mayur Datar
Anatomy of an eCommerce Search Engine by Mayur DatarAnatomy of an eCommerce Search Engine by Mayur Datar
Anatomy of an eCommerce Search Engine by Mayur Datar
 
Recommendation System
Recommendation SystemRecommendation System
Recommendation System
 
Collaborative Recommender System for Music using PyTorch
Collaborative Recommender System for Music using PyTorchCollaborative Recommender System for Music using PyTorch
Collaborative Recommender System for Music using PyTorch
 
AI-driven product innovation: from Recommender Systems to COVID-19
AI-driven product innovation: from Recommender Systems to COVID-19AI-driven product innovation: from Recommender Systems to COVID-19
AI-driven product innovation: from Recommender Systems to COVID-19
 
Item Based Collaborative Filtering Recommendation Algorithms
Item Based Collaborative Filtering Recommendation AlgorithmsItem Based Collaborative Filtering Recommendation Algorithms
Item Based Collaborative Filtering Recommendation Algorithms
 
Recommender Systems: Advances in Collaborative Filtering
Recommender Systems: Advances in Collaborative FilteringRecommender Systems: Advances in Collaborative Filtering
Recommender Systems: Advances in Collaborative Filtering
 
Recommender Systems from A to Z – Model Training
Recommender Systems from A to Z – Model TrainingRecommender Systems from A to Z – Model Training
Recommender Systems from A to Z – Model Training
 

Similar to Fair Recommender Systems

Dasts16 a koene_un_bias
Dasts16 a koene_un_biasDasts16 a koene_un_bias
Dasts16 a koene_un_bias
Ansgar Koene
 
A Comprehensive Review of Relevant Techniques used in Course Recommendation S...
A Comprehensive Review of Relevant Techniques used in Course Recommendation S...A Comprehensive Review of Relevant Techniques used in Course Recommendation S...
A Comprehensive Review of Relevant Techniques used in Course Recommendation S...
IRJET Journal
 
A Survey of Recommender System Techniques and the E-commerce Domain.pptx
A Survey of Recommender System Techniques and the E-commerce Domain.pptxA Survey of Recommender System Techniques and the E-commerce Domain.pptx
A Survey of Recommender System Techniques and the E-commerce Domain.pptx
mansivekaria09
 
IRJET- Survey Paper on Recommendation Systems
IRJET- Survey Paper on Recommendation SystemsIRJET- Survey Paper on Recommendation Systems
IRJET- Survey Paper on Recommendation Systems
IRJET Journal
 
A.hybrid.recommendation.approach.for.a.tourism.system
A.hybrid.recommendation.approach.for.a.tourism.systemA.hybrid.recommendation.approach.for.a.tourism.system
A.hybrid.recommendation.approach.for.a.tourism.system
benny ribeiro
 
Machine learning based recommender system for e-commerce
Machine learning based recommender system for e-commerceMachine learning based recommender system for e-commerce
Machine learning based recommender system for e-commerce
IAESIJAI
 
Introduction to Recommendation Systems
Introduction to Recommendation SystemsIntroduction to Recommendation Systems
Introduction to Recommendation Systems
Zia Babar
 
Human Agency on Algorithmic Systems
Human Agency on Algorithmic SystemsHuman Agency on Algorithmic Systems
Human Agency on Algorithmic Systems
Ansgar Koene
 
iCTRE: The Informal community Transformer into Recommendation Engine
iCTRE: The Informal community Transformer into Recommendation EngineiCTRE: The Informal community Transformer into Recommendation Engine
iCTRE: The Informal community Transformer into Recommendation Engine
IRJET Journal
 
243
243243
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
vivatechijri
 
IRJET- Survey on Facial Feature Detection Methods in Offline Recommendation b...
IRJET- Survey on Facial Feature Detection Methods in Offline Recommendation b...IRJET- Survey on Facial Feature Detection Methods in Offline Recommendation b...
IRJET- Survey on Facial Feature Detection Methods in Offline Recommendation b...
IRJET Journal
 
IRJET-Smart Tourism Recommender System
IRJET-Smart Tourism Recommender SystemIRJET-Smart Tourism Recommender System
IRJET-Smart Tourism Recommender System
IRJET Journal
 
Recommendation System Using Social Networking
Recommendation System Using Social Networking Recommendation System Using Social Networking
Recommendation System Using Social Networking
ijcseit
 
A Novel Jewellery Recommendation System using Machine Learning and Natural La...
A Novel Jewellery Recommendation System using Machine Learning and Natural La...A Novel Jewellery Recommendation System using Machine Learning and Natural La...
A Novel Jewellery Recommendation System using Machine Learning and Natural La...
IRJET Journal
 
An Efficient Content, Collaborative – Based and Hybrid Approach for Movie Rec...
An Efficient Content, Collaborative – Based and Hybrid Approach for Movie Rec...An Efficient Content, Collaborative – Based and Hybrid Approach for Movie Rec...
An Efficient Content, Collaborative – Based and Hybrid Approach for Movie Rec...
ijtsrd
 
MOVIE RECOMMENDATION SYSTEM
MOVIE RECOMMENDATION SYSTEMMOVIE RECOMMENDATION SYSTEM
MOVIE RECOMMENDATION SYSTEM
IRJET Journal
 
IRJET- Hybrid Book Recommendation System
IRJET- Hybrid Book Recommendation SystemIRJET- Hybrid Book Recommendation System
IRJET- Hybrid Book Recommendation System
IRJET Journal
 
“Responsible AI: Tools and Frameworks for Developing AI Solutions,” a Present...
“Responsible AI: Tools and Frameworks for Developing AI Solutions,” a Present...“Responsible AI: Tools and Frameworks for Developing AI Solutions,” a Present...
“Responsible AI: Tools and Frameworks for Developing AI Solutions,” a Present...
Edge AI and Vision Alliance
 
IRJET- Review on Different Recommendation Techniques for GRS in Online Social...
IRJET- Review on Different Recommendation Techniques for GRS in Online Social...IRJET- Review on Different Recommendation Techniques for GRS in Online Social...
IRJET- Review on Different Recommendation Techniques for GRS in Online Social...
IRJET Journal
 

Similar to Fair Recommender Systems (20)

Dasts16 a koene_un_bias
Dasts16 a koene_un_biasDasts16 a koene_un_bias
Dasts16 a koene_un_bias
 
A Comprehensive Review of Relevant Techniques used in Course Recommendation S...
A Comprehensive Review of Relevant Techniques used in Course Recommendation S...A Comprehensive Review of Relevant Techniques used in Course Recommendation S...
A Comprehensive Review of Relevant Techniques used in Course Recommendation S...
 
A Survey of Recommender System Techniques and the E-commerce Domain.pptx
A Survey of Recommender System Techniques and the E-commerce Domain.pptxA Survey of Recommender System Techniques and the E-commerce Domain.pptx
A Survey of Recommender System Techniques and the E-commerce Domain.pptx
 
IRJET- Survey Paper on Recommendation Systems
IRJET- Survey Paper on Recommendation SystemsIRJET- Survey Paper on Recommendation Systems
IRJET- Survey Paper on Recommendation Systems
 
A.hybrid.recommendation.approach.for.a.tourism.system
A.hybrid.recommendation.approach.for.a.tourism.systemA.hybrid.recommendation.approach.for.a.tourism.system
A.hybrid.recommendation.approach.for.a.tourism.system
 
Machine learning based recommender system for e-commerce
Machine learning based recommender system for e-commerceMachine learning based recommender system for e-commerce
Machine learning based recommender system for e-commerce
 
Introduction to Recommendation Systems
Introduction to Recommendation SystemsIntroduction to Recommendation Systems
Introduction to Recommendation Systems
 
Human Agency on Algorithmic Systems
Human Agency on Algorithmic SystemsHuman Agency on Algorithmic Systems
Human Agency on Algorithmic Systems
 
iCTRE: The Informal community Transformer into Recommendation Engine
iCTRE: The Informal community Transformer into Recommendation EngineiCTRE: The Informal community Transformer into Recommendation Engine
iCTRE: The Informal community Transformer into Recommendation Engine
 
243
243243
243
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
IRJET- Survey on Facial Feature Detection Methods in Offline Recommendation b...
IRJET- Survey on Facial Feature Detection Methods in Offline Recommendation b...IRJET- Survey on Facial Feature Detection Methods in Offline Recommendation b...
IRJET- Survey on Facial Feature Detection Methods in Offline Recommendation b...
 
IRJET-Smart Tourism Recommender System
IRJET-Smart Tourism Recommender SystemIRJET-Smart Tourism Recommender System
IRJET-Smart Tourism Recommender System
 
Recommendation System Using Social Networking
Recommendation System Using Social Networking Recommendation System Using Social Networking
Recommendation System Using Social Networking
 
A Novel Jewellery Recommendation System using Machine Learning and Natural La...
A Novel Jewellery Recommendation System using Machine Learning and Natural La...A Novel Jewellery Recommendation System using Machine Learning and Natural La...
A Novel Jewellery Recommendation System using Machine Learning and Natural La...
 
An Efficient Content, Collaborative – Based and Hybrid Approach for Movie Rec...
An Efficient Content, Collaborative – Based and Hybrid Approach for Movie Rec...An Efficient Content, Collaborative – Based and Hybrid Approach for Movie Rec...
An Efficient Content, Collaborative – Based and Hybrid Approach for Movie Rec...
 
MOVIE RECOMMENDATION SYSTEM
MOVIE RECOMMENDATION SYSTEMMOVIE RECOMMENDATION SYSTEM
MOVIE RECOMMENDATION SYSTEM
 
IRJET- Hybrid Book Recommendation System
IRJET- Hybrid Book Recommendation SystemIRJET- Hybrid Book Recommendation System
IRJET- Hybrid Book Recommendation System
 
“Responsible AI: Tools and Frameworks for Developing AI Solutions,” a Present...
“Responsible AI: Tools and Frameworks for Developing AI Solutions,” a Present...“Responsible AI: Tools and Frameworks for Developing AI Solutions,” a Present...
“Responsible AI: Tools and Frameworks for Developing AI Solutions,” a Present...
 
IRJET- Review on Different Recommendation Techniques for GRS in Online Social...
IRJET- Review on Different Recommendation Techniques for GRS in Online Social...IRJET- Review on Different Recommendation Techniques for GRS in Online Social...
IRJET- Review on Different Recommendation Techniques for GRS in Online Social...
 

Recently uploaded

Jpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization SampleJpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization Sample
James Polillo
 
standardisation of garbhpala offhgfffghh
standardisation of garbhpala offhgfffghhstandardisation of garbhpala offhgfffghh
standardisation of garbhpala offhgfffghh
ArpitMalhotra16
 
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Subhajit Sahu
 
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
correoyaya
 
Predicting Product Ad Campaign Performance: A Data Analysis Project Presentation
Predicting Product Ad Campaign Performance: A Data Analysis Project PresentationPredicting Product Ad Campaign Performance: A Data Analysis Project Presentation
Predicting Product Ad Campaign Performance: A Data Analysis Project Presentation
Boston Institute of Analytics
 
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdfCriminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP
 
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
ewymefz
 
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
Ch03-Managing the Object-Oriented Information Systems Project a.pdfCh03-Managing the Object-Oriented Information Systems Project a.pdf
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
haila53
 
Tabula.io Cheatsheet: automate your data workflows
Tabula.io Cheatsheet: automate your data workflowsTabula.io Cheatsheet: automate your data workflows
Tabula.io Cheatsheet: automate your data workflows
alex933524
 
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
vcaxypu
 
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
nscud
 
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
John Andrews
 
Malana- Gimlet Market Analysis (Portfolio 2)
Malana- Gimlet Market Analysis (Portfolio 2)Malana- Gimlet Market Analysis (Portfolio 2)
Malana- Gimlet Market Analysis (Portfolio 2)
TravisMalana
 
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
axoqas
 
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdfCriminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP
 
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...
Subhajit Sahu
 
SOCRadar Germany 2024 Threat Landscape Report
SOCRadar Germany 2024 Threat Landscape ReportSOCRadar Germany 2024 Threat Landscape Report
SOCRadar Germany 2024 Threat Landscape Report
SOCRadar
 
一比一原版(TWU毕业证)西三一大学毕业证成绩单
一比一原版(TWU毕业证)西三一大学毕业证成绩单一比一原版(TWU毕业证)西三一大学毕业证成绩单
一比一原版(TWU毕业证)西三一大学毕业证成绩单
ocavb
 
Investigate & Recover / StarCompliance.io / Crypto_Crimes
Investigate & Recover / StarCompliance.io / Crypto_CrimesInvestigate & Recover / StarCompliance.io / Crypto_Crimes
Investigate & Recover / StarCompliance.io / Crypto_Crimes
StarCompliance.io
 
一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单
enxupq
 

Recently uploaded (20)

Jpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization SampleJpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization Sample
 
standardisation of garbhpala offhgfffghh
standardisation of garbhpala offhgfffghhstandardisation of garbhpala offhgfffghh
standardisation of garbhpala offhgfffghh
 
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
 
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
 
Predicting Product Ad Campaign Performance: A Data Analysis Project Presentation
Predicting Product Ad Campaign Performance: A Data Analysis Project PresentationPredicting Product Ad Campaign Performance: A Data Analysis Project Presentation
Predicting Product Ad Campaign Performance: A Data Analysis Project Presentation
 
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdfCriminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdf
 
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
 
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
Ch03-Managing the Object-Oriented Information Systems Project a.pdfCh03-Managing the Object-Oriented Information Systems Project a.pdf
Ch03-Managing the Object-Oriented Information Systems Project a.pdf
 
Tabula.io Cheatsheet: automate your data workflows
Tabula.io Cheatsheet: automate your data workflowsTabula.io Cheatsheet: automate your data workflows
Tabula.io Cheatsheet: automate your data workflows
 
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
 
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
 
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
 
Malana- Gimlet Market Analysis (Portfolio 2)
Malana- Gimlet Market Analysis (Portfolio 2)Malana- Gimlet Market Analysis (Portfolio 2)
Malana- Gimlet Market Analysis (Portfolio 2)
 
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
 
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdfCriminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdf
 
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...
 
SOCRadar Germany 2024 Threat Landscape Report
SOCRadar Germany 2024 Threat Landscape ReportSOCRadar Germany 2024 Threat Landscape Report
SOCRadar Germany 2024 Threat Landscape Report
 
一比一原版(TWU毕业证)西三一大学毕业证成绩单
一比一原版(TWU毕业证)西三一大学毕业证成绩单一比一原版(TWU毕业证)西三一大学毕业证成绩单
一比一原版(TWU毕业证)西三一大学毕业证成绩单
 
Investigate & Recover / StarCompliance.io / Crypto_Crimes
Investigate & Recover / StarCompliance.io / Crypto_CrimesInvestigate & Recover / StarCompliance.io / Crypto_Crimes
Investigate & Recover / StarCompliance.io / Crypto_Crimes
 
一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单
 

Fair Recommender Systems

  • 1. Fair Recommendation Systems Sharmistha Chatterjee Senior Manager Data Sciences at Publicis Sapient Author | Speaker | GDE for ML @sharmichat82 http://techairesearch.com/ Women in Data Science - Mysuru
  • 2. Key Takeaways Fairness in Machine Learning Need for Fairness in ML based Recommendation Systems Fairness-aware Machine Learning Best Practices Multi-sided Fairness , Platforms and Metrics
  • 3. Why Fairness in Recommendations Fair Housing Act LGBT Fairness Act Disability Status Disparate Impact Source - https://www.nytimes.com/2019/05/07/opinion/google-sundar-pichai-privacy.html Disability Laws and Policies
  • 4. Ethical Artificial Intelligence – Fairness Origin Professor Klaus Schwab - Executive Chairman of WORLD ECONOMIC FORUM “We must address, individually and collectively, moral and ethical issues raised by cutting- edge research in artificial intelligence and biotechnology, which will enable significant life extension, designer babies, and memory extraction.” —Klaus Schwab Source - https://www.statista.com/chart/18805/highest-penalties-in-privacy-enforcement-actions-worldwide/
  • 5. Foundation of Algorithmic Justice League Joy Buolamwini - computer scientist and digital activist based at the MIT Media Lab Whether AI will help us reach our aspirations or reinforce the unjust inequalities is ultimately up to us. If we fail to make ethical and inclusive artificial intelligence, we risk losing gains made in civil rights and gender equity under the guise of machine neutrality Source - https://www.statista.com/chart/18805/highest-penalties-in-privacy-enforcement-actions-worldwide/
  • 6. AI Regulation Sundar Pichai The head of Google and parent company Alphabet has called for artificial intelligence (AI) to be regulated ● Fair Marketplace ● Legal obligation ● Social Responsibility ● Business Requirement /Model Source - https://builtin.com/artificial-intelligence/ai-laws-regulations
  • 7. Unfair Recommendation Systems from Biases Source - Simple Demographics Often Identify People Uniquely Differences in demographics or other user characteristics. Differences across online and offline communities, platforms and contexts Lexical, syntactic, semantic, and structural differences in the contents Connections, interactions, or activities obtained from networks and their attributes Society norms, prejudices, economical status Seasonal, weekly or observed at a certain time Population Behavioral Content Production Linking Social/ Economic Temporal Types of Biases
  • 8. Recommendation Systems Differences in demographics or other user characteristics. Differences across online and offline communities, platforms and contexts Lexical, syntactic, semantic, and structural differences in the contents Connections, interactions, or activities obtained from networks and their attributes Society norms, prejudices, economical status Seasonal, weekly or observed at a certain time Impact of Bias ● Biased customer reviews ● Disparate impact on minority drivers ● Unjust outcomes with low wages
  • 9. Recommend Fairly to All Groups of Users Fairness Escalate Source -https://course.ece.cmu.edu/~ece734/lectures/lecture-2018-10-08-deanonymization.pdf Product Goals Stakeholder Identification Analyze Mitigate TransparencyMonitor Performance Best Practices for Removing Bias
  • 10. Where and How Legal Fairness Experts User Researchers Google’s Responsible AI - Expert Opinion
  • 11. Where and How Social Science Backgrounds Gender and sexual orientation Diverse Identities - Race, Nationality, Religion Google’s Responsible AI - Diversity and Inclusion
  • 12. Fairness for Individual and Groups Pre-processing , Learning Function , Post Processing Source - https://arxiv.org/pdf/1906.08732.pdf FAIRNESS COMMON TERMS Source : https://arxiv.org/pdf/1906.08732.pdf Envy-freeness requires that every user should prefer their own allocation to that of everyone else; it ignores users’ qualifications and considers preferences Individual or metric fairness ignores preferences and requires that similar users should be treated similarly. Multiple task fairness, requires that individual fairness is satisfied separately and simultaneously for all categories Inter-category envy- freeness, which allows users to specify a set of categories that they “care” about, and guarantees that they receive at least that they care about as any other individual
  • 13. Fairness Constraints with 2 latent variables: ProtectedItemRating(i) ⇒ UnProtectedItemRating(i) UnProtectedItemRating(i) ⇒ ProtectedItemRating(i) Protected(u) ∧ RATING(u, i) ∧ ItemGroup(i, g) ⇒ ProtectedItemGroupRating(g) ¬ Protected(u) ∧ Rating(u, i) ∧ ItemGroup(i, g) ⇒ UnProtectedItemGroupRating(g) Recommendation Systems MetricsSL No 1. 2. 3. 4. Statistical independence between recommendation results and the sensitive attribute Value unfairness (estimation error across user groups) Absolute unfairness Underestimation unfairness 5. Overestimation unfairness Fairness Metrics between advantaged and disadvantaged groups 6. Non-parity Fairness Source -https://arxiv.org/pdf/1809.09030.pdf 𝑝𝑖- d-dimensional vector representing the ith user, 𝑞 𝑗 − d−dimensional vector representing the jth item, 𝑢𝑖 𝑎𝑛𝑑 𝑣𝑗. user and item respectively X – observed ratings
  • 14. Recommendation Diversity and User Fairness Ranking algorithms Top-l CF recommendations Sample K items uniformly – K nearest neighbors Greedy algo – New recommendations above threshold ● Individual diversity - Diverse recommendations to the users. ● Aggregate diversity - Improve item diversity by recommending them at least once across all users. ● Limitations – No fairness like differential treatment of two users or two items. ● Impact – Disparity among users increases with Aggregate Diversity CF Output Source - https://www.researchgate.net/publication/324640535_User_Fairness_in_Recommender_Systems
  • 15. Different Types of multi-sided Fairness ● Multi-stakeholder Recommender Systems with multiple goals (e.g. LinkedIn, Etsy) ● Reciprocal Recommendations –Bilateral and acceptable to both parties. (e.g. job, mentor, business partner) ○ Peer-to-peer recommendation –Sharing economy, online advertising and scientific collaboration ● Objective – Maximize system utility Multi-sided Fairness in Recommendation Systems Source - http://proceedings.mlr.press/v81/burke18a/burke18a.pdf
  • 16. Regularization based Sparse Linear Method (SLIM) with Neighborhood balance ● Provider Fairness – Market Diversity , avoid monopoly by recommending minority owned businesses. ● Consumer Fairness – Personalization, Disparate impact of recommendation on protected classes Multi-sided Fairness – User Based Neighborhoods Source -http://proceedings.mlr.press/v81/burke18a/burke18a.pdf Unbalanced Neighbors Balanced Neighbors FairnessFairness ● Regression coefficient <user, item> pair ● Minimize regularized loss function
  • 17. Multi-sided Fairness - Item Based Neighborhoods Source -https://medium.com/@cfpinela/recommender-systems-user-based-and-item-based-collaborative-filtering- 5d5f375a127f ● Use case - Exposure to loans from different geographic regions ● Items in protected group are in neighborhoods that have balanced membership of items from the unprotected group● Balance between protected and non- protected neighbors for each user. Balanced Neighborhood SLIM – Balance Personalization with Fairness
  • 18. Two-sided Fairness Two-sided Platforms ● Fair recommendation -> Fair Allocation ○ Maximin Share (MMS) of exposure for Producers and Envy-Free up to One Good (EF1) fairness for every customer ● Cardinality constrained fair allocation ○ All items are grouped into disjoint categories and no agent receives more than a pre-specified number of items from same category ○ Exactly k items are allocated to each customer Datasets – GL-CUSTOM, (Relevance Scoring) , GL-FACT and LAST.FM DATASET –(Latent Factorization) Source - https://people.mpi-sws.org/~achakrab/papers/patro_FairRec_WWW20.pdf
  • 19. Source - https://people.mpi-sws.org/~achakrab/papers/patro_FairRec_WWW20.pdf FAIR RECOMMENDATION SYSTEMSAdvantages and Metrics MetricsSL No 1. 2. 3. 4. Fraction of Satisfied Producers Inequality of Product Exposures Exposure Loss on Producers Mean-Average Envy 5. Loss and Disparity in Customer Utilities ● Advantages ○ Economic ○ Social ○ Judicial and ○ Long-term sustainability K items
  • 20. Evaluate Fair Recommender Systems - Pairwise Comparisons Original vs Pairwise Regularization ● Pairwise regularization to optimize for inter-group pairwise fairness. ● Corresponds to ranking performance ● Pairwise Fairness - Likelihood of a clicked item being ranked above another relevant unclicked item is the same across both groups (same/opposite), conditioned on the items have safe level of engagement. ● Real world scalable product grade systems Source - http://alexbeutel.com/papers/kdd2019_pairwise_fairness.pdf Intra Group and Inter Group Pairwise Fairness
  • 21. Compositional Fairness Source - https://www.researchgate.net/publication/335755357_RecSim_A_Configurable_Simulation_Platform_for_Recommender_ Systems Each user should be envy-free with respect to categories (inter-category) Combination of inter-category envy-freeness and total-variation fairness into hybrid fairness Individual fairness within each category End-to-end Fair Systems with High Platform Utility
  • 22. Conclusion ● Handle sparse data ● Bias from Advertisers ● Formulate right personalization and key outcome ● Techniques to mitigate harms and mis- behaviors ● Computational enhancement for Multi- stakeholder fairness optimization ● Network structures that define relationships between providers and users ● Sequential notions of fairness into recommender systems with additional time-dependent constraints ● Equal treatment vs equal outcome Source: https://www.researchgate.net/publication/314971193_Optimal_Performance_vs_Fairness_Tradeoff_for_Resource_Allocation_in_Wireless_Systems
  • 24. References ● https://www.ibm.com/blogs/research/2018/09/ai-fairness-360/ ● https://arxiv.org/pdf/1810.01943.pdf ● https://pair.withgoogle.com/explorables/measuring-fairness/ ● https://www.research.ibm.com/artificial-intelligence/trusted-ai/ ● https://pair.withgoogle.com/explorables/measuring-fairness/ ● https://www.researchgate.net/publication/314971193_Optimal_Performance_vs_Fairness_Tradeoff_for_Resource_Alloca tion_in_Wireless_Systems ● https://people.mpi-sws.org/~achakrab/papers/patro_FairRec_WWW20.pdf ● http://alexbeutel.com/papers/kdd2019_pairwise_fairness.pdf ● http://proceedings.mlr.press/v81/burke18a/burke18a.pdf ● https://course.ece.cmu.edu/~ece734/lectures/lecture-2018-10-08-deanonymization.pdf ● https://www.nytimes.com/2019/05/07/opinion/google-sundar-pichai-privacy.html ● https://arxiv.org/pdf/1906.08732.pdf ● Algorithmic Fairness - https://arxiv.org/pdf/2001.09784.pdf ● Fairness Definitions Explained- https://fairware.cs.umass.edu/papers/Verma.pdf ● https://www.researchgate.net/publication/324640535_User_Fairness_in_Recommender_Systems ● https://github.com/gourabkumarpatro/FairRec_www_2020