L 9 Winter annuals.pptx
Your SlideShare is downloading.
×
![11
WIS
Web
Information
Systems
References
[1] R. Epstein and R. E. Robertson. The search engine manipulation effect
(SEME)...](https://image.slidesharecdn.com/dirrankingfairness-201211135217/85/assessing-viewpoint-diversity-in-search-results-using-ranking-fairness-metrics-11-320.jpg?cb=1607694773)
Check these out next
Recommended
More Related Content
Recently uploaded (20)
Featured (20)
Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
- 1. 1 WIS Web Information Systems Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics Tim Draws1, Nava Tintarev1, Ujwal Gadiraju1, Alessandro Bozzon1, and Benjamin Timmermans2 1TU Delft, The Netherlands 2IBM, The Netherlands t.a.draws@tudelft.nl https://timdraws.net
- 2. 2 WIS Web Information Systems Biases in web search “Search Engine Manipulation Effect”1,2 Yes! Yes! Yes! Yes! Yes! No! No! How can we measure viewpoint diversity in search results?
- 3. 3 WIS Web Information Systems Ranking fairness metrics Rank Candidate gender 1 m 2 f 3 m 4 m 5 m … …
- 4. 4 WIS Web Information Systems Our paper RQ: Can ranking fairness metrics be used to assess viewpoint diversity in search results? What we did: • Defined two notions of viewpoint diversity • Conducted two simulation studies to 1. evaluate existing metrics 2. evaluate novel metric that we propose
- 5. 5 WIS Web Information Systems Representing viewpoints Should we all be vegan? Extremely opposing Opposing Somewhat opposing Neutral Somewhat supporting Supporting Extremely supporting -3 -2 -1 0 +1 +2 +3
- 6. 6 WIS Web Information Systems Representing viewpoints Should we all be vegan? Strongly opposing Opposing Somewhat opposing Neutral Somewhat supporting Supporting Strongly supporting protected non-protected Binomial viewpoint fairness -3 -2 -1 0 +1 +2 +3
- 7. 7 WIS Web Information Systems Representing viewpoints Should we all be vegan? Strongly opposing Opposing Somewhat opposing Neutral Somewhat supporting Supporting Strongly supporting protected Multinomial viewpoint fairness -3 -2 -1 0 +1 +2 +3
- 8. 8 WIS Web Information Systems Simulation studies • Three synthetic data sets S1, S2, S3 • Per set created rankings to simulate different levels of viewpoint diversity (ranking bias) • Computed metrics on each simulated ranking
- 9. 9 WIS Web Information Systems Results Considerations: – What is the underlying aim? – How balanced is the data overall? – How strong is the ranking bias? – What is the direction of ranking bias?0.0 0.2 0.4 0.6 0.8 1.0 −1.0 −0.8 −0.6 −0.4 −0.2 0.0 0.2 0.4 0.6 0.8 1.0 Ranking bias MeannDDvalue
- 10. 10 WIS Web Information Systems Take home • Ranking fairness metrics can be used for assessing viewpoint diversity in search results – (when interpreted correctly) • Future work – Appropriate viewpoint labels? – Appropriate level of viewpoint diversity? – Assess viewpoint diversity in real search results – Align different metric and behavioral outcomes t.a.draws@tudelft.nl https://timdraws.net
- 11. 11 WIS Web Information Systems References [1] R. Epstein and R. E. Robertson. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences of the United States of America, 112(33):E4512–E4521, 2015. [2] F. A. Pogacar, A. Ghenai, M. D. Smucker, and C. L. Clarke. The positive and negative influence of search results on people’s decisions about the efficacy of medical treatments. ICTIR 2017 - Proceedings of the 2017 ACM SIGIR International Conference on the Theory of Information Retrieval, pages 209– 216, 2017.
