SlideShare a Scribd company logo

This Is Not What We Ordered: Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics

T
TimDraws

Paper presentation at SIGIR 2021

1 of 13
Download to read offline
1
WIS
Web
Information
Systems
This Is Not What We Ordered
Exploring Why Biased Search Result Rankings
Affect User Attitudes on Debated Topics
Tim Draws1, Nava Tintarev2, Ujwal Gadiraju1,
Alessandro Bozzon1, Benjamin Timmermans3
t.a.draws@tudelft.nl
https://timdraws.net
1Delft University of Technology
2Maastricht University
3IBM Research
2
WIS
Web
Information
Systems
Bias in Web Search
References: Allam, Schulz, and Nakamoto (2014); Baeza-Yates (2018); Epstein and Robertson (2015); Gao and Shah (2020); Pogacar et al. (2017); White (2013)
Yes!
Yes!
Yes!
Yes!
Yes!
No!
No!
Search Engine Manipulation Effect (SEME):
when search results favor a particular viewpoint,
users tend to adopt it (attitude change)
Strongly
opposing
Opposing Somewhat
opposing
Neutral Somewhat
supporting
Supporting Strongly
supporting
3
WIS
Web
Information
Systems
Our Study
RQ1. Ranking of viewpoint-balanced top 10 search results à attitude change?
RQ2. Individual user characteristics (AOT, user engagement) à attitude change?
RQ3. Interaction user factors & search result rankings à attitude change?
RQ4. Ranking of viewpoint-balanced top 10 search results à perceived bias?
Method: (pre-registered) online between-subjects user study
4
WIS
Web
Information
Systems
Method
Crowdsourced viewpoint annotations for search results on five different debated
topics; used those to assemble five viewpoint-balanced top 10 SERPs
1. Are social networking sites good for our society?
2. Should zoos exist?
3. Is cell phone radiation safe?
4. Should bottled water be banned?
5. Is obesity a disease?
à Exposed users to different rankings of these top 10 SERPs and measured
attitude change + user factors, perceived bias
5
WIS
Web
Information
Systems
Procedure
Attitude
Measurement
(7-point Likert scale)
Imagine the government is
seeking informed opinions
from the population related to
a number of debated topics…
6
WIS
Web
Information
Systems
Procedure
Attitude
Measurement
(7-point Likert scale)
Exploration
Topic
Condition
Bias
Direction
Assignment
Ad

Recommended

The Impact of Computing Systems | Causal inference in practice
The Impact of Computing Systems | Causal inference in practiceThe Impact of Computing Systems | Causal inference in practice
The Impact of Computing Systems | Causal inference in practiceAmit Sharma
 
Auditing search engines for differential satisfaction across demographics
Auditing search engines for differential satisfaction across demographicsAuditing search engines for differential satisfaction across demographics
Auditing search engines for differential satisfaction across demographicsAmit Sharma
 
Artificial Intelligence for Societal Impact
Artificial Intelligence for Societal ImpactArtificial Intelligence for Societal Impact
Artificial Intelligence for Societal ImpactAmit Sharma
 
Carma internet research module preparing for manuscript submission
Carma internet research module   preparing for manuscript submissionCarma internet research module   preparing for manuscript submission
Carma internet research module preparing for manuscript submissionSyracuse University
 
Social tags and controlled vocabularies as filters for searching and browsing...
Social tags and controlled vocabularies as filters for searching and browsing...Social tags and controlled vocabularies as filters for searching and browsing...
Social tags and controlled vocabularies as filters for searching and browsing...Michael Zarro, Ph.D.
 
e-Marketing: An Approach to Overcoming Non-Financial Barriers to EHR Adoption
e-Marketing: An Approach to Overcoming Non-Financial Barriers to EHR Adoptione-Marketing: An Approach to Overcoming Non-Financial Barriers to EHR Adoption
e-Marketing: An Approach to Overcoming Non-Financial Barriers to EHR Adoptionkwittman
 
Risk management and the release of microdata:
Risk management and the release of microdata:  Risk management and the release of microdata:
Risk management and the release of microdata: Sonia Whiteley
 
E Marketingposter
E MarketingposterE Marketingposter
E Marketingpostermboufaida
 

More Related Content

What's hot

Development of an Electronic Decision Support tool to optimize performance du...
Development of an Electronic Decision Support tool to optimize performance du...Development of an Electronic Decision Support tool to optimize performance du...
Development of an Electronic Decision Support tool to optimize performance du...INSPIRE_Network
 
AMIA 2013 Social Medica Panel
AMIA 2013 Social Medica PanelAMIA 2013 Social Medica Panel
AMIA 2013 Social Medica PanelGunther Eysenbach
 
How to get your ehealth / mhealth research published
How to get your ehealth / mhealth research publishedHow to get your ehealth / mhealth research published
How to get your ehealth / mhealth research publishedGunther Eysenbach
 
Ethics in Infodemiology and Public Health 2.0
Ethics in Infodemiology and Public Health 2.0Ethics in Infodemiology and Public Health 2.0
Ethics in Infodemiology and Public Health 2.0Gunther Eysenbach
 
Data analytics and the power of creating social impact
Data analytics and the power of creating social impactData analytics and the power of creating social impact
Data analytics and the power of creating social impactTA Telecom
 
Blockchain for a TBI Research Network - Manion
Blockchain for a TBI Research Network - ManionBlockchain for a TBI Research Network - Manion
Blockchain for a TBI Research Network - ManionSean Manion PhD
 
Using Data to Support the Most Vulnerable: An OVC Information Needs Framework
Using Data to Support the Most Vulnerable: An OVC Information Needs FrameworkUsing Data to Support the Most Vulnerable: An OVC Information Needs Framework
Using Data to Support the Most Vulnerable: An OVC Information Needs FrameworkMEASURE Evaluation
 
Theorizing Online News Consumption
Theorizing Online News ConsumptionTheorizing Online News Consumption
Theorizing Online News ConsumptionIris Chyi
 
Statistics in Journalism
Statistics in JournalismStatistics in Journalism
Statistics in JournalismRegina Nuzzo
 
Quality and safety of health information on the Internet: Who decides and ho...
Quality and safety of health information on the Internet: Who decides and ho...Quality and safety of health information on the Internet: Who decides and ho...
Quality and safety of health information on the Internet: Who decides and ho...Francisco J Grajales III
 

What's hot (10)

Development of an Electronic Decision Support tool to optimize performance du...
Development of an Electronic Decision Support tool to optimize performance du...Development of an Electronic Decision Support tool to optimize performance du...
Development of an Electronic Decision Support tool to optimize performance du...
 
AMIA 2013 Social Medica Panel
AMIA 2013 Social Medica PanelAMIA 2013 Social Medica Panel
AMIA 2013 Social Medica Panel
 
How to get your ehealth / mhealth research published
How to get your ehealth / mhealth research publishedHow to get your ehealth / mhealth research published
How to get your ehealth / mhealth research published
 
Ethics in Infodemiology and Public Health 2.0
Ethics in Infodemiology and Public Health 2.0Ethics in Infodemiology and Public Health 2.0
Ethics in Infodemiology and Public Health 2.0
 
Data analytics and the power of creating social impact
Data analytics and the power of creating social impactData analytics and the power of creating social impact
Data analytics and the power of creating social impact
 
Blockchain for a TBI Research Network - Manion
Blockchain for a TBI Research Network - ManionBlockchain for a TBI Research Network - Manion
Blockchain for a TBI Research Network - Manion
 
Using Data to Support the Most Vulnerable: An OVC Information Needs Framework
Using Data to Support the Most Vulnerable: An OVC Information Needs FrameworkUsing Data to Support the Most Vulnerable: An OVC Information Needs Framework
Using Data to Support the Most Vulnerable: An OVC Information Needs Framework
 
Theorizing Online News Consumption
Theorizing Online News ConsumptionTheorizing Online News Consumption
Theorizing Online News Consumption
 
Statistics in Journalism
Statistics in JournalismStatistics in Journalism
Statistics in Journalism
 
Quality and safety of health information on the Internet: Who decides and ho...
Quality and safety of health information on the Internet: Who decides and ho...Quality and safety of health information on the Internet: Who decides and ho...
Quality and safety of health information on the Internet: Who decides and ho...
 

Similar to This Is Not What We Ordered: Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
Assessing Viewpoint Diversity in Search Results Using Ranking Fairness MetricsAssessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
Assessing Viewpoint Diversity in Search Results Using Ranking Fairness MetricsTimDraws
 
Measuring effectiveness of machine learning systems
Measuring effectiveness of machine learning systemsMeasuring effectiveness of machine learning systems
Measuring effectiveness of machine learning systemsAmit Sharma
 
群眾外包應用於藥物服用知識系統之研究
群眾外包應用於藥物服用知識系統之研究群眾外包應用於藥物服用知識系統之研究
群眾外包應用於藥物服用知識系統之研究彥成 林
 
AIRG Presentation
AIRG PresentationAIRG Presentation
AIRG Presentationnirvdrum
 
Attitudinal Impact of Online Reviews on Consumer Purchase Decisions
Attitudinal Impact of Online Reviews on Consumer Purchase DecisionsAttitudinal Impact of Online Reviews on Consumer Purchase Decisions
Attitudinal Impact of Online Reviews on Consumer Purchase DecisionsJIMS Rohini Sector 5
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentationnirvdrum
 
From data to comprehensive prevention for fraternities and sororities
From data to comprehensive prevention for fraternities and sororitiesFrom data to comprehensive prevention for fraternities and sororities
From data to comprehensive prevention for fraternities and sororitiesMichele Collu
 
M&E Systems for Evaluation: Where M meets E
M&E Systems for Evaluation: Where M meets EM&E Systems for Evaluation: Where M meets E
M&E Systems for Evaluation: Where M meets EMEASURE Evaluation
 
Healthcare & digital marketing - Today & Future
Healthcare &  digital marketing - Today & FutureHealthcare &  digital marketing - Today & Future
Healthcare & digital marketing - Today & FutureDeepali Thakur
 
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영Jin Young Kim
 
SERPs: From keyword to click. BrightonSEO (18th September 2015)
SERPs: From keyword to click. BrightonSEO (18th September 2015)SERPs: From keyword to click. BrightonSEO (18th September 2015)
SERPs: From keyword to click. BrightonSEO (18th September 2015)Similarweb
 
Research Using Behavioral Big Data: A Tour and Why Mechanical Engineers Shoul...
Research Using Behavioral Big Data: A Tour and Why Mechanical Engineers Shoul...Research Using Behavioral Big Data: A Tour and Why Mechanical Engineers Shoul...
Research Using Behavioral Big Data: A Tour and Why Mechanical Engineers Shoul...Galit Shmueli
 
Doctor onlineneeds
Doctor onlineneedsDoctor onlineneeds
Doctor onlineneedsAlex Sanchez
 
Doctors’ and patients’ support of automated decision-making in healthcare
Doctors’ and patients’ support of automated decision-making in healthcareDoctors’ and patients’ support of automated decision-making in healthcare
Doctors’ and patients’ support of automated decision-making in healthcareStanislav Ivanov
 
Determinants of Internet Media Abuse In Workplace.
Determinants of Internet Media Abuse In Workplace.Determinants of Internet Media Abuse In Workplace.
Determinants of Internet Media Abuse In Workplace.Fairul Hisyam Mat
 

Similar to This Is Not What We Ordered: Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics (20)

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
Assessing Viewpoint Diversity in Search Results Using Ranking Fairness MetricsAssessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
 
Measuring effectiveness of machine learning systems
Measuring effectiveness of machine learning systemsMeasuring effectiveness of machine learning systems
Measuring effectiveness of machine learning systems
 
群眾外包應用於藥物服用知識系統之研究
群眾外包應用於藥物服用知識系統之研究群眾外包應用於藥物服用知識系統之研究
群眾外包應用於藥物服用知識系統之研究
 
How online social ties and product-related risks influence purchase intention...
How online social ties and product-related risks influence purchase intention...How online social ties and product-related risks influence purchase intention...
How online social ties and product-related risks influence purchase intention...
 
AIRG Presentation
AIRG PresentationAIRG Presentation
AIRG Presentation
 
Attitudinal Impact of Online Reviews on Consumer Purchase Decisions
Attitudinal Impact of Online Reviews on Consumer Purchase DecisionsAttitudinal Impact of Online Reviews on Consumer Purchase Decisions
Attitudinal Impact of Online Reviews on Consumer Purchase Decisions
 
CKX: Social Impact Measurement Around the Globe
CKX: Social Impact Measurement Around the Globe CKX: Social Impact Measurement Around the Globe
CKX: Social Impact Measurement Around the Globe
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentation
 
Multimode Global Scale Usage
Multimode Global Scale UsageMultimode Global Scale Usage
Multimode Global Scale Usage
 
From data to comprehensive prevention for fraternities and sororities
From data to comprehensive prevention for fraternities and sororitiesFrom data to comprehensive prevention for fraternities and sororities
From data to comprehensive prevention for fraternities and sororities
 
M&E Systems for Evaluation: Where M meets E
M&E Systems for Evaluation: Where M meets EM&E Systems for Evaluation: Where M meets E
M&E Systems for Evaluation: Where M meets E
 
Healthcare & digital marketing - Today & Future
Healthcare &  digital marketing - Today & FutureHealthcare &  digital marketing - Today & Future
Healthcare & digital marketing - Today & Future
 
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
Fairness in Search & RecSys 네이버 검색 콜로키움 김진영
 
SERPs: From keyword to click. BrightonSEO (18th September 2015)
SERPs: From keyword to click. BrightonSEO (18th September 2015)SERPs: From keyword to click. BrightonSEO (18th September 2015)
SERPs: From keyword to click. BrightonSEO (18th September 2015)
 
Research Using Behavioral Big Data: A Tour and Why Mechanical Engineers Shoul...
Research Using Behavioral Big Data: A Tour and Why Mechanical Engineers Shoul...Research Using Behavioral Big Data: A Tour and Why Mechanical Engineers Shoul...
Research Using Behavioral Big Data: A Tour and Why Mechanical Engineers Shoul...
 
Recommender systems to help people move forward
Recommender systems to help people move forwardRecommender systems to help people move forward
Recommender systems to help people move forward
 
Doctor onlineneeds
Doctor onlineneedsDoctor onlineneeds
Doctor onlineneeds
 
Doctors’ and patients’ support of automated decision-making in healthcare
Doctors’ and patients’ support of automated decision-making in healthcareDoctors’ and patients’ support of automated decision-making in healthcare
Doctors’ and patients’ support of automated decision-making in healthcare
 
GoodIT2021.pptx
GoodIT2021.pptxGoodIT2021.pptx
GoodIT2021.pptx
 
Determinants of Internet Media Abuse In Workplace.
Determinants of Internet Media Abuse In Workplace.Determinants of Internet Media Abuse In Workplace.
Determinants of Internet Media Abuse In Workplace.
 

Recently uploaded

Model Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfModel Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfgalfinprihardiputra0
 
Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Damar Juniarto
 
Augmented and Mixed Reality Solutions for Aerospace & Defense
Augmented and Mixed Reality Solutions for Aerospace & DefenseAugmented and Mixed Reality Solutions for Aerospace & Defense
Augmented and Mixed Reality Solutions for Aerospace & Defensethirdeyegen65
 
Biometrics Technology Intresting PPT
Biometrics Technology Intresting PPTBiometrics Technology Intresting PPT
Biometrics Technology Intresting PPTPraveenKumarThota7
 
UGB INTERNETBANKING FACILITY LAUNCHED.pptx
UGB INTERNETBANKING FACILITY LAUNCHED.pptxUGB INTERNETBANKING FACILITY LAUNCHED.pptx
UGB INTERNETBANKING FACILITY LAUNCHED.pptxRitesh Sahu
 
AWS Overview of AWS Clarify, Feature Store, Hyper parameter Tuning
AWS Overview of AWS  Clarify, Feature Store, Hyper parameter TuningAWS Overview of AWS  Clarify, Feature Store, Hyper parameter Tuning
AWS Overview of AWS Clarify, Feature Store, Hyper parameter TuningVarun Garg
 
Augmented and Mixed Reality Solutions for Frontline Medical Professionals
Augmented and Mixed Reality Solutions for Frontline Medical ProfessionalsAugmented and Mixed Reality Solutions for Frontline Medical Professionals
Augmented and Mixed Reality Solutions for Frontline Medical Professionalsthirdeyegen65
 
Obstructive jaundice is a medical condition characterized by the yellowing of...
Obstructive jaundice is a medical condition characterized by the yellowing of...Obstructive jaundice is a medical condition characterized by the yellowing of...
Obstructive jaundice is a medical condition characterized by the yellowing of...ssuser7b7f4e
 
Red shadows ringing in Japan's Cyberspace
Red shadows ringing in Japan's CyberspaceRed shadows ringing in Japan's Cyberspace
Red shadows ringing in Japan's Cyberspacesttyk
 

Recently uploaded (9)

Model Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdfModel Jaringan network jaringan komputer.pdf
Model Jaringan network jaringan komputer.pdf
 
Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023Regulation is Coming - Trusted Media Summit 2023
Regulation is Coming - Trusted Media Summit 2023
 
Augmented and Mixed Reality Solutions for Aerospace & Defense
Augmented and Mixed Reality Solutions for Aerospace & DefenseAugmented and Mixed Reality Solutions for Aerospace & Defense
Augmented and Mixed Reality Solutions for Aerospace & Defense
 
Biometrics Technology Intresting PPT
Biometrics Technology Intresting PPTBiometrics Technology Intresting PPT
Biometrics Technology Intresting PPT
 
UGB INTERNETBANKING FACILITY LAUNCHED.pptx
UGB INTERNETBANKING FACILITY LAUNCHED.pptxUGB INTERNETBANKING FACILITY LAUNCHED.pptx
UGB INTERNETBANKING FACILITY LAUNCHED.pptx
 
AWS Overview of AWS Clarify, Feature Store, Hyper parameter Tuning
AWS Overview of AWS  Clarify, Feature Store, Hyper parameter TuningAWS Overview of AWS  Clarify, Feature Store, Hyper parameter Tuning
AWS Overview of AWS Clarify, Feature Store, Hyper parameter Tuning
 
Augmented and Mixed Reality Solutions for Frontline Medical Professionals
Augmented and Mixed Reality Solutions for Frontline Medical ProfessionalsAugmented and Mixed Reality Solutions for Frontline Medical Professionals
Augmented and Mixed Reality Solutions for Frontline Medical Professionals
 
Obstructive jaundice is a medical condition characterized by the yellowing of...
Obstructive jaundice is a medical condition characterized by the yellowing of...Obstructive jaundice is a medical condition characterized by the yellowing of...
Obstructive jaundice is a medical condition characterized by the yellowing of...
 
Red shadows ringing in Japan's Cyberspace
Red shadows ringing in Japan's CyberspaceRed shadows ringing in Japan's Cyberspace
Red shadows ringing in Japan's Cyberspace
 

This Is Not What We Ordered: Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics

  • 1. 1 WIS Web Information Systems This Is Not What We Ordered Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics Tim Draws1, Nava Tintarev2, Ujwal Gadiraju1, Alessandro Bozzon1, Benjamin Timmermans3 t.a.draws@tudelft.nl https://timdraws.net 1Delft University of Technology 2Maastricht University 3IBM Research
  • 2. 2 WIS Web Information Systems Bias in Web Search References: Allam, Schulz, and Nakamoto (2014); Baeza-Yates (2018); Epstein and Robertson (2015); Gao and Shah (2020); Pogacar et al. (2017); White (2013) Yes! Yes! Yes! Yes! Yes! No! No! Search Engine Manipulation Effect (SEME): when search results favor a particular viewpoint, users tend to adopt it (attitude change) Strongly opposing Opposing Somewhat opposing Neutral Somewhat supporting Supporting Strongly supporting
  • 3. 3 WIS Web Information Systems Our Study RQ1. Ranking of viewpoint-balanced top 10 search results à attitude change? RQ2. Individual user characteristics (AOT, user engagement) à attitude change? RQ3. Interaction user factors & search result rankings à attitude change? RQ4. Ranking of viewpoint-balanced top 10 search results à perceived bias? Method: (pre-registered) online between-subjects user study
  • 4. 4 WIS Web Information Systems Method Crowdsourced viewpoint annotations for search results on five different debated topics; used those to assemble five viewpoint-balanced top 10 SERPs 1. Are social networking sites good for our society? 2. Should zoos exist? 3. Is cell phone radiation safe? 4. Should bottled water be banned? 5. Is obesity a disease? à Exposed users to different rankings of these top 10 SERPs and measured attitude change + user factors, perceived bias
  • 5. 5 WIS Web Information Systems Procedure Attitude Measurement (7-point Likert scale) Imagine the government is seeking informed opinions from the population related to a number of debated topics…
  • 7. 7 WIS Web Information Systems Procedure Attitude Measurement (7-point Likert scale) Exploration Attitude Measurement Surveys + search behavior AOT User eng. Perceived Bias Topic Condition Bias Direction Random Assignment (7-point Likert scale)
  • 8. 8 WIS Web Information Systems Results Descriptives: 391 Participants (from Prolific); 70% expressed attitude change Hypothesis tests: • No evidence for a difference in attitude change across conditions (SEME) – Bayesian analysis: moderate evidence (BF = 8.56) in favor of null hypothesis (no difference) • No evidence for individual differences or interactions related to attitude change • No evidence for a difference in perceived bias across conditions – Bayesian analysis: strong evidence (BF = 31.23) in favor of null hypothesis (no difference)
  • 9. 9 WIS Web Information Systems Which Cognitive Biases Cause SEME? References: Azzopardi (2021) Yes! Yes! Yes! Yes! Yes! No! No! Exposure effects: 5/7 search results are supporting veganism Order effects: Higher-ranked search results are supporting veganism
  • 10. 10 WIS Web Information Systems Exploratory Analyses • Order effects – 77 (20% of) users clicked on exactly as many supporting as opposing results; still no evidence for a difference between conditions • Exposure effects – Positive relationship between proportion of supporting results clicked and attitude change (r = 0.34, p < 0.001)
  • 11. 11 WIS Web Information Systems Exploratory Analyses Difference in attitude change between different SERPs in extreme bias condition (t = 2.61, p = 0.01). 0.0 0.5 1.0 1 2 3 4 5 6 7 8 9 10 Ranks Proportion clicked Condition: Little Bias Moderate Bias Extreme Bias
  • 12. 12 WIS Web Information Systems Discussion & Conclusion • Expected to find SEME for viewpoint-balanced top 10 search results + individual differences, found no evidence for that • Exploratory analyses: exposure effects may explain SEME better than order effects • Implications for fair (re-)ranking metrics and algorithms as well as UI interventions – Potential goal: make users read more – Future research: confirm exposure effects t.a.draws@tudelft.nl https://timdraws.net Check our preregistration and supplementary material: https://osf.io/6tbvw/
  • 13. 13 WIS Web Information Systems References • Ahmed Allam, Peter Johannes Schulz, and Kent Nakamoto. 2014. The impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: Two experiments manipulating google output. Journal of Medical Internet Research 16, 4 (2014), e100. https://doi.org/10.2196/jmir.2642 • Leif Azzopardi. 2021. Cognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval (CHIIR ’21). Association for Computing Machinery, New York, NY, USA, 27–37. https://doi.org/10.1145/3406522.3446023 • Ricardo Baeza-Yates. 2018. Bias on the web. Commun. ACM 61, 6 (2018), 54–61. https://doi.org/10.1145/3209581 • Robert Epstein and Ronald E. Robertson. 2015. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences of the United States of America 112, 33 (2015), E4512– E4521. https://doi.org/10.1073/pnas.1419828112 • Ruoyuan Gao and Chirag Shah. 2020. Toward creating a fairer ranking in search engine results. Information Processing and Management 57, 1 (2020), 102138. https://doi.org/10.1016/j.ipm.2019.102138 • Frances A. Pogacar, Amira Ghenai, Mark D. Smucker, and Charles L.A. Clarke. 2017. The Positive and Negative Influence of Search Results on People’s De- cisions about the Efficacy of Medical Treatments. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR ’17). Association for Computing Machinery, New York, NY, USA, 209–216. https: //doi.org/10.1145/3121050.3121074 • Ryen W. White. 2013. Beliefs and biases in web search. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’13). Association for Computing Machinery, New York, NY, USA, 3–12. https://doi.org/10.1145/2484028.2484053