Searching for medical information is both a common and important activity since it influences decisions people make about their healthcare. Using search engine optimization (SEO), content producers seek to increase the visibility of their content. SEO is more likely to be practiced by commercially motivated content producers such as pharmaceutical companies than by non-commercial providers such as governmental bodies. In this study, we ask whether content quality correlates with the presence or absence of SEO measures on a web page. We conducted a user study in which N = 61 participants comprising laypeople as well as experts in health information assessment evaluated health-related web pages classified as either optimized or non-optimized. The subjects rated the expertise of non-optimized web pages as higher than the expertise of optimized pages, justifying their appraisal by the more competent and reputable appearance of non-optimized pages. In addition, comments about the website operators of the non-optimized pages were exclusively positive, while optimized pages tended to receive positive as well as negative assessments. We found no differences between the ratings of laypeople and experts. Since non-optimized, but high-quality content may be outranked by optimized content of lower quality, trusted sources should be prioritized in rankings. https://searchstudies.org/research/seo-effekt/

1. DOES SEARCH ENGINE OPTIMIZATION
COME ALONG WITH HIGH-QUALITY
CONTENT?
A COMPARISON BETWEEN OPTIMIZED
AND NON-OPTIMIZED HEALTH-RELATED
WEB PAGES
Sebastian Schultheiß, Helena Häußler, and Dirk Lewandowski
Hamburg University of Applied Sciences, Germany
searchstudies.org/our-team
ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
March 14–18, 2022

2. 1 INTRODUCTION

3. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
1. INTRODUCTION
2
Users
Search engine optimization (SEO)
 SEO can be seen as an expression of the content producers’ predominantly commercial interests
 Indicators like commercial interests are perceived by the users and influence how they think about those websites
- People use search engines to make
important medical decisions
[1,4]
- Evaluating the quality of health
information is challenging [5,6]
- Laypeople evaluate the quality
based on indicators and criteria, e.g.
- SEO can be defined as „the practice
of optimizing web pages in a way that
improves their ranking in the organic
search results.“ [3]
- SEO is a multi-billion-dollar industry
[7]
- SEO has a measurable influence on
result rankings [2]
a website is
pushing to sell
something
negative indicator
of criterion
“objectivity”

4. 3 RESEARCH QUESTIONS

5. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
3. RESEARCH QUESTIONS
4
RQ1 What differences can be found in the quality assessments of health-related web pages
when the pages are differentiated by presence/absence of SEO?
RQ2 What are the differences between the evaluations of laypeople and those of experts?
RQ3 How do laypeople justify their evaluations of optimized and non-optimized web pages?
Does the presence or absence of
SEO measures allow conclusions to
be drawn about content quality of
those pages?

6. 4 METHODS

7. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
4. METHODS
6

8. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
4. METHODS
optimized pages: clear intention to perform SEO, e.g., using a SEO plugin
non-optimized pages: SEO basics not met, e.g., not using a HTML title
7

9. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
4. METHODS
8

10. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
4. METHODS
9

11. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
4. METHODS
10

12. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
4. METHODS
11

13. 5 KEY RESULTS
WEB PAGE EVALUATION BY THE SUBJECTS

14. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
5. KEY RESULTS: WEB PAGE EVALUATION BY THE SUBJECTS
13
Evaluation by
laypeople
Evaluation of
optimized pages
Evaluation of
non-optimized pages
Evaluation by
experts
vs. vs.

15. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
5. KEY RESULTS: WEB PAGE EVALUATION BY THE SUBJECTS
14
<
Evaluation of
optimized pages
Evaluation of
non-optimized pages
[F(1, 59)= 4.217, p = 0.044]

16. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
5. KEY RESULTS: WEB PAGE EVALUATION BY THE SUBJECTS
15
Evaluation of
optimized pages
Evaluation of
non-optimized pages
[F(1, 59)= 4.217, p = 0.044]
<

17. 5 KEY RESULTS
LAYPEOPLE JUSTIFICATIONS OF WEB PAGE
EVALUATIONS

18. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
5. KEY RESULTS: LAYPEOPLE JUSTIFICATIONS OF WEB PAGE EVALUATIONS
3
6
11
Website Operator appears
competent
optimized
pages
non-optimized
pages
1
4 5
Website Operator seems
reputable
optimized
pages
non-optimized
pages
No. of positive statements
(=agreeing)
No. of negative statements
(=disagreeing)
Most frequent statements for the criterion “expertise“
grouped in pairs of contrast
17
“The website operator is a university
clinic and appears competent.“
“It seems that this is the website of a
software company(?)“

19. 6 DISCUSSION AND CONCLUSION

20. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
6. DISCUSSION AND CONCLUSION
19
We investigated whether quality differences are perceived between health-related web pages
that engage in search engine optimization (SEO) and pages that do not
- Non-optimized pages were attributed a higher level of expertise
- Non-optimized pages are less often commercially motivated
(e.g., web pages of public institutions)
- Laypeople justified their assessments with the more competent and reputable
appearance of non-optimized web pages
- High-quality but non-optimized content could be outranked by optimized content of
lower quality
- Search engine users could miss out on relevant information
- Implications for content producers and search engine operators
- Repetition of the study with doctors
- Investigation of the transferability of the results to other topics, e.g., political
information

21. THANK YOU FOR YOUR
ATTENTION!
Sebastian Schultheiß
searchstudies.org/team/
sebastian-schultheiss
Helena Häußler
searchstudies.org/team/
helena-haeussler
Dirk Lewandowski
searchstudies.org/team/
dirk-lewandowski
ACKNOWLEDGMENTS
This work is funded by the German Research Foundation (DFG - Deutsche Forschungsgemeinschaft), grant number 417552432.

22. 7 REFERENCES

23. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22)
Sebastian Schultheiß, Helena Häußler, Dirk Lewandowski
7. REFERENCES
[1] Eurostat. 2021. One in two EU citizens look for health information online. Retrieved August 25, 2021 from https://ec.europa.eu/eurostat/de/web/products-
eurostat-news/-/edn-20210406-1
[2] Dirk Lewandowski, Sebastian Sünkler, and Nurce Yagci. 2021. The influence of search engine optimization on Google’s results. In 13th ACM Web Science
Conference 2021, ACM, New York, NY, USA, 12–20. DOI:https://doi.org/10.1145/3447535.3462479
[3] Kai Li, Mei Lin, Zhangxi Lin, and Bo Xing. 2014. Running and Chasing -- The Competition between Paid Search Marketing and Search Engine Optimization.
In 2014 47th Hawaii International Conference on System Sciences, IEEE, 3110–3119. DOI:https://doi.org/10.1109/HICSS.2014.640
[4] Lily Ray. 2020. 2020 Google Search Survey: How Much Do Users Trust Their Search Results? Retrieved September 15, 2021 from
https://moz.com/blog/2020-google-search-survey
[5] Doris Schaeffer, Dominique Vogt, Eva-Maria Berens, and Klaus Hurrelmann. 2016. Ergebnisbericht Gesundheitskompetenz der Bevölkerung in Deutschland.
Retrieved from https://pub.uni-bielefeld.de/download/2908111/2908198/Ergebnisbericht_HLS-GER.pdf
[6] Yalin Sun, Yan Zhang, Jacek Gwizdka, and Ciaran B. Trace. 2019. Consumer Evaluation of the Quality of Online Health Information: Systematic Literature
Review of Relevant Criteria and Indicators. J. Med. Internet Res. 21, 5 (May 2019), e12522. DOI:https://doi.org/10.2196/12522
[7] tbrc.info. 2021. Global Search Engine Optimization Services Market Report 2021 – Opportunities And Strategies, Market Forecast And Trends. Retrieved
August 23, 2021 from https://blog.tbrc.info/2021/08/global-search-engine-optimization-services-market-report-2021-opportunities-and-strategies-market-
forecast-and-trends/
project page:
searchstudies.org/research/seo-effekt
22