We carried out a retrieval effectiveness test on the three major web search engines (i.e., Google, Microsoft and Yahoo). In addition to relevance judgments, we classified the results according to their commercial intent and whether or not they carried any advertising. We found that all search engines provide a large number of results with a commercial intent. Google provides significantly more commercial results than the other search engines do. However, the commercial intent of a result did not influence jurors in their relevance judgments.

1. 11.02.2011
The Influence of Commercial Intent of Search Results
on Their Perceived Relevance
Dirk Lewandowski
Hamburg University of Applied Sciences, Germany
Introduction
• Ongoing discussion about results quality
– Did Google’s results really get worse?
– Do search engine results (and especially, Google’s) get more and more
commercial?
• Discussion on biased search results / “search neutrality”
– Does Google prefer its own services in its rankings?
• Objective
– To find out whether commercial results have a negative effect on relevance as
perceived by the users.
1

2. 11.02.2011
Search engine market share (Germany)
Previous work
• Retrieval effectiveness tests
– Lots of work done
– Main objective is to compare results quality of commercial Web search engines
– Tests are highly standardized
• Commercial intent
– Some studies on commercial intent of the searchers
– No studies on the commercial intent of the documents
2

3. 11.02.2011
Research questions
• RQ1: How effective are web search engines in retrieving relevant results?
• RQ2: What ratio of the top 10 search engine results are pages with a
commercial intent?
• RQ3: Are results with a commercial intent comparably relevant to non-
commercial results?
• RQ4: Do banner or textual ads have a negative effect on the perceived
relevance?
Test design
• Retrieval effectiveness test
– To collect relevance judgments
• Results classification
– Classify each result according to its commercial intent
• Commercial intent of the page/website itself
• Commercial intent as expressed through advertisements on the page
3

4. 11.02.2011
Retrieval effectiveness test design
• 50 queries
– “Real life queries” from students asked for their last query entered into a Web
search engine
– Mainly simple queries without an explicit commercial intent
• 3 search engines
– Google, Microsoft, Yahoo (German user interfaces)
• 10 results per query
• Binary relevance judgments
• 50 undergraduate students as jurors
Retrieval effectiveness test: Data collection
• Results from all search engines collected
• Results
– were made anonymous
– were randomized
• Each juror evaluated the complete results set for one query
4

5. 11.02.2011
Results classification (1)
• Commercial
1. A product is sold on the page
2. Webpage of a company
• Non-commercial
3. Page of a public authority, government office, administration, etc.
4. Page of a non-governmental organization, association, club, etc.
5. Private page
6. Page of an institution of higher education
• Other
Results classification (2)
• Number of banner ads
• Number of text-based ads (“sponsored links”)
5

6. 11.02.2011
Limitations
• Results cannot be generalised
– Commercial intent of results does not have a negative influence on their
perceived relevance when asking undergraduate students.
• Relatively small dataset
– 500 URLs per search engine (total of 1,500 URLs)
Results
6

7. 11.02.2011
Results
1. Results from the classification task
2. Results from the retrieval effectiveness test
3. Combined results: Influence of commercial intent on the perceived
relevance
Results
1. Results from the classification task
7

8. 11.02.2011
Results from the classification task (1)
Results with a commercial intent
account for
– 82% (Google)
– 75% (Microsoft)
– 78% (Yahoo)
Results from the classification task (2)
• Lowest ratio of results with a commercial intent on the first position.
8

9. 11.02.2011
Results from the classification task (3)
• Google has a significantly higher ratio of results with sponsored links.
– Google: 23.7% of all results
– Microsoft: 12.3%
– Yahoo: 17.7%
• No significant difference between the ratios of results with banner ads.
– Google: 21.3% of all results
– Microsoft: 23.0%
– Yahoo: 22.0%
Results
• Results from the retrieval effectiveness test
9

10. 11.02.2011
Macro precision
Number of queries answered best by an individual search engine
Mean Average Precision
All search engines, all results
10

11. 11.02.2011
Results
• Combined results: Influence of commercial intent on the perceived
relevance
Relevance of results with a commercial intent
• Relevance of results with commercial intent is comparable to non-
commercial results
– Google: 59.4% of commercial results relevant (vs. 58.7%)
– Microsoft: 34.6% vs. 36.5%
– Yahoo: 51.1% vs. 51.7%
11

12. 11.02.2011
Relevance of results with advertising
• Relevance of results with text-based ads is comparable to results without
text-based ads
– Google: 59.8% vs. 59.3%
– Microsoft: 40.0% vs. 33.9%
– Yahoo: 52.7% vs. 52.0%
• Relevance of results with banner ads is comparable to results without
banner ads
– Google: 58.5% vs. 59.8%
– Microsoft: 33.3 vs.35.1%
– Yahoo: 55.8% vs. 49.1%
Discussion
12

13. 11.02.2011
Discussion
• Most striking result is that Google has significantly more results with text-
based ads.
• Search Engine Optimization (SEO) may be a reason
– Many pages are highly optimised for Google.
• Wouldn’t it be clever to prefer pages carrying ads from the search engine’s
own ads system?
– However, Google says it doesn’t do this.
Google‘s advertising model
13

14. 11.02.2011
Conclusion & further research
Conclusion
1. In terms of retrieval effectiveness, Google outperforms the other two
search engines (RQ1).
2. There is only a slight variation in the ratio of commercial results from
search engine to search engine (RQ2).
3. The results classified as “commercial” are comparably relevant to the
results classified as “non-commercial” (RQ3).
4. Textual or banner ads do not have a negative effect on the perceived
relevance of the corresponding result (RQ4).
14

15. 11.02.2011
Further research
1. Study should be repeated using other user groups
– Relevance judgments came from a comparably homogeneous user group.
– Data already collected for other user groups (another student group;
librarians; crowd-sourcing).
2. Large ratio of results with text-based ads in Google deserves further
investigation.
– Currently, we conduct a large-scale study where we automatically detect
text-based ads on search results, comparing Google and Bing.
• 30,000 queries, 2 search engines, 30 results per query  1.8 Million URLs.
15