1. 11.02.2011
The Influence of Commercial Intent of Search Results
on Their Perceived Relevance
Dirk Lewandowski
Hamburg University of Applied Sciences, Germany
Introduction
• Ongoing discussion about results quality
– Did Google’s results really get worse?
– Do search engine results (and especially, Google’s) get more and more
commercial?
• Discussion on biased search results / “search neutrality”
– Does Google prefer its own services in its rankings?
• Objective
– To find out whether commercial results have a negative effect on relevance as
perceived by the users.
1

2. 11.02.2011
Search engine market share (Germany)
Previous work
• Retrieval effectiveness tests
– Lots of work done
– Main objective is to compare results quality of commercial Web search engines
– Tests are highly standardized
• Commercial intent
– Some studies on commercial intent of the searchers
– No studies on the commercial intent of the documents
2

3. 11.02.2011
Research questions
• RQ1: How effective are web search engines in retrieving relevant results?
• RQ2: What ratio of the top 10 search engine results are pages with a
commercial intent?
• RQ3: Are results with a commercial intent comparably relevant to non-
commercial results?
• RQ4: Do banner or textual ads have a negative effect on the perceived
relevance?
Test design
• Retrieval effectiveness test
– To collect relevance judgments
• Results classification
– Classify each result according to its commercial intent
• Commercial intent of the page/website itself
• Commercial intent as expressed through advertisements on the page
3

4. 11.02.2011
Retrieval effectiveness test design
• 50 queries
– “Real life queries” from students asked for their last query entered into a Web
search engine
– Mainly simple queries without an explicit commercial intent
• 3 search engines
– Google, Microsoft, Yahoo (German user interfaces)
• 10 results per query
• Binary relevance judgments
• 50 undergraduate students as jurors
Retrieval effectiveness test: Data collection
• Results from all search engines collected
• Results
– were made anonymous
– were randomized
• Each juror evaluated the complete results set for one query
4

5. 11.02.2011
Results classification (1)
• Commercial
1. A product is sold on the page
2. Webpage of a company
• Non-commercial
3. Page of a public authority, government office, administration, etc.
4. Page of a non-governmental organization, association, club, etc.
5. Private page
6. Page of an institution of higher education
• Other
Results classification (2)
• Number of banner ads
• Number of text-based ads (“sponsored links”)
5

6. 11.02.2011
Limitations
• Results cannot be generalised
– Commercial intent of results does not have a negative influence on their
perceived relevance when asking undergraduate students.
• Relatively small dataset
– 500 URLs per search engine (total of 1,500 URLs)
Results
6

7. 11.02.2011
Results
1. Results from the classification task
2. Results from the retrieval effectiveness test
3. Combined results: Influence of commercial intent on the perceived
relevance
Results
1. Results from the classification task
7

8. 11.02.2011
Results from the classification task (1)
Results with a commercial intent
account for
– 82% (Google)
– 75% (Microsoft)
– 78% (Yahoo)
Results from the classification task (2)
• Lowest ratio of results with a commercial intent on the first position.
8

9. 11.02.2011
Results from the classification task (3)
• Google has a significantly higher ratio of results with sponsored links.
– Google: 23.7% of all results
– Microsoft: 12.3%
– Yahoo: 17.7%
• No significant difference between the ratios of results with banner ads.
– Google: 21.3% of all results
– Microsoft: 23.0%
– Yahoo: 22.0%
Results
• Results from the retrieval effectiveness test
9

10. 11.02.2011
Macro precision
Number of queries answered best by an individual search engine
Mean Average Precision
All search engines, all results
10

11. 11.02.2011
Results
• Combined results: Influence of commercial intent on the perceived
relevance
Relevance of results with a commercial intent
• Relevance of results with commercial intent is comparable to non-
commercial results
– Google: 59.4% of commercial results relevant (vs. 58.7%)
– Microsoft: 34.6% vs. 36.5%
– Yahoo: 51.1% vs. 51.7%
11

12. 11.02.2011
Relevance of results with advertising
• Relevance of results with text-based ads is comparable to results without
text-based ads
– Google: 59.8% vs. 59.3%
– Microsoft: 40.0% vs. 33.9%
– Yahoo: 52.7% vs. 52.0%
• Relevance of results with banner ads is comparable to results without
banner ads
– Google: 58.5% vs. 59.8%
– Microsoft: 33.3 vs.35.1%
– Yahoo: 55.8% vs. 49.1%
Discussion
12

13. 11.02.2011
Discussion
• Most striking result is that Google has significantly more results with text-
based ads.
• Search Engine Optimization (SEO) may be a reason
– Many pages are highly optimised for Google.
• Wouldn’t it be clever to prefer pages carrying ads from the search engine’s
own ads system?
– However, Google says it doesn’t do this.
Google‘s advertising model
13

14. 11.02.2011
Conclusion & further research
Conclusion
1. In terms of retrieval effectiveness, Google outperforms the other two
search engines (RQ1).
2. There is only a slight variation in the ratio of commercial results from
search engine to search engine (RQ2).
3. The results classified as “commercial” are comparably relevant to the
results classified as “non-commercial” (RQ3).
4. Textual or banner ads do not have a negative effect on the perceived
relevance of the corresponding result (RQ4).
14

15. 11.02.2011
Further research
1. Study should be repeated using other user groups
– Relevance judgments came from a comparably homogeneous user group.
– Data already collected for other user groups (another student group;
librarians; crowd-sourcing).
2. Large ratio of results with text-based ads in Google deserves further
investigation.
– Currently, we conduct a large-scale study where we automatically detect
text-based ads on search results, comparing Google and Bing.
• 30,000 queries, 2 search engines, 30 results per query  1.8 Million URLs.
15