• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
The Influence of Commercial Intent of Search Results on Their Perceived Relevance
 

The Influence of Commercial Intent of Search Results on Their Perceived Relevance

on

  • 387 views

We carried out a retrieval effectiveness test on the three major web search engines (i.e., Google, Microsoft and Yahoo). In addition to relevance judgments, we classified the results according to ...

We carried out a retrieval effectiveness test on the three major web search engines (i.e., Google, Microsoft and Yahoo). In addition to relevance judgments, we classified the results according to their commercial intent and whether or not they carried any advertising. We found that all search engines provide a large number of results with a commercial intent. Google provides significantly more commercial results than the other search engines do. However, the commercial intent of a result did not influence jurors in their relevance judgments.

Statistics

Views

Total Views
387
Views on SlideShare
386
Embed Views
1

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 1

http://www.slideshare.net 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    The Influence of Commercial Intent of Search Results on Their Perceived Relevance The Influence of Commercial Intent of Search Results on Their Perceived Relevance Document Transcript

    • 11.02.2011 The Influence of Commercial Intent of Search Results on Their Perceived Relevance Dirk Lewandowski Hamburg University of Applied Sciences, GermanyIntroduction•  Ongoing discussion about results quality –  Did Google’s results really get worse? –  Do search engine results (and especially, Google’s) get more and more commercial?•  Discussion on biased search results / “search neutrality” –  Does Google prefer its own services in its rankings?•  Objective –  To find out whether commercial results have a negative effect on relevance as perceived by the users. 1
    • 11.02.2011Search engine market share (Germany)Previous work•  Retrieval effectiveness tests –  Lots of work done –  Main objective is to compare results quality of commercial Web search engines –  Tests are highly standardized•  Commercial intent –  Some studies on commercial intent of the searchers –  No studies on the commercial intent of the documents 2
    • 11.02.2011Research questions•  RQ1: How effective are web search engines in retrieving relevant results?•  RQ2: What ratio of the top 10 search engine results are pages with a commercial intent?•  RQ3: Are results with a commercial intent comparably relevant to non- commercial results?•  RQ4: Do banner or textual ads have a negative effect on the perceived relevance?Test design•  Retrieval effectiveness test –  To collect relevance judgments•  Results classification –  Classify each result according to its commercial intent •  Commercial intent of the page/website itself •  Commercial intent as expressed through advertisements on the page 3
    • 11.02.2011Retrieval effectiveness test design•  50 queries –  “Real life queries” from students asked for their last query entered into a Web search engine –  Mainly simple queries without an explicit commercial intent•  3 search engines –  Google, Microsoft, Yahoo (German user interfaces)•  10 results per query•  Binary relevance judgments•  50 undergraduate students as jurorsRetrieval effectiveness test: Data collection•  Results from all search engines collected•  Results –  were made anonymous –  were randomized•  Each juror evaluated the complete results set for one query 4
    • 11.02.2011Results classification (1)•  Commercial 1.  A product is sold on the page 2.  Webpage of a company •  Non-commercial 3.  Page of a public authority, government office, administration, etc. 4.  Page of a non-governmental organization, association, club, etc. 5.  Private page 6.  Page of an institution of higher education •  OtherResults classification (2)•  Number of banner ads •  Number of text-based ads (“sponsored links”) 5
    • 11.02.2011Limitations•  Results cannot be generalised –  Commercial intent of results does not have a negative influence on their perceived relevance when asking undergraduate students.•  Relatively small dataset –  500 URLs per search engine (total of 1,500 URLs) Results 6
    • 11.02.2011Results1.  Results from the classification task2.  Results from the retrieval effectiveness test3.  Combined results: Influence of commercial intent on the perceived relevanceResults1.  Results from the classification task 7
    • 11.02.2011Results from the classification task (1)Results with a commercial intent account for –  82% (Google) –  75% (Microsoft) –  78% (Yahoo)Results from the classification task (2)•  Lowest ratio of results with a commercial intent on the first position. 8
    • 11.02.2011Results from the classification task (3)•  Google has a significantly higher ratio of results with sponsored links. –  Google: 23.7% of all results –  Microsoft: 12.3% –  Yahoo: 17.7%•  No significant difference between the ratios of results with banner ads. –  Google: 21.3% of all results –  Microsoft: 23.0% –  Yahoo: 22.0%Results•  Results from the retrieval effectiveness test 9
    • 11.02.2011Macro precisionNumber of queries answered best by an individual search engineMean Average PrecisionAll search engines, all results 10
    • 11.02.2011Results•  Combined results: Influence of commercial intent on the perceived relevanceRelevance of results with a commercial intent•  Relevance of results with commercial intent is comparable to non- commercial results –  Google: 59.4% of commercial results relevant (vs. 58.7%) –  Microsoft: 34.6% vs. 36.5% –  Yahoo: 51.1% vs. 51.7% 11
    • 11.02.2011Relevance of results with advertising•  Relevance of results with text-based ads is comparable to results without text-based ads –  Google: 59.8% vs. 59.3% –  Microsoft: 40.0% vs. 33.9% –  Yahoo: 52.7% vs. 52.0%•  Relevance of results with banner ads is comparable to results without banner ads –  Google: 58.5% vs. 59.8% –  Microsoft: 33.3 vs.35.1% –  Yahoo: 55.8% vs. 49.1% Discussion 12
    • 11.02.2011Discussion•  Most striking result is that Google has significantly more results with text- based ads.•  Search Engine Optimization (SEO) may be a reason –  Many pages are highly optimised for Google.•  Wouldn’t it be clever to prefer pages carrying ads from the search engine’s own ads system? –  However, Google says it doesn’t do this.Google‘s advertising model 13
    • 11.02.2011 Conclusion & further researchConclusion 1.  In terms of retrieval effectiveness, Google outperforms the other two search engines (RQ1). 2.  There is only a slight variation in the ratio of commercial results from search engine to search engine (RQ2). 3.  The results classified as “commercial” are comparably relevant to the results classified as “non-commercial” (RQ3). 4.  Textual or banner ads do not have a negative effect on the perceived relevance of the corresponding result (RQ4). 14
    • 11.02.2011Further research 1.  Study should be repeated using other user groups –  Relevance judgments came from a comparably homogeneous user group. –  Data already collected for other user groups (another student group; librarians; crowd-sourcing). 2.  Large ratio of results with text-based ads in Google deserves further investigation. –  Currently, we conduct a large-scale study where we automatically detect text-based ads on search results, comparing Google and Bing. •  30,000 queries, 2 search engines, 30 results per query  1.8 Million URLs. 15