Your SlideShare is downloading. ×
Why we need an independent index of the Web
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Why we need an independent index of the Web

74
views

Published on

Published in: Internet, Technology, Design

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
74
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Why we need an independent index of the Web Dirk Lewandowski dirk.lewandowski@haw-hamburg.de http://www.bui.haw-hamburg.de/lewandowski.html @Dirk_Lew Society of the Query Conference, Amsterdam, 7/11/2013
  • 2. The “local copy” of the Web •  Web Indexing –  New, changed, deleted document –  “Holy grail” of keeping the index complete and current Risvik, K. M., & Michelsen, R. (2002). Search engines and web dynamics. Computer Networks, 39(3), 289–302.
  • 3. Representation of documents in a search engine Referring documents à Document à Metadata (examplex) heading1 heading 2 Anchor text Anchor text Anchor text From the source code - Title - Description - Keywords - Author From the document (document info) - Length - Date - Decay - Name of the author From the Web - PageRank - Number of citations
  • 4. The User’s Perspective •  Everyone uses search engines (Purcell, Brenner & Raine, 2012; van Eimeren & Frees, 2012) •  Market is dominated by Google (ComScore data) •  Users rely on –  Google’s method of ordering results –  Google’s method of collecting data à If Google hasn’t seen it — and indexed it — or kept it up to date, it can’t be found with a search query.
  • 5. Freshness of Web search engines (see Lewandowski, Wahlig & Meyer-Bautor, 2006; Lewandowski, 2008) Original (as of yesterday) Google‘s copy (as of yesterday)
  • 6. What about the alternatives to Google? •  Many “seems to be” search engines –  Accessing the data of another search engine –  Representing nothing more than an alternative user interface to one of the more well-known engines –  In many cases, that turns out to be Google –  E.g., in Germany, we can see that the major internet portals T-Online, GMX, AOL, and web.de all display results obtained from Google
  • 7. Why is one search engine not enough? •  We need more than one search engine to ensure that a broad range of opinions are represented in the search market. •  Users should have the choice between different worldviews which originate as a product of algorithm-based search result generation •  Ideology-free search algorithms are simply not possible
  • 8. Alternative Search Engine Indexes •  There are only a handful of search engines that operate their own indexes, due to costs and technical complexity •  Search engines start-ups –  Use an existing external index –  Focus on a specialised topic (which requires only a small index) –  Aggregate data from different search engines (meta search engine) •  Actual search engine startups like Blekko and Duck Duck Go are more the exception than the rule
  • 9. Partner model •  “Real” search engine providers such as Google and Bing operate their own search engines but also provide their search results to partners •  All the major web portals have now embraced this model. •  Income through ads; revenue-sharing •  Attractiveness of the model –  The search engine provider encounters only minimal costs –  The operator of the portal no longer needs to go to the great expense of running its own search engine. –  The partner index model has served to thin out the competition in the search industry.
  • 10. Access to Search Engine Indexes •  Application programming interfaces (APIs) –  No direct access to the search engine index –  Limited number of top results which have already been ranked by the search engine provider –  Access via APIs is similar to what is occurring at the meta-search engines –  The representation of the document in the source search engine is also not included
  • 11. Alternative Search Engines •  What constitutes an “alternative search engine”? –  All search engines that are not Google? (“Google Killers“, e.g., Cuil) –  Some alternatives are not perceived as such because they are considered to be simply the same as Google (e.g., Bing) –  Search engines which explicitly position themselves as an alternative to Google through a regional approach (e.g., Seekport) –  New approaches to search / “Real alternatives”: Alternative approaches to gathering and representing web content
  • 12. Public Support for Search Engine Technology? •  Quaero/Theseus: Funding a “Google Killer”? –  Quaero: Technologies for multimedia searching. –  Theseus: Semantic technologies for business-to-business applications (without focusing exclusively on search). •  The proposal to provide government funding for search engine technology has been subject to intense criticism in the past •  Establish a single alternative? •  A number of factors which would cause it to fail –  Poor marketing –  Graphic design of the user interface –  ... •  Regardless of the reason, a failure of the new search engine would result in the entire publicly funded initiative failing.
  • 13. Economic perspective •  Only the largest internet companies are able to afford large indexes. •  Microsoft is the only company besides Google to possess a comprehensive search engine index. •  Yahoo gave up on its own index several years ago •  It appears as though operating a dedicated index is attractive to practically no one — and there are hardly any candidates with the necessary financial resources in any case
  • 14. The Solution •  Create the conditions that will make establishing alternative search engines possible •  We can expect that the possibilities it presents would benefit a number of different companies, individuals, and institutions. •  The result will be fair competition to develop the best concepts for using the data provided by the index.
  • 15. Vision •  “An index of the web that can be accessed at fair conditions for everyone” –  “Everyone” means that anyone who is interested can access the index. –  “Fair conditions” does not mean that access to the index must be free of charge for everyone. A certain number of document requests per day should be available at no cost in order to promote non-profit projects. –  “Access” to the index can be defined as the ability to automatically query the index with ease. –  The concept “index of the web” is intended to cover as much of the web as possible
  • 16. Funding and operation •  Funding –  This type of project cannot be supported by any one country alone. The only feasible option is a pan-European initiative. •  Who would operate the index? –  Existing research institution or newly-founded institution –  The operator of the index should not obtain the exclusive right to determine the way in which the documents are used or made available (à Board of trustees)
  • 17. Conclusion: Advantages of an independent index of the web •  Motivate companies, institutions, and developers pursuing personal projects to create their own search applications. •  The data available on the web is so boundless that it lends itself to countless applications in a broad range of fields. •  Enable applications we are not yet capable of even imagining. •  An open structure, transparency with respect to access, and the assurance of permanent availability thanks to state sponsorship would lay the groundwork for innovation.
  • 18. Thank you Prof. Dr. Dirk Lewandowski Hochschule für Angewandte Wissenschaften Hamburg dirk.lewandowski@haw-hamburg,de Twitter: Dirk_Lew http://www.bui.haw-hamburg.de/lewandowski.html http://www.searchstudies.org

×