This document discusses modern web search and Google's search engine architecture specifically. It describes how search engines work by crawling the web to create an index rather than searching the live web. It then details Google's crawling system, how it indexes pages by creating an inverted index, and how it ranks pages using factors like PageRank to determine relevance. The document provides technical details on Google's systems for scalability and speeding up query processing.

1. Modern Web Search
Web Information Systems -
2015

2. Module Outline
● Story so far…
● This presentation talks about:
o Search Engine (Google as a case study)
 Crawlers
 Web Indexing

3. Web Search Basics
● Web search engines don’t search the web
o They search a copy of the web
o They crawl or spider documents from the web
o They index the documents, and provide a
search interface based on that index
o Users click on links to visit the actual, original
web document
slide taken from Hugh E. Williams Ebay Inc.

4. Google Search Engine Architecture
Taken from: http://infolab.stanford.edu/~backrub/google.html

5. Google Search Engine Architecture
Taken from: http://infolab.stanford.edu/~backrub/google.html

6. Google Search Engine Architecture
Taken from: http://infolab.stanford.edu/~backrub/google.html

7. Web Crawlers
Main idea:
● Start with known sites
● Record information for these sites
● Follow the links from each site
● Record information found at new sites
● Repeat
Reference: http://web.mst.edu/~ercal/253/Papers/WebSearchEngines-1.pdf

8. Web Crawlers
● Issues with this:
o Speed
o Politeness
o Excluded content
o Spamming
o Continuous Crawling - e.g. “current weather”
Reference: http://web.mst.edu/~ercal/253/Papers/WebSearchEngines-1.pdf

9. Some more Crawler Challenges
Crawler challenges:
● HTTP HEAD and GET requests sometimes return different
headers
● A pages can redirect to itself, or into a cycle (more in a
moment)
● Pages can look different to end-user browsers and crawlers
● Pages can require JavaScript processing
● Pages can require cookies
● Pages can be built in non-HTML environments

10. The Processing Involved ...A Google Case Study
“In order to scale to hundreds of millions of web pages, Google
has a fast distributed crawling system. A single URLserver
serves lists of URLs to a number of crawlers (we typically ran
about 3). Each crawler keeps roughly 300 connections open at
once. This is necessary to retrieve web pages at a fast enough
pace. At peak speeds, the system can crawl over 100 web pages
per second using four crawlers. This amounts to roughly 600K
per second of data. ”
Taken from: http://infolab.stanford.edu/~backrub/google.html

11. Web Indexing
● Given a set of keywords, how do we find relevant documents?
● Looked at inverted index in the previous lecture
Taken from: http://infolab.stanford.edu/~backrub/google.html and
http://www.google.com/insidesearch/howsearchworks/crawling-indexing.html

12. Inverted Index
● Simple (one occurrence per doc):
word (e.g. “knoesis”): DocID, DocID, DocID
● Detailed (all occurrences in docs):
word (e.g. “knoesis”):: DocID, Position, Position, DocID, Position.
. .
● Even more detailed: Position annotated with info (heading,
boldface, . .). Useful for ranking.
● Adding another dimension: spatial e.g. Google’s new Pigeon
rank search
Reference: http://www.google.com/technology/pigeonrank.html

13. Google’s Inverted Index

14. Indexing Tricks
● Scale Up: “Barrels”
● Phrase Indexing: sublists
o “electrical” is the union of the sets containing terms
{“electrical apparatus”, “electrical current”, ..}
● Anchor text: text linking to other web pages. These links can
also provide a summary about the linked page
o This is Tanvi Banerjee’s web page
References: Web Search Engines: Part 2, David Hawking

15. Page Ranking Using Google as Case Study
“Google maintains much more information about web documents
than typical search engines. Every hitlist includes position, font,
and capitalization information. Additionally, we factor in hits from
anchor text and the PageRank of the document. Combining all of
this information into a rank is difficult. We designed our ranking
function so that no particular factor can have too much influence.”
Reference: http://infolab.stanford.edu/~backrub/google.html

16. Query Processing
● Current search engines return only documents containing all
the query words
● Improve search quality by incorporating more factors in the
page ranking algorithm => makes it slow
● Speed up Query Processing
o Clever numbering of search documents in decreasing
order of importance to current query
o Caching
References: Web Search Engines: Part 2, David Hawking

17. Other Aspects of Search Engines
● Numerous other aspects of a search engine
o query specific advertisement
o image, video, audio search
● Many challenges to building a search engine
o Continuous room for improvement (Google’s Pigeon
rank..)