The document discusses the history and impact of the World Wide Web on information retrieval and search engines. It covers: 1) How Berners-Lee invented the World Wide Web in 1990 by creating HTTP, HTML, and the first browser and server. The Web has since grown enormously. 2) How the Web changed search by requiring crawling to collect documents in a central repository before indexing, and by increasing the scale, size, and difficulty of relevance prediction due to the large collection. 3) The basic architecture of centralized crawling and indexing used by most early search engines, and how distributed and cluster-based architectures were developed to handle the Web's massive growth.

1. UNIT-IV
IV Year / VIII Semester
By
K.Karthick AP/CSE
KNCET.
KONGUNADU COLLEGE OF ENGINEERING AND
TECHNOLOGY
(Autonomous)
NAMAKKAL- TRICHY MAIN ROAD, THOTTIAM
DEPARTMENT OF COMPUTER SCIENCE AND
ENGINEERING
CS8080 – Information Retrieval Techniques

2. Syllabus

3. The Web
• A Brief History
• In 1990, Berners-Lee Wrote the HTTP protocol
• Defined the HTML language
• Wrote the first browser, which he called World Wide Web
• Wrote the first Web server
• In 1991, he made his browser and server software available
in the Internet. The Web was born!
• Since its inception, the Web became a huge success
• Well over 20 billion pages are now available and accessible
in the Web
• More than one fourth of humanity now access the Web on a
regular basis

4. How the Web Changed Search
• Web search is today the most prominent application of
IR and its techniques—the ranking and indexing
components of any search engine are fundamentally IR
pieces of technology
• The first major impact of the Web on search is related
to the characteristics of the document collection itself
• The Web is composed of pages distributed over
millions of sites and connected through hyperlinks
• This requires collecting all documents and storing
copies of them in a central repository, prior to indexing
• This new phase in the IR process, introduced by the
Web, is called crawling

5. • The second major impact of the Web on search is
related to:
• The size of the collection
• The volume of user queries submitted on a daily basis
• As a consequence, performance and scalability have
become critical characteristics of the IR system
• The third major impact: in a very large collection,
predicting
• relevance is much harder than before
• Fortunately, the Web also includes new sources of
• evidence
• Ex: hyperlinks and user clicks in documents in the
answer set

6. • The fourth major impact derives from the fact
that the Web is also a medium to do business
• Search problem has been extended beyond the
seeking of text information to also encompass
other user needs
• Ex: the price of a book, the phone number of a
hotel, the link fo r downloading a software
• The fifth major impact of the Web on search is
• Web spam Web spam: abusive availability of
commercial information disguised in the form of
informational content
• This difficulty is so large that today we talk of
Adversarial Web Retrieval

7. Web structure
• The web graph
• We can view the static Web consisting of static HTML pages
together with the hyperlinks between them as a directed graph in
which each web page is a node and each hyperlink a directed edge.
• We refer to the hyperlinks into a page as in-links and those out of a
page as out-links.
• The number of in-links to a page (also known as its in-degree) has
averaged from roughly 8 to 15, in a range of studies. We similarly
define the out-degree of a web page to be the number of links out
of it. These notions are represented in Figure 4.1.
• The web can be represented as a graph:
• • Webpage ≡ node
• • hyperlink ≡ directed edge
• • # in-links ≡ in-degree of a node
• • # out-links ≡ out-degree of a node

9. Search Engine Architectures
• A software architecture consists of software
components, the interfaces provided by those
components, and the relationships between
them
– describes a system at a particular level of
abstraction
• Architecture of a search engine determined by
2 requirements
– effectiveness (quality of results) and efficiency
(response time and throughput)

10. • Each Search Engine architecture has its own
components. The architecture of search
engines falls into the following categories:
• Basic architecture- Centralized crawler
• Cluster based architecture
• Distributed Architectures
• Multi site Architecture

11. Basic architecture- Centralized
crawler
– used by most engines
– crawlers are software agents that traverse the Web copying pages
– pages crawled are stored in a central repository and then indexed
– index is used in a centralized fashion to answer queries
– most search engines use indices based on the inverted index
– only a logical, rather than a literal, view of the text needs to be
indexed
• Normalization operations
– removal of punctuation
– substitution of multiple consecutive spaces by a single space
– conversion of uppercase to lowercase letters
• Some engines eliminate stopwords to reduce index size
• Index is complemented with metadata associated with pages
– Creation date, size, title, etc.

12. • Given a query
– 10 results shown are subset of complete result set
– if user requests more results, search engine can
• recompute the query to generate the next 10 results
• obtain them from a partial result set maintained in
main memory
– In any case, a search engine never computes the
full answer set for the whole Web

14. • Main problem faced by this architecture
– gathering of the data, and
– sheer volume
• Crawler-indexer architecture could not cope
with Web growth (end of 1990s)
– solution: distribute and parallelize computation

15. Cluster-based Architecture
• Current engines adopt a massively parallel cluster-
based architecture
– document partitioning is used
– replicated to handle the overall query load
– cluster replicas maintained in various geographical
locations to decrease latency time (for nearby users)
• Many crucial details need to be addressed
– good balance between the internal and external activities
– good load balancing among different clusters
– fault tolerance at software level to protect against
hardware failures

16. • Caching
– Search engines need to be fast
• whenever possible, execute tasks in main memory
• caching is highly recommended and extensively used
– provides for shorter average response time
– significantly reduces workload on back-end servers
– decreases the overall amount of bandwidth utilized
– In the Web, caching can be done both at the client
or the server side

17. – Caching of answers
• the most effective caching technique in search engines
• query distribution follows a power law
– small cache can answer a large percentage of queries
• with a 30% hit-rate, capacity of search engine increases
by almost 43%
– Still, in any time window a large fraction of queries
will be unique
• hence, those queries will not be in the cache
• 50% in Baeza-Yates et al this can be improved by also
caching inverted lists at the search cluster level

19. Distributed Architectures
– There exist several variants of the crawler-indexer
architecture
– We describe here the most important ones
• most significant early example is Harvest
• among newest proposals, we distinguish the multi-site
architecture proposed by Baeza-Yates et al
– Harvest
• Harvest uses a distributed architecture to gather and distribute
data
• interestingly, it does not suffer from some of common problems of
the crawler-indexer architectures, such as
– increased servers load caused by reception of simultaneous requests
from different crawlers
– increased Web traffic, due to crawlers retrieving entire objects,
• while most content is not retained eventually
• lack of coordination between engines, as information is
gathered independently by each crawler

20. • To avoid these issues, two components are
introduced: gatherers and brokers
• gatherer: collects and extracts indexing
information from one or more Web servers
• broker: provides indexing mechanism and
query interface to the data gathered

22. Multi-site Architecture
– As the document collection grows
• capacity of query processors has to grow as well
• unlikely that growth in size of single processors can
match growth of very large collections
– even if a large number of servers is used
– main reasons are physical constraints such as size of single
data center and power and cooling requirements

23. – Distributed resolution of queries using different
query processors is a viable approach
• enables a more scalable solution
• but also imposes new challenges
• one such challenge is the routing of queries to
appropriate query processors
– to utilize more efficiently available resources and provide
more precise results
– factors affecting query routing include geographical proximity,
query topic, or language of the query

24. – Geographical proximity: reduce network latency
by using resources close to the user posing the
query
– Possible implementation is DNS redirection
• according to IP address of client, the DNS service routes
query to appropriate Web server
– usually the closest in terms of network distance
– As another example, DNS service can use the
geographical location to determine where to route
queries to
– There is a fluctuation in submitted queries from a
particular geographic region during a day
• possible to offload a server from a busy region by
rerouting some queries to query servers in a less busy
region

25. Search Engine Ranking
• Ranking is the hardest and most important function of
a search engine
• A key first challenge
• devise an adequate process of evaluating the ranking,
in terms of relevance of results to the user
• without such evaluation, it is close to impossible to fine
tune the ranking function
• without fine tuning the ranking, there is no state-of-
the-art engine—this is an empirical field of science
• A second critical challenge
• identification of quality content in the Web

26. • Evidence of quality can be indicated by several
signals such as:
• domain names
• text content
• links (like Page Rank)
• Web page access patterns as monitored by the
search engine
• Additional useful signals are provided by the
layout of the Web page, its title, metadata,
font sizes, etc.

27. Third critical challenge
• avoiding, preventing, managing Web spam
• spammers are malicious users who try to trick
search engines by
• artificially inflating signals used for ranking
• a consequence of the economic incentives of the
current advertising model adopted by search
engines
• A fourth major challenge
• defining the ranking function and computing it

28. Link-based Ranking
• Number of hyperlinks that point to a page
provides a measure of its popularity and
quality
• Many links in common among pages are
indicative of page relations with potential
value for ranking purposes
• Examples of ranking techniques that exploit
links are discussed next

29. Early Algorithms
• Use incoming links for ranking Web pages
• But, just counting links was not a very reliable measure of
authoritativeness
• Yuwono and Lee proposed three early ranking algorithms (in
addition to the classic TF-IDF vector ranking)
• Boolean spread
• given page p in result set
• extend result set with pages that point to and are pointed by page p
• Vector spread
• given page p in result set
• extend result set with pages that point to and are pointed by page p
• most-cited
• a page p is assigned a score given by the sum of number of query
words contained in other pages that point to page p
• WebQuery is an early algorithm that allows visual browsing of Web
pages

30. Page Rank
• The basic idea is that good pages point to good pages
• Let p, r be two variables for pages and L a set of links
• Is the link analysis method that uses link structure to
rank the web pages by search engines
• PAGERANK: the web graphs a numerical score between
0 and 1, known as its Page Rank. The Page Rank of a
node will depend on the link structure of the web
graph
• Page Rank: scoring measure based only on the link
structure of web pages
– Every node in the web graph is given a score between 0
and1, depending on its in and out-links
– Given a query, a search engine combines the Page Rank
score with other values to compute the ranked retrieval
(e.g. cosine similarity, relevance feedback, etc.)

31. The PageRank Citation Ranking: Bring
Order to the web
• Why is Page Importance Rating important?
– New challenges for information retrieval on the World Wide
Web.
• Huge number of web pages: 150 million by1998
1000 billion by 2008
• Diversity of web pages: different topics, different quality, etc.
• What is PageRank?
• A method for rating the importance of web pages
objectively and mechanically using the link structure of the
web.

32. The History of PageRank
• PageRank was developed by Larry Page (hence
the name Page-Rank) and Sergey Brin.
• It is first as part of a research project about a
new kind of search engine. That project
started in 1995 and led to a functional
prototype in 1998.
• Shortly after, Page and Brin founded Google.
• 16 billion…

33. • There are some news about that PageRank
will be canceled by Google.
• There are large numbers of Search Engine
Optimization (SEO).
• SEO use different trick methods to make a
web page more important under the rating of
PageRank.

34. • 150 million web pages  1.7 billion links
• Backlinks and Forward links:
A and B are C’s backlinks
C is A and B’s forward link

46. • Underlying idea: computing a score reflecting
the “visitability” of a page
– When surfing the web, a user may visit some
pages more often than others (e.g. more in-links)
– Pages that are often visited are more likely to
contain relevant information
– When a dead-end page is reached, the user may
teleport (e.g. type an address in the browser)

47. • Page Rank: Strengths & Weaknesses
• Strengths
– Elegant theoretical foundation with nice
interpretations (e.g., stochastic matrices, principal
Eigenvectors, stationary state probabilities in random
walk model/ Markov-chain process)
– Can be extended to topic-based and personalized
models
– Static measure (i.e., can be precomputed)
– Relatively straight-forward to scale
– PageRank scores are relatively stable to graph
perturbations
• Weaknesses
– Rank is not stable to perturbations

48. HITS
• Hypertext Induced Topic Selection (HITS) is a link
analysis method developed by John Kleinberg in 1999
using Hub and Authority scores.
• considers set of pages S that point to or are pointed by
pages in answer set
• pages that have many links pointing to it are called
authorities
• pages that have many outgoing links are called hubs
• positive two-way feedback
• better authority pages come from incoming edges from
good hubs
• better hub pages come from outgoing edges to good
authorities

49. – Start with a “root set” of pages relevant to a topic
(or information need)
– Create a “base sub graph” by adding to the root
set
– –All pages that have incoming links from pages in the root set
– –For each page in the root set: up to 𝑘 pages pointing to it
– Compute authority and hubness scores for all
pages in the base sub graph
• Rank pages by decreasing authority scores

51. • Let
• H(p): hub value of page p
• A(p): authority value of page p
• H(p) and A(p) are defined such that

52. Simple Ranking Functions
• use a global ranking function such as PageRank
• In that case, quality of a Web page in the result set is
independent of the query
• the query only selects pages to be ranked
• use a linear combination of different ranking signals
• for instance, combine BM25 (text-based ranking) with
PageRank (link-based ranking)
• To illustrate, consider the pages p that satisfy query Q
• Rank score R(p,Q) of page p with regard to query Q can
be computed as
• R(p,Q) = ∝ BM25(p,Q) + (1-∝ ) PR(p)