The document discusses how search engines work by describing their main components and processes. It explains that search engines crawl websites to index their content, then use that index to match users' search queries and return relevant results. The document outlines the key steps search engines go through, including crawling, indexing, processing searches, retrieving matches, ranking results by relevance, and displaying them to users. It also notes some of the challenges of making search engines return high-quality results.

1. How SearcH engineS
work
Presentation
by
cHinna

2. What is Search Engine
Search engine is a software program that
searches for sites based on the words that you
designate as search terms.
"Search engine" is the popular term for an
Information Retrieval (IR) system.
2

3. Motto of search engines
A web search engine is designed to search for
information on the World Wide Web and
FTP servers. The search results are generally
presented in a list of results often referred to
as SERPS, or "search engine results pages".
The information may consist of web pages,
images, information and other types of files.
3

4. Purpose of Search Engines
Helping people find what they’re looking
for
• Starts with an "information need"
• Convert to a query
• Gets results
In the materials available
• Web pages
• Other formats
• Deep Web
4

5. HISTORY
Archie – First search tool for the Internet
Gopher – indexed plain text documents
Jughead – searched the files stored in
Gopher index systems
Wandex – First Web search engine
5

6. How web search engines work
search engine operates in the following
order:
Web Crawling
Indexing
Searching
6

7. How do Search Engine Works
 Spiders
 Robots
7

8. Search is Not a Panacea
Search can’t find what’s not there
• The content is hugely important
Information Architecture is vital
Usable sites have good navigation and
structure
8

9. Search Engine Modules
A query processor
A search and matching function
A ranking capability
Summarizing and Presenting documents.
9

10. Search Engines Mode of Working in
Earlier Days
From 1990-1998 (1st Generation of search
tools):
• Looked at title of web pages
• Ranking was based on page content
• Looked at number of times the search term
appeared on the page
• Looked at metatags
10

11. SEO (Search Engine Optimization)
Used by companies to get a higher result in
search engines
White hat: Using legitimate techniques
Black hat: Using illegal techniques to trick
the search engine, like paying sites to link
to you.
11

12. Search Processing
12

13. Search is Only as Good as the Content
Users blame the search engine
• Even when the content is unavailable
Understand the scope of site or intranet
• Kinds of information
• Divided sites: products / corporate info
• Dates
• Languages
• Sources and data silos: databases...
• Update processes
13

14. Making a Searchable Index
Store text to search it later
Many ways to gather text
• Crawl (spider) via HTTP
• Read files on file servers
• Access databases (HTTP or API)
• Data silos via local APIs
• Applications, CMSs, via Web Services
Security and Access Control
14

15. Robot Indexing Diagram
Sour
15

16. What the Index Needs
Basic information for document or record
• File name / URL / record ID
• Title or equivalent
• Size, date, MIME type
Full text of item
More metadata
• Product name, picture ID
• Category, topic, or subject
• Other attributes, for relevance ranking and display
16

17. Simple Index Diagram
17

18. Index Issues
Stopwords
Stemming
Metadata
• Explicit (tags)
• Implicit (context)
Semantics
• CMS and Database fields
• XML tags and attributes
18

19. Search Query Processing
What happens after you click the search
button, and before retrieval starts.
Usually in this order
• Handle character set, maybe language
• Look for operators and organize the query
• Look for field names or metadata
• Extract words (just like the indexer)
• Deal with letter casing
19

20. Search and Retrieval
Retrieval: find files with query terms
Not the same as relevance ranking
Recall: find all
relevant items
Precision: find only
relevant items
Increasing one
decreases the other
20

21. Retrieval = Matching
Single-word queries
• Find items containing that word
Multi-word queries: combine lists
• Any: every item with any query word
• All: only items with every word
• Phrases: find only items with all words in
order
Boolean and complex queries
• Use algorithm to combine lists
21

22. Why Searches Fail
Empty search
Nothing on the site on that topic (scope)
Misspelling or typing mistakes
Vocabulary differences
Restrictive search defaults
Restrictive search choices
Software failure
22

23. Relevance Ranking
Theory: sort the matching items, so the most
relevant ones appear first
Can't really know what the user wants
Relevance is hard to define and situational
Short queries tend to be deeply ambiguous
• What do people mean when they type “bank”?
First 10 results are the most important
23

24. Relevance Processing
Sorting documents on various criteria
Start with words matching query terms
Citation and link analysis
• Like old library Citation Indexes
• Not only hypertext, but the links
• Google PageRank
• Incoming links
• Authority of linkers
Taxonomies and external metadata
24

25. Search Results Interface
What users see after they click the Search
button
The most visible part of search
Elements of the results page
• Page layout and navigation
• Results header
• List of results items
• Results footer
25

26. Search Suggestions
Human judgment beats algorithms
Great for frequent, ambiguous searches
• Use search log to identify best candidates
Recommend good starting pages
• Product information, FAQs, etc.
Requires human resources
• That means money and time
More static than algorithmic search
26

27. Search Metrics
Number of searches
Number of matches searches
Traffic from search to high-value pages
Relate search changes to other metrics
27

28. Query Example
Consider the Query Mahendra Singh Dhoni
A good answer contains all the three words, and more
frequently the better, we call this Term Frequency(TF)
Some Query terms are more important those have better
discriminating power than others
For example an answer containing only "Dhoni" is likely to
be better than an answer containing only “Mahendra“
We call this Inverse Document Frequency (IDF)
28

29. Search Will Never Be Perfect
Search engines can’t read minds
• User queries are short and ambiguous
Some things will help
• Design a usable interface
• Show match words in context
• Keep index current and complete
• Adjust heuristic weighting
• Maintain suggestions and synonyms
• Consider faceted metadata search
29