Search Engine working, Crawlers working, Search Engine mechanism
•Download as PPTX, PDF•
2 likes•144 views
This PPT tell us about how the search engine works and search the entered text, and retrieve information by the techniques involved behind it.
More Related Content
What's hot
What's hot (20)
Similar to Search Engine working, Crawlers working, Search Engine mechanism
Similar to Search Engine working, Crawlers working, Search Engine mechanism (20)
Recently uploaded
Recently uploaded (20)
Search Engine working, Crawlers working, Search Engine mechanism
- 1. Working of web search engine work Umang Mishra Roll no -1631 College – Panjab University (P.U.S.S.G.R.C)
- 2. What is a search engine? A web search engine is a software system that is designed to search for information on the World Wide Web. The search results are generally presented in a line of results often referred to as search engine results pages(SERPs). The information may be a mix of web pages, images, and other types of files
- 4. Brief History of Search engines The first basic search engine was created by three computer science students at McGill University in 1990.The tool was named Archie (a play on the word “Archive”). Archie created a searchable database of public file names on FTP (File Transfer Protocol) sites. Since the tool was launched before the Internet took off on a large-scale basis, the database was small enough to be searched manually. Google began in 1996 as a research project by Larry Page and Sergey Brin Ph.D. students at Stanford University and is now today's biggest search engine. Larry and Sergey begin collaborating on a search engine called BackRub. BackRub operates on Stanford servers for more than a year—eventually taking up too much bandwidth.
- 6. 3 Core Parts Crawling The web ‘spider’ crawls across all the pages on the internet indexing Like a librarian categorising fetched web pages Ranking Set the websites links according to their popularity
- 7. But sometimes it gets complicated… The internet is huge (trillions of webpages) Useless information (old, poorly written, advertising, duplicates data) Homophonic words search (words which are same in sound and sometimes spellings) like Inappropriate stuff Different languages Spam
- 8. Servers
- 9. cooling
- 11. let’s take a little look at how each part works…
- 12. 1. Crawling Definition : A web crawler is a computer program that browses the world wide web(www) Utilities: Gather pages from the web Support a search engine, perform data mining and so on.
- 13. Why we use web crawlers Internet has a wide expense of information Finding relevant information requires efficient mechanism. Web crawler provides that scope to the search engine.
- 14. WebCrawlers How do the web search engines get all of the items they index Main idea: Start with known sites Record information for these sites Follow the links from each site Record information found at new sites Repeat BFS algorithm
- 18. Crawler Working Pick a URL from the frontier Fetch the document at the URL Parse the URL Extract links from it to other docs (URLs) Check if URL has content already seen If not, add to indexes For each extracted URL Ensure it passes certain URL filter tests Check if it is already in the frontier (duplicate URL elimination
- 19. Basic crawler algorithm Create queue with “seed” pages Repeat 1. Fetch each URL on the queue 2. Parse fetched pages 3. Extract URLs they point to 4. Place the extracted URLs on a queue Until empty queue or out of time
- 20. 2. Indexing Indexing is the process of adding Webpages data into search engine index. Depending upon what meta tag we used for our webpage for crawl and index our pages. After a web page or document has been detected by crawlers, all its accessible data is stored (cached) on search engine servers so it can be retrieved when a user performs a search query. Indexing serves two purposes: to return results related to a search engine user’s query to rank those results in order of importance and relevancy
- 21. To understand indexing consider what a crawler and a scraper might identifyfrom a web page and how they might store it.
- 23. 3. Ranking PageRank is an algorithm, first published by Google’s founders in 1998. According to the authors, PageRank is a method for computing a ranking for every web page based on the graph of the web. The graph of the web being referred to looks at the hyperlinks between web pages, and how that creates a web of pages with links. • Sites with thousands of links with are surely more important than sites with only a handful of backlinks
- 24. Motivation and Introduction Why is Page Importance Rating important? New challenges for information retrieval on the World Wide Web. Huge number of web pages: 150 million by1998 1000 billion by 2008 Diversity of web pages: different topics, different quality, etc. What is PageRank? A method for rating the importance of web pages objectively and mechanically using the link structure of the web.
- 25. There are about 200+ factors about page ranking for a webpage ranking
- 26. Some interesting facts about Google search engine Google is the world biggest results finding search engine. Google now processes over 40,000 search queries every second on average (visualize them here), which translates to over 3.5 billion searches per day 1.2 trillion searches per year worldwide.The chart below shows the number of searches per year.
- 27. Thank You Umang Mishra Roll no -1631 College – Panjab University (P.U.S.S.G.R.C)
Editor's Notes
- Different search engines; We know about Google and Bing, can you list any others? Yahoo etc. What kinds of things do you search for? Amazon is also a search engine, because you search for items you might want to buy. It finds them from all over the world and returns the results to you. Push/pull search: talk about how Amazon allows you to search for the item you really want and then gives you suggestions about other products that you may wish to buy at the same time. Offering you items that similar people have also wanted. Search is about patterns and trends in what people want … we all have unique needs, but there are many similarities in interests when you consider segments of people across the globe
- Introduce the most common search engine that we might use. When we start typing, what happens? It begins to try and predict what we are looking for... How? Why? The big challenge in search engines is to keep up to date so that you find the information you are looking for, quickly, otherwise you might go to a different service.
- * The idea sounds very simple in principle, take a picture of the internet, sort and order it, give people what they want.
- Ever wondered what a search engine looks like? The outside of a search engine building is probably not the most impressive thing you have ever seen...
- Lots of searching, very importantly needs lots of cooling. If we were to leave the Air Con off in the ICT suite, the temperature would rise considerably because of 20 PCs on for a few hours a day. Thousands of computers working 24/7/365 creates a lot of heat.
- Broadly covers the principle if web link graphs, composed of the in- and out-links from web pages. ‘Authoritative’ pages tend to have many more in-links (i.e. pages linking to them) – and so if you were clicking randomly forever, you’d be much more likely to arrive at them. These pages are naturally considered better, and so will rank higher in search results (this is the foundation of the PageRank algorithm invented by the founders of Google).
- ** We will propose that a user has searched for the word ‘TIGER’ using a search engine. ** We can immediately retrieve the index that we have created of the top pages. ** Now we need to do the job of the search engine. Is this order a good one, are some of the pages at the top particularly useful? Can we see any pages from well known companies/charities which have ended up further down the index. Are there any videos which might be relevant? ** Re-sort the index based upon what, having read the pages, we feel is the best order. (This could involve actually going onto the pages themselves and deciding for ourselves)