Search engines are designed to help users find information stored digitally. They aim to minimize the time and amount of information needed to find what users are looking for. Major methods of information retrieval for search engines include Boolean, vector space model, probabilistic, and meta search. Designing the perfect search engine requires dealing with challenges like the web's huge and constantly changing document set that is loosely organized through hyperlinks. Effective search requires components like crawlers to discover pages, repositories to store them, indexes for efficient searching, and ranking algorithms to order results.

1. Understanding Search Engines

2. Basic Defintions: Search Engine Search engines are information retrieval (IR) systems designed to help find specific information stored in digital server and database systems. Search engines are meant to minimize both the time required to find information, and the amount of information which must be searched.

3. Our focus is on Web Information Retrieval, not traditional IR · Web IR means “search within the world’s largest and linked document collection.” · This document collection is growing at a rate that is almost impossible to know. · Links arise and disappear at an unknown rate.

4. Methods of IR and Search · Boolean Search · Vector Space Model Search · Probabilistic Model Search · Meta Search

5. Boolean Search · One of the earliest and simplest computerized IR methods. · Applies Boolean algebraic operations (AND, OR, NOT) to user keywords. · AND = x and y satisfied (both conditions, I) · OR= x or y condition (either condition, U) · NOT= only x, not y (specific subset, S)

6. Boolean Search 2 +’s · Simple. Fast. Manageable. —’ s · Simplistic; car+maintenance≠ auto care (polysemy and synonymy) Assumes user has strong familiarity with the topic domain. · Limited; best used for specific topics with small vocabulary.

7. Vector Space Model Search · Developed in the early 1960s by Gerard Salton. · Transforms text into numeric vectors and matrices, then uses matrix analysis techniques to discern features and semantic relationships.(!)

8. Vector Space Model 2 +’s Incredibly powerful tool for keeping track of evolving meanings and shifting vocabularies. Automatically includes relevance scores thereby returning ranked search results.(!) —’ s Computationally intense; requires massive computing power and cannot scale up to deal with massive (web-sized) document sets.

9. Probabilistic Model Search Uses a probability model to guess which documents a user will find relevant. They key to this model’s effectiveness is the set of initial conditions. One of the most powerful initial conditions is an index of a user’s search history/search tendency. Another initial condition is the search term. Some powerful search algorithms begin by broadening the search terms to include conceptually related documents. Most appropriate for enterprises where complete understanding of an evolving topic domain or wordspace is mission critical. Grapeshot

10. Probabilistic Model 2 +’s Very powerful tool. Uses evolving meanings and shifting vocabularies to expand the search vectors. Cutting edge. This is the area of greatest research interest, and greatest value generation. In other words, this is where the money is. —’ s When there is no history, you have to start with assumptions; that can be devastating to relevance. Very hard to build, therefore, very expensive. Like, unbelievably expensive. Megabucks.

11. Meta Search If one search engine is good (but has drawbacks) why not combine them?! That’s a MetaSearch engine. Queries are sent to multiple engines, or multiple processors. As you would expect, this can be very accurate, but very slow. When they’re wrong, they’re monumentally wrong.

12. To make the perfect Web Search Engine, you must deal with the web’s externalities: You will have to search through the largest document set in the known universe. That document set is changing The set is self-organizing; or more accurately, the set is completely disorganized. It is hyperlinked

13. The perfect web search engine: A Huge Document Set The web is, in fact, too big to accurately measure. JAN 2004: 10,000,000,000+ pages FEB 2007: 25,000,000,000+ pages Surface web counts, not Deep Web.

14. The perfect web search engine: A Changing Document Set Cho and Molina, 2000. The evolution of the Web and implications for an incremental crawler . Proceedings of the 26 th International Conference on Very Large Databases 40% of pages in sample changed w/in 7 days 23% changed w/in 24 hours * Growth rate is unknown, but significant

15. The perfect web search engine: A Self-Organizing Set There are no standards for content, minimal control over structure, no rules for formats. The data are volatile subject to error, dishonesty, link-rot, and file disappearance. Data exist in multiple formats; in duplicate; or they don’t exist until a specific request. Data are re-created for many different uses and conditions (shopping, research, entertainment, way-finding).

16. The perfect web search engine: A Hyperlinked Set Thank God. The availability of hyperlinks creates an additional layer of meaning. This also places the web document set into a relational framework that can be very accurately described using a branch of mathematics called topology. Hyperlinks (the only new form of punctuation created in the last 500 years) allow us to do ranked searches.

17. Designing a precise* search mechanism. Crawler Module Page Repository Indexing Module Indexes Query Module Ranking Module

18. The Pieces (Google style)

19. The Crawler Module (CM) A distributed system of software robots (bots, spiders) designed to examine and record the content and structure of pages within a site within a defined domain. CM gives bots root URLs Spiders consume resources! (bandwidth, quotas) Should conform to ethical crawling (robots.txt)

20. The Page Repository Temporary storage for full page contents and link structure. Valuable and popular pages can be stored for longer term.

21. Indexing Module · A software processor that applies a compression algorithm. · For content, the algorithm generates an inverted file index. · Also yields Structure Indexes, and Special-purpose Indexes (for PDFs and video) Software, 2 Processor, 3 Compression, 7 Algorithm, 8, 12 Index, 17 Indexes, 21, 25

22. Indexes Storage area for inverted files and other processed page results. These are the valuable assets of an Internet Search company.

23. The Query Module The software that handles user queries. Interacts with the ranking module, the indexes, and the page repository. Must be fast! Feb 2003, Google reported serving 250,000,000 searches per day. (2,894 queries per second) Langville & Meyer, 2006

24. The Ranking Module The software that examines the hyperlink structure and calculates a page’s value.

25. 3 guys and 2 theses Sergey Brin, Larry Page and Jon Kleinberg HITS and PageRank™ More on this next class.

26. An Excellent History (the key reference text) Amy Langville, Carl Meyer, Google’s Page Rank and Beyond: The Science of Search Engine Rankings. Princeton University Pres, 2006

27. Questions & Discussion Ask questions now.