Role of Text Mining in Search Engines IST 345 Term Project Team 3 Fall 2008 Role of Text Mining in Search Engine

Agenda Role of Text Mining in Search Engine Text Mining in Search Engine Case Study 1 Introduction 2 Current Trends 3 4 Future Trend 5 6 Conclusions

Role of Text Mining in Search Engine

Role of Text Mining in Search Engine

Role of Text Mining in Search Engine

Role of Text Mining in Search Engine

Role of Text Mining in Search Engine

Role of Text Mining in Search Engine

Role of Text Mining in Search Engine

Role of Text Mining in Search Engine

Really RESULTS How do Search Engine Work? Role of Text Mining in Search Engine

1. New Website is posted, Linked to, or has its content altered Role of Text Mining in Search Engine

2. Search Engine’s Spider Crawl the page Role of Text Mining in Search Engine Goglebot: Google Slurp: Yahoo MSNbot: MSN

Skims text, image descriptions, meta data, page titles and URL Role of Text Mining in Search Engine

Follow links, count links In and Out Role of Text Mining in Search Engine

Search Wiki Engine SEO SEM Web Role of Text Mining in Search Engine Index key terms, count word frequency

Why Mining in Search Engine? Overwhelming information in typical user query results Results are only partly related to each other Many users investigate only the two or three top ranked documents Traditional lists of ranked documents do not seem to be sufficient for the exploratory search tasks Role of Text Mining in Search Engine

Overwhelming information in typical user query results

Results are only partly related to each other

Many users investigate only the two or three top ranked documents

Traditional lists of ranked documents do not seem to be sufficient for the exploratory search tasks

Additional techniques are needed to help the users analyze the search results efficiently and drill down to the information they are looking for. Users need to explore the information Discovering new patterns, New entities, and Knowledge they do not even realize they needed Role of Text Mining in Search Engine Why Mining in Search Engine?

Additional techniques are needed to help the users analyze the search results efficiently and drill down to the information they are looking for.

Users need to explore the information

Discovering new patterns,

New entities, and

Knowledge they do not even realize they needed

Text Mining Text mining applications today draw on a wide range of techniques and serve many purposes in information management and business intelligence. TM techniques can be organized into four categories: Classification techniques Association analysis Information extraction techniques Clustering techniques Role of Text Mining in Search Engine

Text mining applications today draw on a wide range of techniques and serve many purposes in information management and business intelligence.

TM techniques can be organized into four categories:

Classification techniques

Association analysis

Information extraction techniques

Clustering techniques

Use of Text Mining in Search Engine Text categorization (faceted search systems) Using multi-dimensional categories to describe (groups of) documents Richer descriptions Expensive to develop the categories Semantic Web Search Linguistic analysis of text Addition to purely statistical techniques Role of Text Mining in Search Engine

Text categorization (faceted search systems)

Using multi-dimensional categories to describe (groups of) documents

Richer descriptions

Expensive to develop the categories

Semantic Web Search

Linguistic analysis of text

Addition to purely statistical techniques

Role of Text Mining in Search Engine Contextualized clustering Group the search results by topic Clustering of documents according to terms found in the documents Use of Text Mining in Search Engine

Contextualized clustering

Group the search results by topic

Clustering of documents according to terms found in the documents

Clustering Engines Scatter/Gather Grouper The Lingo system The Clusty/Vivisimo engine (www.clusty.com and www.vivisimo.com) SnakeT HOBSearch Role of Text Mining in Search Engine

Scatter/Gather

Grouper

The Lingo system

The Clusty/Vivisimo engine (www.clusty.com and www.vivisimo.com)

SnakeT

HOBSearch

Clusty / Vivisimo Engine Role of Text Mining in Search Engine

Role of Text Mining in Search Engine Future Trends Anticipated to expand 1000 times With the introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos, and Northern Light, the Web can be viewed as a searchable 15-billion-word encyclopedia. Nutch engine is the future search engine Fetch several billion pages per month Maintain an index of these pages Search the index up to 1000 times per second Provide very high quality search results Operate at minimal cost

Anticipated to expand 1000 times

With the introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos, and Northern Light, the Web can be viewed as a searchable 15-billion-word encyclopedia.

Nutch engine is the future search engine

Fetch several billion pages per month

Maintain an index of these pages

Search the index up to 1000 times per second

Provide very high quality search results

Operate at minimal cost

Future Trend Introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos, and Northern Light Summarization of online documents More efficient Categorization/Clustering for the search results Entity Extraction by using linguistics and pattern detection Answering intelligent questions Role of Text Mining in Search Engine

Introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos, and Northern Light

Summarization of online documents

More efficient Categorization/Clustering for the search results

Entity Extraction by using linguistics and pattern detection

Answering intelligent questions

Role of Text Mining in Search Engine Case Study: Data Crow

Role of Text Mining in Search Engine Case Study: Data Crow

Role of Text Mining in Search Engine Customized Search Music Data Crow contains two separate modules The Music Album and Audio CD module Use one of the online services (MusicBrainz, Amazon, Discogs and others) to find information on your CD and or music files Parse information from your mp3, flac, ape and or ogg file and fill missing information using an online service

Music

Data Crow contains two separate modules

The Music Album and

Audio CD module

Use one of the online services (MusicBrainz, Amazon, Discogs and others) to find information on your CD and or music files

Parse information from your mp3, flac, ape and or ogg file and fill missing information using an online service

Role of Text Mining in Search Engine Conclusion Learnt the evolution of Search Engine Efficiency of Search Engine can be increased by using mining techniques Increased demand from customized search

Learnt the evolution of Search Engine

Efficiency of Search Engine can be increased by using mining techniques

Increased demand from customized search

Role of Text Mining in Search Engine Which of the following is not a search engine? Google Open Directory Yahoo search Lycos Open Directory

Which of the following is not a search engine?

Google

Open Directory

Yahoo search

Lycos

Open Directory

Role of Text Mining in Search Engine What search engine has the largest index of listings on the web? Yahoo Google MSN Microsoft Google

What search engine has the largest index of listings on the web?

Yahoo

Google

MSN

Microsoft

Google

Role of Text Mining in Search Engine Search Engines and directory are both the same thing because: They both index information They’re both on the internet They’re not the same thing They both search for information They’re not the same thing

Search Engines and directory are both the same thing because:

They both index information

They’re both on the internet

They’re not the same thing

They both search for information

They’re not the same thing

Role of Text Mining in Search Engine What was the first search engine ever created? WWW Wanderer Google; created by Larry and Sergey in 1995 Yahoo; created in 1994 MSN Search; created in 1982 WWW Wanderer: technically not really a search engine, but a pioneer of the crawling process

What was the first search engine ever created?

WWW Wanderer

Google; created by Larry and Sergey in 1995

Yahoo; created in 1994

MSN Search; created in 1982

WWW Wanderer: technically not really a search engine, but a pioneer of the crawling process

Q: Google is limited to how many search terms in one query: 16 18 13 15 15 Role of Text Mining in Search Engine

Q: Google is limited to how many search terms in one query:

16

18

13

15

15

Q: How do search engines find out about sites on the Web? Osmosis Search engines automatically know everything on the Web. Two different ways: search engine spiders index the information, or site owners submit it manually Search engines have special features that enable them to know when your site is uploaded. It's called "crystal ball technology." Two different ways: search engine spiders index the information, or site owners submit it manually. Role of Text Mining in Search Engine

Q: How do search engines find out about sites on the Web?

Osmosis

Search engines automatically know everything on the Web.

Two different ways: search engine spiders index the information, or site owners submit it manually

Search engines have special features that enable them to know when your site is uploaded. It's called "crystal ball technology."

Two different ways: search engine spiders index the information, or site owners submit it manually.

Role of Text Mining in Search Engine