The document proposes a multi-agent based web mining model to improve search engine efficiency. It divides the mining task among several parallel agents that coordinate to improve mining efficiency greatly. The model crawls websites using multiple autonomous web crawler agents in parallel to index pages faster than a single crawler. Experimental results showed the average seed website mining time dropped by 20% and search speeds increased compared to a non-distributed approach.

1. Multi-Agent Based Web
Mining Model

2.  “A multi-agent based Web mining model is designed
for the improvement of the efficiency of keywords
based search engine.”
 “The model divides mining task into several parallel
agents which coordinately work together, and the
mining efficiency is improved greatly.”

3. What is Search Engine
“A program that searches documents for
specified keywords and returns a list of the
documents where the keywords were found”.
“A search engine is a web site that collects and
organizes content from all over the internet.Those
wishing to locate something would enter a query about
what they'd like to find and the engine provides links to
content that matches what they want.”

5. How web search engines work
A search engine operates in the following order
Web crawling
Indexing
Searching

6.  Search engines index web pages by web
spiders(also called web crawler or web
robot).
 A Web crawler is a computer program
that browses the World Wide Web in a
methodical, automated manner or in an
orderly fashion.
 A crawler is a program that visits Web
sites and reads their pages and other
information in order to create entries for a
search engine index.

8. Yahoo! Slurp
Bingbot
Googlebot
Scooter
ArchitextSpider

9. Why Need Web Spider as Multi Agent Mode
 Web information expands rapidly.
 Web search engines are facing severe situation.
 Web pages increased by over 7 million everyday.
 Web information retrieval has defects such as low coverage, low
quality inquires.
 The lack of effective information classification.
 Bad understanding of user behavior based on keywords.
 70% of the information people demand comes from search
engines but only 60% of Web information can be indexed by six
big search engines such as Yahoo, Google etc.

10. Multi-agent technology
Research of agent and the multi-agent systems (MAS)
has become a hot spot in Distributed Artificial Intelligence
(DAI). Agent technology provides a new calculation model
and problem solving method, as well as a good remote
intelligent program design method. The multi-agent
system relaxes the limit of centralizing, planning and
sequence control, and provides a decentralized control,
emergency and parallel processing. It can also reduce the
cost of hardware and software. So MAS provides more
rapid problem solving speed.

11. Autonomy
Interactivity
Responsiveness
Initiative
Flexibility
 Scalability

12. There are three kinds of methods to acquire web
information
 Pure link analysis method(such as PageRank )
Text analysis method
 Combination of link analysis and text analysis
method

13. Intelligent grabber model expands the user’s concept space
and determines the user’s interest degree based on the user’s
search demand and thus gets Web resources more quickly
and accurately. It doesn’t do the whole text analysis of the
result pages but only guarantees that gathered pages are
related with users’ themes.

16.  Parallel execution between multiple agents has
improved system efficiency.
The average time mining seeds website has been
dropped by 20%.
 It improves the quality of page extraction.
 It Improve Searching speed.
 It Reduce web page search time.
 Classified Indexing.

17. Using two different algorithms (old algorithm and
newly multi-agent designed algorithm, ie Grabber)
to crawl the page, since new algorithm is multi-
agent designed, system performance has been
improved greatly than before.
Mining algorithm of search engines needs further
improvement, especially the improvement of mining
intelligent degree.

18. Pierre Baldi, Paolo Frasconi, Padhraic Smyth. Modeling the Internet
and the Web[M]. Chichester: John Wiley & Sons Ltd, 2003.
 Wang Chuan, Chang Gui-ran,Ma Yan, et al. Research on search engines
based on the user behavior rmodel [J].Journal of Information.
Shi Zhongzhi. Intelligent agent and its application [M]. Beijing: science
press, 2000.
 Brin S, Page L. The anatomy of a large-scale hypertextual Web search
engine[A]. In: Thistlewaite P, et al., eds. Proceedings of the 7th ACM-
WWW International Conference[C]. Brisbane: ACM Press, 1998, pp. 107-
117.
 Kleinberg J. Authoritative sources in a hyperlinked environment [A].
Tarjan RE, et al., eds. Proceedings of the 9th ACM-SIAM Symposium on
Discrete Algorithms[C]. New Orleans: ACM Press,1997, pp. 668-677.
http://en.wikipedia.org/wiki/Web_crawler
http://searchsoa.techtarget.com/definition/crawler
http://en.wikipedia.org/wiki/Agent-based_model
http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=5989730