This project aims to develop an efficient web crawler to browse the World Wide Web in an automated manner. The web crawler will be created by students Atul Singh and Mayur Garg under the guidance of their mentor Mrs. Deepika. A web crawler systematically visits websites to create copies of pages for search engines to index, starting with an initial list of URLs. This specific crawler will be developed to have a high performance using a computer with 640MB memory, 100Mbps internet connection, and running Windows XP/Vista with Java SDK 1.6 and a database client.

1. Y.M.C.A UniversitY of sCienCe And
teChnologY, fAridAbAd
ProJeCt sYnoPsis
_______________________________________________________
Web CrAWler
_______________________________________________________
Team Members: - ATUL SINGH,
MAYUR GARG
Roll Numbers: - IT-2327-2k7,
IT-2337-2k7
Mentor: - Mrs DEEPIKA
Email: - a2025singh@gmail.com
Mayurgarg2@gmail.com

2. Objective/ Aim:
This project aims at developing a highly efficient WEB CRAWLER that
browses the World Wide Web in a methodical, automated manner.
Introduction:
A Web crawler is a computer program that browses the World Wide Web
in a methodical, automated manner. Other terms for Web crawlers are ants,
automatic indexers, bots, and worms or Web spider, Web robot, or—
especially in the FOAF community—Web scutter.
The process is called Web crawling or spidering. Many sites, in particular
search engines, use spidering as a means of providing up-to-date data. Web
crawlers are mainly used to create a copy of all the visited pages for later
processing by a search engine that will index the downloaded pages to
provide fast searches. A Web crawler is one type of bot, or software agent.
In general, it starts with a list of URLs to visit, called the seeds. As the
crawler visits these URLs, it identifies all the hyperlinks in the page and
adds them to the list of URLs to visit, called the crawl frontier. URLs from
the frontier are recursively visited according to a set of policies.
Machine specification:
640MB Memory,
100Mbps Ether, Win XP/VISTA,
Jdk1.6, Database Client.