The document describes a generic crawler that can crawl websites without APIs by using rules to extract data. It discusses the crawler's infrastructure, introduction to crawler rules using XPATH and CSS expressions, the crawl procedure of generating links, crawling based on links and saving data to a local DB, and limitations such as not working on AJAX sites. The goal is to build a multipurpose crawler powered by cloud computing that can extract information from various websites.

1. Generic Crawler
CRAWL THE WORLD!

2. Content
• Introduction to Generic Crawler
• The Infrastructure
• Introduction to Crawler Rule
• Crawl Procedure
• Data Fact Sheet
• Limitation
• Future Work

3. Introduction to Generic Crawler
• Information are not only in Social Media
• Some websites do not provide API
Proposed Solution
• Multi purpose Crawler
• Rule based crawler
• The power of cloud

4. The Infrastructure

5. Introduction to Crawler Rule
• XPATH or CSS Expression
• Tree Data Structure
• Deep First Search Algorithm
XPATH
• Search HTML Tag
• String and Array basic function
• Text extraction (Remove HTML tag)

6. Introduction to Crawler Rule
XPATH: //div[@class=‘detail_content]

7. Crawler Procedure
Link
Generation
• Schedule Auto Runner task
• Schedule Auto Pusher task
Crawl
• Crawl based of the links
• Save the crawled data to local DB
On-Demand
Central DB
Pusher
• Keyword Matching
• Push to Central DB

8. Data Fact Sheet
Average Crawling
Time
15s
* Based on 1,000 links
New Links Generation
Time
3/min
* From 5 sources

9. Limitation
• AJAX Website
• Depends on Rule
• High CPU and Bandwidth demand
• Robot.txt
Links
Viva.co.id 724
Detik.com 418
Beritajatim.com 120
Hukumonline.com 13
* Last update: 27 January 2016 – 16:00

10. Future Work
• Input URL to scrap
• Scheduler for Auto Crawl
• Crawler Health Monitoring System

11. THANK YOU
GENERIC CRAWLER