Roll your own web crawler. RubyDay 2013


Published on

It is all about data.
Having the right data at the right time might make the difference between you and your competitor. Google can show you just what it can catch. If you know where to find the data of your interest, let's go deeper and roll your own web crawler framework.
Taking the advantage of the latest cool technologies I will show you how to build your distributed web crawler based on Redis and Mongo

Published in: Technology, Design
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Roll your own web crawler. RubyDay 2013

  1. 1. Get the Data you want, because you want the Data now!Francesco LauritaRubyDay 2013, Milan - ItalyRoll you own WebCrawlerFriday, June 14, 13
  2. 2. What a web crawler is?“A Web crawler is an Internet bot that systematically browses the WorldWide Web, typically for the purpose of Web indexing.”, June 14, 13
  3. 3. How does it work?1.Starts with a list of urls to visit (seeds)2.Get all of the hyperlinks in the page andadds them to the list of urls to visit (push)1. The page content is stored somewhere2.The visited url is marked as visited3.Urls are recursively visitedDirected graphQueue (FIFO)Friday, June 14, 13
  4. 4. How does it work?Web Crawler is able to “walk” a“WebGraph”A WebGraph is a directed graph whosevertices are pages and a direct edgeconnects page A to page B if there is a linkbetween A and BDirected graphQueue (FIFO)Friday, June 14, 13
  5. 5. Generic Web Crawler InfrastructureWhile it’s fairly easy to build and write a standalone single-instance Crawler,building a distribute and scalable system that can download millions ofpages over weeks is notFriday, June 14, 13
  6. 6. Why should you roll your own Web Crawler?Universal Crawlers:* General purpose* Most interested contents (page rank)Focused Crawlers:* Better accuracy* Only certain topic* Highly selective* Not only for search enginesReady to be used for Machine Learning Engine as a serviceData warehouse and so onFriday, June 14, 13
  7. 7. Sentiment AnalysisFriday, June 14, 13
  8. 8. FinanceFriday, June 14, 13
  9. 9. A.I, Machine Learning, RecommendationEngine as A ServiceFriday, June 14, 13
  10. 10. Last but not least....Friday, June 14, 13
  11. 11. Polipus (because octopus was taken)Friday, June 14, 13
  12. 12. Polipus (because octopus was taken)A distributed easy-to-use DSL-ish web crawler framework writtenin ruby* Distributed and scalable* Easy to use inspired to Anemone* Well designed* Easy to use* Not distributed* Not Scalable, June 14, 13
  13. 13. Polipus in actionFriday, June 14, 13
  14. 14. Polipus: Under the hoodRedis(What is it?)* Is a NoSQL DB* Is an advanced Key/Value Store* Is a caching server* Is a lot of things...Friday, June 14, 13
  15. 15. Polipus: Under the hoodRedis(What is it?)* It is a way to share Memory over TCP/IPCan share memory (data structure) between different processes* List (LinkedList) --> queue.pop, queue.push* Hash --> {}* Set --> Set* SortedSet -->* ....Friday, June 14, 13
  16. 16. Polipus: Under the hoodRedis* Reliable and Distributed Queue1) A producer pushes an URL to visit into the QueueRPUSH2) A consumer fetches the URL and at the same time pushesit into a processing LISTRPOPLPUSH (Non blocking)/BRPOPLPUSH (blocking)An additional client may monitor the processing list foritems that remain there for too much time, and will pushthose timed out items into the queue again if needed.Friday, June 14, 13
  17. 17. Polipus: Under the hoodRedis* Reliable and Distributed Queue, June 14, 13
  18. 18. Polipus: Under the hoodRedis* URL TrackerA crawler should know if an URL has been already visited or itabout to be visited* SET(a =, a << url ; a.include?(url))* Bloom Filter (SETBIT / GETBIT)Friday, June 14, 13
  19. 19. Polipus: Under the hoodRedisBloom Filter:“A Bloom filter, is a space-efficient probabilistic data structure that is usedto test whether an element is a member of a set.”, June 14, 13
  20. 20. Polipus: Under the hoodRedisBloom Filter:* Very space efficient! 1.000.000 of elements ~2Mb on Redis* With a cost: False positive retrieval are possible, while negative are notWith a probability of 0.1% of false positive, every 1M of pages, 1k of themmight be marked erroneously as already visitedUsing SET : No errors at all but 1.000.000 of elements are ~150MBoccupied on Redis, June 14, 13
  21. 21. Polipus: Under the hoodMongoDB1) MongoDB is used mainly for storing pages2) Pages are stored using upsert command so that a document can be easilyupdated during a fresh crawling on the same contents3) By default the body of the page is compressed in order to save disk space4) No query() is needed because of bloom filterFriday, June 14, 13
  22. 22. Polipus: The infrastructureFriday, June 14, 13
  23. 23. Is it so easy?!Not really...1) Redis is an in-memory database2) A queue of URLs can grow very fast3) A queue of 1M of URLs is about 370MB occupied on Redis (about 400 charsfor each entry)4) MongoDB will eat your disk space: 50M of saved pages are around 400GBSuggested Redis conf:maxmemory 2.5GB (or whatever your instance can handle)maxmemory-policy noevictionAfter 6M Igot Redis torefuse writesFriday, June 14, 13
  24. 24. An experiment using the current available codeSetup:6x t1.micro (web crawlers, 5 workers each)1x m1.medium (Redis and MongoDB)MongoDB with default settingsRedismaxmemory 2.5GBmaxmemory-policy noeviction~4.700.000 of Pages downloaded in 24h...then I ran out of disk because of MongoDBFriday, June 14, 13
  25. 25. TODO•Redis memory Guard• Should be able to move items from the Redis queue to MongoDB if thequeue size hits a threshold and move items back on Redis at somepoint•Honor the robot.txt file• So that we can be respect Disallow directives if any•Add support for Ruby Mechanize• Maintain browsing sessions• Filling and submitting formsFriday, June 14, 13
  26. 26. Questions?, June 14, 13