Web crawl with Elixir
•
3 likes•719 views
This document summarizes the key steps taken to optimize a web crawler built with Elixir. It describes: 1) Moving from a producer-driven to a demand-driven architecture using GenStage to prevent memory exhaustion from the crawler producing items faster than the database could save them. 2) Adding rate limiting via TokenBucket to avoid exceeding site rate limits when crawling multiple pages concurrently. 3) Using a priority queue and supervision tree to introduce fault tolerance and allow crawling multiple sites in parallel while respecting each site's rate limits.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Web crawl with Elixir
Similar to Web crawl with Elixir (20)
Recently uploaded
Recently uploaded (20)
Web crawl with Elixir
- 2. Who am I? • Jechol Lee (mr.jechol@gmail.com) • Software engineer at Skelterlabs • Loves elixir, elm, ruby • We are HIRING!
- 3. First try
- 4. First try
- 5. First try Database Crawler JSONHTTP Crawler HTMLHTTP Products Products Save Queue Products
- 6. SQL Site A ES SaveQueue Task Page Task Item Task Item product req Sup Sup (site A) Sup (simple 1 for 1) Task.SupTask.Sup Sup (site B) progress start_link
- 7. Crawler speed > Site rate limit Crawler WebsiteHTTP 10/s
- 9. DB speed < Crawler speed Save QueueDatabase Crawler100/s 200/s
- 10. Memory exhaustion Save QueueDatabase Crawler100/s 200/s SaveQueue Out of Memory
- 11. Solution
- 12. OOM Problem : Producer driven 200/s Crawler Save QueueDatabase 100/s
- 15. Elixir GenStage (2016 / 7)
- 16. GenStage : Demand-driven Crawler Save Queue DEMAND 100 100Database 100/s
- 17. Memory usage Producer driven Demand driven
- 18. Site rate limit :TokenBucket
- 19. Site rate limit :TokenBucket Crawler WebsiteHTTP Burst Token Bucket 60/min
- 21. Network Requests Overflow Page 2 Page 1 item Page 1 item Page 1 item
- 22. Network Requests Overflow Page 1 item Page 2 item Page 3 Page 1 item Page 2 item Page 2 item
- 23. Network Requests Overflow Can't depend on random processing order. Page 1 item Page 2 item Page 4 Page 3 item Page 1 item Page 2 item Page 3 item Page 3 item
- 24. Priority Queue Page 1 item Page 2 item Page 3 Page 1 item Page 1 item Page 2 item Page 2 item
- 28. Rate limit byTokenBucket twotap.com c21stores.com WebProxy PriorityQueue + TokenBucket HTTP SPAWN HTML Parser C21 HTML Parser C21 HTML Crawler C21 JSON Crawler Lego
- 30. Fault-tolerance by SupervisionTree Supervisor Supervisor C21 Task Supervisor ErrorMonitor WebProxy PriorityQueue + TokenBucket HTML Parser C21 HTML Parser C21 SaveQueue HTML Crawler C21 JSON Crawler Lego
- 31. Tree for Multiple Crawlers JSON Crawler GNC Supervisor JCPenney Supervisor Supervisor C21 Task Supervisor ErrorMonitor WebProxy PriorityQueue + TokenBucket HTML Parser C21 HTML Parser C21 SaveQueue HTML Crawler C21 JSON Crawler Lego
- 32. Final JSON Crawler GNC Supervisor JCPenney Supervisor Supervisor C21 Task Supervisor ErrorMonitor sentry.io MONITOR {:DOWN, :page_not_found} twotap.com c21stores.com WebProxy PriorityQueue + TokenBucket HTTP SPAWN HTML Parser C21 HTML Parser C21 PRODUCT DEMAND DEMAND PRODUCT SQL ES SaveQueue HTML Crawler C21 JSON Crawler Lego
- 33. Building Blocks
- 35. GenServer vs Task • Tasks don't provide services. → No handle_call, etc. • Just run a function and exit.
- 36. Task.Supervisor.async • Not trap exit. • Caller process dies together. • Not restart task.
- 37. Task.async vs Task.Supervisor.async Only later builds supervision relationship so that visible using observer.
- 38. End