Downloaded 10 times
Uploaded by이재철
Web crawl with Elixir
This document summarizes the key steps taken to optimize a web crawler built with Elixir. It describes: 1) Moving from a producer-driven to a demand-driven architecture using GenStage to prevent memory exhaustion from the crawler producing items faster than the database could save them. 2) Adding rate limiting via TokenBucket to avoid exceeding site rate limits when crawling multiple pages concurrently. 3) Using a priority queue and supervision tree to introduce fault tolerance and allow crawling multiple sites in parallel while respecting each site's rate limits.
More Related Content
![[AWSKRUG&JAWS-UG Meetup #1] 70% Cost Reduction with On-demand resizing](https://cdn.slidesharecdn.com/ss_thumbnails/awsjawsondemandresizing-160526114219-thumbnail.jpg?width=640&height=640&fit=bounds)
Web crawl with Elixir
- 1.
- 2. Who am I? •Jechol Lee (mr.jechol@gmail.com) • Software engineer at Skelterlabs • Loves elixir, elm, ruby • We are HIRING!
- 3.
- 4.
- 5. First try Database Crawler JSONHTTP CrawlerHTMLHTTP Products Products Save Queue Products
- 6. SQL Site A ES SaveQueue Task Page Task Item Task Item product req Sup Sup (site A) Sup (simple 1for 1) Task.SupTask.Sup Sup (site B) progress start_link
- 7. Crawler speed >Site rate limit Crawler WebsiteHTTP 10/s
- 8.
- 9. DB speed <Crawler speed Save QueueDatabase Crawler100/s 200/s
- 10. Memory exhaustion Save QueueDatabase Crawler100/s200/s SaveQueue Out of Memory
- 11.
- 12. OOM Problem :Producer driven 200/s Crawler Save QueueDatabase 100/s
- 13.
- 14.
- 15.
- 16. GenStage : Demand-driven Crawler Save Queue DEMAND100 100Database 100/s
- 17.
- 18. Site rate limit:TokenBucket
- 19. Site rate limit:TokenBucket Crawler WebsiteHTTP Burst Token Bucket 60/min
- 20.
- 21. Network Requests Overflow Page2 Page 1 item Page 1 item Page 1 item
- 22. Network Requests Overflow Page1 item Page 2 item Page 3 Page 1 item Page 2 item Page 2 item
- 23. Network Requests Overflow Can'tdepend on random processing order. Page 1 item Page 2 item Page 4 Page 3 item Page 1 item Page 2 item Page 3 item Page 3 item
- 24. Priority Queue Page 1item Page 2 item Page 3 Page 1 item Page 1 item Page 2 item Page 2 item
- 25.
- 26.
- 27.
- 28. Rate limit byTokenBucket twotap.com c21stores.com WebProxy PriorityQueue + TokenBucket HTTP SPAWN HTMLParser C21 HTML Parser C21 HTML Crawler C21 JSON Crawler Lego
- 29.
- 30. Fault-tolerance by SupervisionTree Supervisor Supervisor C21 Task Supervisor ErrorMonitor WebProxy PriorityQueue + TokenBucket HTMLParser C21 HTML Parser C21 SaveQueue HTML Crawler C21 JSON Crawler Lego
- 31. Tree for MultipleCrawlers JSON Crawler GNC Supervisor JCPenney Supervisor Supervisor C21 Task Supervisor ErrorMonitor WebProxy PriorityQueue + TokenBucket HTML Parser C21 HTML Parser C21 SaveQueue HTML Crawler C21 JSON Crawler Lego
- 32. Final JSON Crawler GNC Supervisor JCPenney Supervisor Supervisor C21 Task Supervisor ErrorMonitor sentry.io MONITOR {:DOWN, :page_not_found} twotap.com c21stores.com WebProxy PriorityQueue + TokenBucket HTTP SPAWN HTML Parser C21 HTMLParser C21 PRODUCT DEMAND DEMAND PRODUCT SQL ES SaveQueue HTML Crawler C21 JSON Crawler Lego
- 33.
- 34.
- 35. GenServer vs Task •Tasks don't provide services. → No handle_call, etc. • Just run a function and exit.
- 36. Task.Supervisor.async • Not trapexit. • Caller process dies together. • Not restart task.
- 37. Task.async vs Task.Supervisor.async Only later buildssupervision relationship so that visible using observer.
- 38.