Download as PDF, PPTX
![Twisted event loop
https://stackoverflow.com/questions/35111265/how-does-pythons-twisted-reactor-work
https://www.cs.cmu.edu/~adamchik/15-121/lectures/Binary%20Heaps/heaps.html
events:
[e1: Event, e2: Event, … eN]
Event:
func, args, desired_time](https://image.slidesharecdn.com/205-scrapyinternals-170724105004/85/Scrapy-internals-Scrapinghub-33-320.jpg)
![Twisted event loop
https://stackoverflow.com/questions/35111265/how-does-pythons-twisted-reactor-work
https://www.cs.cmu.edu/~adamchik/15-121/lectures/Binary%20Heaps/heaps.html
events:
[e1: Event, e2: Event, … eN]
Event:
func, args, desired_time min: O(1)](https://image.slidesharecdn.com/205-scrapyinternals-170724105004/85/Scrapy-internals-Scrapinghub-34-320.jpg)
Uploaded byit-people
«Scrapy internals» Александр Сибиряков, Scrapinghub
- Scrapy is a framework for web scraping that allows for extraction of structured data from HTML/XML through selectors like CSS and XPath. It provides features like an interactive shell, feed exports, encoding support, and more. - Scrapy is built on top of the Twisted asynchronous networking framework, which provides an event loop and deferreds. It handles protocols and transports like TCP, HTTP, and more across platforms. - Scrapy architecture includes components like the downloader, scraper, and item pipelines that communicate internally. Flow control is needed between these to limit memory usage and scheduling through techniques like concurrent item limits, memory limits, and delays between calls.
More Related Content
![[245] presto 내부구조 파헤치기](https://cdn.slidesharecdn.com/ss_thumbnails/245presto-150915054242-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to «Scrapy internals» Александр Сибиряков, Scrapinghub
«Scrapy internals» Александр Сибиряков, Scrapinghub
- 1. Scrapy internals Alexander Sibiryakov,16-17 July 2017, PyConRU 2017 sibiryakov@scrapinghub.com made by
- 2.
- 3. Talk scope • Designof complex asynchronous application,
- 4. Talk scope • Designof complex asynchronous application, • Flow-control issues,
- 5. Talk scope • Designof complex asynchronous application, • Flow-control issues, • open source life.
- 6.
- 7. Scrapy: web scraping •extraction of structured data,
- 8. Scrapy: web scraping •extraction of structured data, • Selecting and extracting data from HTML/XML (CSS, Xpath, regexps) → Parsel
- 9. Scrapy: web scraping •extraction of structured data, • Selecting and extracting data from HTML/XML (CSS, Xpath, regexps) → Parsel • Interactive shell,
- 10. Scrapy: web scraping •extraction of structured data, • Selecting and extracting data from HTML/XML (CSS, Xpath, regexps) → Parsel • Interactive shell, • Feed exports in JSON, CSV, XML and storing in FTP, S3, local fs,
- 11. Scrapy: web scraping •extraction of structured data, • Selecting and extracting data from HTML/XML (CSS, Xpath, regexps) → Parsel • Interactive shell, • Feed exports in JSON, CSV, XML and storing in FTP, S3, local fs, • Robust encoding support and auto-detection,
- 12.
- 13. Main features • Extensible:spider, signals, middlewares, extensions, and pipelines,
- 14. Main features • Extensible:spider, signals, middlewares, extensions, and pipelines, Telnet console
- 15. Main features • Extensible:spider, signals, middlewares, extensions, and pipelines, Form submission Telnet console
- 16. Main features • Extensible:spider, signals, middlewares, extensions, and pipelines, COOKIES Form submission Telnet console
- 17. Main features • Extensible:spider, signals, middlewares, extensions, and pipelines, COOKIES Form submission Telnet console Graceful shutdown by signal
- 18. Main features • Extensible:spider, signals, middlewares, extensions, and pipelines, COOKIES Robots.txt Form submission Telnet console Graceful shutdown by signal
- 19.
- 20.
- 21. Twisted • Event-driven networkprogramming framework
- 22. Twisted • Event-driven networkprogramming framework • Event loop and Deferreds («Обещания»)
- 23. Twisted • Event-driven networkprogramming framework • Event loop and Deferreds («Обещания») • Protocols and transport:
- 24. Twisted • Event-driven networkprogramming framework • Event loop and Deferreds («Обещания») • Protocols and transport: • TCP, UDP, SSL, UNIX sockets
- 25. Twisted • Event-driven networkprogramming framework • Event loop and Deferreds («Обещания») • Protocols and transport: • TCP, UDP, SSL, UNIX sockets • HTTP, DNS, SMTP/IMAP, IRC
- 26. Twisted • Event-driven networkprogramming framework • Event loop and Deferreds («Обещания») • Protocols and transport: • TCP, UDP, SSL, UNIX sockets • HTTP, DNS, SMTP/IMAP, IRC • Cross platform
- 27. Twisted • Event-driven networkprogramming framework • Event loop and Deferreds («Обещания») • Protocols and transport: • TCP, UDP, SSL, UNIX sockets • HTTP, DNS, SMTP/IMAP, IRC • Cross platform
- 28.
- 29. Glyph Lefkowitz Creator ofTwisted
- 30. –Twisted source code self._nameResolver= _SimpleResolverComplexifier(resolver)
- 31.
- 32.
- 33. Twisted event loop https://stackoverflow.com/questions/35111265/how-does-pythons-twisted-reactor-work https://www.cs.cmu.edu/~adamchik/15-121/lectures/Binary%20Heaps/heaps.html events: [e1:Event, e2: Event, … eN] Event: func, args, desired_time
- 34. Twisted event loop https://stackoverflow.com/questions/35111265/how-does-pythons-twisted-reactor-work https://www.cs.cmu.edu/~adamchik/15-121/lectures/Binary%20Heaps/heaps.html events: [e1:Event, e2: Event, … eN] Event: func, args, desired_time min: O(1)
- 35.
- 36. x86 time sources •Real Time Clock - absolute time, 1 sec. precision,
- 37. x86 time sources •Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously,
- 38. x86 time sources •Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz
- 39. x86 time sources •Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode,
- 40. x86 time sources •Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode, • many for one-shot mode,
- 41. x86 time sources •Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode, • many for one-shot mode, • compares actual timer value and target
- 42. x86 time sources •Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode, • many for one-shot mode, • compares actual timer value and target • RDTSC/RDTSCP - CPU clock cycles
- 43. x86 time sources •Real Time Clock - absolute time, 1 sec. precision, • 8254 chip previously, • HPET (High Precision Event Timer), at least 10Mhz • single counter for periodic mode, • many for one-shot mode, • compares actual timer value and target • RDTSC/RDTSCP - CPU clock cycles • Proprietary timers
- 44.
- 45.
- 46.
- 47. Twisted.Deferred • callback • errback •addCallback, addErrback
- 48. Twisted.Deferred • callback • errback •addCallback, addErrback • cancel
- 49. Twisted.Deferred • callback • errback •addCallback, addErrback • cancel • addTimeout
- 50. Twisted.Deferred • callback • errback •addCallback, addErrback • cancel • addTimeout • pause/unpause
- 51.
- 52.
- 53.
- 54.
- 55.
- 56. Throttling between internal components •Downloader,
- 57. Throttling between internal components •Downloader, • Scraper
- 58. Throttling between internal components •Downloader, • Scraper • Item pipelines (cleansing, validating, dups, storing,..)
- 59. Throttling between internal components •Downloader, • Scraper • Item pipelines (cleansing, validating, dups, storing,..) • Feed exports (serialization + disk/network IO)
- 60. Throttling between internal components •Downloader, • Scraper • Item pipelines (cleansing, validating, dups, storing,..) • Feed exports (serialization + disk/network IO) • ?
- 61.
- 62.
- 63. Flow control: memory •Unlimited downloading -> unlimited items growth from cascading feed pages.
- 64. Flow control: memory •Unlimited downloading -> unlimited items growth from cascading feed pages. • maintain limit per amount of memory used for Responses in queue (~5Mb)
- 65. Flow control: CPU spendingmore time on than > reactor.callLater( 0.1 , d.errback, _failure) an artificial delay in 100ms Callbacks-> CPU io
- 66.
- 67.
- 68. Summarizing • concurrent itemslimits, • memory consumption limits,
- 69. Summarizing • concurrent itemslimits, • memory consumption limits, • scheduling of new calls with delays.
- 70. Summarizing • concurrent itemslimits, • memory consumption limits, • scheduling of new calls with delays. if limit is reached ->
- 71. Summarizing • concurrent itemslimits, • memory consumption limits, • scheduling of new calls with delays. if limit is reached -> don’t pickup new request from scheduler
- 72.
- 73.
- 74. It just stopped… •Why? • Some Deferred was lost?
- 75. It just stopped… •Why? • Some Deferred was lost? • Where?
- 76. It just stopped… •Why? • Some Deferred was lost? • Where? • How to debug?
- 77. It just stopped… •Why? • Some Deferred was lost? • Where? • How to debug? No silver bullet.
- 78. It just stopped… •Why? • Some Deferred was lost? • Where? • How to debug? No silver bullet. > self.heartbeat = task.LoopingCall(nextcall.schedule)
- 79. It just stopped… •Why? • Some Deferred was lost? • Where? • How to debug? No silver bullet. > self.heartbeat = task.LoopingCall(nextcall.schedule) + extensive logging
- 81.
- 82. Design your async applicationwell Iterations
- 83. Design your async applicationwell Iterations State diagrams
- 84. Вопросы Alexander Sibiryakov, ScrapinghubLtd., sibiryakov@scrapinghub.com