This document discusses the evolution of spider frameworks from a simple spider to the Scrapy framework. It describes how Scrapy abstracts workflows, tasks, and platforms. It then discusses how spider architectures evolved from single process to distributed processes and machines. Finally, it outlines potential modules like renderers, proxies, extractors, schedulers, and processors that could continue evolving spider frameworks.

1. The Evolution Theory of Spider
逐浪@淘宝北京研发中心

2. Topic
• Simplest Spider
• Framework(Scrapy)
– Abstraction
– IO Model
• Evolution
– Architecture
– Module
• Simplify
• Do it

3. Simplest Spider
import urllib, lxml, MySQLdb
urls = [...]
for url in urls:
html = urllib.urlopen(url).read()
item = parse(html)
save(item)

4. Framework(scrapy)---Abstraction
• Workflow abstraction
– Work abstraction
– Flow abstraction
• Task abstraction
– Request/Response
– Task
• Platform abstraction
– Linux
– Windows

5. Framework(scrapy)---Abstraction
• Workflow abstraction
– Work abstraction
• Schedule
• Download
• Extract
• Pipeline
– Immutable and variable
• Scrapy perspective
• My perspective
– What is spider class?
• Variable works abstraction

6. Framework(scrapy)---Abstraction)
• Workflow abstraction
– Flow abstraction
• Apache vs Scrapy
– Why control center
• Control ability
– Error Retry
• Extensibility
• Module independency

7. Framework(scrapy)---Abstraction)
• Task abstraction
– Request/Response
– Task
• Platform abstraction
– Linux
– Windows

8. Framework(scrapy)---IO Model)
• Concepts
– Synchronous/Asynchronous(IO state consistency)
– Block/Nonblock(Process/Thread status)
• IO Model
– Synchronous Block(urllilb)
– Asynchronous Block(spynner, gevent, nginx_lua)
– Asynchronous
NonBlock(twisted, reactor, proactor)
– Synchronous NonBlock(mistery)

9. Evolution---Architecture
• Why
– Scrapy
• Single Process
– Etao Spider v1
Etao Spider v1

10. Evolution---Architecture
• Distributed on Processes
• Distributed on Machines
• How
– Thrift/HSF
– Interact
• Direction
– Dependent
• Task queue
– Stateless

11. Evolution---Module
• Downloader
– Render
• Webkit(Javascript)
• Webkit(AJAX):click simulation, event notify
• Webkit(CSS): css feature
– ADSL Proxy
• How to get
– Why scan by ourselves
• How to use
– Why nginx

12. Evolution---Module
• Extractor
– Wrapper induction
• Semi automation
– Firefox extensions
– How to improve
– Templates management
• Full automation
– Scrapy extract tool
• Cascade extraction supported

13. Evolution---Module
• Scheduler
– FIFO Queue
– Priority Queue
• Seed weight
• Smallest interval
• User Query distribution
• User Query importance
• Webpage change characteristics

14. Evolution---Module
• Processor
– Mysql
– Redis
– Hadoop

15. Simplify
• IO module
– Synchronous block
• No Middleware supported
• No Item Loader
• No Framework
• No …

16. Do it
• Time Estimation
– Basic 1-2 month
– Improve