TwistedMichal Sedlakmichal6103@gmail.com
Based on           Dave Peticolas: Twisted Introduction            http://krondo.com/?page_id=1327Orestis Markou: Asynchro...
Outline●   What is Twisted?●   Why Twisted?    ●   Async vs. Sync    ●   Blocking vs. Non Blocking●   Parts:    ●   Reacto...
What is TwistedTwisted is a networking engine written in Python,         supporting numerous protocols.It contains a web s...
Why use Twisted●   Python... 2 :(●   Asynchronous and event-based●   Full-featured    ●   Mail, web, news, chat, DNS, SSH,...
Async? Eventbased?●   Single thread    ●   Synchronous    ●   Asynchronous●   Multi-thread                                ...
Synchronous model         ●   3 tasks         ●   One after another         ●   Simple flow                               ...
Threaded model●   Parallel execution●   Complex flow coordination    ●   IPC, critical section, race condition, synchronis...
Asynchronous model         ●   Interleaved tasks         ●   Sequence of small steps         ●   More complex than sync   ...
Typical network server blocksimport sockets = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.bind((HOST, PORT))s.liste...
Blocking ●   Some function calls blocks     and sync programs waits ●   I/O ●   CPU much faster then:     ●   Disk     ●  ...
When and why async?●   Many tasks             ●   Non-blocking●   Interactive            ●   Responsive●   Independent tas...
How?●   Low level I/O - Platform dependent    ●   Socket – setblocking(0)    ●   Async Disk I/O        –   Python 3 - http...
Twisted basic parts●   Reactor●   Endpoint●   Factory●   Protocol●   Transfer●   Deferred                                 ...
15
Reactor    ●   Wait for I/O    ●   Handle Event    ●   Q: What if event        handler blocks?                           16
Blocking event handler   stops application                         17
●   potentially blocking operations    ●   reading or writing from a non-socket file descriptor        (pipe)    ●   waiti...
First serverfrom twisted.internet import reactorreactor.run()●   The reactor isn’t created explicitly, just imported.●   r...
Reactor                         Status       T S U Threading        Processes     Sched    Platforms                      ...
Change reactorfrom twisted.internet import pollreactorpollreactor.install()from twisted.internet import reactorreactor.run...
22
Factory●   persistent configuration common to all    connections●   instantiates “protocol”●   access persistence: protoco...
24
Protocol●   Per connection instance●   Protocol handling●   DataReceived(): Called whenever data is received.●   Connectio...
26
Transport●   Represents physical connection●   write(): send data over connection●   GetPeer()●   GetHost()●   loseConnect...
28
Endpoint ●   Listener or connector abstractioncep = clientFromString(reactor                      "tcp:host=www.example.co...
sep = serverFromString(reactor,                     "tcp:80:interface=127.0.0.1")sep.listen(factory)sep = serverFromString...
Summary●   Reactor is most important●   reactor.run()●   Appropriate endpoint.●   Protocol factory to create protocol inst...
Whats next?●   Server example●   Deferred - promise●   Database handling – Async?●   Error handling – reverse error flow● ...
First server●   Proxy at localhost:8000●   Input: URL●   Output: Fetched data                                 33
First serverimport timeimport urllib2from twisted.protocols import basicclass ProxyProtocol(basic.LineReceiver):    def li...
from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory):    protocol = ProxyProtocolfrom twisted.i...
RunSite   Server Client   Why same download=====  ====== ======   times?A      0.771  3.817B      0.567  2.026C      1.457...
We need async url data fetcher●   twisted.web.client.getPage●   http://launchpad.net/tx                                   ...
import timefrom twisted.web import clientfrom twisted.protocols import basicclass ProxyProtocol(basic.LineReceiver):    de...
from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory):    protocol = ProxyProtocolfrom twisted.i...
Site  Server Client   Better :)===== ====== ======A      0.850  0.853B      0.486  0.488C      1.582  1.584D      0.999  1...
Upcoming SlideShare
Loading in …5
×

Twisted

2,587 views

Published on

Published in: Technology, Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,587
On SlideShare
0
From Embeds
0
Number of Embeds
107
Actions
Shares
0
Downloads
32
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Twisted

  1. 1. TwistedMichal Sedlakmichal6103@gmail.com
  2. 2. Based on Dave Peticolas: Twisted Introduction http://krondo.com/?page_id=1327Orestis Markou: Asynchronous programming with Twistedhttp://ep2011.europython.eu/conference/talks/asynchronous-pro https://github.com/orestis/twisted-tutorial Twisted documentation http://twistedmatrix.com/documents 2
  3. 3. Outline● What is Twisted?● Why Twisted? ● Async vs. Sync ● Blocking vs. Non Blocking● Parts: ● Reactor ● Factory ● Protocol ● Transfer 3
  4. 4. What is TwistedTwisted is a networking engine written in Python, supporting numerous protocols.It contains a web server, numerous chat clients, chat servers, mail servers, and more.http://twistedmatrix.com/trac/wiki/TwistedProjects 4
  5. 5. Why use Twisted● Python... 2 :(● Asynchronous and event-based● Full-featured ● Mail, web, news, chat, DNS, SSH, Telnet, RPC, database access, and more● Flexible● Open source 5 http://www.eecho.info/Echo/python/why-use-twisted/
  6. 6. Async? Eventbased?● Single thread ● Synchronous ● Asynchronous● Multi-thread 6
  7. 7. Synchronous model ● 3 tasks ● One after another ● Simple flow 7
  8. 8. Threaded model● Parallel execution● Complex flow coordination ● IPC, critical section, race condition, synchronisation 8
  9. 9. Asynchronous model ● Interleaved tasks ● Sequence of small steps ● More complex than sync No parallelism! Q: Same execution time as in sync model? 9
  10. 10. Typical network server blocksimport sockets = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.bind((HOST, PORT))s.listen(1)while 1:    conn, addr = s.accept()    while 1:        data = conn.recv(1024)        if not data: break        conn.send(data)    conn.close() 10
  11. 11. Blocking ● Some function calls blocks and sync programs waits ● I/O ● CPU much faster then: ● Disk ● Network 11
  12. 12. When and why async?● Many tasks ● Non-blocking● Interactive ● Responsive● Independent tasks ● Less interprocess communication means less waiting for processes Network services 12
  13. 13. How?● Low level I/O - Platform dependent ● Socket – setblocking(0) ● Async Disk I/O – Python 3 - http://www.python.org/dev/peps/pep-3116/ – Twisted http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.f● Reactor pattern and async libs - Twisted 13
  14. 14. Twisted basic parts● Reactor● Endpoint● Factory● Protocol● Transfer● Deferred 14
  15. 15. 15
  16. 16. Reactor ● Wait for I/O ● Handle Event ● Q: What if event handler blocks? 16
  17. 17. Blocking event handler stops application 17
  18. 18. ● potentially blocking operations ● reading or writing from a non-socket file descriptor (pipe) ● waiting for a subprocess to finish● Switch from blocking to non-blocking? ● Many standard Python functions are blocking only ● Example: os.system – always block ● Use Twisted API :) 18
  19. 19. First serverfrom twisted.internet import reactorreactor.run()● The reactor isn’t created explicitly, just imported.● reactor.run(), reactor.stop()● The reactor loop runs in the same thread it was started in.● Once the loop starts up, it just keeps going.● If it doesn’t have anything to do, the reactor loop does not consume CPU.● Default reactor is twisted.internet.selectreactor 19
  20. 20. Reactor Status T S U Threading Processes Sched Platforms C S D uling P L Pselect() Stable Y Y Y Y Y Y Unix, Win32poll Stable Y Y Y Y Y Y UnixWaitForMultipleO Experimental Y Y Y Y Y Y Win32bjectsInput/Output Experimental Y Y N N N Y Win32Completion PortCoreFoundation Unmaintained Y Y Y Y Y Y Mac OS Xepoll Stable Y Y Y Y Y Y Linux 2.6GTK+ Stable Y Y Y Y Y Y Unix, Win32wx Experimental Y Y Y Y Y Y Unix, Win32kqueue Experimental Y Y Y Y Y Y FreeBSD 20 http://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html
  21. 21. Change reactorfrom twisted.internet import pollreactorpollreactor.install()from twisted.internet import reactorreactor.run() 21
  22. 22. 22
  23. 23. Factory● persistent configuration common to all connections● instantiates “protocol”● access persistence: protocol_instance.factory● factory does not listen to connections● same service on multiple ports or network addresses 23
  24. 24. 24
  25. 25. Protocol● Per connection instance● Protocol handling● DataReceived(): Called whenever data is received.● ConnectionLost(): Called when the connection is shut down.● MakeConnection(): Make a connection to a transport and a server.● ConnectionMade(): Called when a connection is made. 25
  26. 26. 26
  27. 27. Transport● Represents physical connection● write(): send data over connection● GetPeer()● GetHost()● loseConnection() 27
  28. 28. 28
  29. 29. Endpoint ● Listener or connector abstractioncep = clientFromString(reactor  "tcp:host=www.example.com:port=80")cep.connect(factory)cep = clientFromString(reactor,      "ssl:host=web.example.com:port=443:"      "privateKey=foo.pem:certKey=foo.pem")cep.connect(factory) 29
  30. 30. sep = serverFromString(reactor,  "tcp:80:interface=127.0.0.1")sep.listen(factory)sep = serverFromString(reactor,  "unix:/var/run/finger:mode=660")sep.listen(factory) 30
  31. 31. Summary● Reactor is most important● reactor.run()● Appropriate endpoint.● Protocol factory to create protocol instances● Protocol instances only deal with a single connection● Shared state in factory● Data arrive in chunks● Implementing protocols is hard. Reuse as much code as possible 31
  32. 32. Whats next?● Server example● Deferred - promise● Database handling – Async?● Error handling – reverse error flow● Trial 32
  33. 33. First server● Proxy at localhost:8000● Input: URL● Output: Fetched data 33
  34. 34. First serverimport timeimport urllib2from twisted.protocols import basicclass ProxyProtocol(basic.LineReceiver):    def lineReceived(self, url):        if not url.startswith(http://):            return        start = time.time()        print fetching, url        connection = urllib2.urlopen(url)        data = connection.read()        print fetched, url,        self.transport.write(data)        self.transport.loseConnection() 34        print in, time.time() ­ start
  35. 35. from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory):    protocol = ProxyProtocolfrom twisted.internet import reactor, endpointsendpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)factory = ProxyFactory()endpoint.listen(factory)reactor.run() 35
  36. 36. RunSite   Server Client Why same download=====  ====== ====== times?A      0.771  3.817B      0.567  2.026C      1.457  2.026 c = urllib2.urlopen(url)D      1.019  3.815 data=c.read()­­­­­  ­­­­­­ ­­­­­­Total: 3.813  3.817 36
  37. 37. We need async url data fetcher● twisted.web.client.getPage● http://launchpad.net/tx 37
  38. 38. import timefrom twisted.web import clientfrom twisted.protocols import basicclass ProxyProtocol(basic.LineReceiver):    def lineReceived(self, url):        if not url.startswith(http://):            return        start = time.time()        print fetching, url        deferredData = client.getPage(url)        def urlFetched(data):            self.transport.write(data)            self.transport.loseConnection()            print fetched, url,            print in, time.time() ­ start        deferredData.addCallback(urlFetched) 38
  39. 39. from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory):    protocol = ProxyProtocolfrom twisted.internet import reactor, endpointsendpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)factory = ProxyFactory()endpoint.listen(factory)reactor.run() 39
  40. 40. Site  Server Client Better :)===== ====== ======A      0.850  0.853B      0.486  0.488C      1.582  1.584D      0.999  1.000­­­­­ ­­­­­­ ­­­­­­Total: 3.918  1.585 40

×