Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Twisted

2,705 views

Published on

Published in: Technology, Education
  • Be the first to comment

Twisted

  1. 1. TwistedMichal Sedlakmichal6103@gmail.com
  2. 2. Based on Dave Peticolas: Twisted Introduction http://krondo.com/?page_id=1327Orestis Markou: Asynchronous programming with Twistedhttp://ep2011.europython.eu/conference/talks/asynchronous-pro https://github.com/orestis/twisted-tutorial Twisted documentation http://twistedmatrix.com/documents 2
  3. 3. Outline● What is Twisted?● Why Twisted? ● Async vs. Sync ● Blocking vs. Non Blocking● Parts: ● Reactor ● Factory ● Protocol ● Transfer 3
  4. 4. What is TwistedTwisted is a networking engine written in Python, supporting numerous protocols.It contains a web server, numerous chat clients, chat servers, mail servers, and more.http://twistedmatrix.com/trac/wiki/TwistedProjects 4
  5. 5. Why use Twisted● Python... 2 :(● Asynchronous and event-based● Full-featured ● Mail, web, news, chat, DNS, SSH, Telnet, RPC, database access, and more● Flexible● Open source 5 http://www.eecho.info/Echo/python/why-use-twisted/
  6. 6. Async? Eventbased?● Single thread ● Synchronous ● Asynchronous● Multi-thread 6
  7. 7. Synchronous model ● 3 tasks ● One after another ● Simple flow 7
  8. 8. Threaded model● Parallel execution● Complex flow coordination ● IPC, critical section, race condition, synchronisation 8
  9. 9. Asynchronous model ● Interleaved tasks ● Sequence of small steps ● More complex than sync No parallelism! Q: Same execution time as in sync model? 9
  10. 10. Typical network server blocksimport sockets = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.bind((HOST, PORT))s.listen(1)while 1:    conn, addr = s.accept()    while 1:        data = conn.recv(1024)        if not data: break        conn.send(data)    conn.close() 10
  11. 11. Blocking ● Some function calls blocks and sync programs waits ● I/O ● CPU much faster then: ● Disk ● Network 11
  12. 12. When and why async?● Many tasks ● Non-blocking● Interactive ● Responsive● Independent tasks ● Less interprocess communication means less waiting for processes Network services 12
  13. 13. How?● Low level I/O - Platform dependent ● Socket – setblocking(0) ● Async Disk I/O – Python 3 - http://www.python.org/dev/peps/pep-3116/ – Twisted http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.f● Reactor pattern and async libs - Twisted 13
  14. 14. Twisted basic parts● Reactor● Endpoint● Factory● Protocol● Transfer● Deferred 14
  15. 15. 15
  16. 16. Reactor ● Wait for I/O ● Handle Event ● Q: What if event handler blocks? 16
  17. 17. Blocking event handler stops application 17
  18. 18. ● potentially blocking operations ● reading or writing from a non-socket file descriptor (pipe) ● waiting for a subprocess to finish● Switch from blocking to non-blocking? ● Many standard Python functions are blocking only ● Example: os.system – always block ● Use Twisted API :) 18
  19. 19. First serverfrom twisted.internet import reactorreactor.run()● The reactor isn’t created explicitly, just imported.● reactor.run(), reactor.stop()● The reactor loop runs in the same thread it was started in.● Once the loop starts up, it just keeps going.● If it doesn’t have anything to do, the reactor loop does not consume CPU.● Default reactor is twisted.internet.selectreactor 19
  20. 20. Reactor Status T S U Threading Processes Sched Platforms C S D uling P L Pselect() Stable Y Y Y Y Y Y Unix, Win32poll Stable Y Y Y Y Y Y UnixWaitForMultipleO Experimental Y Y Y Y Y Y Win32bjectsInput/Output Experimental Y Y N N N Y Win32Completion PortCoreFoundation Unmaintained Y Y Y Y Y Y Mac OS Xepoll Stable Y Y Y Y Y Y Linux 2.6GTK+ Stable Y Y Y Y Y Y Unix, Win32wx Experimental Y Y Y Y Y Y Unix, Win32kqueue Experimental Y Y Y Y Y Y FreeBSD 20 http://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html
  21. 21. Change reactorfrom twisted.internet import pollreactorpollreactor.install()from twisted.internet import reactorreactor.run() 21
  22. 22. 22
  23. 23. Factory● persistent configuration common to all connections● instantiates “protocol”● access persistence: protocol_instance.factory● factory does not listen to connections● same service on multiple ports or network addresses 23
  24. 24. 24
  25. 25. Protocol● Per connection instance● Protocol handling● DataReceived(): Called whenever data is received.● ConnectionLost(): Called when the connection is shut down.● MakeConnection(): Make a connection to a transport and a server.● ConnectionMade(): Called when a connection is made. 25
  26. 26. 26
  27. 27. Transport● Represents physical connection● write(): send data over connection● GetPeer()● GetHost()● loseConnection() 27
  28. 28. 28
  29. 29. Endpoint ● Listener or connector abstractioncep = clientFromString(reactor  "tcp:host=www.example.com:port=80")cep.connect(factory)cep = clientFromString(reactor,      "ssl:host=web.example.com:port=443:"      "privateKey=foo.pem:certKey=foo.pem")cep.connect(factory) 29
  30. 30. sep = serverFromString(reactor,  "tcp:80:interface=127.0.0.1")sep.listen(factory)sep = serverFromString(reactor,  "unix:/var/run/finger:mode=660")sep.listen(factory) 30
  31. 31. Summary● Reactor is most important● reactor.run()● Appropriate endpoint.● Protocol factory to create protocol instances● Protocol instances only deal with a single connection● Shared state in factory● Data arrive in chunks● Implementing protocols is hard. Reuse as much code as possible 31
  32. 32. Whats next?● Server example● Deferred - promise● Database handling – Async?● Error handling – reverse error flow● Trial 32
  33. 33. First server● Proxy at localhost:8000● Input: URL● Output: Fetched data 33
  34. 34. First serverimport timeimport urllib2from twisted.protocols import basicclass ProxyProtocol(basic.LineReceiver):    def lineReceived(self, url):        if not url.startswith(http://):            return        start = time.time()        print fetching, url        connection = urllib2.urlopen(url)        data = connection.read()        print fetched, url,        self.transport.write(data)        self.transport.loseConnection() 34        print in, time.time() ­ start
  35. 35. from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory):    protocol = ProxyProtocolfrom twisted.internet import reactor, endpointsendpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)factory = ProxyFactory()endpoint.listen(factory)reactor.run() 35
  36. 36. RunSite   Server Client Why same download=====  ====== ====== times?A      0.771  3.817B      0.567  2.026C      1.457  2.026 c = urllib2.urlopen(url)D      1.019  3.815 data=c.read()­­­­­  ­­­­­­ ­­­­­­Total: 3.813  3.817 36
  37. 37. We need async url data fetcher● twisted.web.client.getPage● http://launchpad.net/tx 37
  38. 38. import timefrom twisted.web import clientfrom twisted.protocols import basicclass ProxyProtocol(basic.LineReceiver):    def lineReceived(self, url):        if not url.startswith(http://):            return        start = time.time()        print fetching, url        deferredData = client.getPage(url)        def urlFetched(data):            self.transport.write(data)            self.transport.loseConnection()            print fetched, url,            print in, time.time() ­ start        deferredData.addCallback(urlFetched) 38
  39. 39. from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory):    protocol = ProxyProtocolfrom twisted.internet import reactor, endpointsendpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)factory = ProxyFactory()endpoint.listen(factory)reactor.run() 39
  40. 40. Site  Server Client Better :)===== ====== ======A      0.850  0.853B      0.486  0.488C      1.582  1.584D      0.999  1.000­­­­­ ­­­­­­ ­­­­­­Total: 3.918  1.585 40

×