• Like
Twisted
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Twisted

  • 2,008 views
Published

 

Published in Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,008
On SlideShare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
21
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. TwistedMichal Sedlakmichal6103@gmail.com
  • 2. Based on Dave Peticolas: Twisted Introduction http://krondo.com/?page_id=1327Orestis Markou: Asynchronous programming with Twistedhttp://ep2011.europython.eu/conference/talks/asynchronous-pro https://github.com/orestis/twisted-tutorial Twisted documentation http://twistedmatrix.com/documents 2
  • 3. Outline● What is Twisted?● Why Twisted? ● Async vs. Sync ● Blocking vs. Non Blocking● Parts: ● Reactor ● Factory ● Protocol ● Transfer 3
  • 4. What is TwistedTwisted is a networking engine written in Python, supporting numerous protocols.It contains a web server, numerous chat clients, chat servers, mail servers, and more.http://twistedmatrix.com/trac/wiki/TwistedProjects 4
  • 5. Why use Twisted● Python... 2 :(● Asynchronous and event-based● Full-featured ● Mail, web, news, chat, DNS, SSH, Telnet, RPC, database access, and more● Flexible● Open source 5 http://www.eecho.info/Echo/python/why-use-twisted/
  • 6. Async? Eventbased?● Single thread ● Synchronous ● Asynchronous● Multi-thread 6
  • 7. Synchronous model ● 3 tasks ● One after another ● Simple flow 7
  • 8. Threaded model● Parallel execution● Complex flow coordination ● IPC, critical section, race condition, synchronisation 8
  • 9. Asynchronous model ● Interleaved tasks ● Sequence of small steps ● More complex than sync No parallelism! Q: Same execution time as in sync model? 9
  • 10. Typical network server blocksimport sockets = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.bind((HOST, PORT))s.listen(1)while 1:    conn, addr = s.accept()    while 1:        data = conn.recv(1024)        if not data: break        conn.send(data)    conn.close() 10
  • 11. Blocking ● Some function calls blocks and sync programs waits ● I/O ● CPU much faster then: ● Disk ● Network 11
  • 12. When and why async?● Many tasks ● Non-blocking● Interactive ● Responsive● Independent tasks ● Less interprocess communication means less waiting for processes Network services 12
  • 13. How?● Low level I/O - Platform dependent ● Socket – setblocking(0) ● Async Disk I/O – Python 3 - http://www.python.org/dev/peps/pep-3116/ – Twisted http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.f● Reactor pattern and async libs - Twisted 13
  • 14. Twisted basic parts● Reactor● Endpoint● Factory● Protocol● Transfer● Deferred 14
  • 15. 15
  • 16. Reactor ● Wait for I/O ● Handle Event ● Q: What if event handler blocks? 16
  • 17. Blocking event handler stops application 17
  • 18. ● potentially blocking operations ● reading or writing from a non-socket file descriptor (pipe) ● waiting for a subprocess to finish● Switch from blocking to non-blocking? ● Many standard Python functions are blocking only ● Example: os.system – always block ● Use Twisted API :) 18
  • 19. First serverfrom twisted.internet import reactorreactor.run()● The reactor isn’t created explicitly, just imported.● reactor.run(), reactor.stop()● The reactor loop runs in the same thread it was started in.● Once the loop starts up, it just keeps going.● If it doesn’t have anything to do, the reactor loop does not consume CPU.● Default reactor is twisted.internet.selectreactor 19
  • 20. Reactor Status T S U Threading Processes Sched Platforms C S D uling P L Pselect() Stable Y Y Y Y Y Y Unix, Win32poll Stable Y Y Y Y Y Y UnixWaitForMultipleO Experimental Y Y Y Y Y Y Win32bjectsInput/Output Experimental Y Y N N N Y Win32Completion PortCoreFoundation Unmaintained Y Y Y Y Y Y Mac OS Xepoll Stable Y Y Y Y Y Y Linux 2.6GTK+ Stable Y Y Y Y Y Y Unix, Win32wx Experimental Y Y Y Y Y Y Unix, Win32kqueue Experimental Y Y Y Y Y Y FreeBSD 20 http://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html
  • 21. Change reactorfrom twisted.internet import pollreactorpollreactor.install()from twisted.internet import reactorreactor.run() 21
  • 22. 22
  • 23. Factory● persistent configuration common to all connections● instantiates “protocol”● access persistence: protocol_instance.factory● factory does not listen to connections● same service on multiple ports or network addresses 23
  • 24. 24
  • 25. Protocol● Per connection instance● Protocol handling● DataReceived(): Called whenever data is received.● ConnectionLost(): Called when the connection is shut down.● MakeConnection(): Make a connection to a transport and a server.● ConnectionMade(): Called when a connection is made. 25
  • 26. 26
  • 27. Transport● Represents physical connection● write(): send data over connection● GetPeer()● GetHost()● loseConnection() 27
  • 28. 28
  • 29. Endpoint ● Listener or connector abstractioncep = clientFromString(reactor  "tcp:host=www.example.com:port=80")cep.connect(factory)cep = clientFromString(reactor,      "ssl:host=web.example.com:port=443:"      "privateKey=foo.pem:certKey=foo.pem")cep.connect(factory) 29
  • 30. sep = serverFromString(reactor,  "tcp:80:interface=127.0.0.1")sep.listen(factory)sep = serverFromString(reactor,  "unix:/var/run/finger:mode=660")sep.listen(factory) 30
  • 31. Summary● Reactor is most important● reactor.run()● Appropriate endpoint.● Protocol factory to create protocol instances● Protocol instances only deal with a single connection● Shared state in factory● Data arrive in chunks● Implementing protocols is hard. Reuse as much code as possible 31
  • 32. Whats next?● Server example● Deferred - promise● Database handling – Async?● Error handling – reverse error flow● Trial 32
  • 33. First server● Proxy at localhost:8000● Input: URL● Output: Fetched data 33
  • 34. First serverimport timeimport urllib2from twisted.protocols import basicclass ProxyProtocol(basic.LineReceiver):    def lineReceived(self, url):        if not url.startswith(http://):            return        start = time.time()        print fetching, url        connection = urllib2.urlopen(url)        data = connection.read()        print fetched, url,        self.transport.write(data)        self.transport.loseConnection() 34        print in, time.time() ­ start
  • 35. from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory):    protocol = ProxyProtocolfrom twisted.internet import reactor, endpointsendpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)factory = ProxyFactory()endpoint.listen(factory)reactor.run() 35
  • 36. RunSite   Server Client Why same download=====  ====== ====== times?A      0.771  3.817B      0.567  2.026C      1.457  2.026 c = urllib2.urlopen(url)D      1.019  3.815 data=c.read()­­­­­  ­­­­­­ ­­­­­­Total: 3.813  3.817 36
  • 37. We need async url data fetcher● twisted.web.client.getPage● http://launchpad.net/tx 37
  • 38. import timefrom twisted.web import clientfrom twisted.protocols import basicclass ProxyProtocol(basic.LineReceiver):    def lineReceived(self, url):        if not url.startswith(http://):            return        start = time.time()        print fetching, url        deferredData = client.getPage(url)        def urlFetched(data):            self.transport.write(data)            self.transport.loseConnection()            print fetched, url,            print in, time.time() ­ start        deferredData.addCallback(urlFetched) 38
  • 39. from twisted.internet import protocolclass ProxyFactory(protocol.ServerFactory):    protocol = ProxyProtocolfrom twisted.internet import reactor, endpointsendpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)factory = ProxyFactory()endpoint.listen(factory)reactor.run() 39
  • 40. Site  Server Client Better :)===== ====== ======A      0.850  0.853B      0.486  0.488C      1.582  1.584D      0.999  1.000­­­­­ ­­­­­­ ­­­­­­Total: 3.918  1.585 40