Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Asynchronous Python with Twisted

1,426 views

Published on

An overview of asynchronous programming in the Python development language utilizing the Twisted framework.

Published in: Technology
  • Be the first to comment

Asynchronous Python with Twisted

  1. 1. Who Am I? Developer/Evangelist at LaunchKey Organizer of Las Vegas Python User Group Founder/Co-Organizer of Las Vegas PHP UG Co-Organizer of Las Vegas Developer UG National Junior Basketball Coach
  2. 2. What is Asynchronous Programming? • Non-Linear • Event Driven • Functional • Depends on non-blocking code
  3. 3. Why Asynchronous? One thread for all connections with the following benefits: Shared application level objects and data reduces memory footprint per connection Limited threads allows for better CPU utilization of each core thus fewer cores overall. Both of the above enable large applications to run on smaller virtual machines in the cloud.
  4. 4. What’s the Catch? Functional non-blocking code development has a learning curve. Developing applications with objects that persist beyond a single request can be more difficult than standard WSGI applications. Requires asynchronous libraries which can be scarce in Python.
  5. 5. Twisted Established Asynchronous Python Framework 1.0 Release was in March 2004 Utilizes asynchronous I/O Server runs in Python 2.6+ and 3.2+
  6. 6. Deferred = Asynchronous Analogous to promises/futures Returned by asynchronous and concurrent calls Use callbacks to process result of fulfillment Process callbacks in the order they are registered with each being passed the result of the predecessor Can be quickly created for static data with defer.sucess and defer.failure functions
  7. 7. Callback Example d = defer.Deferred() d.addCallback(lambda x: x + “ is an “) d.addCallback(lambda x: x + “ example!“) d.callback(“This”) >>> This is an example!
  8. 8. Deferred Error Handling Deferred objects catch all callback Exceptions Exceptions are wrapped in a Failure object which can also be returned by a callback or errback If no errbacks remain, exception is swallowed and processing is stopped Remaining callbacks before next errback are skipped If errback returns non-Failure, callback processing resumes
  9. 9. Callback Non-Failure Flow Deferred callback(result) • callback - success • callback – success • errback - skipped • errback – skipped
  10. 10. Errback Non-Failure Flow Deferred callback(result) • callback - success • errback - failure • callback – success • errback – skipped
  11. 11. Errback Failure Flow Deferred callback(result) • callback - success • callback – failure • errback - failure • errback – success
  12. 12. Threads = Concurrent Threads run in thread pool managed by reactor callInThread executes function in its own thread with supplied data deferToThread works like callInThread but returns a Deferred object and processes the callbacks when complete Use threads for blocking code
  13. 13. Reactor Manages the main event loop Waits for and dispatches events or messages May have predefined services to provide access to network communications, threading, and event dispatching. Available services are operating system dependent
  14. 14. Application Hierarchy Protocols Service Application Mail SSL Web Mail SMTP UDP StatsD
  15. 15. Application Starts and stops the reactor Manages services Handles logging Can run as daemon
  16. 16. Service Registers with the reactor Processed in the main event loop Manages connections Communicates via Protocols
  17. 17. Predefined Services File Descriptor Process Threads Time (scheduling) SSL TCP UDP UNIX (UNIX sockets) Socket (3rd Party Sockets)
  18. 18. Protocol Utilized for interaction with services Services with stateful connections utilize protocol factories to generate a protocol instance per connection. Provided with method to send data to connection in the form of transports or producers
  19. 19. Protocol Factory Builds protocol instances Injected into every protocol instance Meant to perform inter-instance interaction Can be used as testable registry for registry pattern implementations
  20. 20. Included Protocols AMP Basic Dict Finger FTP GPS HTB HTTP Ident Loopback Memcache Mice PCP Portforward Postfix Shoutcast SIP SOCKSv4 Stateful Telnet TLS Wire
  21. 21. Available Protocols Mongo LDAP Dict Finger FTP GPS HTB HTTP Ident Loopback Memcache Mice PCP Portforward Postfix Shoutcast SIP SOCKSv4 Stateful Telnet TLS Wire MongoDB* LDAP* * External packages maintained by Twisted Matrix Labs
  22. 22. Twisted Web Web server built on Twisted HTTP Protocol Can be stand alone web server Can embedded as part of an application Supports all of the features of most modern web servers
  23. 23. Web Server Features Virtual hosts Proxy/Reverse proxy Scripts (hello.py) Static files CGI WSGI URL rewrites MIME processors Sessions
  24. 24. Simple Web Application Site object is root Resource Resource object represents URL segment Tree hierarchy of resources Can directly return string or write response via request object for deferred processing
  25. 25. Example Hierarchy / foo/ bar/ bat/ fizz/buz
  26. 26. Example Hierarchy Code root = RootResource() foo = FooResource() foo.putChild(“bar”, BarResource()) foo.putChild(“bat”, BatResource()) root.putChild(“foo”, foo) root.putChild(“fiz/buzz”, FizBuzzResource()) reactor.listenTCP(8080, Site(root)) reactor.start()
  27. 27. Concurrent Example from twisted.web import resource class DemoResource(resource.Resource): def render_GET(self, request): return "<html>foo</html>"
  28. 28. Deferred Example def render_GET(self, request): def process_resp(html, request): request.write(html) request.finish() deferred = self.async_client.request() deferred.addCallback(process_resp, request) return NOT_DONE_YET
  29. 29. Demo Time
  30. 30. Example on GitHub: https://github.com/aenglander/demo-python-twisted- websocket @adam_englander #launchkey on freenode.net #vegastech on freenode.net adam@launchkey.com

×