Your SlideShare is downloading. ×
  • Like
Python twisted
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Python twisted


Introduction to twisted programming (Pycon 2010)

Introduction to twisted programming (Pycon 2010)

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Python TwistedMahendra M@mahendra    
  • 2. Methods of concurrency Workers  Threads and processes Event driven Let us examine this with the case of a web server    
  • 3. Worker modelrequest dispatch() worker_1() read(fp) db_rd() db_wr() sock_wr() worker_n()    
  • 4. Worker modelrequest dispatch() worker_1() read(fp) BLOCKING!!db_rd() db_wr() sock_wr() worker_n()    
  • 5. How does it scale ? A worker gets CPU when it is not blocking When it makes a blocking call, it sleeps till the sys­ call is requested At this time another worker gets CPU Worker might block before it completes its  allocated timeslice. This model has worked great and still works great Eg: Apache, Squid    
  • 6. Overheads Worker management  Thread creation  Process creation and management  Synchronization  Scheduling (though this is left to OS) More system calls for blocking operations    
  • 7. Event Driven Code Non blocking code blocks Code execution on events  Data on sockets, timer, new connection Execution triggered from an event loop Full use of CPU timeslice    
  • 8. Visually ... Non blocking functionsevent_1 hdler_1()event_2 block_on_events( .. ) hdler_2() Events are postedevent_n hdler_n()    
  • 9. Visually ... Non blocking functionsevent_1 hdler_1() ev()event_2 block_on_events() hdler_2() Events are postedevent_n hdler_n()    
  • 10. Web Server Non blocking functions request open(fp) reg() opened parse() event_loop() read_sql() reg()sql_read wri_sql() reg()sql_writ sock_wr() reg()responded close()    
  • 11. Event Driven Designs Nginx, Tornado Varnish Memcached OS support  epoll – Linux  kqueue – BSDs    
  • 12. Event Driven Libraries Libevent Python­twisted Java NIO  Apache MINA, Tomcat (not default)  Jetty QT    
  • 13. Drawbacks Tougher to code, design and maintain Workers required to make use of multiple CPUs All callbacks must be non­blocking  Tough to get non­blocking libraries for all modules No isolation  A block in any event loop can freeze everything    
  • 14. Python Twisted Event driven programming framework MIT licensed 8 years old and stable Support for large number of protocols  Client and server support  HTTP – SOAP, REST, CouchDB, XMLRPC, ....  Sockets, TCP/IP, Multicast, TLS, SSH, IMAP …  SMTP, NNTP, FTP, Memcached, AMQP, XMPP, ...    
  • 15. Deferred The central concept of twisted A callback returns a deferred to indicate that the job  is not done yet. The caller can add callbacks to a deferred. The callbacks are invoked then the job is done Eh ?    
  • 16. Deferred examplefrom twisted.internet import reactor# Define a success callbackdef cb( arg1, arg2 ): print ”Timeout after %d %s” % ( arg1, arg2 )# Define an error callbackdef eb( error ): Print ”error %s” % error# Invoke a non blocking taskdeferred = someTimeout( 4 )# Register the callbacks on the returned deferreddeferred.addCallback( cb, 4, twisted is great )deferred.addErrback( eb )# Run the event    
  • 17. Twisted Serverfrom twisted.internet.protocol import Protocol, Factoryfrom twisted.internet import reactorclass QOTD(Protocol): def connectionMade(self): self.transport.write("Welcomern") self.transport.loseConnection()# Next lines are magic:factory = Factory()factory.protocol = QOTD# 8007 is the port you want to run under.reactor.listenTCP(8007, factory)    
  • 18. Deferred chaining A callback can register and return another deferred This callback can return another deferred In short we have a deferred chain … Web server example:    
  • 19. Deferred Chaining   
  • 20. Advanced twisted Twisted application support  Mixing and matching twisted applications Command line twistd runner  Pre­defined twisted apps  Web, telnet, xmpp, dns, conch (ssh), mail, … Plugin architecture Live debugging ADBAPI – for RDBMS    
  • 21. Advanced twisted Perspective Broker  RPC and object sharing  Spreading out servers Cred – Authentication framework Deferring to threads External loops (GTK, QT) Streaming support MVC framework (Complex. Very complex)    
  • 22. Drawbacks Single threaded by design Makes use of only one core/CPU Need external modules for using multiple CPU  Run multiple instances of twisted on a box  num_instances = num_cpus  Use nginx (HTTP), HAProxy for load balancing    
  • 23. Demos   
  • 24. Links