Python twisted

2,464 views
2,234 views

Published on

Introduction to twisted programming (Pycon 2010)

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,464
On SlideShare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
21
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Python twisted

  1. 1. Python TwistedMahendra M@mahendra http://creativecommons.org/licenses/by-sa/3.0/    
  2. 2. Methods of concurrency Workers  Threads and processes Event driven Let us examine this with the case of a web server    
  3. 3. Worker modelrequest dispatch() worker_1() read(fp) db_rd() db_wr() sock_wr() worker_n()    
  4. 4. Worker modelrequest dispatch() worker_1() read(fp) BLOCKING!!db_rd() db_wr() sock_wr() worker_n()    
  5. 5. How does it scale ? A worker gets CPU when it is not blocking When it makes a blocking call, it sleeps till the sys­ call is requested At this time another worker gets CPU Worker might block before it completes its  allocated timeslice. This model has worked great and still works great Eg: Apache, Squid    
  6. 6. Overheads Worker management  Thread creation  Process creation and management  Synchronization  Scheduling (though this is left to OS) More system calls for blocking operations    
  7. 7. Event Driven Code Non blocking code blocks Code execution on events  Data on sockets, timer, new connection Execution triggered from an event loop Full use of CPU timeslice    
  8. 8. Visually ... Non blocking functionsevent_1 hdler_1()event_2 block_on_events( .. ) hdler_2() Events are postedevent_n hdler_n()    
  9. 9. Visually ... Non blocking functionsevent_1 hdler_1() ev()event_2 block_on_events() hdler_2() Events are postedevent_n hdler_n()    
  10. 10. Web Server Non blocking functions request open(fp) reg() opened parse() event_loop() read_sql() reg()sql_read wri_sql() reg()sql_writ sock_wr() reg()responded close()    
  11. 11. Event Driven Designs Nginx, Tornado Varnish Memcached OS support  epoll – Linux  kqueue – BSDs    
  12. 12. Event Driven Libraries Libevent Python­twisted Java NIO  Apache MINA, Tomcat (not default)  Jetty QT    
  13. 13. Drawbacks Tougher to code, design and maintain Workers required to make use of multiple CPUs All callbacks must be non­blocking  Tough to get non­blocking libraries for all modules No isolation  A block in any event loop can freeze everything    
  14. 14. Python Twisted Event driven programming framework MIT licensed 8 years old and stable Support for large number of protocols  Client and server support  HTTP – SOAP, REST, CouchDB, XMLRPC, ....  Sockets, TCP/IP, Multicast, TLS, SSH, IMAP …  SMTP, NNTP, FTP, Memcached, AMQP, XMPP, ...    
  15. 15. Deferred The central concept of twisted A callback returns a deferred to indicate that the job  is not done yet. The caller can add callbacks to a deferred. The callbacks are invoked then the job is done Eh ?    
  16. 16. Deferred examplefrom twisted.internet import reactor# Define a success callbackdef cb( arg1, arg2 ): print ”Timeout after %d %s” % ( arg1, arg2 )# Define an error callbackdef eb( error ): Print ”error %s” % error# Invoke a non blocking taskdeferred = someTimeout( 4 )# Register the callbacks on the returned deferreddeferred.addCallback( cb, 4, twisted is great )deferred.addErrback( eb )# Run the event loopreactor.run()    
  17. 17. Twisted Serverfrom twisted.internet.protocol import Protocol, Factoryfrom twisted.internet import reactorclass QOTD(Protocol): def connectionMade(self): self.transport.write("Welcomern") self.transport.loseConnection()# Next lines are magic:factory = Factory()factory.protocol = QOTD# 8007 is the port you want to run under.reactor.listenTCP(8007, factory)reactor.run()    
  18. 18. Deferred chaining A callback can register and return another deferred This callback can return another deferred In short we have a deferred chain … Web server example:    
  19. 19. Deferred Chaining   
  20. 20. Advanced twisted Twisted application support  Mixing and matching twisted applications Command line twistd runner  Pre­defined twisted apps  Web, telnet, xmpp, dns, conch (ssh), mail, … Plugin architecture Live debugging ADBAPI – for RDBMS    
  21. 21. Advanced twisted Perspective Broker  RPC and object sharing  Spreading out servers Cred – Authentication framework Deferring to threads External loops (GTK, QT) Streaming support MVC framework (Complex. Very complex)    
  22. 22. Drawbacks Single threaded by design Makes use of only one core/CPU Need external modules for using multiple CPU  Run multiple instances of twisted on a box  num_instances = num_cpus  Use nginx (HTTP), HAProxy for load balancing    
  23. 23. Demos   
  24. 24. Links http://twistedmatrix.com/trac/    

×