Your SlideShare is downloading. ×

Python twisted

1,629
views

Published on

Introduction to twisted programming (Pycon 2010)

Introduction to twisted programming (Pycon 2010)

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,629
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Python TwistedMahendra M@mahendra http://creativecommons.org/licenses/by-sa/3.0/    
  • 2. Methods of concurrency Workers  Threads and processes Event driven Let us examine this with the case of a web server    
  • 3. Worker modelrequest dispatch() worker_1() read(fp) db_rd() db_wr() sock_wr() worker_n()    
  • 4. Worker modelrequest dispatch() worker_1() read(fp) BLOCKING!!db_rd() db_wr() sock_wr() worker_n()    
  • 5. How does it scale ? A worker gets CPU when it is not blocking When it makes a blocking call, it sleeps till the sys­ call is requested At this time another worker gets CPU Worker might block before it completes its  allocated timeslice. This model has worked great and still works great Eg: Apache, Squid    
  • 6. Overheads Worker management  Thread creation  Process creation and management  Synchronization  Scheduling (though this is left to OS) More system calls for blocking operations    
  • 7. Event Driven Code Non blocking code blocks Code execution on events  Data on sockets, timer, new connection Execution triggered from an event loop Full use of CPU timeslice    
  • 8. Visually ... Non blocking functionsevent_1 hdler_1()event_2 block_on_events( .. ) hdler_2() Events are postedevent_n hdler_n()    
  • 9. Visually ... Non blocking functionsevent_1 hdler_1() ev()event_2 block_on_events() hdler_2() Events are postedevent_n hdler_n()    
  • 10. Web Server Non blocking functions request open(fp) reg() opened parse() event_loop() read_sql() reg()sql_read wri_sql() reg()sql_writ sock_wr() reg()responded close()    
  • 11. Event Driven Designs Nginx, Tornado Varnish Memcached OS support  epoll – Linux  kqueue – BSDs    
  • 12. Event Driven Libraries Libevent Python­twisted Java NIO  Apache MINA, Tomcat (not default)  Jetty QT    
  • 13. Drawbacks Tougher to code, design and maintain Workers required to make use of multiple CPUs All callbacks must be non­blocking  Tough to get non­blocking libraries for all modules No isolation  A block in any event loop can freeze everything    
  • 14. Python Twisted Event driven programming framework MIT licensed 8 years old and stable Support for large number of protocols  Client and server support  HTTP – SOAP, REST, CouchDB, XMLRPC, ....  Sockets, TCP/IP, Multicast, TLS, SSH, IMAP …  SMTP, NNTP, FTP, Memcached, AMQP, XMPP, ...    
  • 15. Deferred The central concept of twisted A callback returns a deferred to indicate that the job  is not done yet. The caller can add callbacks to a deferred. The callbacks are invoked then the job is done Eh ?    
  • 16. Deferred examplefrom twisted.internet import reactor# Define a success callbackdef cb( arg1, arg2 ): print ”Timeout after %d %s” % ( arg1, arg2 )# Define an error callbackdef eb( error ): Print ”error %s” % error# Invoke a non blocking taskdeferred = someTimeout( 4 )# Register the callbacks on the returned deferreddeferred.addCallback( cb, 4, twisted is great )deferred.addErrback( eb )# Run the event loopreactor.run()    
  • 17. Twisted Serverfrom twisted.internet.protocol import Protocol, Factoryfrom twisted.internet import reactorclass QOTD(Protocol): def connectionMade(self): self.transport.write("Welcomern") self.transport.loseConnection()# Next lines are magic:factory = Factory()factory.protocol = QOTD# 8007 is the port you want to run under.reactor.listenTCP(8007, factory)reactor.run()    
  • 18. Deferred chaining A callback can register and return another deferred This callback can return another deferred In short we have a deferred chain … Web server example:    
  • 19. Deferred Chaining   
  • 20. Advanced twisted Twisted application support  Mixing and matching twisted applications Command line twistd runner  Pre­defined twisted apps  Web, telnet, xmpp, dns, conch (ssh), mail, … Plugin architecture Live debugging ADBAPI – for RDBMS    
  • 21. Advanced twisted Perspective Broker  RPC and object sharing  Spreading out servers Cred – Authentication framework Deferring to threads External loops (GTK, QT) Streaming support MVC framework (Complex. Very complex)    
  • 22. Drawbacks Single threaded by design Makes use of only one core/CPU Need external modules for using multiple CPU  Run multiple instances of twisted on a box  num_instances = num_cpus  Use nginx (HTTP), HAProxy for load balancing    
  • 23. Demos   
  • 24. Links http://twistedmatrix.com/trac/    

×