Twisted Introduction

542 views

Published on

An introduction to Twisted, an event-driven networking engine.

Presentation I gave at a mind share session at Rackspace.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
542
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Twisted Introduction

  1. 1. Twisted“An event-driven networking engine written in Python” Ying Li (ying.li@rackspace.com)
  2. 2. Twisted“An event-driven networking engine written in Python”
  3. 3. OS / System Transport ProtocolApplication (e.g. server)Runner / Daemonization
  4. 4. OS / System Transport ProtocolApplication (e.g. server)Runner / Daemonization
  5. 5. Transport•manages socket or pipe
  6. 6. Transport•manages socket or pipe•writes data
  7. 7. Transport•manages socket or pipe•writes data•reads data•notifies protocol
  8. 8. Transport•TCP •AF_UNIX•UDP •Pipes
  9. 9. Transport•TCP •AF_UNIX•UDP •Pipes•TLS •memory
  10. 10. OS / System Transport ProtocolApplication (e.g. server)Runner / Daemonization
  11. 11. Incoming data Transport Protocol•handle data received
  12. 12. Outgoing data Incoming data Transport Protocol •handle data received •write data to transport
  13. 13. Outgoing data Incoming data Transport Protocol •handle data received •write data to transport •handle connections
  14. 14. TransportProtocol
  15. 15. TCPTLS
  16. 16. Process TLS
  17. 17. Bluetooth? TLS
  18. 18. OS / System Transport ProtocolApplication (e.g. server)Runner / Daemonization
  19. 19. Outgoing data / System OS Incoming data Transport Protocol Transport ... Protocol Transport Protocol Application (e.g. server)
  20. 20. Outgoing data Incoming data TCP TLS (protocol) TLS (transport) HTTP
  21. 21. Outgoing data Incoming data UDP TCP (protocol) TCP (transport) TLS (protocol) TLS (transport) HTTP
  22. 22. Protocol•TLS •IMAP •SOCKS•HTTP •SMTP •TELNET•IRC •POP •SIP (voip)•XMPP •DNS •NMEA (gps)•OSCAR •SSH •...more
  23. 23. OS / System Transport ProtocolApplication (e.g. server)Runner / Daemonization
  24. 24. wait... that’s it?
  25. 25. ProtocolApplication (e.g. server)
  26. 26. Application / Server•IRC •NNTP•SMTP •POP•SSH •Generic•SFTP •...more•HTTP
  27. 27. Application / Clients•IRC •NNTP•SMTP •POP•SSH •...more•SFTP•HTTP
  28. 28. server and/or client
  29. 29. Protocol Application•builds protocol•hooks up protocol
  30. 30. Protocol Application•builds multiple protocols•database•authentication•hooks up protocol
  31. 31. Authentication•unix user/password•ssh key•user/password list file•in-memory user/password
  32. 32. OS / System Transport ProtocolApplication (e.g. server)Runner / Daemonization
  33. 33. python myapplication.py Application (e.g. server) Runner / Daemonization
  34. 34. twistd [options] <myplugin>
  35. 35. twistd•daemonization•logging•privilege dropping•chroot•non-default reactor•profiling
  36. 36. twistd [options] <myplugin> [my_plugin_options]
  37. 37. twistdtwistd web --port 80 --path /srv/web
  38. 38. twistdtwistd web --port 8080 --path /srv/web twistd telnet --port 4040
  39. 39. twistd•twistd web •twistd ftp•twistd telnet •twistd mail•twistd dns •twistd conch (ssh)•...more (see `twistd --help`)
  40. 40. Twisted“An event-driven networking engine written in Python”
  41. 41. Concurrency Models (A story of 3 HTTP requests)
  42. 42. TimeHTTP Request
  43. 43. TimeAwait response
  44. 44. TimeProcess response
  45. 45. TimeRequest 2 Request 3
  46. 46. Time Interpolate Responses
  47. 47. Time Multi-Process Process 1 Process 2 Process 3 Blocking I/O Request 1 Interpolation Request 2 Request 3
  48. 48. Time Multi-Threaded Thread 1 Thread 2 Thread 3 Blocking I/O Request 1 Interpolation Request 2 Request 3
  49. 49. Time Multi-Threaded Thread 1 Thread 2 Thread 3 Blocking I/O Request 1 Interpolation Request 2 Request 3
  50. 50. Time Multi-Threaded Thread 1 Thread 2 Thread 3 Blocking I/O Request 1 Interpolation Request 2 Request 3
  51. 51. Time Multi-Threaded Thread 1 Thread 2 Thread 3 Blocking I/O Request 1 Interpolation Request 2 Request 3
  52. 52. Time Multi-Threaded Thread 1 Thread 2 Thread 3 Blocking I/O Request 1 Interpolation Request 2 Request 3
  53. 53. Threads• Locking• Re-entrancy• Debugging
  54. 54. Event-Driven (not to scale)
  55. 55. TimeCallbacks: CB
  56. 56. Time First HTTP RequestCallbacks: CB CB
  57. 57. Time Second HTTP RequestCallbacks: CB CB CB
  58. 58. Time Event!Callbacks: CB CB CB
  59. 59. Time Event!Callbacks: CB CB CB
  60. 60. Time CBCallbacks: CB CB
  61. 61. Time CB Event!Callbacks: CB CB CB
  62. 62. Time CB Event!Callbacks: CB CB CB
  63. 63. Time CB CB Event!Callbacks: CB CB
  64. 64. Time CB CB CBCallbacks: CB
  65. 65. Time CB CB CB CBCallbacks:
  66. 66. Events Handlers
  67. 67. Events Handlers• file descriptors• timed events
  68. 68. • select • CoreFoundation• poll • IOCP• epoll • ...more• kqueue
  69. 69. Deferreds
  70. 70. Deferreds(promises of future results)
  71. 71. DR1 CBR2
  72. 72. DR1 CBR2
  73. 73. D F1CB EB R2
  74. 74. D R1 CB1 EB CB2 CB3CB3 (CB 2 (CB 1 (R1)))
  75. 75. DCB EB A EB B EBC
  76. 76. D try: ... F1CB1 EB except: return R2CB2 R2
  77. 77. D try: ... F1=CCB1 EB A except A: ... EB B except B: ... EBC except C: return R2
  78. 78. D try: ... F1=CCB1 EB A except A: ... EB B except B: ... EBC except C: return R2CB2 R2
  79. 79. D D D D DCB EB CB EB EBCB EB CB
  80. 80. courtesy of http://callbackhell.comfs.readdir(source, function(err, files) { if (err) { console.log(Error finding files: + err) } else { files.forEach(function(filename, fileIndex) { console.log(filename) gm(source + filename).size(function(err, values) { if (err) { console.log(Error identifying file size: + err) } else { console.log(filename + : + values) aspect = (values.width / values.height) widths.forEach(function(width, widthIndex) { height = Math.round(width / aspect) console.log(resizing + filename + to + height + x + height) this.resize(width, height).write(destination + w + width + _ + filename, function(err) { if (err) console.log(Error writing file: + err) }) }.bind(this)) } }) }) }})
  81. 81. • jQuery Deferreds• MochiKit Deferreds• Dojo Deferreds• Google Closure Deferreds
  82. 82. Twisted•easier to reason about concurrency•lots of components•useful abstractions

×