node.js, javascript and the future

5,474 views
5,370 views

Published on

Published in: Technology, Design

node.js, javascript and the future

  1. 1. javascript & the future
  2. 2. Jeff MiccolisDevelopment Seed
  3. 3. Open AtriumFeaturesContextStrongarm
  4. 4. Jeff MiccolisMapBox
  5. 5. First, a brief apology.
  6. 6. Three “war” storiesabout things that arehard with php & drupal
  7. 7. ...and easier in node.js
  8. 8. Two lessons I learned
  9. 9. ...from Drupal andnode.js work
  10. 10. SERVER SIDEJAVASCRIPT
  11. 11. V8
  12. 12. CHROME ‘SJAVASCRIPT ENGINE
  13. 13. ON YOUR SERVER
  14. 14. 1. Sending lots of email
  15. 15. OpenAtrium
  16. 16. Single emailsDigests email
  17. 17. SMSXMPP (jabber)Twitteretc, etc, etc...
  18. 18. channel independent messagesdon’t send e-mails to users, send them messages delivered by mail, IM, SMS, etc...d.o/project/Messaging
  19. 19. Problems
  20. 20. Send an email to one user - 50 ms 28 users - 1.4 seconds 600 users - 30 secondsProblems
  21. 21. Run cron every 5 minutes and you can send a lot more emails but if one run doesn’t complete...Problems
  22. 22. <?php$addresses = array( jeff@mapbox.com, alex@mapbox.com, eric@mapbox.com);$message = I am writing to inform you of a...;$count = 0;foreach($addresses as $sucker) { if (mail($sucker, New opportunity, $message)) { $count++; };}echo "$count messages sent";
  23. 23. var mail = require(mail);var addresses = [ jeff@mapbox.com, alex@mapbox.com, eric@mapbox.com];var message = I am writing to inform you of a...;var count = 0;addresses.forEach(function(sucker, i) { mail(sucker, New opportunity, message, function(err) { if (!err) { count++; } if (i == addresses.length - 1) { console.log(count +" messages sent"); } });});
  24. 24. // Attempt to send email.if (mail($sucker, New opportunity, $message)) { $count++};// Once email is sent, get the next address.echo next;// Attempt to send email.mail(sucker, New opportunity, message, function(err) { if (!err) { count++; }});// Dont wait, get the next address NOW.console.log(next);
  25. 25. Send an email ~ 5 ms processing, 45 ms waiting 28 emails ~ 140 ms to process, done 45 ms later.600 users ~ 3 seconds to process, done 45 ms later.Stop waiting around
  26. 26. 600 emails w/Php: ~30 seconds 600 emails w/node.js: ~3 secondsStop waiting around
  27. 27. DEMOStop waiting around
  28. 28. 2. Handling feeds
  29. 29. Fetch RSS feeds Fetch original article Tag items (calais) Tag items (watchlist) Tag items (wikipedia) Geocode itemsThe task
  30. 30. Thousands of feeds. Millions of items. Gigs and Gigs of data.The scale
  31. 31. cron.phpProblems
  32. 32. maggie
  33. 33. multi-threaded python daemonmaggied
  34. 34. 4 “retriever” workers get batches of 50 items. they fetch/tag/geocode each item.maggied
  35. 35. retrieve original story: 300ms tag: 100ms geocode: 150ms TOTAL: 550msmaggied
  36. 36. Nearly all of that 550ms is spent idle.Stop waiting around
  37. 37. ...but we could run “packs” of retrievers!
  38. 38. Is that really the bestidea?
  39. 39. Replace the retrieverswith a singleHYPERACTIVE SQUID!
  40. 40. The squid runs up and down as fast as it can dealing with each item in turn.It fires off any long running I/O operations and then moves on to the next item.When the I/O operation reports progress,it does a little more work on behalf of the corresponding item.
  41. 41. Event loop100% async
  42. 42. Anything that leaves v8 takes a callback.100% async
  43. 43. filesystem, network, stdio, timers, child processes100% async
  44. 44. var fs = require(fs);fs.readFile(/etc/passwd, function (err, data) { if (err) throw err; console.log(data);});var request = require(request);request(http://example.com, function (err, resp, body) { if (!err && resp.statusCode == 200) { console.log(body); }});
  45. 45. Limiting factors change: how many open sockets are you allowed? how much bandwidth can you grab? how fast can you issue requests?100% async
  46. 46. 3. Big files, long sessions.
  47. 47. gigabyteWhat’s big?
  48. 48. hoursWhat’s long?
  49. 49. HTTPWhat’s a session?
  50. 50. A category of stories...
  51. 51. upload_max_filesize post_max_size max_input_time max_execution_timeBig uploads in php
  52. 52. this approach caps out ~500mbBig uploads in php
  53. 53. Opens the door to DOS. Tolerates application bloat. Problems in production can get really bad. Never gonna get gigabyte uploads.Problems
  54. 54. Php makes you look elsewhere.Problems
  55. 55. If only we could stream...
  56. 56. If only we could stream...
  57. 57. Deal with things bucket by bucket and you don’t need all that memory.Streaming
  58. 58. write that files to disk as it comes in ...or stream it off to s3!Streaming
  59. 59. var formidable = require(formidable);var http = require(http);http.createServer(function(req, res) { if (req.url == /upload && req.method == POST) { var form = new formidable.IncomingForm(); form.parse(req); form.onPart = function(part) { part.addListener(data, function(chunk) { // Do cool stuff, like streaming! console.log(chunk.toString(utf8)); }); }; }}).listen(80);
  60. 60. _changes
  61. 61. A persistent connection to your database, which feeds you new data when it has some._changes
  62. 62. it’s amazing_changes
  63. 63. 1. Map uploads to s3 2. Save a record to CouchDB 3. “Downloader” listens for new mapsMapBox uploads
  64. 64. node.js process very long-lived http connection to CouchDBMapBox uploads
  65. 65. non-blocking I/O & single event loopEverything is different
  66. 66. 4. Package Management
  67. 67. Package management for Drupaldrush make
  68. 68. d.o - project namespace d.o - inclusive project policydrush make
  69. 69. a way to stay sane.drush make
  70. 70. ...and is part of drush proper now!drush make
  71. 71. But I’d been using Drupal for YEARS by thendrush make
  72. 72. pear
  73. 73. PHP Extension and Application Repositorypear
  74. 74. high threshold for new projectspear
  75. 75. wildly inclusive awesomely useful awesomely successfulimagine if pear was...
  76. 76. wildly inclusive awesomely useful awesomely successfulnpm
  77. 77. node package managernpm
  78. 78. pear: 584 d.o: 15, 296 (~3,600 for D7) npm: 7,976 (2 years old!)packages
  79. 79. { "author": "Jeff Miccolis <jeff@miccolis.net>", "name": "portal", "description": "Data Catalog", "version": "0.0.0", "engines": { "node": "~v0.6.6" }, "dependencies": { "couchapp": "https://github.com/.../attachment_operators", "underscore": "= 1.2.0", "request": "= 2.1.1" }} npm - package.json
  80. 80. you’ll love it.npm
  81. 81. 5. Nice hammer
  82. 82. “Javascript, really?”Nice hammer...
  83. 83. “Clearly he’s overly excited about this async stuff”Nice hammer...
  84. 84. “...and thinks it’ll work for everything.”Nice hammer...
  85. 85. “Do it with Drupal”, eh?
  86. 86. computationally heavy tasks databasesnode.js is bad for...
  87. 87. interacting with other services.node.js is awesome for...
  88. 88. databases, mail servers, web services, web clients..services like...
  89. 89. http://substack.net/posts/b96642http://blog.nelhage.com/2012/03/why-node-js-is-cool/Other people’s words
  90. 90. “The primary focus of most node modules is onusing, not extending... A big part of what makesnode modules so great is how they tend to havereally obvious entry points as a consequence of focusing on usability and limited surface area”Limited surface area
  91. 91. “Instead of the http server being an external service that we configure to run our code, it becomes just another tool in our arsenal”Callback Austerity
  92. 92. "The upshot of this pressure is that, since essentially every node.js library works this way, you can pick and choose arbitrary node.js libraries and combine them in the sameprogram, without even having to think about the fact that you’re doing so."Async by default
  93. 93. If nothing else you’ll get better at javascript.Try it!
  94. 94. But I bet you’ll like it.Try it!
  95. 95. Thanks!
  96. 96. Photo credit due to these wonderful people who offertheir photos on flickr under a creative commonslicense:Spam - http://www.flickr.com/photos/olivabassa/Cows - http://www.flickr.com/photos/lynndombrowski/Dogs - http://www.flickr.com/photos/photosightfaces/Squid - http://www.flickr.com/photos/laughingsquid/Ants - http://www.flickr.com/photos/fuzzcat/Stream - http://www.flickr.com/photos/universalpops/Boxes - http://www.flickr.com/photos/sillydog/Pear - http://www.flickr.com/photos/reebob/Hammer - http://www.flickr.com/photos/kefraya
  97. 97. What did you think? Locate this session on the DrupalCon Denver websitehttp://denver2012.drupal.org/program Click the “Take the Survey” link. Thank You!

×