I’m Shane Hansen, a software engineer here in Salt Lake. By day I run a team of engineers doing j2ee for Backcountry.com, an online etailer.
Chances are you won’t get your product right the first time. If you can decrease the cost of an iteration, you can save a lot of wasted money. Being first to market can be worth a lot.
Lets not forget the most important part, RAD let developers have more fun developing. I say this in all seriousness, if you aren’t having fun developing, you will not be great.
The web is doing I/O wrong. This is a graph showing the execution path of a typical request in ecommerce land. What’s the process doing while the database is loading the product data? Jack shit, that’s what. We can do better, and we’re not alone. The right way to handle this is to parallelize and create an execution graph that mirrors the dependency graph. Parallelization is easy on modern hardware, but we lack good infrastructure (languages and libraries) to simply express our intent. Facebook created tornado, a high level asynchronous framework for making remote service requests.
Here’s a picture of what our execution graph should look like. We receive a request, Fetch the reviews and products in parallel, process the results and return the request. Our problem now becomes, “how can we transparently handle concurrenct in database calls, web services calls, and I/O “ to achive this?
The venerable fork syscall has been the workhorse of concurrency for a long time. Apache, CGI, postgres just to name a few technologies that use this sort of model to handle concurrency. It works well when your worker threads are largely independent. Becomes an order of magnitude more complex if you have to share data between processes.
For those who don’t know, threads of execution share the same address space and therefore the same data structures, but don’t have the same stack. For our point of view today, threads are not the most efficient concurrency mechanism we can use, and they are frankly a lot harder to use correctly than people give them credit for.
Technologies like epoll and select start to get more esotoric. The basic idea behind epoll is that you register a set of events you’re interested in (client a is ready for data, the database query is done, etc) and let the kernel tell you when the events are occurring. This is the approach Node.js takes, and the foundation of it’s crazy scalability. By leveraging epoll and event loops, a single node.js process can service thousands of concurrent clients.
Of course, I’ve just explained how easy it is to build scalable apps on node, but I haven’t explained how you’re going To get your apps built any faster. Well, a funny thing happened to node in the last year and a half. It’s community caught Fire and started pumping out libraries and tools like there’s no tomorrow. There’s over 2k projects on github tagged with node.js And over 500 packages installable via node package manager including xml parsing, datbase drivers, application server middleware, and testing frameworks to name a very few.
Express is a sinatra like framework for controlling url and application routing. It utilizes the larger Connect middleware framework. It provides cruft free url routing.
Socket-io is a cross browser library for full bidirectional low latency communication. It utilizes Websockets when available and progressively degrades to flash sockets, ajax long polling, etc. This is the guts of a browser based chat application. Socket-io is writeen by Guillermo, a friend of a friend.
Real time web analytics. I might have led you to believe I was going to show you how to build one Of these on node.js, that was before I discovered someone beat me to the punch. Go node.js and the gilt group for contributing to this project.
The amazon product api provides a way to build online stores. The entire search, browse, and cart api for amazon is exposed via soap webservices which node-apac can consume. Thanks Dustin McQuay!
The YUI library provides apis and classes for event buses, dom manipulation, html generation, testing, ajax, Internationalization, even YQL (yahoo query language) integration.
Rapid Application with Node.js
A Love Story: Presented in II acts Shane Hansen Software Engineer Whitane Technologies, LLC
Shane Hansen Utah based Software Engineer Likes it when things are easy
<ul><li>Rapid Development Frameworks are important because they: </li></ul><ul><ul><li>Encourage exploration </li></ul></ul><ul><ul><li>Decrease costs associated with each iteration </li></ul></ul><ul><ul><li>Decrease time to market </li></ul></ul>
<ul><li>The web is doing I/O wrong </li></ul><ul><li>Serial I/O is bad for business </li></ul><ul><li>Facebook knows this (tornado) </li></ul>
<ul><li>We need to start thinking like this </li></ul><ul><li>But how can we parallelize? </li></ul>
<ul><li>Fork worker processes, free up the main process </li></ul><ul><li>Postgres does it so it can’t be all bad </li></ul><ul><li>Processes can’t share data except through expensive IPC mechanisms. </li></ul><ul><li>Variations – cgi/preforking/fastcgi </li></ul>
<ul><li>Run jobs in separate thread of execution </li></ul><ul><li>MySQL does it so it can’t be all bad </li></ul><ul><li>Threads can share data easily </li></ul><ul><li>Deadlocking, thread safety, other headaches. </li></ul>
<ul><li>Event driven </li></ul><ul><li>memcached does it, so it can’t be all bad </li></ul><ul><li>One address space, One thread, no headaches </li></ul><ul><li>Sucessors (epoll/kqueue) are the basis for solving the C10K problem. </li></ul><ul><li>Paired with event loops </li></ul><ul><li>Node does this. </li></ul>
<ul><li>The language you love to hate </li></ul><ul><li>Lisp in c’s clothing – anonymous functions, closures. </li></ul><ul><li>Culture of asynchnonous, callback driven code patterns </li></ul><ul><li>Google made it fast. Like really fast. </li></ul><ul><li>Node does this too. </li></ul>
<ul><li>2k+ projects on github </li></ul><ul><li>500+ packages available in npm </li></ul><ul><li>Database drivers </li></ul><ul><li>Web frameworks </li></ul><ul><li>Worker threads </li></ul><ul><li>Parsers </li></ul><ul><li>Testing frameworks </li></ul><ul><li>Payment gateway integrations </li></ul><ul><li>Web analytics </li></ul>