Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Firehose

3,689 views

Published on

how we built a firehose with NodeJS and ZeroMQ

Published in: Technology, Business
  • Be the first to comment

Firehose

  1. 1. FirehoseBuilding a real-time data stream with NodeJS Luca Grulla@Forward London, 2/8/2011
  2. 2. IntroductionOr what we wanted to build
  3. 3. We like to have a (near) real-time view of the data stream
  4. 4. Resilient
  5. 5. Scalable
  6. 6. Accurate(but not necessarily precise)
  7. 7. Verse and Chorus Or how we built it
  8. 8. Stack• NodeJS (0.4.9)• CoffeeScript• ZeroMQ
  9. 9. Architecture
  10. 10. InterludeOr joys and pitfalls of building a tail library for node
  11. 11. Living the asynchronous dream
  12. 12. Living the asynchronous dream• Error manifests itself only at high throughput (7MB/secs)• The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)
  13. 13. Breathing in an asynchronous way
  14. 14. Learnings• You need to think asynchronous to write asynchronous code• Dispatch internal events if you need to keep control of the flow• “Internal queues” to scale(up to a certain limit)
  15. 15. Performance• Beyond 5MB we start to fill the internal queue• We are limited by IO, not from CPU/memory• Consistent throughput
  16. 16. Questions ?
  17. 17. References• npm install tail• https://github.com/forward/node-tail

×