Firehose

3,244 views

Published on

how we built a firehose with NodeJS and ZeroMQ

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,244
On SlideShare
0
From Embeds
0
Number of Embeds
115
Actions
Shares
0
Downloads
35
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Because you can do cool stuff with this !!Previous implementation was with Flume+zeroMQ..flume was unraliable
  • Firehose

    1. 1. FirehoseBuilding a real-time data stream with NodeJS Luca Grulla@Forward London, 2/8/2011
    2. 2. IntroductionOr what we wanted to build
    3. 3. We like to have a (near) real-time view of the data stream
    4. 4. Resilient
    5. 5. Scalable
    6. 6. Accurate(but not necessarily precise)
    7. 7. Verse and Chorus Or how we built it
    8. 8. Stack• NodeJS (0.4.9)• CoffeeScript• ZeroMQ
    9. 9. Architecture
    10. 10. InterludeOr joys and pitfalls of building a tail library for node
    11. 11. Living the asynchronous dream
    12. 12. Living the asynchronous dream• Error manifests itself only at high throughput (7MB/secs)• The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)
    13. 13. Breathing in an asynchronous way
    14. 14. Learnings• You need to think asynchronous to write asynchronous code• Dispatch internal events if you need to keep control of the flow• “Internal queues” to scale(up to a certain limit)
    15. 15. Performance• Beyond 5MB we start to fill the internal queue• We are limited by IO, not from CPU/memory• Consistent throughput
    16. 16. Questions ?
    17. 17. References• npm install tail• https://github.com/forward/node-tail

    ×