Firehose

FirehoseBuilding a real-time data stream with NodeJS Luca Grulla@Forward London, 2/8/2011

IntroductionOr what we wanted to build

We like to have a (near) real-time view of the data stream

Resilient

Scalable

Accurate(but not necessarily precise)

Verse and Chorus Or how we built it

Stack• NodeJS (0.4.9)• CoffeeScript• ZeroMQ

Architecture

InterludeOr joys and pitfalls of building a tail library for node

Living the asynchronous dream

Living the asynchronous dream• Error manifests itself only at high throughput (7MB/secs)• The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)

Breathing in an asynchronous way

Learnings• You need to think asynchronous to write asynchronous code• Dispatch internal events if you need to keep control of the flow• “Internal queues” to scale(up to a certain limit)

Performance• Beyond 5MB we start to fill the internal queue• We are limited by IO, not from CPU/memory• Consistent throughput

Questions ?

References• npm install tail• https://github.com/forward/node-tail

http://coderwall.com	37
http://paper.li	1
http://a0.twimg.com	1

Firehose

by Luca Grulla on Nov 25, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 39

Statistics

Firehose — Presentation Transcript