• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Firehose
 

Firehose

on

  • 3,062 views

how we built a firehose with NodeJS and ZeroMQ

how we built a firehose with NodeJS and ZeroMQ

Statistics

Views

Total Views
3,062
Views on SlideShare
2,981
Embed Views
81

Actions

Likes
3
Downloads
33
Comments
0

7 Embeds 81

http://coderwall.com 61
http://localhost.dev 14
http://www.linkedin.com 2
http://paper.li 1
http://a0.twimg.com 1
http://localhost 1
https://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Because you can do cool stuff with this !!Previous implementation was with Flume+zeroMQ..flume was unraliable

Firehose Firehose Presentation Transcript

  • FirehoseBuilding a real-time data stream with NodeJS Luca Grulla@Forward London, 2/8/2011
  • IntroductionOr what we wanted to build
  • We like to have a (near) real-time view of the data stream
  • Resilient
  • Scalable
  • Accurate(but not necessarily precise)
  • Verse and Chorus Or how we built it
  • Stack• NodeJS (0.4.9)• CoffeeScript• ZeroMQ
  • Architecture
  • InterludeOr joys and pitfalls of building a tail library for node
  • Living the asynchronous dream
  • Living the asynchronous dream• Error manifests itself only at high throughput (7MB/secs)• The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)
  • Breathing in an asynchronous way
  • Learnings• You need to think asynchronous to write asynchronous code• Dispatch internal events if you need to keep control of the flow• “Internal queues” to scale(up to a certain limit)
  • Performance• Beyond 5MB we start to fill the internal queue• We are limited by IO, not from CPU/memory• Consistent throughput
  • Questions ?
  • References• npm install tail• https://github.com/forward/node-tail