Recommended
More Related Content
What's hot
What's hot (11)
Viewers also liked
Viewers also liked (19)
Similar to LNUG January 2013 Streaming Data Windows
Similar to LNUG January 2013 Streaming Data Windows (20)
Recently uploaded
Recently uploaded (20)
LNUG January 2013 Streaming Data Windows
- 1. LNUG London January 2013 Copyright Push Technology 2012
- 2. • Distributed Systems / HPC guy. • Chief Scien*st :-‐ at Push Technology • Responds to: Guinness, Whisky • TwiOer: @darachennis About me? Copyright Push Technology 2012 Darach@PushTechnology.com
- 3. Streamy Pipy Analy*cy Copyright Push Technology 2012
- 4. EEP + ‘Streams & Pipes’= CEP w w S C Q w w eep.js: Func*onal Opera*ons on Streaming Data Windows • An experiment in Embedded Event Processing • Sliding, Tumbling, Monotonic and Periodic windows • Separate ‘window’ definiYon from operaYon • Aggregate funcYons. Window of data produces scalar result • But? No filtering, branching or combinators, no flows … • That’s a job for Streams & Pipes. Let’s add that. Copyright Push Technology 2012
- 5. Windows Copyright Push Technology 2012
- 6. Windows + Aggregate FuncYons What is this? Example • A window of data is a slice of data over Yme, number of • Average – Aggregate Func*on events or some other dimension • CPU – Data (events) • An aggregate funcYon is something you do in the context of • On a second by second basis a window. -‐ Periodic Yme window Copyright Push Technology 2012
- 7. Tumbling Windows x() x() x() x() emit() x() x() x() x() emit() 1 2 3 4 x() x() x() x() emit() 2 3 4 5 init() 2 3 4 5 init() init() t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 ... What is a tumbling window? • Every N events, give me an average of the last N events • Does not overlap windows • ‘Closing’ a window, ‘Emits’ a result (the average) • Closing a window, Opens a new window Copyright Push Technology 2012
- 8. Sliding Windows init() 1 2 3 4 5 .. .. .. .. x() 1 2 3 4 .. .. .. .. x() 1 2 3 .. .. .. .. x() 1 2 .. .. .. .. t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 ... What is a sliding window? • Like tumbling, except can overlap. • But typically O(N2), Keep N small. Except EEP.js. O(N) perf. • Every event opens a new window. • Ader N events, every subsequent event emits a result. • Like all windows, cost of calculaYon amorYzed over events Copyright Push Technology 2012
- 9. Periodic Windows x() x() x() x() emit() x() x() x() x() emit() 1 2 3 4 x() x() x() x() emit() 2 3 4 5 init() 2 3 4 5 init() init() t0 t1 t2 t3 ... What is a periodic window? • Driven by ‘wall clock Yme’ in milliseconds • Not monotonic, natch. Beware of NTP Copyright Push Technology 2012
- 10. Monotonic Windows my my my x() x() x() x() emit() x() x() x() x() emit() 1 2 3 4 x() x() x() x() emit() 2 3 4 5 init() 2 3 4 5 init() init() t0 t1 t2 t3 ... What is a monotonic window? • Driven mad by ‘wall clock Yme’? Need a logical clock? • No worries. Provide your own clock! Eg. Vector clock Copyright Push Technology 2012
- 11. Slide beOer with CompensaYng Aggregates init() 1 2 3 4 5 .. .. .. .. x() 1 2 3 4 .. .. .. .. do { x() 1 2 3 … .. .. .. .. } while (…) x() 1 2 .. .. .. .. 1 compensate() t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 ... Copyright Push Technology 2012
- 12. Bad Sliding -‐ O(N2) Copyright Push Technology 2012
- 13. Good Sliding • Takes us from O(N2) to O(N) for Sliding windows Copyright Push Technology 2012
- 14. EEP.js is fast Copyright Push Technology 2012
- 15. Using Sliding, Tumbling Windows Copyright Push Technology 2012
- 16. Using Periodic, Monotonic Windows Copyright Push Technology 2012
- 17. Custom clocks (noYon of Yme) Copyright Push Technology 2012
- 18. EEP.js v0.1, v0.2 were ugly babies. Sorry! Swear, the next version will be just as funcYonal but preOy… Copyright Push Technology 2012
- 19. Streams & Pipes Copyright Push Technology 2012
- 20. What about Streams & Pipes? ???? S C Q + w w eep w w Copyright Push Technology 2012
- 21. Streams & Pipes: Origins • Do one thing. Do it well • Compose sophisYcated behaviors from simple parts • Maximize reuse • Unix, ‘Chain of Responsibility’ (GoF), Interceptor (POSA2), XPipe, Builder, … • The ‘Assembly Line Principle’ is nothing new Copyright Push Technology 2012
- 22. Streams & Pipes: Node.JS • var events = require(‘events’) • Publish/Subscribe to event (streams) • var stream = require(‘stream’) • Readable – Consume a (finite) set of events • Writable – Produce a (finite) set of events • readable.pipe(writeable) • writeable.pipe(readable) Copyright Push Technology 2012
- 23. Streams & Pipes: streams2 • Transform – Compress, Encrypt, Encode, … • Duplex – Readable and Writable • Passthrough – The canonical ‘noop’ transform • Node.js Streams history (so far) hOp://bit.ly/XupqkO -‐ by @izs Copyright Push Technology 2012
- 24. Streams & Pipes: but … • Oriented for IO, not compute/analy*cs • Array-‐like buffers not individual datums • @dominictarr event-‐streams? Array based • ASCII, UTF-‐8, Binary -‐ not JS types • Oden require copying, parsing, … (slow) • So, streams & pipes for JS types? Yes! • Do one thing. Do it well • Compose sophisYcated simple parts • Maximize reuse Copyright Push Technology 2012
- 25. Introducing Beam.js Copyright Push Technology 2012
- 26. Beams, Pipes • Streams & Pipes for analyYcs • Not designed for IO. Use Streams for that • Not concerned with CEP. • … Use EEP for that? J • Not concerned with arrays of things • … Use Dominic Tarr’s event-‐stream for that • Beam • Crunch events • Pipeline, Branch & Combine Copyright Push Technology 2012
- 27. Beams & Pipes. • Streams & Pipes, reconsidered for JS types • var Beam = require(‘beam’); • Beam.Source -‐-‐ Push data in • Beam.Sink -‐-‐ Suck analysis out • Beam.Operator -‐-‐ OODA / PDCA • Really Simple: ~150 LOC Copyright Push Technology 2012
- 28. Beams & Pipes: Operators • Three types of operator • Transform • 1 in, 1 out. Output data/type may differ • Filter • 1 in, 1 or none out. Output data/type same as input • Custom • May transform, filter Copyright Push Technology 2012
- 29. Example: Defini*ons Copyright Push Technology 2012
- 30. Example: Usage Copyright Push Technology 2012
- 31. Example: Easy to debug … Copyright Push Technology 2012
- 32. Example: Streams & Beams Copyright Push Technology 2012
- 33. Branch • You can define 1 or many • They can overlap or not as you see fit • It’s just an applicaYon of predicate (boolean) filters • Simple Copyright Push Technology 2012
- 34. Combine? • You can combine many sources or branches into one • Works like a union. First in, first out. • You can write your own. It’s just an Operator • You can branch from, combine to … any beam Copyright Push Technology 2012
- 35. Streams & Pipes, ++ • In Node.js the definiYon and usage of streams in a pipe are entangled. • Typically, with Streams & Pipes for IO, you only ever want one. • In algorithms you may want to reuse. • Think about it … • Event EmiOer. 1 square … 2 branches? Copyright Push Technology 2012
- 36. Pipes ++ • Beam Pipes are different (& really really really simple) • You can define a filter once • You can store it in a module • Store like opera*ons together • Make libraries • Use ‘em. Share ‘em. Copyright Push Technology 2012
- 37. EEP based on Beam soon! Copyright Push Technology 2012
- 38. Un*l then? • npm install beam • Filter data events • Transform data events • Analyze, crunch all the things • Branch all the things • Combine all the things Copyright Push Technology 2012
- 39. Beam futures? • Taps – Convert events into beams • Drain – Convert beams into events • Beams • Write Beam operators in ‘beam’ • Beams ‘inside’ beams • Source.pipe(op).compile(); // Maybe? Copyright Push Technology 2012
- 40. Ques*ons Copyright Push Technology 2012
- 41. • Thank you for listening to, having me • Le twiOer: @darachennis • hOps://github.com/darach/beam-‐js hOps://github.com/darach/eep-‐js • npm install eep npm install beam • EEP built on beam? EEP in other langs? Soon • Fork it, Port it, Enjoy it! QuesYons? Copyright Push Technology 2012 Darach@PushTechnology.com