Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Streaming, effects and beautiful folds: a winning trilogy

1,379 views

Published on

Streaming libraries are kind of complex. Why is that so?
They have to handle many different concerns: streaming, composition, concurrency, resources management,... Is it possible to isolate those concerns into smaller, composable libraries?

Published in: Technology
  • Be the first to comment

Streaming, effects and beautiful folds: a winning trilogy

  1. 1. Streams, effects and Beautiful folds Eric Torreborre
  2. 2. About Zalando
  3. 3. Every application ever written can be viewed as some sort of transformation on data. Daniel Spiewak
  4. 4. ???
  5. 5. Streams! or iteratees... or observables...
  6. 6. data ConduitM i o m r This type represents a general component which can • consume a stream of input values i • produce a stream of output values o • perform actions in the m monad • and produce a final result r
  7. 7. type Pipe a b = Proxy () a () b A Proxy is a monad transformer that receives and sends information on both an upstream and downstream interface data Proxy a' a b' b m r
  8. 8. Machines are demand-driven coroutines ... they are entirely pure. A Machine can be driven by a Driver which can have effects when feeding the Machine input or when reading its output • you can construct a `Plan` and then `compile` it to a `Machine` • a `Plan[K, O, A]` is a specification for a pure `Machine` that reads inputs selected by `K` and writes values of type `O` • the `Plan` has intermediate results of type `A` which are placeholders for further plans trait Plan[+K, +O, +A] type Machine[+K, +O] = Plan[K, O, Nothing]
  9. 9. Motivation • data too large • actors offer no static guarantee • back-pressure
  10. 10. class Flow[-In, +Out, +Mat](m: Module) extends Graph[FlowShape[In, Out], Mat] A Flow is a set of stream processing steps that has one open input and one open output. Materialized: 'a value that can be obtained from the materialized processing pipeline'
  11. 11. The design goals are • compositionality • expressiveness • resource safety • speed
  12. 12. StreamCore / Stack Segment (Fail, Emit, Handler, Append) Scope Chunk / Pull Stream[+F[_],+O](core: CoreRef[F,O]) A stream producing output of type `O` and which may evaluate `F` effects
  13. 13. Motivation • high-performance • lightweight • conceptually simple • debuggable • type-safe and concise
  14. 14. class Pipe[-A, +B] extends StreamOps[B] Simple transformations are stream graph components with one input port and one output port trait StreamOps[A] { type Repr[T] <: StreamOps[T] { type Repr[X] <: self.Repr[X] } } } FanIn, Spout, shapeless
  15. 15. Motivation • resource management • streaming • fusion • error recovery • not for concurrency
  16. 16. Motivation • asynchronous • reactive • streaming • back-pressure
  17. 17. class Observable[A] class Consumer[A, B] extends (Observable[A] => Task[B]) • Observable implements the "Reactive Pattern" = subscribe Subscribers • Consumer is a specification of how to consume an Observable to return a result
  18. 18. Differences?
  19. 19. Streaming Composition Safety Concurrency Backpressure Topology
  20. 20. Producer EffectsFolds
  21. 21. Streaming Composition Backpressure
  22. 22. ...
  23. 23. Create
  24. 24. Transform
  25. 25. Monad
  26. 26. Transform
  27. 27. Nice way to the room?
  28. 28. Run
  29. 29. Streaming Composition Backpressure
  30. 30. Streaming Composition Topology
  31. 31. def count(): current = 0 while True: (yield current) current = current + 1
  32. 32. c = count() next(c) print(c.send(1)) // 1 print(c.send(1)) // 2 print(c.send(1)) // 3 c.close()
  33. 33. Run
  34. 34. Folds count minimum maximum all any headlast firstN lastN plus times varianceaverage quantiles countMinSketch
  35. 35. Applicative
  36. 36. Topology
  37. 37. Sinks
  38. 38. Streaming Composition Topology
  39. 39. Safety Concurrency Backpressure
  40. 40. TimedFutureEffect TaskEffect TaskEffect TwitterFutureEffect TaskEffect
  41. 41. Short-circuit
  42. 42. Safety
  43. 43. Safety Concurrency Backpressure
  44. 44. Backpressure
  45. 45. Merging queues
  46. 46. Producer EffectsFolds
  47. 47. tricky
  48. 48. Producer Fold Safe? Safe Producer
  49. 49. tricky II

×