Downloaded 12 times
Reactive Streams
- 1. Reactive Streams Non-blocking asynchronousstream processing with back-pressure Jordan Parmer @profparmer
- 2.
- 3. “Common mechanism to move dataacross an asynchronous boundary without losses, buffering, or resource exhaustion3
- 4. Common Use Cases ▪Data pipeline processing ▫ Unbounded data volume ▪ Message queue streaming ▪ Network streaming ▪ File streaming 4
- 5.
- 6.
- 7. Reactive Programming Reactive Systems ▪Application-view ▪ CPU’s, threads, local resources ▪ Per-process execution ▪ Event-driven ▪ Systems-view ▪ Process interaction ▪ Cohesion of parts ▪ Message-driven 7 https://www.oreilly.com/ideas/reactive-programming-vs-reactive-systems
- 8. Reactive Programming Reactive Systems ▪Application-view ▪ CPU’s, threads, local resources ▪ Per-process execution ▪ Event-driven ▪ Systems-view ▪ Process interaction ▪ Cohesion of parts ▪ Message-driven 8 https://www.oreilly.com/ideas/reactive-programming-vs-reactive-systems We are talking about this
- 9. How We UseIt 9
- 10. Oseberg Use Case 10 KafkaKafkaMicro-Service Messages Messages
- 11. Oseberg Use Case 11 KafkaKafkaMicro-Service Messages Messages Source SinkTransformations Reactive Programming Workflow
- 12. Oseberg Use Case 12 KafkaSvc Kafka Svc Svc SvcKafka DB DB Each service internally uses Reactive Streams API to read from Kafka, transform messages, write to storage (if necessary), and write to Kafka.
- 13.
- 14. General Workflow 14 Source or Publisher Sinkor Subscriber
- 15. General Workflow 15 Source or Publisher Sinkor Subscriber Connected to collection, stream, iterator, etc. Lazily emits records on demand Subscribes to upstream flow and gathers records Produces demand Series of chained operations applied to records
- 16. Push-Based Systems Without Back-Pressure 16 Fast Publisher Slow Subscriber Pushrecords downstream 1000 ops/second 10 ops/second
- 17. Push-Based Systems Without Back-Pressure 17 Fast Publisher Slow Subscriber ●Must manually buffer and/or throttle ● Out-of-Memory risk 1000 ops/second 10 ops/second Push records downstream
- 18. Push-Based Systems Without Back-Pressure 18 Fast Publisher Slow Subscriber 100ops/second 1 op/second Flow of pushed records What if buffer overflows? https://www.lightbend.com/blog/understanding-akka-streams-back-pressure-and-asynchronous-architectures
- 19. Push-Based Systems Without Back-Pressure 19 Fast Publisher Slow Subscriber 100ops/second 1 op/second Flow of pushed records Drop messages and re-send? https://www.lightbend.com/blog/understanding-akka-streams-back-pressure-and-asynchronous-architectures
- 20. Push-Based Systems Without Back-Pressure 20 Fast Publisher Slow Subscriber 100ops/second 1 op/second Flow of pushed records Extend buffer size? https://www.lightbend.com/blog/understanding-akka-streams-back-pressure-and-asynchronous-architectures
- 21. Push-Based Systems Without Back-Pressure 21 Fast Publisher Slow Subscriber 100ops/second 1 op/second Flow of pushed records Extend buffer size? https://www.lightbend.com/blog/understanding-akka-streams-back-pressure-and-asynchronous-architectures OOM PENDING!!!
- 22. Reactive With Back-Pressure 22 Fast Publisher Slow Subscriber Pull-basedon demand If possible, publisher only emits on pull
- 23. Reactive With Back-Pressure 23 Fast Publisher Slow Subscriber Pull-basedon demand If possible, publisher only emits on pull Request/signal for more records (aka demand)
- 24. Reactive With Back-Pressure 24 Fast Publisher Slow Subscriber Pull-basedon demand If possible, publisher only emits on pull Request/signal for more records (aka demand) Respond with more records
- 25. Reactive With Back-Pressure 25 Fast Publisher Slow Subscriber Request/signalfor 2 more records Respond up to 2 more records Demand
- 26.
- 27. Composability 27 Images fromhttp://doc.akka.io/docs/akka/current/scala/stream/stream-composition.html
- 28. Composability 28 Images fromhttp://doc.akka.io/docs/akka/current/scala/stream/stream-composition.html
- 29. Composability 29 Images fromhttp://doc.akka.io/docs/akka/current/scala/stream/stream-composition.html
- 30.
- 31. Asynchronous Boundaries 31 Source SinkFlowFlow Flow Async Boundaries
- 32. Asynchronous Boundaries 32 Source SinkFlowFlow Flow Async Boundaries Actor Actor Actor
- 33.
- 34.
- 35.
- 36.
- 37.
- 38. Questions? Twitter @profparmer 38 SlideDesign by Slides Carnival Creative Commons Attribution License
Editor's Notes
- #2 How many of you have used Reactive Streams? How many of you work with data processing? ETL? Big data techniques? Network communication?
- #4 What are some techniques or challenges streaming data today? Requires lots of boiler-plate Tricky to manage system resources What happens when the stream is interrupted or fails on delivery?