Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Not Only Streams for Akademia JLabs

504 views

Published on

Talk explaining that Reactive is quite a lot of things, and also reactive streams in the end

Published in: Technology
  • Be the first to comment

Not Only Streams for Akademia JLabs

  1. 1. Konrad `ktoso` Malawski @ ITAkademia J-Labs, 2017 Reactive more than just streams
  2. 2. Underlying motto for this talk, and our continued research. “We can do better than that.”
  3. 3. Konrad `ktoso` Malawski Akka Team, Reactive Streams TCK, Persistence, Streams & HTTP, Core, Remoting
  4. 4. Konrad `@ktosopl` Malawski work: akka.io lightbend.com personal blog: http://kto.so communities: geecon.org Java.pl / KrakowScala.pl sckrk.com GDGKrakow.pl lambdakrk.pl
  5. 5. Konrad `@ktosopl` Malawski work: akka.io lightbend.com personal blog: http://kto.so communities: geecon.org Java.pl / KrakowScala.pl sckrk.com GDGKrakow.pl lambdakrk.pl
  6. 6. The concurrent & distributed applications toolkit Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM
  7. 7. The concurrent & distributed applications toolkit Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM
  8. 8. Actors – Concurrency / high perf. messaging Cluster – Location transparent, resilient clusters Persistence – EventSourcing support (many DBs) HTTP – Fully Async & Reactive Http Server …and much more (kafka, cassandra, testing, …)
  9. 9. And the many meanings it carries. Reactive
  10. 10. And the many meanings it carries. Reactive
  11. 11. The many meanings of Reactive reactivemanifesto.org
  12. 12. The many meanings of Reactive
  13. 13. So what are Reactive Streams actually?
  14. 14. So what are Reactive Streams actually?
  15. 15. So what are Reactive Streams actually?
  16. 16. Getting the complete picture
  17. 17. Getting the complete picture (yet… not the topic of today’s talk)
  18. 18. How to think about these techniques? bit.ly/why-reactive
  19. 19. ”The actor model in computer science is a mathematical model of concurrent computation that treats actors as the universal primitives of concurrent computation. ” Wikipedia The Actor Model
  20. 20. An Actor and acts on them by: • Sending messages • Changing its state / behaviour • Creating more actors receives messages
  21. 21. An Actor A concurrency and distribution construct. an addressable, location-transparent, entity Show diagram showing people interacting with each other.
  22. 22. Not Only Server-Client
  23. 23. HTTP/2 “push” still needs a “pull” https://blog.cloudflare.com/announcing-support-for-http-2-server-push-2
  24. 24. Clusters, direct, p2p communication
  25. 25. Clusters, direct, p2p communication
  26. 26. Clusters, direct, p2p communication
  27. 27. Clusters, direct, p2p communication
  28. 28. Not Only JSON “The Internet is running in debug mode.” — Rüdiger Möller http://java-is-the-new-c.blogspot.de/2014/10/why-protocols-are-messy-concept.html
  29. 29. Not Only JSON: Example data MediaContent { media = Media { uri = "http://javaone.com/keynote.mpg" title = "Javaone Keynote" width = 640 height = 480 format = "video/mpg4" duration = 18000000 size = 58982400 bitrate = 262144 persons = ["Bill Gates", "Steve Jobs"] player = JAVA copyright = null } } images = [ Image { uri = "http://javaone.com/keynote_large.jpg" title = "Javaone Keynote" width = 1024 height = 768 size = LARGE } Image { uri = "http://javaone.com/keynote_small.jpg" title = "Javaone Keynote" width = 320 height = 240 size = SMALL } ]
  30. 30. create ser deser total size protostuff 68 433 634 1067ns 239bytes protobuf 121 1173 719 1891 239 kryo-serializer 53 1480 1331 2810 286 thrift 95 1455 731 2186 349 . . . json/jackson/manual 52 1039 1228 2267 468 json/jackson/databind 54 1164 1866 3030 485 json/gson/databind 56 4667 4403 9070 486 xml/xstream+c-aalto 54 3310 6732 10042 525 xml/JAXB 54 4354 141333 145686 719 java-built-in 53 5046 23279 28325 889 Not Only JSON: Benchmarks Slide only to illustrate order-of-magniture differences. Don’t over-focus on numbers. All details here: https://github.com/eishay/jvm-serializers/wiki
  31. 31. Not Only JSON: Benchmarks create ser deser total size protostuff 68 433 634 1067ns 239bytes protobuf 121 1173 719 1891 239 kryo-serializer 53 1480 1331 2810 286 thrift 95 1455 731 2186 349 . . . json/jackson/manual 52 1039 1228 2267 468 json/jackson/databind 54 1164 1866 3030 485 json/gson/databind 56 4667 4403 9070 486 xml/xstream+c-aalto 54 3310 6732 10042 525 xml/JAXB 54 4354 141333 145686 719 java-built-in 53 5046 23279 28325 889 Slide only to illustrate order-of-magniture differences. Don’t over-focus on numbers. All details here: https://github.com/eishay/jvm-serializers/wiki
  32. 32. Not Only JSON: Benchmarks Slide only to illustrate order-of-magniture differences. Don’t over-focus on numbers. All details here: https://github.com/eishay/jvm-serializers/wiki create ser deser total size protostuff 68 433 634 1067ns 239bytes protobuf 121 1173 719 1891 239 kryo-serializer 53 1480 1331 2810 286 thrift 95 1455 731 2186 349 . . . json/jackson/manual 52 1039 1228 2267 468 json/jackson/databind 54 1164 1866 3030 485 json/gson/databind 56 4667 4403 9070 486 xml/xstream+c-aalto 54 3310 6732 10042 525 xml/JAXB 54 4354 141333 145686 719 java-built-in 53 5046 23279 28325 889
  33. 33. Advantages of Async Messaging
  34. 34. Time holding Connection
  35. 35. Time holding Connection
  36. 36. Time holding Connection
  37. 37. Time holding Connection AFAIK: Addressed by HTTP/2, since you can open “subStreams.”
  38. 38. Long running task, sync vs. async
  39. 39. Long running task, sync vs. async
  40. 40. Long running task, sync vs. async
  41. 41. Single Reactive App Any benefits?
  42. 42. Reactive on the Application level https://speakerdeck.com/benjchristensen/applying-rxjava-to-existing-applications-at-philly-ete-2015
  43. 43. “Not-quite-Reactive-System” The reason we started researching into transparent to users flow control.
  44. 44. “Best practices are solutions to yesterdays problems.” https://twitter.com/FrankBuytendijk/status/795555578592555008 Circuit breaking as substitute of flow-control
  45. 45. See also, Nitesh Kant, Netflix @ Reactive Summit https://www.youtube.com/watch?v=5FE6xnH5Lak
  46. 46. See also, Nitesh Kant, Netflix @ Reactive Summit https://www.youtube.com/watch?v=5FE6xnH5Lak
  47. 47. HTTP/1.1 503 Service Unavailable HTTP/1.1 503 Service Unavailable Throttling as represented by 503 responses. Client will back-off… but how? What if most of the fleet is throttling?
  48. 48. http://doc.akka.io/docs/akka/2.4/common/circuitbreaker.html HTTP/1.1 503 Service Unavailable HTTP/1.1 503 Service Unavailable
  49. 49. http://doc.akka.io/docs/akka/2.4/common/circuitbreaker.html
  50. 50. See also, Nitesh Kant, Netflix @ Reactive Summit https://www.youtube.com/watch?v=5FE6xnH5Lak “slamming the breaks”
  51. 51. See also, Nitesh Kant, Netflix @ Reactive Summit https://www.youtube.com/watch?v=5FE6xnH5Lak “slamming the breaks”
  52. 52. See also, Nitesh Kant, Netflix @ Reactive Summit https://www.youtube.com/watch?v=5FE6xnH5Lak “slamming the breaks”
  53. 53. See also, Nitesh Kant, Netflix @ Reactive Summit https://www.youtube.com/watch?v=5FE6xnH5Lak “slamming the breaks”
  54. 54. See also, Nitesh Kant, Netflix @ Reactive Summit https://www.youtube.com/watch?v=5FE6xnH5Lak “slamming the breaks”
  55. 55. See also, Nitesh Kant, Netflix @ Reactive Summit https://www.youtube.com/watch?v=5FE6xnH5Lak We’ll re-visit this specific case in a bit :-) “slamming the breaks”
  56. 56. Are absolutely useful! Still… “We can do better than that.” Circuit Breakers
  57. 57. But we’ll need everyone to understand some shared semantics… We can do better.
  58. 58. A fundamental building block. Not end-user API by itself. reactive-streams.org Reactive Streams
  59. 59. Reactive Streams More of an SPI (Service Provider Interface), than API. reactive-streams.org
  60. 60. “Stream”
  61. 61. “Stream” What does it mean?!
  62. 62. Suddenly everyone jumped on the word “Stream”. Akka Streams / Reactive Streams started end-of-2013. “Streams” * when put in “” the word does not appear in project name, but is present in examples / style of APIs / wording.
  63. 63. Suddenly everyone jumped on the word “Stream”. Akka Streams / Reactive Streams started end-of-2013. “Streams” Akka Streams Reactive Streams RxJava “streams”* Spark Streaming Apache Storm “streams”* Java Steams (JDK8) Reactor “streams”* Kafka Streams ztellman / Manifold (Clojure) * when put in “” the word does not appear in project name, but is present in examples / style of APIs / wording. Apache GearPump “streams” Apache [I] Streams (!) Apache [I] Beam “streams” Apache [I] Quarks “streams” Apache [I] Airflow “streams” (dead?) Apache [I] Samza Scala Stream Scalaz Streams, now known as FS2 Swave.io Java InputStream / OutputStream / … :-)
  64. 64. The specification. Reactive Streams Origins of
  65. 65. Reactive Streams – why it all started
  66. 66. Reactive Streams – why it all started
  67. 67. What is back-pressure? ?
  68. 68. What is back-pressure?
  69. 69. No no no…! Not THAT Back-pressure! Also known as: flow control. What is back-pressure?
  70. 70. No no no…! Not THAT Back-pressure! Also known as: application level flow control. What is back-pressure?
  71. 71. Reactive Streams - story: 2013’s impls ~2013: Reactive Programming becoming widely adopted on JVM. - Play introduced “Iteratees” - Akka (2009) had Akka-IO (TCP etc.) - Ben starts work on RxJava http://blogs.msdn.com/b/rxteam/archive/2009/11/17/announcing-reactive-extensions-rx-for-net-silverlight.aspx http://infoscience.epfl.ch/record/176887/files/DeprecatingObservers2012.pdf - Ingo Maier, Martin Odersky https://github.com/ReactiveX/RxJava/graphs/contributors https://github.com/reactor/reactor/graphs/contributors https://medium.com/@viktorklang/reactive-streams-1-0-0-interview-faaca2c00bec#.69st3rndy Teams discuss need for back-pressure in simple user API. Play’s Iteratee / Akka’s NACK in IO. }
  72. 72. Reactive Streams - story: 2013’s impls 2014–2015: Reactive Streams Spec & TCK development, and implementations. 1.0 released on April 28th 2015, with 5+ accompanying implementations. 2015 Proposed to be included with JDK9 by Doug Lea via JEP-266 “More Concurrency Updates” http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/6e50b992bef4/src/java.base/share/classes/java/util/concurrent/Flow.java
  73. 73. Reactive Streams - story: 2013’s impls 2014–2015: Reactive Streams Spec & TCK development, and implementations. 1.0 released on April 28th 2015, with 5+ accompanying implementations. 2015 Proposed to be included with JDK9 by Doug Lea via JEP-266 “More Concurrency Updates” http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/6e50b992bef4/src/java.base/share/classes/java/util/concurrent/Flow.java
  74. 74. But what does it do!? Reactive Streams
  75. 75. Publisher[T] Subscriber[T] Back-pressure explained
  76. 76. Fast Publisher Slow Subscriber Push model
  77. 77. Subscriber usually has some kind of buffer. Push model
  78. 78. Push model
  79. 79. Push model
  80. 80. What if the buffer overflows? Push model
  81. 81. Use bounded buffer, drop messages + require re-sending Push model
  82. 82. Kernel does this! Routers do this! (TCP) Use bounded buffer, drop messages + require re-sending Push model
  83. 83. Increase buffer size… Well, while you have memory available! Push model
  84. 84. Push model
  85. 85. Reactive Streams explained Reactive Streams explained in 1 slide
  86. 86. Fast Publisher will send at-most 3 elements. This is pull-based-backpressure. Reactive Streams: “dynamic push/pull”
  87. 87. JEP-266 – soon…! public final class Flow { private Flow() {} // uninstantiable @FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); } public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); } public static interface Subscription { public void request(long n); public void cancel(); } public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> { } }
  88. 88. JEP-266 – soon…! public final class Flow { private Flow() {} // uninstantiable @FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); } public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); } public static interface Subscription { public void request(long n); public void cancel(); } public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> { } } Single basic (helper) implementation available in JDK: SubmissionPublisher
  89. 89. Reactive Streams: goals 1) Avoiding unbounded buffering across async boundaries 2)Inter-op interfaces between various libraries
  90. 90. Reactive Streams: goals 1) Avoiding unbounded buffering across async boundaries 2)Inter-op interfaces between various libraries Argh, implementing a correct RS Publisher or Subscriber is so hard!
  91. 91. 1) Avoiding unbounded buffering across async boundaries 2)Inter-op interfaces between various libraries Reactive Streams: goals Argh, implementing a correct RS Publisher or Subscriber is so hard!
  92. 92. Reactive Streams: goals Argh, implementing a correct RS Publisher or Subscriber is so hard! You should be using Akka Streams instead! 1) Avoiding unbounded buffering across async boundaries 2)Inter-op interfaces between various libraries
  93. 93. Already made a huge industry impact Reactive Streams
  94. 94. Back-pressure as a feature
  95. 95. Inspiring other technologies
  96. 96. Inspiring other technologies It’s been a while since Java inspired other modern technologies, hasn’t it?
  97. 97. The implementation. Complete and awesome Java and Scala APIs. As everything since day 1 in Akka. Akka Streams
  98. 98. Akka Streams in 20 seconds: // types: Source[Out, Mat] Flow[In, Out, Mat] Sink[In, Mat] // generally speaking, it's always: val ready = Source.from…(???).via(flow).map(i => i * 2).to(sink) val mat: Mat = ready.run() // the usual example: val f: Future[String] = Source.single(1).map(i => i.toString).runWith(Sink.head) Proper static typing!
  99. 99. Akka Streams in 20 seconds: // types: _ Source[Int, NotUsed] Flow[Int, String, NotUsed] Sink[String, Future[String]] Source.single(1).map(i => i / 0).runWith(Sink.head)
  100. 100. Source.single(1).map(i => i.toString).runWith(Sink.head) // types: _ Source[Int, NotUsed] Flow[Int, String, NotUsed] Sink[String, Future[String]] Akka Streams in 20 seconds:
  101. 101. Materialization Gears from GeeCON.org, did I mention it’s an awesome conf?
  102. 102. What is “materialization” really?
  103. 103. What is “materialization” really?
  104. 104. What is “materialization” really?
  105. 105. What is “materialization” really?
  106. 106. Akka Streams & HTTP streams & HTTP
  107. 107. A core feature not obvious to the untrained eye…! Quiz time! TCP is a ______ protocol? Akka Streams & HTTP
  108. 108. A core feature not obvious to the untrained eye…! Quiz time! TCP is a STREAMING protocol! Akka Streams & HTTP
  109. 109. Streaming in Akka HTTP DEMO HttpServer as a: Flow[HttpRequest, HttpResponse] http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” https://github.com/akka/akka/pull/20778
  110. 110. Streaming in Akka HTTP DEMO HttpServer as a: Flow[HttpRequest, HttpResponse] HTTP Entity as a: Source[ByteString, _] http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” https://github.com/akka/akka/pull/20778
  111. 111. Streaming in Akka HTTP DEMO http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” https://github.com/akka/akka/pull/20778 HttpServer as a: Flow[HttpRequest, HttpResponse] HTTP Entity as a: Source[ByteString, _] Websocket connection as a: Flow[ws.Message, ws.Message]
  112. 112. It’s turtles buffers all the way down!
  113. 113. Streaming from Akka HTTP
  114. 114. Streaming from Akka HTTP
  115. 115. Streaming from Akka HTTP
  116. 116. Streaming from Akka HTTP (Scala) object Example extends App with SprayJsonSupport with DefaultJsonProtocol { import akka.http.scaladsl.server.Directives._ implicit val system = ActorSystem() implicit val mat = ActorMaterializer() implicit val jsonRenderingMode = EntityStreamingSupport.json() implicit val TweetFormat = jsonFormat1(Tweet) def tweetsStreamRoutes = path("tweets") { complete { Source.repeat(Tweet("")) } } Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080) System.out.println("Running at http://localhost:8080"); }
  117. 117. Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system); final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world")); final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) ); final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer); http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080"); }
  118. 118. Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system); final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world")); final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) ); final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer); http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080"); }
  119. 119. Ecosystem that solves problems > (is greater than) solving all the problems ourselves
  120. 120. Codename: Alpakka // these are “Alpacasso”
  121. 121. A community for Streams connectors Alpakka – a community for Stream connectors Alp
  122. 122. Alpakka – a community for Stream connectors Threading & Concurrency in Akka Streams Explained (part I) Mastering GraphStages (part I, Introduction) Akka Streams Integration, codename Alpakka A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II) Writing Akka Streams Connectors for existing APIs Flow control at the boundary of Akka Streams and a data provider Akka Streams Kafka 0.11
  123. 123. Alpakka – a community for Stream connectors Existing examples: MQTT / AMQP Streaming HTTP Streaming TCP Streaming FileIO Amazon S3, SQS & more Cassandra Queries Reactive Kafka (akka-stream-kafka) S3, SQS & other Amazon APIs Streaming JSON Parsing Streaming XML Parsing
  124. 124. Reactive Streams / Akka Streams Is now the time to adopt?
  125. 125. Totally, go for it.
  126. 126. ReactiveSocket.io Taking it to the next level:
  127. 127. ReactiveSocket.io Taking it to the next level: A collaboration similar in spirit, and continuing from where Reactive Streams brought us today.
  128. 128. Reactive Streams over network boundaries http://reactivesocket.io/ Reactive Streams = async boundaries Reactive Socket = RS + network boundaries
  129. 129. Reactive Streams over network boundaries http://reactivesocket.io/ Reactive Streams = async boundaries Reactive Socket = RS + network boundaries Primarily led by: - Ben Christensen, Todd Montgomery (Facebook) & team - Nitesh Kant (Netflix) & team Lightbend on board as well – right now we’re prototyping with it.
  130. 130. Reactive Streams over network boundaries “ReactiveSocket is an application protocol providing Reactive Streams semantics over an asynchronous, binary boundary.” http://reactivesocket.io/
  131. 131. Reactive Streams over network boundaries Binary Support various platforms: - java - c++ - js - …
  132. 132. Reactive Streams over network boundaries Async Obviously we want it to be async and properly bi-directional.
  133. 133. Reactive Streams over network boundaries Application protocol Again, bridging app-level semantics to wire semantics. Reactive Streams semantics: - “you can do 10 requests”
  134. 134. Reactive Streams over network boundaries Application protocol Again, bridging app semantics to wire semantics. Reactive Streams semantics: - “you can do 10 requests” Extra Lease semantics: - “you can do 10 reqs in 30secs”
  135. 135. Lease semantics, “flipping the problem”
  136. 136. Lease semantics, “flipping the problem”
  137. 137. Lease semantics, “flipping the problem”
  138. 138. Exciting times ahead!
  139. 139. State and Future[_] of Reactive Reactive Systems – well established “goal” architecture …excellent building blocks available, and getting even better with: Reactive-Streams eco-system blooming! … as very important building block of the puzzle. Akka Streams driving implementation of Reactive Streams (first passing TCK, prime contributor to spec, strong ecosystem) Reactive Socket continuing to improve app-level flow-control semantics. More control than “just use HTTP/2”. … considering resumability for streams as well. The best is yet to come: combining all these components into resilient, scalable systems!
  140. 140. Happy hAkking!
  141. 141. We <3 contributions • Easy to contribute: • https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute • https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22 • Akka: akka.io && github.com/akka • Reactive Streams: reactive-streams.org • Reactive Socket: reactivesocket.io • Mailing list: • https://groups.google.com/group/akka-user • Public chat rooms: • http://gitter.im/akka/dev developing Akka • http://gitter.im/akka/akka using Akka
  142. 142. Pics Gundam pictures from: http://www.wallpaperup.com/tag/gundam/3
  143. 143. Free e-book and printed report. bit.ly/why-reactive Covers what reactive actually is. Implementing in existing architectures. Thoughts from the team that’s building reactive apps since more than 6 years. Obligatory “read my book!” slide :-)
  144. 144. Q/A ktoso @ lightbend.com twitter: ktosopl github: ktoso team blog: blog.akka.io home: akka.io myself: kto.so

×