Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

10,265 views

Published on

Since its stable release in 2016, Akka Streams is quickly becoming the de facto standard integration layer between various Streaming systems and products. Enterprises like PayPal, Intel, Samsung and Norwegian Cruise Lines see this is a game changer in terms of designing Reactive streaming applications by connecting pipelines of back-pressured asynchronous processing stages.

This comes from the Reactive Streams initiative in part, which has been long led by Lightbend and others, allowing multiple streaming libraries to inter-operate between each other in a performant and resilient fashion, providing back-pressure all the way. But perhaps even more so thanks to the various integration drivers that have sprung up in the community and the Akka team—including drivers for Apache Kafka, Apache Cassandra, Streaming HTTP, Websockets and much more.

In this webinar for JVM Architects, Konrad Malawski explores the what and why of Reactive integrations, with examples featuring technologies like Akka Streams, Apache Kafka, and Alpakka, a new community project for building Streaming connectors that seeks to “back-pressurize” traditional Apache Camel endpoints.

* An overview of Reactive Streams and what it will look like in JDK 9, and the Akka Streams API implementation for Java and Scala.

* Introduction to Alpakka, a modern, Reactive version of Apache Camel, and its growing community of Streams connectors (e.g. Akka Streams Kafka, MQTT, AMQP, Streaming HTTP/TCP/FileIO and more).

* How Akka Streams and Akka HTTP work with Websockets, HTTP and TCP, with examples in both in Java and Scala.

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

  1. 1. akka streams, Exploring Reactive Integrations with Alpakka and Kafka Konrad `ktoso` Malawski
  2. 2. Konrad `ktoso` Malawski Akka Team, Reactive Streams TCK, Persistence, HTTP, Remoting / Cluster
  3. 3. Make building powerful concurrent & distributed applications simple. Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM
  4. 4. Actors – simple & high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community Complete Java & Scala APIs for all features What’s in the toolkit?
  5. 5. “Stream” has many meanings
  6. 6. akka streams Asynchronous back pressured stream processing Source Sink Flow
  7. 7. akka streams Asynchronous back pressured stream processing Source Sink (possible) asynchronous boundaries Flow
  8. 8. akka streams Asynchronous back pressured stream processing Source Sink 10 msg/s 1 msg/s OutOfMemoryError!! Flow
  9. 9. akka streams Asynchronous back pressured stream processing Source Sink 10 msg/s 1 msg/s hand me 3 morehand me 3 more 1 msg/s Flow
  10. 10. akka streams Not only linear streams Source SinkFlow Source Sink Flow Flow
  11. 11. And the many meanings it carries. Reactive
  12. 12. The many meanings of Reactive reactivemanifesto.org
  13. 13. The many meanings of Reactive
  14. 14. “Not-quite-Reactive-System” The reason we started researching into transparent to users flow control.
  15. 15. Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols http://www.reactive-streams.org
  16. 16. Reactive Streams A buiding-block of Reactive Systems, not the “entire story”.
  17. 17. Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols http://www.reactive-streams.org
  18. 18. Part of JDK 9 java.util.concurrent.Flow http://openjdk.java.net/projects/jdk9/
  19. 19. JEP-266 – soon…! public final class Flow { private Flow() {} // uninstantiable @FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); } public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); } public static interface Subscription { public void request(long n); public void cancel(); } public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> { } }
  20. 20. Reactive Streams RS Library A RS library B async boundary
  21. 21. Reactive Streams RS Library A RS library B async boundary Make building powerful concurrent & distributed applications simple.
  22. 22. The API Akka Streams Complete and awesome Java and Scala APIs (Just like everything in Akka)
  23. 23. Akka Streams in 20 seconds: Source<Integer, NotUsed> source = null;
 
 Flow<Integer, String, NotUsed> flow =
 Flow.<Integer>create().map((Integer n) -> n.toString());
 
 Sink<String, CompletionStage<Done>> sink =
 Sink.foreach(str -> System.out.println(str));
 
 RunnableGraph<NotUsed> runnable = source.via(flow).to(sink);
 
 runnable.run(materializer);

  24. 24. Akka Streams in 20 seconds: CompletionStage<String> firstString =
 Source.single(1)
 .map(n -> n.toString())
 .runWith(Sink.head(), materializer);

  25. 25. Source.single(1).map(i -> i.toString).runWith(Sink.head()) // types: _ Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>> Akka Streams in 20 seconds:
  26. 26. Source.single(1).map(i -> i.toString).runWith(Sink.head()) // types: _ Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>> Akka Streams in 20 seconds:
  27. 27. Materialization Gears from GeeCON.org,(it’s an awesome conf)
  28. 28. What is “materialization” really?
  29. 29. What is “materialization” really?
  30. 30. What is “materialization” really?
  31. 31. Introspection, and more coming… Highly sophisticated stream introspection capabilities.
  32. 32. AlpakkaA community for Streams connectors http://blog.akka.io/integrations/2016/08/23/intro-alpakka
  33. 33. Alpakka – a community for Stream connectors Threading & Concurrency in Akka Streams Explained (part I) Mastering GraphStages (part I, Introduction) Akka Streams Integration, codename Alpakka A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II) Writing Akka Streams Connectors for existing APIs Flow control at the boundary of Akka Streams and a data provider Akka Streams Kafka 0.11
  34. 34. Alpakka – a community for Stream connectors http://developer.lightbend.com/docs/alpakka/current/
  35. 35. Alpakka – a community for Stream connectors Demo
  36. 36. Akka Streams & HTTP streams & HTTP
  37. 37. A core feature not obvious to the untrained eye…! Akka Streams / HTTP Quiz time! TCP is a ______ protocol?
  38. 38. A core feature not obvious to the untrained eye…! Akka Streams / HTTP Quiz time! TCP is a STREAMING protocol!
  39. 39. Streaming in Akka HTTP http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html HttpServer as a: Flow[HttpRequest, HttpResponse]
  40. 40. Streaming in Akka HTTP HttpServer as a: Flow[HttpRequest, HttpResponse] HTTP Entity as a: Source[ByteString, _] http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
  41. 41. Streaming in Akka HTTP HttpServer as a: Flow[HttpRequest, HttpResponse] HTTP Entity as a: Source[ByteString, _] Websocket connection as a: Flow[ws.Message, ws.Message] http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
  42. 42. Resource efficiency and dynamic fan-out
  43. 43. Resource efficiency and dynamic fan-out
  44. 44. It’s turtles buffers all the way down! xkcd.com
  45. 45. Streaming from Akka HTTP
  46. 46. Streaming from Akka HTTP
  47. 47. Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system); final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world")); final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) ); final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer); http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080"); }
  48. 48. Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system); final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world")); final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) ); final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer); http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080"); }
  49. 49. Streaming from Akka HTTP (Scala) object Example extends App with SprayJsonSupport with DefaultJsonProtocol { import akka.http.scaladsl.server.Directives._ implicit val system = ActorSystem() implicit val mat = ActorMaterializer() implicit val jsonRenderingMode = EntityStreamingSupport.json() implicit val TweetFormat = jsonFormat1(Tweet) def tweetsStreamRoutes = path("tweets") { complete { Source.repeat(Tweet("")) } } Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080) System.out.println("Running at http://localhost:8080"); }
  50. 50. Next steps for Akka Completely new Akka Remoting (goal: 700.000+ msg/s (!)), (it is built using Akka Streams, Aeron). More integrations for Akka Streams stages, project Alpakka. Reactive Kafka polishing with SoftwareMill, Krzysiek Ciesielski Akka Typed progressing again, likely towards 3.0. Akka HTTP 2.0 Proof of Concept in progress. Collaboration with Reactive Sockets
  51. 51. Ready to adopt on prod?
  52. 52. Totally, go for it.
  53. 53. Akka <3 contributions Easy to contribute tickets: https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22 Akka Stream Contrib https://github.com/akka/akka-stream-contrib Mailing list: https://groups.google.com/group/akka-user Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka
  54. 54. More resources: Read more tutorials and deep-dives: http://blog.akka.io/ https://www.lightbend.com/resources
  55. 55. Reactive Platform Reactive Platform Reactive Platform
  56. 56. Further reading: Reactive Streams: reactive-streams.org Akka documentation: akka.io/docs Free O’Reilly report – bit.ly/why-reactive Example Sources: ktoso/akka-streams-alpakka-talk-demos-2016 Contact: Konrad ktoso@lightbend.com Malawski http://kto.so / @ktosopl
  57. 57. Thanks! Questions? @apnylle johan.andren@lightbend.com @ktosopl konrad.malawski@lightbend.com

×