Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reactive Kafka with Akka Streams

10,733 views

Published on

A walk through the features of akka-stream-kafka, a library for representing Apache Kafka as stages of Akka Streams.

Published in: Software
  • Sex in your area is here: ❤❤❤ http://bit.ly/369VOVb ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ❶❶❶ http://bit.ly/369VOVb ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Reactive Kafka with Akka Streams

  1. 1. Reactive Kafka with Akka Streams Krzysztof Ciesielski Reactive Summit 2016, Austin, TX
  2. 2. about.me • Scala developer @
 • Editor of ScalaTimes
 • Open Source (akka-stream-kafka, scala-clippy and other) Premier Consulting Partner
  3. 3. akka-stream-kafka • Formely reactive-kafka
 
 • A library for accessing Kafka as stages of Akka Streams
  4. 4. Apache Kafka a message log 8 7 6 5 4 3 2 1 Producer Producer Consumer Consumer topic “topic1”
  5. 5. Apache Kafka a distributed message log 4 3 2 1 Producer 4 3 2 1 partition 1 partition 2 “topic1” “topic1”
  6. 6. Apache Kafka a distributed message log Consumer balancing 4 3 2 1 Producer 4 3 2 1 partition 1 partition 2 “topic1” “topic1”
  7. 7. Apache Kafka a distributed message log Consumer1 Consumer2 4 3 2 1 Producer 4 3 2 1 partition 1 partition 2 “topic1” “topic1”
  8. 8. Apache Kafka a distributed message log 4 3 2 1 4 3 2 1 partition 1 partition 3 “topic1” “topic1” Consumer1 Consumer2 4 3 2 1 partition 2“topic1” manual assignment
  9. 9. Apache Kafka Uses Zookeeper Leader election Cluster coordination Shared configuration
  10. 10. Apache Kafka Decoupling of producers and consumers
  11. 11. Apache Kafka Commit 8 7 6 5 4 3 2 1 Consumer commit(p1, o3) msg1, msg2
  12. 12. 8 7 6 5 4 3 2 1 Consumer Apache Kafka Commit
  13. 13. Apache Kafka Commit 8 7 6 5 4 3 2 1 New consumer msg3
  14. 14. Akka Streams • DSL for describing data transformation pipelines.
 • Uses Actor Model.
 • Implements the Reactive Streams specification.
  15. 15. Akka Streams • Focused on backpressure and async processing.
 • Comes with a powerful test kit.
 • Extensible.
  16. 16. Alpakka Initiative to build a rich set of connectors HTTP Streaming TCP Streaming File IO Reactive Streams (MongoDB Reactive Streams Java Driver)
  17. 17. Akka Streams + Kafka
  18. 18. Akka Streams + Kafka
  19. 19. Akka Streams + Kafka
  20. 20. Akka Streams + Kafka
  21. 21. akka-stream-kafka • Started as reactive-kafka @ SoftwareMill • First users • Alexey Romanchuk has rebuilt the library • Akka Team joined (Patrik Nordwall, Endre Varga)
  22. 22. Example: plain consumer
  23. 23. Example: plain consumer
  24. 24. Example: plain consumer +async processing +error handling +backpressure
  25. 25. Consumer Actor SourceStage ConsumerActor consumer (Kafka API) RequestMessages demand messages Messages Scheduled Poll poll() dispatcher
  26. 26. Plain consumer performance 1,202,081 944,746
  27. 27. Apache Kafka Commit 8 7 6 5 4 3 2 1 Consumer commit(p1, o3) msg1, msg2
  28. 28. Consumer Actor and commit SourceStage ConsumerActor consumer (Kafka API) ? Commit commit() Scheduled Poll poll() dispatcher msg msg commitAsync() callback Request Messages
  29. 29. Committable Source
  30. 30. Committable Source
  31. 31. Batched Committable Source
  32. 32. Batched Committable Source
  33. 33. At-least-once delivery 456,695 1,194,331
  34. 34. External commit source
  35. 35. Producer
  36. 36. Producer Vanilla API
  37. 37. Producer as a Flow
  38. 38. Producer as a Flow
  39. 39. From Kafka to Kafka
  40. 40. From Kafka to Kafka SourceStage Producer Stage msg send() callback() msg msg mapAsync Consumer Actor
  41. 41. From Kafka to Kafka
  42. 42. It’s streaming streams backpressure per partition
  43. 43. Source[(TopicPartition, Source[CommittableMessage[Array[Byte], String], NotUsed]), Control] It’s streaming streams
  44. 44. Error handling
  45. 45. Java API
  46. 46. Kafka Streams
  47. 47. Kafka Streams akka-stream-kafka Kafka Streams Scala + Java, backpressure, async processing, interop with Akka Streams Java, stateful processors, windowing, joining, aggregation operations
  48. 48. Thank you! Join us! https://gitter.im/akka/reactive-kafka @kpciesielski

×