Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Reactive Kafka with Akka Streams Slide 1 Reactive Kafka with Akka Streams Slide 2 Reactive Kafka with Akka Streams Slide 3 Reactive Kafka with Akka Streams Slide 4 Reactive Kafka with Akka Streams Slide 5 Reactive Kafka with Akka Streams Slide 6 Reactive Kafka with Akka Streams Slide 7 Reactive Kafka with Akka Streams Slide 8 Reactive Kafka with Akka Streams Slide 9 Reactive Kafka with Akka Streams Slide 10 Reactive Kafka with Akka Streams Slide 11 Reactive Kafka with Akka Streams Slide 12 Reactive Kafka with Akka Streams Slide 13 Reactive Kafka with Akka Streams Slide 14 Reactive Kafka with Akka Streams Slide 15 Reactive Kafka with Akka Streams Slide 16 Reactive Kafka with Akka Streams Slide 17 Reactive Kafka with Akka Streams Slide 18 Reactive Kafka with Akka Streams Slide 19 Reactive Kafka with Akka Streams Slide 20 Reactive Kafka with Akka Streams Slide 21 Reactive Kafka with Akka Streams Slide 22 Reactive Kafka with Akka Streams Slide 23 Reactive Kafka with Akka Streams Slide 24 Reactive Kafka with Akka Streams Slide 25 Reactive Kafka with Akka Streams Slide 26 Reactive Kafka with Akka Streams Slide 27 Reactive Kafka with Akka Streams Slide 28 Reactive Kafka with Akka Streams Slide 29 Reactive Kafka with Akka Streams Slide 30 Reactive Kafka with Akka Streams Slide 31 Reactive Kafka with Akka Streams Slide 32 Reactive Kafka with Akka Streams Slide 33 Reactive Kafka with Akka Streams Slide 34 Reactive Kafka with Akka Streams Slide 35 Reactive Kafka with Akka Streams Slide 36 Reactive Kafka with Akka Streams Slide 37 Reactive Kafka with Akka Streams Slide 38 Reactive Kafka with Akka Streams Slide 39 Reactive Kafka with Akka Streams Slide 40 Reactive Kafka with Akka Streams Slide 41 Reactive Kafka with Akka Streams Slide 42 Reactive Kafka with Akka Streams Slide 43 Reactive Kafka with Akka Streams Slide 44 Reactive Kafka with Akka Streams Slide 45 Reactive Kafka with Akka Streams Slide 46 Reactive Kafka with Akka Streams Slide 47 Reactive Kafka with Akka Streams Slide 48 Reactive Kafka with Akka Streams Slide 49
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

2 Likes

Share

Download to read offline

Reactive Kafka with Akka Streams

Download to read offline

http://2017.scalamatsuri.org/

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Reactive Kafka with Akka Streams

  1. 1. Reactive Kafka with Akka Streams Krzysztof Ciesielski
  2. 2. about.me • Scala developer @
 • Editor of ScalaTimes
 • Open Source (akka-stream-kafka, scala-clippy and other) Premier Consulting Partner
  3. 3. akka-stream-kafka • Formely reactive-kafka
 
 • A library for accessing Kafka as stages of Akka Streams akka-stream-kafka は Akka Streams のストレージとして Kafka にアクセスするラ イブラリ
  4. 4. Apache Kafka a message log 8 7 6 5 4 3 2 1 Producer Producer Consumer Consumer topic “topic1”
  5. 5. Apache Kafka a distributed message log 4 3 2 1 Producer 4 3 2 1 partition 1 partition 2 “topic1” “topic1” 分散メッセージログ
  6. 6. Apache Kafka a distributed message log Consumer balancing 4 3 2 1 Producer 4 3 2 1 partition 1 partition 2 “topic1” “topic1” 分散メッセージログ
  7. 7. Apache Kafka a distributed message log Consumer1 Consumer2 4 3 2 1 Producer 4 3 2 1 partition 1 partition 2 “topic1” “topic1” 分散メッセージログ
  8. 8. Apache Kafka a distributed message log 4 3 2 1 4 3 2 1 partition 1 partition 3 “topic1” “topic1” Consumer1 Consumer2 4 3 2 1 partition 2“topic1” manual assignment 分散メッセージログ
  9. 9. Apache Kafka Uses Zookeeper Leader election Cluster coordination Shared configuration リーダー選出、クラスタ協調、共有された設定に Zookeeper を使う
  10. 10. Apache Kafka Decoupling of producers and consumers プロデューサとコンシューマの分離
  11. 11. Apache Kafka Commit 8 7 6 5 4 3 2 1 Consumer commit(p1, o3) msg1, msg2
  12. 12. 8 7 6 5 4 3 2 1 Consumer Apache Kafka Commit
  13. 13. Apache Kafka Commit 8 7 6 5 4 3 2 1 New consumer msg3
  14. 14. Akka Streams • DSL for describing data transformation pipelines.
 • Uses Actor Model.
 • Implements the Reactive Streams specification. アクターモデルを使用した、データ変換パイプラインを記述するためのDSL。 Reactive Streams仕様を実装。
  15. 15. Akka Streams • Focused on backpressure and async processing.
 • Comes with a powerful test kit.
 • Extensible. バックプレッシャーと非同期処理にフォーカス。強力な非同期コード用のテストツー ル。拡張可能。
  16. 16. Alpakka Initiative to build a rich set of connectors HTTP Streaming TCP Streaming File IO Reactive Streams (MongoDB Reactive Streams Java Driver) 豊富なコネクタを構築する取組み
  17. 17. Akka Streams + Kafka
  18. 18. Akka Streams + Kafka
  19. 19. Akka Streams + Kafka
  20. 20. Akka Streams + Kafka
  21. 21. akka-stream-kafka • Started as reactive-kafka @ SoftwareMill • First users • Alexey Romanchuk has rebuilt the library • Akka Team joined (Patrik Nordwall, Endre Varga) reactive-kafka として SoftwareMill で開発を始める。Alexey Romanchuk が再構 築し、その後 Akka Team が参加。
  22. 22. Example: plain consumer plain consumer (素のコンシューマ) の例
  23. 23. Example: plain consumer plain consumer (素のコンシューマ) の例
  24. 24. Example: plain consumer +async processing +error handling +backpressure plain consumer (素のコンシューマ) の例
  25. 25. Consumer Actor SourceStage ConsumerActor consumer (Kafka API) RequestMessages demand messages Messages Scheduled Poll poll() dispatcher
  26. 26. Plain consumer performance 1,202,081 944,746 plain consumer は直接 Java API を叩くぐらいの性能が出ている
  27. 27. Apache Kafka Commit 8 7 6 5 4 3 2 1 Consumer commit(p1, o3) msg1, msg2 次に commit の性能を比較する
  28. 28. Consumer Actor and commit SourceStage ConsumerActor consumer (Kafka API) ? Commit commit() Scheduled Poll poll() dispatcher msg msg commitAsync() callback Request Messages
  29. 29. Committable Source commit 機能を持った Source
  30. 30. Committable Source commit 機能を持った Source
  31. 31. Batched Committable Source commit を呼ぶと重いのでバッチ化する
  32. 32. Batched Committable Source groupedWithin でメッセージを 100 個ごと、もしくは 5 秒経過するごとにグループ としてまとめる
  33. 33. At-least-once delivery 456,695 1,194,331
  34. 34. External commit source オフセットを独自に保存することで exactly-once セマンティックを実現する
  35. 35. Producer Sink に流す形のプロデューサ
  36. 36. Producer Vanilla API
  37. 37. Producer as a Flow Flow としてのプロデューサ
  38. 38. Producer as a Flow Flow としてのプロデューサ
  39. 39. From Kafka to Kafka Kafka からのログを Kafka へ流す
  40. 40. From Kafka to Kafka SourceStage Producer Stage msg send() callback() msg msg mapAsync Consumer Actor Kafka からのログを Kafka へ流す
  41. 41. From Kafka to Kafka Kafka からのログを Kafka へ流す
  42. 42. It’s streaming streams backpressure per partition パーティションごとにバックプレッシャーをかける
  43. 43. Source[(TopicPartition, Source[CommittableMessage[Array[Byte], String], NotUsed]), Control] It’s streaming streams パーティションごとにバックプレッシャーをかける
  44. 44. Error handling run() から Future を取得してエラー処理を行う
  45. 45. Java API
  46. 46. Kafka Streams Apache Kafka 製の Kafka Stream という別物のライブラリもある
  47. 47. Kafka Streams akka-stream-kafka Kafka Streams Scala + Java, backpressure, async processing, interop with Akka Streams Java, stateful processors, windowing, joining, aggregation operations Apache Kafka 製の Kafka Stream という別物のライブラリもある
  48. 48. Thank you! Join us! https://gitter.im/akka/reactive-kafka @kpciesielski
  • clbsteria

    Sep. 6, 2017
  • KenTakehara

    Aug. 18, 2017

http://2017.scalamatsuri.org/

Views

Total views

572

On Slideshare

0

From embeds

0

Number of embeds

4

Actions

Downloads

2

Shares

0

Comments

0

Likes

2

×