Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)

1,454 views

Published on

D&S Data Night vol.04

Published in: Internet

Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)

  1. 1. Apache Kafka & Kafka Connect ( ETL ) Future Architect ,Inc Keigo Suda 2016/11/25 D&S Data Night vol.04
  2. 2. 12 ! Kafka Connect ETL ( ) ( ) /
  3. 3. 13 ! Future Architect ,Inc ( ) ! ! IoT
  4. 4. 14
  5. 5. 15 Kafka Connect
  6. 6. 16 ! Kafka ver 0.9 ! Kafka ! ! ! / Kafka Connect
  7. 7. 17 Connectors https://www.confluent.io/product/connectors/
  8. 8. Kafka Connect
  9. 9. 21
  10. 10. 22 ! ! ! Kafka Connect ( )
  11. 11. 23 ETL Kakfa Connect( ) P T // / / AC KM I M KM
  12. 12. Kafka Connect
  13. 13. 25http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started Kakfa Connect Kafka Connect
  14. 14. 26 Kakfa Connect http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
  15. 15. 27 Stream & Partition(RDB ) http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
  16. 16. 28 N:1 Stream & Partition(RDB ) http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect
  17. 17. 29 Kakfa Connect http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
  18. 18. 30 Worker & Connector http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started
  19. 19. 31 CONNECTER WORKER STREAM PARTITION STANDALONE MODE DISTRIBUTED MODE TASK
  20. 20. 32 $ bin/connect-standalone.sh config/connect-standalone.properties connector1.properties (Standalone mode) ! ! ! Worker ! Connector Worker Connector
  21. 21. 33 name=local-file-source connector.class=FileStreamSource tasks.max=1 file=test.txt topic=connect-test bootstrap.servers=localhost:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter ... Worker Connector
  22. 22. 35 ! ! ! ETL
  23. 23. 36 ! Kafka Connect Source Sink ! Sink/Source Sink Kafka Connect Source Sink
  24. 24. 37 ! Connector/Task OK ! API OK ( ) ( )
  25. 25. 38 Connector( )
  26. 26. 39 !!(source pull) Task
  27. 27. 40 ! Task Connector ! Deploy name=sample-sink connector.class=SampleSinkConnector tasks.max=1 topic=connect-test Connector
  28. 28. 42 ! Kafka ! Kafka Connect
  29. 29. 43 ! API Kafka Connect
  30. 30. 44 (kafka-connect-jdbc)
  31. 31. 45 ! Confluent Confuluent Platform Kafka Connect
  32. 32. 46 ! Kakfa ( ) ! Confluent….

×