Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Queryable State for Kafka Streamsを使ってみた

895 views

Published on

Apache Kafka Meetup Japan #2 https://kafka-apache-jp.connpass.com/event/45923/ 発表資料

Published in: Technology
  • Be the first to like this

Queryable State for Kafka Streamsを使ってみた

  1. 1. Queryable State for Kafka Streams / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • • Kafka Streams • State for Kafka Streams • Queryable State for Kafka Streams • • 2
  3. 3. • 2011/04 • 2015/09 • • Druid (KDP, 2015) • RDB NoSQL ( , 2016; : HBase ) • ESP8266 Wi-Fi IoT (KDP, 2016) • • (DEIM 2014) • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) 3
  4. 4. Apache Storm PipelineDB Druid Apache Kafka 4
  5. 5. Apache Storm Memcached Redis Apache Kafka 5
  6. 6. Apache Kafka 6
  7. 7. Kafka 7
  8. 8. Apache Kafka Version 0.10.1.0 • New Features • [KAFKA-1464] - Add a throttling option to the Kafka replication tool • [KAFKA-3176] - Allow console consumer to consume from particular partitions when new consumer is used. • [KAFKA-3492] - support quota based on authenticated user name • [KAFKA-3776] - Unify store and downstream caching in streams • [KAFKA-3858] - Add functions to print stream topologies • [KAFKA-3909] - Queryable state for Kafka Streams • [KAFKA-4015] - Change cleanup.policy config to accept a list of valid policies • [KAFKA-4093] - Cluster id 8
  9. 9. Kafka Streams • Version 0.10.0.0 • Kafka API 9
  10. 10. Kafka Streams Stateless transformations Stateful transformations 10
  11. 11. State for Kafka Streams • Stateful transformations • 11
  12. 12. State • Kafka Streams • 2 • State Store RocksDB • Kafka StateChangelog • Kafka Streams • Key Value • Window 12
  13. 13. • Record Stream • Changelog Stream • State (Table) Record Changelog State 13
  14. 14. Queryable State for Kafka Streams • Version 0.10.1.0 • State Store API 14
  15. 15. Queryable State • ReadOnlyKeyValueStore • Window Streams State • Key • Value • ReadOnlyWindowStore • Window Streams State • Key, From (Long), To (Long) • Iterator<KeyValue<Long, Value>> 15
  16. 16. Kafka Streams • Stateful transformations count KafkaStreams streams = null; KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> textLines = builder.stream(stringSerde, stringSerde, inputStream); KStream<String, Long> wordCounts = textLines .flatMapValues(value -> ...) .groupBy((key, word) -> word) .count(storeName) .toStream(); wordCounts.to(stringSerde, longSerde, outputStream); streams = new KafkaStreams(builder, conf); 17
  17. 17. Queryable State Streams • Streams o.a.k.streams.KafkaStreams KafkaStreams streams = ...; String key = "kafka"; ReadOnlyKeyValueStore<String, Long> store = streams.store( storeName, QueryableStoreTypes.<String, Long>keyValueStore()); Long value = store.get(key); return value; 18
  18. 18. Queryable State Streams • API KafkaStreams streams = ...; get("/results/:key", (req, res) -> { String key = req.params(":key"); ReadOnlyKeyValueStore<String, Long> store = streams.store( storeName, QueryableStoreTypes.<String, Long>keyValueStore()); Long value = store.get(key); return value; }); 19
  19. 19. github:laclefyoshi/kafka_streams_example
  20. 20. Apache Kafka 21 HTTP
  21. 21. $ java -cp target/kafka_streams_example-1.0-SNAPSHOT.jar org.saekiyoshiyasu.App kafka -> null ... kafka -> 5 $ ./bin/kafka-console-producer.sh --topic input-stream --broker-list 127.0.0.1:9092 What is Kafka good for? ... 22
  22. 22. Queryable State for Kafka Streams • 
 Kafka Streams ? 23
  23. 23. Queryable State for Kafka Streams • 
 Kafka Streams ? State 24
  24. 24. • Queryable State • Kafka Streams 25

×