Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ストリーム処理を支えるキューイングシステムの選び方

20,832 views

Published on

ビッグデータのリアルタイム処理技術勉強会 http://futureofdata.connpass.com/event/40077/ 発表資料

Published in: Data & Analytics

ストリーム処理を支えるキューイングシステムの選び方

  1. 1. / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • • • • • • 2
  3. 3. • 2011/04 • 2015/09 • • Druid (KDP, 2015) • RDB NoSQL ( , 2016; : HBase ) • ESP8266 Wi-Fi IoT (KDP, 2016) • • (DEIM 2014) • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • Kinesis Family (R-JAWS Bigdata; 2016) 3
  4. 4. 5
  5. 5. • Queue Queueing system • • Message • 1 • Broker • • Producer • • Consumer • • Protocol • 7
  6. 6. • • • 8
  7. 7. 9 Apache ActiveMQ Apache Kafka RabbitMQ NATS NSQ Redis ZeroMQ Nanomsg
  8. 8. Apache Kafka A distributed streaming platform 10
  9. 9. Apache ActiveMQ The most popular and powerful open source
 messaging and Integration Patterns server 11
  10. 10. RabbitMQ Robust messaging for applications Easy to use 12
  11. 11. NATS For cloud native applications, IoT messaging, and microservices architectures 13
  12. 12. NSQ A realtime distributed messaging platform 14
  13. 13. Redis In-memory data structure store, used as database, cache and message broker 15
  14. 14. ZeroMQ An embeddable networking library but acts like a concurrency framework 16
  15. 15. Nanomsg A socket library that provides several common communication patterns 17
  16. 16. 19
  17. 17. 20
  18. 18. : • • • Lambda Architecture 
 
 : • • 21
  19. 19. 22
  20. 20. : • • 
 : • 23
  21. 21. 24
  22. 22. : • 
 • 
 
 : • 
 25
  23. 23. 27
  24. 24. : …… https://cloud.google.com/blog/big-data/2016/09/ apache-kafka-for-gcp-users-connectors-for-pubsub-dataflow-and-bigquery 28 Pub/Sub Dataflow/Apache Beam BigQuery Apache Kafka
  25. 25. 29
  26. 26. 30 : • 
 : •
  27. 27. 31
  28. 28. 32 : • 
 : • 

  29. 29. 
 2016/10/25 official / 3rd party 33
  30. 30. • JIRA Github • Apache Flink Apache Spark 
 Apache Bahir (3rd party) • Apache Kafka 34
  31. 31. 1000 • [ 1] → [ 1 → 1] • AWS: EC2 /t2.medium x 2 ( VPC) • github:laclefyoshi/queues-benchmark • Docker • Python 36
  32. 32. 1000 37 • • • • 1000 Byte 100000 Byte 1000 Byte • 1000
  33. 33. 38
  34. 34. 40
  35. 35. 41
  36. 36. Queue makes everyone happy!!

×