Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Kafka AWS Kinesis
/ @laclefyoshi / ysaeki@r.recruit.co.jp
•
• Kafka Kinesis
•
•
•
•
2
• 2011/04
• 2015/09
•
• Druid (2015; Kindle Direct Publishing)
• RDB NoSQL (2016; ; HBase )
•
• (WebDB Forum 2014)
• Spark...
Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Fireho...
Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Fireho...
:
5
: <100ms
: ~8000msgs/sec
Kinesis
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL 7
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL
OSS
•Kafka Streams
•Pipeline...
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[ ]
http://insightdataengineering.com/blog/ingestion-comparison/
X
Kafka Kinesis
• Kafka or Kinesis
• Kafka OSS
Kinesis
•
9
Kafka Kinesis
• (< 5000msgs/sec) Kinesis
• Kafka
• Kinesis
• Hadoop Kafka
• Kinesis
• Kafka
10
• OSS → Kafka
• → Kinesis
• API
•
• SNS Bot IoT
•
11
Upcoming SlideShare
Loading in …5
×

KafkaとAWS Kinesisの比較

8,239 views

Published on

Apache Kafka Meetup Japan #1 http://connpass.com/event/32293/ LT発表資料

Published in: Technology
  • Be the first to comment

KafkaとAWS Kinesisの比較

  1. 1. Kafka AWS Kinesis / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • • Kafka Kinesis • • • • 2
  3. 3. • 2011/04 • 2015/09 • • Druid (2015; Kindle Direct Publishing) • RDB NoSQL (2016; ; HBase ) • • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) 3
  4. 4. Kafka Kinesis • Apache • Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
  5. 5. Kafka Kinesis • Apache • Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
  6. 6. : 5 : <100ms : ~8000msgs/sec Kinesis
  7. 7. [Kafka] [Kinesis] 6
  8. 8. [Kafka] [Kinesis] 6
  9. 9. [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS API Gateway HTTP API ETL ETL 7
  10. 10. [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS API Gateway HTTP API ETL ETL OSS •Kafka Streams •PipelineDB AWS •Kinesis Analytics 7
  11. 11. [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ← → ← → ← → 8
  12. 12. [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ← → ← → ← → 8
  13. 13. [ ] http://insightdataengineering.com/blog/ingestion-comparison/ X
  14. 14. Kafka Kinesis • Kafka or Kinesis • Kafka OSS Kinesis • 9
  15. 15. Kafka Kinesis • (< 5000msgs/sec) Kinesis • Kafka • Kinesis • Hadoop Kafka • Kinesis • Kafka 10
  16. 16. • OSS → Kafka • → Kinesis • API • • SNS Bot IoT • 11

×