This document compares Apache Kafka and AWS Kinesis for message streaming. It outlines that Kafka is an open source publish-subscribe messaging system designed as a distributed commit log, while Kinesis provides streaming data services. It also notes some key differences like Kafka typically handling over 8000 messages/second while Kinesis can handle under 100 messages/second.

1. Kafka AWS Kinesis
/ @laclefyoshi / ysaeki@r.recruit.co.jp

2. •
• Kafka Kinesis
•
•
•
•
2

3. • 2011/04
• 2015/09
•
• Druid (2015; Kindle Direct Publishing)
• RDB NoSQL (2016; ; HBase )
•
• (WebDB Forum 2014)
• Spark Streaming (Spark Meetup
December 2015)
3

4. Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Firehose
• Analytics
: Pub/Sub
:
: (SQL)
4

5. Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Firehose
• Analytics
: Pub/Sub
:
: (SQL)
4

6. :
5
: <100ms
: ~8000msgs/sec
Kinesis

7. [Kafka] [Kinesis]
6

8. [Kafka] [Kinesis]
6

9. [Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL 7

10. [Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL
OSS
•Kafka Streams
•PipelineDB
AWS
•Kinesis Analytics
7

11. [Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8

12. [Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8

13. [ ]
http://insightdataengineering.com/blog/ingestion-comparison/
X

14. Kafka Kinesis
• Kafka or Kinesis
• Kafka OSS
Kinesis
•
9

15. Kafka Kinesis
• (< 5000msgs/sec) Kinesis
• Kafka
• Kinesis
• Hadoop Kafka
• Kinesis
• Kafka
10

16. • OSS → Kafka
• → Kinesis
• API
•
• SNS Bot IoT
•
11