Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Kafka AWS Kinesis
/ @laclefyoshi / ysaeki@r.recruit.co.jp
•
• Kafka Kinesis
•
•
•
•
2
• 2011/04
• 2015/09
•
• Druid (2015; Kindle Direct Publishing)
• RDB NoSQL (2016; ; HBase )
•
• (WebDB Forum 2014)
• Spark...
Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Fireho...
Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Fireho...
:
5
: <100ms
: ~8000msgs/sec
Kinesis
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL 7
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL
OSS
•Kafka Streams
•Pipeline...
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[ ]
http://insightdataengineering.com/blog/ingestion-comparison/
X
Kafka Kinesis
• Kafka or Kinesis
• Kafka OSS
Kinesis
•
9
Kafka Kinesis
• (< 5000msgs/sec) Kinesis
• Kafka
• Kinesis
• Hadoop Kafka
• Kinesis
• Kafka
10
• OSS → Kafka
• → Kinesis
• API
•
• SNS Bot IoT
•
11
Upcoming SlideShare
Loading in …5
×

KafkaとAWS Kinesisの比較

9,114 views

Published on

Apache Kafka Meetup Japan #1 http://connpass.com/event/32293/ LT発表資料

Published in: Technology
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes ,Download or read Ebooks here ... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m6jJ5M }
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

KafkaとAWS Kinesisの比較

  1. 1. Kafka AWS Kinesis / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • • Kafka Kinesis • • • • 2
  3. 3. • 2011/04 • 2015/09 • • Druid (2015; Kindle Direct Publishing) • RDB NoSQL (2016; ; HBase ) • • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) 3
  4. 4. Kafka Kinesis • Apache • Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
  5. 5. Kafka Kinesis • Apache • Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
  6. 6. : 5 : <100ms : ~8000msgs/sec Kinesis
  7. 7. [Kafka] [Kinesis] 6
  8. 8. [Kafka] [Kinesis] 6
  9. 9. [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS API Gateway HTTP API ETL ETL 7
  10. 10. [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS API Gateway HTTP API ETL ETL OSS •Kafka Streams •PipelineDB AWS •Kinesis Analytics 7
  11. 11. [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ← → ← → ← → 8
  12. 12. [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ← → ← → ← → 8
  13. 13. [ ] http://insightdataengineering.com/blog/ingestion-comparison/ X
  14. 14. Kafka Kinesis • Kafka or Kinesis • Kafka OSS Kinesis • 9
  15. 15. Kafka Kinesis • (< 5000msgs/sec) Kinesis • Kafka • Kinesis • Hadoop Kafka • Kinesis • Kafka 10
  16. 16. • OSS → Kafka • → Kinesis • API • • SNS Bot IoT • 11

×