Kafkaによるリアルタイム処理
Upcoming SlideShare
Loading in...5
×
 

Kafkaによるリアルタイム処理

on

  • 19,998 views

 

Statistics

Views

Total Views
19,998
Views on SlideShare
13,764
Embed Views
6,234

Actions

Likes
50
Downloads
124
Comments
0

16 Embeds 6,234

http://d.hatena.ne.jp 4696
http://open-groove.net 1218
http://narupeko.soup.io 213
http://paper.li 53
http://a0.twimg.com 17
https://twitter.com 11
http://webcache.googleusercontent.com 10
http://takeoverapp.com 5
https://www.google.co.jp 3
http://wiki.onakasuita.org 2
https://www.facebook.com 1
https://si0.twimg.com 1
http://translate.googleusercontent.com 1
http://us-w1.rockmelt.com 1
http://tweetedtimes.com 1
http://www.google.co.jp 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Kafkaによるリアルタイム処理 Kafkaによるリアルタイム処理 Presentation Transcript

  • Kafka Twitter: yanaoki 2011/11/27 16 +WEB http://www.flickr.com/photos/devnull/19765635/2011 11 27
  • • • • Kafka • Kafka2011 11 27
  • • Twitter: @yanaoki • • • Java Ruby Hadoop/Mahout Cassandra2011 11 27
  • • • • 20112011 11 27
  • Facebook Insights • 2011 3 • “Like” ”Share” CTR • Facebook2011 11 27
  • Google Analytics • 2011 9 • • PV UU •2011 11 27
  • Twitter Web Analytics • 2011 09 • Twitter • Twitter •2011 11 27
  • Linktedin • Kafka • PV • • • •2011 11 27
  • • Facebook Insight • PUMA Scribe / HDFS / pTail / HBase /Thrift • http://slidesha.re/ijWfPh • Twitter Promoted Tweets Reporting • Rainbird ZooKeeper / Cassandra • http://slidesha.re/dRxtIp • Twitter Web Analytics • Storm Zookieper • http://slidesha.re/qbpKbY • Google Analytics • • LinktedIn • Kafka ZooKeeper2011 11 27
  • Kafka • Kafka • • Linkedin • http://incubator.apache.org •2011 11 27
  • Linkedin • • SNS • • 2011 11 • Kafka • 2010 11 • 2011 07 Apache incubator project • a Distributed Messaging System for Log Processing • http://research.microsoft.com/en-us/um/people/ srikanth/netdb11/netdb11papers/netdb11-final12.pdf2011 11 27
  • Kafka • • Kafka • • • • Hadoop/HDFS • DWH •2011 11 27
  • 2011 11 27
  • • • ZooKeeper2011 11 27
  • • • SPOF • ZooKeeper •2011 11 27
  • Push or Pull • Push • scribe flume • Pull • Kafka •2011 11 27
  • • Pub/Sub • • Publish • • Subscribe • subscribe • Kafka2011 11 27
  • • • • • • Kafka • Zookeeper •2011 11 27
  • • • • • • ZooKeeper • • •2011 11 27
  • 2011 11 27
  • 2011 11 27
  • A B D C2011 11 27
  • A B D C2011 11 27
  • A B D C2011 11 27
  • ● ● ― ― ● ● ● ― ● ● ● ● ● ● ZK ● ● ● ZooKeeper ― ●2011 11 27
  • • • Kafka O(log n) O(1) • OS • Java GC • BTree2011 11 27
  • • • • • Java NIO (※ Linux sendfile )2011 11 27
  • HUG January 2011 Kafka Presentation http://www.slideshare.net/ydn/hug-january-2011-kafka-presentation2011 11 27
  • 2011 11 27
  • Facebook • MapReduce(Not HadoopMR) • Scribe/PTail/Puma Map • HBase Reduce http://www.slideshare.net/tatsuya6502/tokyo-hbase-meetup-realtime-big-data-at-facebook-ja Map Reduce2011 11 27
  • Kafka HBase • Twitter • (en ja lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 Kafka Broker2011 11 27
  • Kafka HBase • Twitter lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 client web → 100 iPhoone→10 Kafka Broker Android→102011 11 27
  • Hadoop • Hadoop • Hadoop Map Kafka • KafkaETLJob / KafkaETLInputFormat / KafkaETLRecordReader • MapReuduce API2011 11 27
  • Hadoop Offset Offset HDFS Map Offset Offset Limit map → Kafka Mapper Kafka Reducer HDFS2011 11 27
  • 2011 11 27