Advertisement
Kafkaによるリアルタイム処理
Nov. 26, 2011•0 likes•29,967 views
92 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Technology
Naoki YanaiFollow
Product Manager at FreakOut, Inc.Advertisement
Advertisement
Advertisement
Recommended
Apache Kafkaって本当に大丈夫？～故障検証のオーバービューと興味深い挙動の紹介～NTT DATA OSS Professional Services
入門 Kubeflow ～Kubernetesで機械学習をはじめるために～ （NTT Tech Conference #4 講演資料）NTT DATA Technology & Innovation
Apache Spark on Kubernetes入門（Open Source Conference 2021 Online Hiroshima 発表資料）NTT DATA Technology & Innovation
Kinesis + Elasticsearchでつくるさいきょうのログ分析基盤Amazon Web Services Japan
AWSで作る分析基盤Yu Otsubo
ホットペッパービューティーにおけるモバイルアプリ向けAPIのBFF/Backend分割Recruit Lifestyle Co., Ltd.
Kafkaによるリアルタイム処理
- Kafka Twitter: yanaoki 2011/11/27 16 +WEB http://www.flickr.com/photos/devnull/19765635/ 2011 11 27
- • • • Kafka • Kafka 2011 11 27
- • Twitter: @yanaoki • • • Java Ruby Hadoop/Mahout Cassandra 2011 11 27
- • • • 2011 2011 11 27
- Facebook Insights • 2011 3 • “Like” ”Share” CTR • Facebook 2011 11 27
- Google Analytics • 2011 9 • • PV UU • 2011 11 27
- Twitter Web Analytics • 2011 09 • Twitter • Twitter • 2011 11 27
- Linktedin • Kafka • PV • • • • 2011 11 27
- • Facebook Insight • PUMA Scribe / HDFS / pTail / HBase /Thrift • http://slidesha.re/ijWfPh • Twitter Promoted Tweets Reporting • Rainbird ZooKeeper / Cassandra • http://slidesha.re/dRxtIp • Twitter Web Analytics • Storm Zookieper • http://slidesha.re/qbpKbY • Google Analytics • • LinktedIn • Kafka ZooKeeper 2011 11 27
- Kafka • Kafka • • Linkedin • http://incubator.apache.org • 2011 11 27
- Linkedin • • SNS • • 2011 11 • Kafka • 2010 11 • 2011 07 Apache incubator project • a Distributed Messaging System for Log Processing • http://research.microsoft.com/en-us/um/people/ srikanth/netdb11/netdb11papers/netdb11-final12.pdf 2011 11 27
- Kafka • • Kafka • • • • Hadoop/HDFS • DWH • 2011 11 27
- 2011 11 27
- • • ZooKeeper 2011 11 27
- • • SPOF • ZooKeeper • 2011 11 27
- Push or Pull • Push • scribe flume • Pull • Kafka • 2011 11 27
- • Pub/Sub • • Publish • • Subscribe • subscribe • Kafka 2011 11 27
- • • • • • Kafka • Zookeeper • 2011 11 27
- • • • • • ZooKeeper • • • 2011 11 27
- 2011 11 27
- 2011 11 27
- A B D C 2011 11 27
- A B D C 2011 11 27
- A B D C 2011 11 27
- ● ● ― ― ● ● ● ― ● ● ● ● ● ● ZK ● ● ● ZooKeeper ― ● 2011 11 27
- • • Kafka O(log n) O(1) • OS • Java GC • BTree 2011 11 27
- • • • • Java NIO (※ Linux sendfile ) 2011 11 27
- HUG January 2011 Kafka Presentation http://www.slideshare.net/ydn/hug-january-2011-kafka-presentation 2011 11 27
- 2011 11 27
- Facebook • MapReduce(Not HadoopMR) • Scribe/PTail/Puma Map • HBase Reduce http://www.slideshare.net/tatsuya6502/tokyo-hbase-meetup-realtime-big-data-at-facebook-ja Map Reduce 2011 11 27
- Kafka HBase • Twitter • (en ja lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 Kafka Broker 2011 11 27
- Kafka HBase • Twitter lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 client web → 100 iPhoone→10 Kafka Broker Android→10 2011 11 27
- Hadoop • Hadoop • Hadoop Map Kafka • KafkaETLJob / KafkaETLInputFormat / KafkaETLRecordReader • MapReuduce API 2011 11 27
- Hadoop Offset Offset HDFS Map Offset Offset Limit map → Kafka Mapper Kafka Reducer HDFS 2011 11 27
- 2011 11 27
Advertisement