Download as PDF, PPTX
Uploaded byNaoki Yanai
Kafkaによるリアルタイム処理
This document discusses Kafka, a distributed messaging system originally created by LinkedIn and now an Apache project. It provides an overview of how Kafka works and how various companies use it, including for log processing (LinkedIn), analytics (Facebook, Twitter, Google), and integrating with other technologies like Hadoop, Zookeeper, HBase and Storm. It also covers Kafka's scalability, performance and architecture.
Kafkaによるリアルタイム処理
- 1. Kafka Twitter: yanaoki 2011/11/27 16 +WEB http://www.flickr.com/photos/devnull/19765635/ 2011 11 27
- 2. • • • Kafka • Kafka 2011 11 27
- 3. • Twitter: @yanaoki • • • Java Ruby Hadoop/Mahout Cassandra 2011 11 27
- 4. • • • 2011 2011 11 27
- 5. Facebook Insights • 2011 3 • “Like” ”Share” CTR • Facebook 2011 11 27
- 6. Google Analytics • 2011 9 • • PV UU • 2011 11 27
- 7. Twitter Web Analytics • 2011 09 • Twitter • Twitter • 2011 11 27
- 8. Linktedin • Kafka • PV • • • • 2011 11 27
- 9. • Facebook Insight • PUMA Scribe / HDFS / pTail / HBase /Thrift • http://slidesha.re/ijWfPh • Twitter Promoted Tweets Reporting • Rainbird ZooKeeper / Cassandra • http://slidesha.re/dRxtIp • Twitter Web Analytics • Storm Zookieper • http://slidesha.re/qbpKbY • Google Analytics • • LinktedIn • Kafka ZooKeeper 2011 11 27
- 10. Kafka • Kafka • • Linkedin • http://incubator.apache.org • 2011 11 27
- 11. Linkedin • • SNS • • 2011 11 • Kafka • 2010 11 • 2011 07 Apache incubator project • a Distributed Messaging System for Log Processing • http://research.microsoft.com/en-us/um/people/ srikanth/netdb11/netdb11papers/netdb11-final12.pdf 2011 11 27
- 12. Kafka • • Kafka • • • • Hadoop/HDFS • DWH • 2011 11 27
- 13. 2011 11 27
- 14. • • ZooKeeper 2011 11 27
- 15. • • SPOF • ZooKeeper • 2011 11 27
- 16. Push or Pull • Push • scribe flume • Pull • Kafka • 2011 11 27
- 17. • Pub/Sub • • Publish • • Subscribe • subscribe • Kafka 2011 11 27
- 18. • • • • • Kafka • Zookeeper • 2011 11 27
- 19. • • • • • ZooKeeper • • • 2011 11 27
- 20. 2011 11 27
- 21. 2011 11 27
- 22. A B D C 2011 11 27
- 23. A B D C 2011 11 27
- 24. A B D C 2011 11 27
- 25. ● ● ― ― ● ● ● ― ● ● ● ● ● ● ZK ● ● ● ZooKeeper ― ● 2011 11 27
- 26. • • Kafka O(log n) O(1) • OS • Java GC • BTree 2011 11 27
- 27. • • • • Java NIO (※ Linux sendfile ) 2011 11 27
- 28. HUG January 2011Kafka Presentation http://www.slideshare.net/ydn/hug-january-2011-kafka-presentation 2011 11 27
- 29. 2011 11 27
- 30. Facebook • MapReduce(Not HadoopMR) • Scribe/PTail/Puma Map • HBase Reduce http://www.slideshare.net/tatsuya6502/tokyo-hbase-meetup-realtime-big-data-at-facebook-ja Map Reduce 2011 11 27
- 31. Kafka HBase • Twitter • (en ja lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 Kafka Broker 2011 11 27
- 32. Kafka HBase • Twitter lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 client web → 100 iPhoone→10 Kafka Broker Android→10 2011 11 27
- 33. Hadoop • Hadoop • Hadoop Map Kafka • KafkaETLJob / KafkaETLInputFormat / KafkaETLRecordReader • MapReuduce API 2011 11 27
- 34. Hadoop Offset Offset HDFS Map Offset Offset Limit map → Kafka Mapper Kafka Reducer HDFS 2011 11 27
- 35. 2011 11 27