Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kafkaによるリアルタイム処理

37,463 views

Published on

Published in: Technology

Kafkaによるリアルタイム処理

  1. 1. Kafka Twitter: yanaoki 2011/11/27 16 +WEB http://www.flickr.com/photos/devnull/19765635/2011 11 27
  2. 2. • • • Kafka • Kafka2011 11 27
  3. 3. • Twitter: @yanaoki • • • Java Ruby Hadoop/Mahout Cassandra2011 11 27
  4. 4. • • • 20112011 11 27
  5. 5. Facebook Insights • 2011 3 • “Like” ”Share” CTR • Facebook2011 11 27
  6. 6. Google Analytics • 2011 9 • • PV UU •2011 11 27
  7. 7. Twitter Web Analytics • 2011 09 • Twitter • Twitter •2011 11 27
  8. 8. Linktedin • Kafka • PV • • • •2011 11 27
  9. 9. • Facebook Insight • PUMA Scribe / HDFS / pTail / HBase /Thrift • http://slidesha.re/ijWfPh • Twitter Promoted Tweets Reporting • Rainbird ZooKeeper / Cassandra • http://slidesha.re/dRxtIp • Twitter Web Analytics • Storm Zookieper • http://slidesha.re/qbpKbY • Google Analytics • • LinktedIn • Kafka ZooKeeper2011 11 27
  10. 10. Kafka • Kafka • • Linkedin • http://incubator.apache.org •2011 11 27
  11. 11. Linkedin • • SNS • • 2011 11 • Kafka • 2010 11 • 2011 07 Apache incubator project • a Distributed Messaging System for Log Processing • http://research.microsoft.com/en-us/um/people/ srikanth/netdb11/netdb11papers/netdb11-final12.pdf2011 11 27
  12. 12. Kafka • • Kafka • • • • Hadoop/HDFS • DWH •2011 11 27
  13. 13. 2011 11 27
  14. 14. • • ZooKeeper2011 11 27
  15. 15. • • SPOF • ZooKeeper •2011 11 27
  16. 16. Push or Pull • Push • scribe flume • Pull • Kafka •2011 11 27
  17. 17. • Pub/Sub • • Publish • • Subscribe • subscribe • Kafka2011 11 27
  18. 18. • • • • • Kafka • Zookeeper •2011 11 27
  19. 19. • • • • • ZooKeeper • • •2011 11 27
  20. 20. 2011 11 27
  21. 21. 2011 11 27
  22. 22. A B D C2011 11 27
  23. 23. A B D C2011 11 27
  24. 24. A B D C2011 11 27
  25. 25. ● ● ― ― ● ● ● ― ● ● ● ● ● ● ZK ● ● ● ZooKeeper ― ●2011 11 27
  26. 26. • • Kafka O(log n) O(1) • OS • Java GC • BTree2011 11 27
  27. 27. • • • • Java NIO (※ Linux sendfile )2011 11 27
  28. 28. HUG January 2011 Kafka Presentation http://www.slideshare.net/ydn/hug-january-2011-kafka-presentation2011 11 27
  29. 29. 2011 11 27
  30. 30. Facebook • MapReduce(Not HadoopMR) • Scribe/PTail/Puma Map • HBase Reduce http://www.slideshare.net/tatsuya6502/tokyo-hbase-meetup-realtime-big-data-at-facebook-ja Map Reduce2011 11 27
  31. 31. Kafka HBase • Twitter • (en ja lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 Kafka Broker2011 11 27
  32. 32. Kafka HBase • Twitter lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 client web → 100 iPhoone→10 Kafka Broker Android→102011 11 27
  33. 33. Hadoop • Hadoop • Hadoop Map Kafka • KafkaETLJob / KafkaETLInputFormat / KafkaETLRecordReader • MapReuduce API2011 11 27
  34. 34. Hadoop Offset Offset HDFS Map Offset Offset Limit map → Kafka Mapper Kafka Reducer HDFS2011 11 27
  35. 35. 2011 11 27

×