Kafka                      Twitter: yanaoki                        2011/11/27                 16               +WEB       ...
•                 •                 •   Kafka                 •   Kafka2011   11   27
•   Twitter: @yanaoki                 •                     •                 •   Java Ruby   Hadoop/Mahout Cassandra2011 ...
•                 •                 •   20112011   11   27
Facebook Insights                 •   2011   3                     •   “Like” ”Share”   CTR                     •   Facebo...
Google Analytics                 •   2011   9                     •                     •   PV UU                     •201...
Twitter Web Analytics                 •   2011    09                     •             Twitter                     •   Twi...
Linktedin                 •                    Kafka                     •   PV                     •                     ...
•   Facebook Insight                     •   PUMA        Scribe / HDFS / pTail / HBase /Thrift                     •   htt...
Kafka            •    Kafka                 •                 •   Linkedin                 •              http://incubator...
Linkedin                 •                 •                              SNS                 •                 •   2011  ...
Kafka            •                 •          Kafka                 •                     •                 •             ...
2011   11   27
•            •    ZooKeeper2011   11   27
•                     •               SPOF                     •   ZooKeeper                     •2011   11   27
Push or Pull                 •   Push                     •               scribe   flume                 •   Pull          ...
•        Pub/Sub                 •                 •   Publish                     •                 •   Subscribe        ...
•                     •                         •                         •                 •   Kafka                     ...
•                     •                         •                     •                         •   ZooKeeper             ...
2011   11   27
2011   11   27
A   B       D                         C2011   11   27
A   B       D                         C2011   11   27
A   B       D                         C2011   11   27
●       ●                                        ―            ―                        ●       ●           ●              ...
•                     •   Kafka           O(log n)   O(1)                 •   OS                 •   Java        GC       ...
•                     •                 •                     •                                Java                       ...
HUG January 2011 Kafka Presentation                        http://www.slideshare.net/ydn/hug-january-2011-kafka-presentati...
2011   11   27
Facebook             •                                                MapReduce(Not HadoopMR)                 •   Scribe/P...
Kafka HBase                 •   Twitter                     •                                     (en       ja            ...
Kafka HBase                 •                                      Twitter                                                ...
Hadoop                 •                                     Hadoop                     •   Hadoop    Map        Kafka    ...
Hadoop                  Offset                                                      Offset                                ...
2011   11   27
Upcoming SlideShare
Loading in...5
×

Kafkaによるリアルタイム処理

26,990

Published on

Published in: Technology
0 Comments
78 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
26,990
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
175
Comments
0
Likes
78
Embeds 0
No embeds

No notes for slide

Kafkaによるリアルタイム処理

  1. 1. Kafka Twitter: yanaoki 2011/11/27 16 +WEB http://www.flickr.com/photos/devnull/19765635/2011 11 27
  2. 2. • • • Kafka • Kafka2011 11 27
  3. 3. • Twitter: @yanaoki • • • Java Ruby Hadoop/Mahout Cassandra2011 11 27
  4. 4. • • • 20112011 11 27
  5. 5. Facebook Insights • 2011 3 • “Like” ”Share” CTR • Facebook2011 11 27
  6. 6. Google Analytics • 2011 9 • • PV UU •2011 11 27
  7. 7. Twitter Web Analytics • 2011 09 • Twitter • Twitter •2011 11 27
  8. 8. Linktedin • Kafka • PV • • • •2011 11 27
  9. 9. • Facebook Insight • PUMA Scribe / HDFS / pTail / HBase /Thrift • http://slidesha.re/ijWfPh • Twitter Promoted Tweets Reporting • Rainbird ZooKeeper / Cassandra • http://slidesha.re/dRxtIp • Twitter Web Analytics • Storm Zookieper • http://slidesha.re/qbpKbY • Google Analytics • • LinktedIn • Kafka ZooKeeper2011 11 27
  10. 10. Kafka • Kafka • • Linkedin • http://incubator.apache.org •2011 11 27
  11. 11. Linkedin • • SNS • • 2011 11 • Kafka • 2010 11 • 2011 07 Apache incubator project • a Distributed Messaging System for Log Processing • http://research.microsoft.com/en-us/um/people/ srikanth/netdb11/netdb11papers/netdb11-final12.pdf2011 11 27
  12. 12. Kafka • • Kafka • • • • Hadoop/HDFS • DWH •2011 11 27
  13. 13. 2011 11 27
  14. 14. • • ZooKeeper2011 11 27
  15. 15. • • SPOF • ZooKeeper •2011 11 27
  16. 16. Push or Pull • Push • scribe flume • Pull • Kafka •2011 11 27
  17. 17. • Pub/Sub • • Publish • • Subscribe • subscribe • Kafka2011 11 27
  18. 18. • • • • • Kafka • Zookeeper •2011 11 27
  19. 19. • • • • • ZooKeeper • • •2011 11 27
  20. 20. 2011 11 27
  21. 21. 2011 11 27
  22. 22. A B D C2011 11 27
  23. 23. A B D C2011 11 27
  24. 24. A B D C2011 11 27
  25. 25. ● ● ― ― ● ● ● ― ● ● ● ● ● ● ZK ● ● ● ZooKeeper ― ●2011 11 27
  26. 26. • • Kafka O(log n) O(1) • OS • Java GC • BTree2011 11 27
  27. 27. • • • • Java NIO (※ Linux sendfile )2011 11 27
  28. 28. HUG January 2011 Kafka Presentation http://www.slideshare.net/ydn/hug-january-2011-kafka-presentation2011 11 27
  29. 29. 2011 11 27
  30. 30. Facebook • MapReduce(Not HadoopMR) • Scribe/PTail/Puma Map • HBase Reduce http://www.slideshare.net/tatsuya6502/tokyo-hbase-meetup-realtime-big-data-at-facebook-ja Map Reduce2011 11 27
  31. 31. Kafka HBase • Twitter • (en ja lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 Kafka Broker2011 11 27
  32. 32. Kafka HBase • Twitter lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 client web → 100 iPhoone→10 Kafka Broker Android→102011 11 27
  33. 33. Hadoop • Hadoop • Hadoop Map Kafka • KafkaETLJob / KafkaETLInputFormat / KafkaETLRecordReader • MapReuduce API2011 11 27
  34. 34. Hadoop Offset Offset HDFS Map Offset Offset Limit map → Kafka Mapper Kafka Reducer HDFS2011 11 27
  35. 35. 2011 11 27
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×