Kafkaによるリアルタイム処理
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
21,215
On Slideshare
14,828
From Embeds
6,387
Number of Embeds
16

Actions

Shares
Downloads
127
Comments
0
Likes
51

Embeds 6,387

http://d.hatena.ne.jp 4,707
http://open-groove.net 1,325
http://narupeko.soup.io 247
http://paper.li 53
http://a0.twimg.com 17
https://twitter.com 12
http://webcache.googleusercontent.com 10
http://takeoverapp.com 5
https://www.google.co.jp 3
http://wiki.onakasuita.org 2
https://www.facebook.com 1
https://si0.twimg.com 1
http://translate.googleusercontent.com 1
http://us-w1.rockmelt.com 1
http://tweetedtimes.com 1
http://www.google.co.jp 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Kafka Twitter: yanaoki 2011/11/27 16 +WEB http://www.flickr.com/photos/devnull/19765635/2011 11 27
  • 2. • • • Kafka • Kafka2011 11 27
  • 3. • Twitter: @yanaoki • • • Java Ruby Hadoop/Mahout Cassandra2011 11 27
  • 4. • • • 20112011 11 27
  • 5. Facebook Insights • 2011 3 • “Like” ”Share” CTR • Facebook2011 11 27
  • 6. Google Analytics • 2011 9 • • PV UU •2011 11 27
  • 7. Twitter Web Analytics • 2011 09 • Twitter • Twitter •2011 11 27
  • 8. Linktedin • Kafka • PV • • • •2011 11 27
  • 9. • Facebook Insight • PUMA Scribe / HDFS / pTail / HBase /Thrift • http://slidesha.re/ijWfPh • Twitter Promoted Tweets Reporting • Rainbird ZooKeeper / Cassandra • http://slidesha.re/dRxtIp • Twitter Web Analytics • Storm Zookieper • http://slidesha.re/qbpKbY • Google Analytics • • LinktedIn • Kafka ZooKeeper2011 11 27
  • 10. Kafka • Kafka • • Linkedin • http://incubator.apache.org •2011 11 27
  • 11. Linkedin • • SNS • • 2011 11 • Kafka • 2010 11 • 2011 07 Apache incubator project • a Distributed Messaging System for Log Processing • http://research.microsoft.com/en-us/um/people/ srikanth/netdb11/netdb11papers/netdb11-final12.pdf2011 11 27
  • 12. Kafka • • Kafka • • • • Hadoop/HDFS • DWH •2011 11 27
  • 13. 2011 11 27
  • 14. • • ZooKeeper2011 11 27
  • 15. • • SPOF • ZooKeeper •2011 11 27
  • 16. Push or Pull • Push • scribe flume • Pull • Kafka •2011 11 27
  • 17. • Pub/Sub • • Publish • • Subscribe • subscribe • Kafka2011 11 27
  • 18. • • • • • Kafka • Zookeeper •2011 11 27
  • 19. • • • • • ZooKeeper • • •2011 11 27
  • 20. 2011 11 27
  • 21. 2011 11 27
  • 22. A B D C2011 11 27
  • 23. A B D C2011 11 27
  • 24. A B D C2011 11 27
  • 25. ● ● ― ― ● ● ● ― ● ● ● ● ● ● ZK ● ● ● ZooKeeper ― ●2011 11 27
  • 26. • • Kafka O(log n) O(1) • OS • Java GC • BTree2011 11 27
  • 27. • • • • Java NIO (※ Linux sendfile )2011 11 27
  • 28. HUG January 2011 Kafka Presentation http://www.slideshare.net/ydn/hug-january-2011-kafka-presentation2011 11 27
  • 29. 2011 11 27
  • 30. Facebook • MapReduce(Not HadoopMR) • Scribe/PTail/Puma Map • HBase Reduce http://www.slideshare.net/tatsuya6502/tokyo-hbase-meetup-realtime-big-data-at-facebook-ja Map Reduce2011 11 27
  • 31. Kafka HBase • Twitter • (en ja lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 Kafka Broker2011 11 27
  • 32. Kafka HBase • Twitter lang client lang HBaseImport Comsumer TwitterStreaming Producer ja→10 en→32 client web → 100 iPhoone→10 Kafka Broker Android→102011 11 27
  • 33. Hadoop • Hadoop • Hadoop Map Kafka • KafkaETLJob / KafkaETLInputFormat / KafkaETLRecordReader • MapReuduce API2011 11 27
  • 34. Hadoop Offset Offset HDFS Map Offset Offset Limit map → Kafka Mapper Kafka Reducer HDFS2011 11 27
  • 35. 2011 11 27