Your SlideShare is downloading. ×
Hadoop Conference Japan 2011 Fallに行ってきました
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Hadoop Conference Japan 2011 Fallに行ってきました

4,394
views

Published on

Published in: Technology, Business

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,394
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
21
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. @just_do_neet 1
  • 2. Hadoop Conference Japan 2011 fall• 2011/9/26( ) http://hadoop-conference-japan-2011-fall.eventbrite.com/• Hadoop 1000• Cloudera HortonWorks MapR• NTT mixi Yahoo 2
  • 3. http://itpro.nikkeibp.co.jp/article/NEWS/20110926/369421/?SS=imgview&FD=-821521671&ST=cloud 3
  • 4. 4
  • 5. 5
  • 6. Hadoop T 6
  • 7. Hadoop 7
  • 8. Hadoop• Apache Software Foundation OSS• • :MapReduce • :HDFS/HBase 8
  • 9. Hadoop• Google GFS(ACM ’03) “The Google File System” → HDFS MapReduce(OSDI ’04) “Simplified Data Processing on Large Clusters” → MapReduce BigTable(OSDI ’06) “A Distributed Storage System for Structured Data” → HBase• Yahoo! Inc. Cloudera Doug Cutting Google 2004 Java OSS ”Hadoop” Doug 9
  • 10. Hadoop• index index etc... ※Google ※ Doug Lucene Nutch OSS• Hadoop• Hadoop 10
  • 11. Hadoop• HBase BigTable• ZooKeeper Chubby• Hive HDFS SQL• Flume• and more.... 11
  • 12. • sorry, confidential... 12
  • 13. 13
  • 14. Hadoop Conference Japan 2011 Fall• Hadoop 0.23•• 14
  • 15. Hadoop 0.23• 2011 Q4 beta Integration Test Pig• Hadoop NameNode SPoF (NameNode• MapReduce v2 : MapReduce MPI(C Giraph Hama Graph Spark etc• MapReduce shuffle 30% / 10,000 4000 / etc 15
  • 16. Giraph• Hadoop https://github.com/aching/Giraph Graph → Google PageRank Google Pregel(SIGMOD ’10)• BSP(Bulk Synchronous Parallel 16
  • 17. Spark• Scala http://www.spark-project.org/ → ex.• Mesos http://www.mesosproject.org/ Hadoop 17
  • 18. H• Cloudera http://www.cloudera.com/ OSS (CDH Sqoop Hue• HortonWorks http://www.hortonworks.com/ Yahoo! Inc. Hadoop• MapR http://www.mapr.com/ Hadoop• Hadoop 18
  • 19. H Cloudera• CDH Linux• OSS Sqoop SQL HDFS Hue Web Console etc..• SCM Express Web Hadoop 50 free• Cloudera Enterprise Cloudera Management Suite• 19
  • 20. H HortonWorks• Yahoo! Inc. Hadoop 22 Hadoop ※ ( 500,000• 2011/7• Yahoo! Inc. Yahoo! Mail 42,000 Hadoop• Horton the Elephant http://en.wikipedia.org/wiki/Horton_the_Elephant 20
  • 21. H MapR• Hadoop EMC OEM Greenplum http://www.greenplum.com/• HDFS C++ HDFS random read/write NIC bonding snapshot etc.. NFS Hadoop R• OSS Hadoop 21
  • 22. • Hadoop LSP etc..• Asakusa Framework OSS https://github.com/asakusafw DSL 22
  • 23. Yahoo! Japan• Hadoop Geohash Hadoop Iterable Twitter ※ MPI ... 23
  • 24. 24
  • 25. • Hadoop BtoC Hadoop 0.23 MapReduce v2 Hadoop• Hadoop Hadoop Map/Reduce + Graph BI 25
  • 26. 26