Hadoop Conference Japan 2011 Fallに行ってきました

4,909 views

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,909
On SlideShare
0
From Embeds
0
Number of Embeds
458
Actions
Shares
0
Downloads
22
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Hadoop Conference Japan 2011 Fallに行ってきました

  1. 1. @just_do_neet 1
  2. 2. Hadoop Conference Japan 2011 fall• 2011/9/26( ) http://hadoop-conference-japan-2011-fall.eventbrite.com/• Hadoop 1000• Cloudera HortonWorks MapR• NTT mixi Yahoo 2
  3. 3. http://itpro.nikkeibp.co.jp/article/NEWS/20110926/369421/?SS=imgview&FD=-821521671&ST=cloud 3
  4. 4. 4
  5. 5. 5
  6. 6. Hadoop T 6
  7. 7. Hadoop 7
  8. 8. Hadoop• Apache Software Foundation OSS• • :MapReduce • :HDFS/HBase 8
  9. 9. Hadoop• Google GFS(ACM ’03) “The Google File System” → HDFS MapReduce(OSDI ’04) “Simplified Data Processing on Large Clusters” → MapReduce BigTable(OSDI ’06) “A Distributed Storage System for Structured Data” → HBase• Yahoo! Inc. Cloudera Doug Cutting Google 2004 Java OSS ”Hadoop” Doug 9
  10. 10. Hadoop• index index etc... ※Google ※ Doug Lucene Nutch OSS• Hadoop• Hadoop 10
  11. 11. Hadoop• HBase BigTable• ZooKeeper Chubby• Hive HDFS SQL• Flume• and more.... 11
  12. 12. • sorry, confidential... 12
  13. 13. 13
  14. 14. Hadoop Conference Japan 2011 Fall• Hadoop 0.23•• 14
  15. 15. Hadoop 0.23• 2011 Q4 beta Integration Test Pig• Hadoop NameNode SPoF (NameNode• MapReduce v2 : MapReduce MPI(C Giraph Hama Graph Spark etc• MapReduce shuffle 30% / 10,000 4000 / etc 15
  16. 16. Giraph• Hadoop https://github.com/aching/Giraph Graph → Google PageRank Google Pregel(SIGMOD ’10)• BSP(Bulk Synchronous Parallel 16
  17. 17. Spark• Scala http://www.spark-project.org/ → ex.• Mesos http://www.mesosproject.org/ Hadoop 17
  18. 18. H• Cloudera http://www.cloudera.com/ OSS (CDH Sqoop Hue• HortonWorks http://www.hortonworks.com/ Yahoo! Inc. Hadoop• MapR http://www.mapr.com/ Hadoop• Hadoop 18
  19. 19. H Cloudera• CDH Linux• OSS Sqoop SQL HDFS Hue Web Console etc..• SCM Express Web Hadoop 50 free• Cloudera Enterprise Cloudera Management Suite• 19
  20. 20. H HortonWorks• Yahoo! Inc. Hadoop 22 Hadoop ※ ( 500,000• 2011/7• Yahoo! Inc. Yahoo! Mail 42,000 Hadoop• Horton the Elephant http://en.wikipedia.org/wiki/Horton_the_Elephant 20
  21. 21. H MapR• Hadoop EMC OEM Greenplum http://www.greenplum.com/• HDFS C++ HDFS random read/write NIC bonding snapshot etc.. NFS Hadoop R• OSS Hadoop 21
  22. 22. • Hadoop LSP etc..• Asakusa Framework OSS https://github.com/asakusafw DSL 22
  23. 23. Yahoo! Japan• Hadoop Geohash Hadoop Iterable Twitter ※ MPI ... 23
  24. 24. 24
  25. 25. • Hadoop BtoC Hadoop 0.23 MapReduce v2 Hadoop• Hadoop Hadoop Map/Reduce + Graph BI 25
  26. 26. 26

×