@just_do_neet                1
Hadoop Conference Japan 2011 fall• 2011/9/26( )    http://hadoop-conference-japan-2011-fall.eventbrite.com/•              ...
http://itpro.nikkeibp.co.jp/article/NEWS/20110926/369421/?SS=imgview&FD=-821521671&ST=cloud                               ...
4
5
Hadoop   T             6
Hadoop         7
Hadoop• Apache Software Foundation   OSS•    •        :MapReduce    •        :HDFS/HBase                                  ...
Hadoop• Google    GFS(ACM ’03) “The Google File System” → HDFS    MapReduce(OSDI ’04) “Simplified Data Processing on Large ...
Hadoop•                                           index                index                              etc...    ※Googl...
Hadoop• HBase   BigTable• ZooKeeper    Chubby• Hive    HDFS         SQL• Flume• and more....                       11
• sorry, confidential...                          12
13
Hadoop Conference Japan 2011 Fall• Hadoop         0.23••                                    14
Hadoop 0.23• 2011 Q4 beta                   Integration Test   Pig•           Hadoop    NameNode    SPoF       (NameNode• ...
Giraph• Hadoop https://github.com/aching/Giraph   Graph → Google PageRank   Google   Pregel(SIGMOD ’10)• BSP(Bulk Synchron...
Spark• Scala http://www.spark-project.org/              →     ex.• Mesos http://www.mesosproject.org/          Hadoop     ...
H• Cloudera  http://www.cloudera.com/                                OSS   (CDH Sqoop   Hue• HortonWorks  http://www.horto...
H                               Cloudera•                                        CDH    Linux•                          OS...
H                               HortonWorks•     Yahoo! Inc.       Hadoop                  22    Hadoop                   ...
H                              MapR•          Hadoop    EMC OEM              Greenplum    http://www.greenplum.com/•    HD...
•                         Hadoop                                   LSP   etc..•                                           ...
Yahoo! Japan•              Hadoop    Geohash              Hadoop     Iterable    Twitter    ※            MPI          ... ...
24
• Hadoop                  BtoC  Hadoop 0.23   MapReduce v2            Hadoop• Hadoop                       Hadoop  Map/Red...
26
Upcoming SlideShare
Loading in...5
×

Hadoop Conference Japan 2011 Fallに行ってきました

4,473

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,473
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
21
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Hadoop Conference Japan 2011 Fallに行ってきました

  1. 1. @just_do_neet 1
  2. 2. Hadoop Conference Japan 2011 fall• 2011/9/26( ) http://hadoop-conference-japan-2011-fall.eventbrite.com/• Hadoop 1000• Cloudera HortonWorks MapR• NTT mixi Yahoo 2
  3. 3. http://itpro.nikkeibp.co.jp/article/NEWS/20110926/369421/?SS=imgview&FD=-821521671&ST=cloud 3
  4. 4. 4
  5. 5. 5
  6. 6. Hadoop T 6
  7. 7. Hadoop 7
  8. 8. Hadoop• Apache Software Foundation OSS• • :MapReduce • :HDFS/HBase 8
  9. 9. Hadoop• Google GFS(ACM ’03) “The Google File System” → HDFS MapReduce(OSDI ’04) “Simplified Data Processing on Large Clusters” → MapReduce BigTable(OSDI ’06) “A Distributed Storage System for Structured Data” → HBase• Yahoo! Inc. Cloudera Doug Cutting Google 2004 Java OSS ”Hadoop” Doug 9
  10. 10. Hadoop• index index etc... ※Google ※ Doug Lucene Nutch OSS• Hadoop• Hadoop 10
  11. 11. Hadoop• HBase BigTable• ZooKeeper Chubby• Hive HDFS SQL• Flume• and more.... 11
  12. 12. • sorry, confidential... 12
  13. 13. 13
  14. 14. Hadoop Conference Japan 2011 Fall• Hadoop 0.23•• 14
  15. 15. Hadoop 0.23• 2011 Q4 beta Integration Test Pig• Hadoop NameNode SPoF (NameNode• MapReduce v2 : MapReduce MPI(C Giraph Hama Graph Spark etc• MapReduce shuffle 30% / 10,000 4000 / etc 15
  16. 16. Giraph• Hadoop https://github.com/aching/Giraph Graph → Google PageRank Google Pregel(SIGMOD ’10)• BSP(Bulk Synchronous Parallel 16
  17. 17. Spark• Scala http://www.spark-project.org/ → ex.• Mesos http://www.mesosproject.org/ Hadoop 17
  18. 18. H• Cloudera http://www.cloudera.com/ OSS (CDH Sqoop Hue• HortonWorks http://www.hortonworks.com/ Yahoo! Inc. Hadoop• MapR http://www.mapr.com/ Hadoop• Hadoop 18
  19. 19. H Cloudera• CDH Linux• OSS Sqoop SQL HDFS Hue Web Console etc..• SCM Express Web Hadoop 50 free• Cloudera Enterprise Cloudera Management Suite• 19
  20. 20. H HortonWorks• Yahoo! Inc. Hadoop 22 Hadoop ※ ( 500,000• 2011/7• Yahoo! Inc. Yahoo! Mail 42,000 Hadoop• Horton the Elephant http://en.wikipedia.org/wiki/Horton_the_Elephant 20
  21. 21. H MapR• Hadoop EMC OEM Greenplum http://www.greenplum.com/• HDFS C++ HDFS random read/write NIC bonding snapshot etc.. NFS Hadoop R• OSS Hadoop 21
  22. 22. • Hadoop LSP etc..• Asakusa Framework OSS https://github.com/asakusafw DSL 22
  23. 23. Yahoo! Japan• Hadoop Geohash Hadoop Iterable Twitter ※ MPI ... 23
  24. 24. 24
  25. 25. • Hadoop BtoC Hadoop 0.23 MapReduce v2 Hadoop• Hadoop Hadoop Map/Reduce + Graph BI 25
  26. 26. 26
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×