Hadoop Conference Japan 2011 Fallに行ってきました
Upcoming SlideShare
Loading in...5
×
 

Hadoop Conference Japan 2011 Fallに行ってきました

on

  • 4,056 views

 

Statistics

Views

Total Views
4,056
Views on SlideShare
3,622
Embed Views
434

Actions

Likes
2
Downloads
20
Comments
0

5 Embeds 434

http://dayafterneet.blogspot.com 338
http://dayafterneet.blogspot.jp 91
http://paper.li 2
http://dayafterneet.blogspot.sg 2
http://webcache.googleusercontent.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Hadoop Conference Japan 2011 Fallに行ってきました Hadoop Conference Japan 2011 Fallに行ってきました Presentation Transcript

    • @just_do_neet 1
    • Hadoop Conference Japan 2011 fall• 2011/9/26( ) http://hadoop-conference-japan-2011-fall.eventbrite.com/• Hadoop 1000• Cloudera HortonWorks MapR• NTT mixi Yahoo 2
    • http://itpro.nikkeibp.co.jp/article/NEWS/20110926/369421/?SS=imgview&FD=-821521671&ST=cloud 3
    • 4
    • 5
    • Hadoop T 6
    • Hadoop 7
    • Hadoop• Apache Software Foundation OSS• • :MapReduce • :HDFS/HBase 8
    • Hadoop• Google GFS(ACM ’03) “The Google File System” → HDFS MapReduce(OSDI ’04) “Simplified Data Processing on Large Clusters” → MapReduce BigTable(OSDI ’06) “A Distributed Storage System for Structured Data” → HBase• Yahoo! Inc. Cloudera Doug Cutting Google 2004 Java OSS ”Hadoop” Doug 9
    • Hadoop• index index etc... ※Google ※ Doug Lucene Nutch OSS• Hadoop• Hadoop 10
    • Hadoop• HBase BigTable• ZooKeeper Chubby• Hive HDFS SQL• Flume• and more.... 11
    • • sorry, confidential... 12
    • 13
    • Hadoop Conference Japan 2011 Fall• Hadoop 0.23•• 14
    • Hadoop 0.23• 2011 Q4 beta Integration Test Pig• Hadoop NameNode SPoF (NameNode• MapReduce v2 : MapReduce MPI(C Giraph Hama Graph Spark etc• MapReduce shuffle 30% / 10,000 4000 / etc 15
    • Giraph• Hadoop https://github.com/aching/Giraph Graph → Google PageRank Google Pregel(SIGMOD ’10)• BSP(Bulk Synchronous Parallel 16
    • Spark• Scala http://www.spark-project.org/ → ex.• Mesos http://www.mesosproject.org/ Hadoop 17
    • H• Cloudera http://www.cloudera.com/ OSS (CDH Sqoop Hue• HortonWorks http://www.hortonworks.com/ Yahoo! Inc. Hadoop• MapR http://www.mapr.com/ Hadoop• Hadoop 18
    • H Cloudera• CDH Linux• OSS Sqoop SQL HDFS Hue Web Console etc..• SCM Express Web Hadoop 50 free• Cloudera Enterprise Cloudera Management Suite• 19
    • H HortonWorks• Yahoo! Inc. Hadoop 22 Hadoop ※ ( 500,000• 2011/7• Yahoo! Inc. Yahoo! Mail 42,000 Hadoop• Horton the Elephant http://en.wikipedia.org/wiki/Horton_the_Elephant 20
    • H MapR• Hadoop EMC OEM Greenplum http://www.greenplum.com/• HDFS C++ HDFS random read/write NIC bonding snapshot etc.. NFS Hadoop R• OSS Hadoop 21
    • • Hadoop LSP etc..• Asakusa Framework OSS https://github.com/asakusafw DSL 22
    • Yahoo! Japan• Hadoop Geohash Hadoop Iterable Twitter ※ MPI ... 23
    • 24
    • • Hadoop BtoC Hadoop 0.23 MapReduce v2 Hadoop• Hadoop Hadoop Map/Reduce + Graph BI 25
    • 26