Hadoop　～Yahoo! JAPANの活用について～

オープンソースカンファレンス 2010 Tokyo/Fall Hadoop ～ Yahoo! JAPAN の活用について～ 2010/9/10 ヤフー株式会社 R&D 統括本部角田直行、吉田一星

自己紹介 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop 関連プロダクト Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止大規模データ処理用スクリプト言語 Pig Hive Facebook が開発扱いが一般データベースに似ている A = load 'passwd' using PigStorage(':'); B = foreach A generate $0 as id; dump B; CREATE TABLE pokes (foo INT, bar STRING); SELECT a.foo FROM pokes a;

Hadoop 関連プロダクト Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Hadoop 上に構築された列指向データベース Google BigTable のクローン HBase 機械学習ライブラリ Hadoop でスケール可 Mahout 複数の MapReduce ジョブなどを実行制御するワークフロー Oozie

The New York Times ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止

簡単に使える環境 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止

その他の事例 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止

Hadoop Client ,[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 $ hadoop fs –put localfile hdfs://nn.example.com:9000/hadoop/hadoopfile $ hadoop fs –get hdfs://nn.example.com/user/hadoop/file localfile

Hadoop Client ,[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 <property> <name>fs.default.name</name> <value> hdfs://nn.example.com:9000 </value> </property> - core-site.xml <property> <name>mapred.job.tracker</name> <value> nn.example.com:9000 </value> </property> - mapred-site.xml

Hadoop の user/group ,[object Object],[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止　 public final static String USER_NAME_COMMAND = " whoami "; 　 public static String[] getGROUPS_COMMAND() { return new String[]{" bash ", " -c ", " groups "}; } static String getUnixUserName() throws IOException { String[] result = executeShellCommand( new String[]{Shell.USER_NAME_COMMAND}); return result[0]; } UnixUserGroupInformation.java Shell.java

任意の user/group を指定 ,[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 <property> <name> hadoop.job.ugi </name> <value> user,group </value> </property> JobConf conf = new JobConf(Test.class); conf.set(UnixUserGroupInformation.UGI_PROPERTY_NAME,” user,group ”); - core-site.xml - MapReduce

Hadoop with security ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止

パラメータチューニング ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止

ソートのチューニング ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Copyright © 2010 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止

Hadoop　～Yahoo! JAPANの活用について～

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (18)

Similar to Hadoop　～Yahoo! JAPANの活用について～

Similar to Hadoop　～Yahoo! JAPANの活用について～ (20)

More from Yahoo!デベロッパーネットワーク

More from Yahoo!デベロッパーネットワーク (20)

Recently uploaded

Recently uploaded (10)