• Save
Jubatus workshop - 分散処理の仕組み
Upcoming SlideShare
Loading in...5
×
 

Jubatus workshop - 分散処理の仕組み

on

  • 3,818 views

 

Statistics

Views

Total Views
3,818
Views on SlideShare
3,809
Embed Views
9

Actions

Likes
11
Downloads
0
Comments
0

2 Embeds 9

https://twitter.com 8
https://si0.twimg.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Jubatus workshop - 分散処理の仕組み Jubatus workshop - 分散処理の仕組み Presentation Transcript

  • Copyright(c)2011 NTT CORPORATION. All Rights Reserved.分散処理の仕組み NTT 情報流通プラットフォーム研究所 上西康太 2011/11/7 © 2011 NTT Information Sharing Platform Laboratories
  • いろいろ• Pythonクライアント refined by repeatedly• https://github.com/jubatus/jubatus-python-client• Google Groupsつくりました• http://groups.google.com/group/jubatus2 © 2011 NTT Information Sharing Platform Laboratories
  • ありがとうございます• Rubyクライアント• https://github.com/pfi/jubatus-ruby-client• PHPクライアント by @oxycaster• https://github.com/oxalis-gps/jubatus-php-client• MacPorts (!)• http://d.hatena.ne.jp/hjym_u/20111107/132064755 73 © 2011 NTT Information Sharing Platform Laboratories
  • プロセス構成と通信 RPC JubaClassifier クライアント JubaKeeper mix RPC 死活監視4 © 2011 NTT Information Sharing Platform Laboratories
  • 学習・推論時のクライアントのシーケンス JubaClassifier クライアント JubaKeeper5 © 2011 NTT Information Sharing Platform Laboratories
  • それぞれの計算の擬似コードtypedef String Labeltrain:: Model-> {Label, Datum} -> Modelclassify :: Model-> Datum -> [{Label, Double}]mix :: [Model] -> [Model]mix models = let mixees = map (fun Model ->Mixee) models let mixee = fold_left(lambda Mixee->Mixee->Mixee) 0 mixees let apply = (lambda Mixee-> Model-> Model) mixee map apply models6 © 2011 NTT Information Sharing Platform Laboratories
  • MIX JubaClassifier クライアント JubaKeeper 1.学習がたまるor 一定時間経過 2.ZooKeeperで ロックをとる 3.全員から get_diff 4.XXを計算 5.全員にput_diff7 © 2011 NTT Information Sharing Platform Laboratories
  • Mixのシーケンス(今のところ) get_diff ZK lock みっくす put_diff ZK unlock8 © 2011 NTT Information Sharing Platform Laboratories
  • 将来の話: Streaming もしたいストリーム処理のロジック抽象化と物理リソースの分離(したい)→状況に応じて最適化集計、フィルタなど機械学習以外の処理も追加できる(ようにしたい)Applicativeをコンビネータでつなぐ、とか… fv_convert train/classify fv_convert train/classify fv_convert train/classify fv_convert train/classifyclient client client client client aggregate fv_convert train/classify fv_convert train/classify fv_convert train/classify FilterX train/classify9 © 2011 NTT Information Sharing Platform Laboratories
  • 将来のはなし • Distributed RPC • Horizontal Scalability • Complex Event Processing DSL10 © 2011 NTT Information Sharing Platform Laboratories
  • Thank you11 © 2011 NTT Information Sharing Platform Laboratories