Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spark GraphX で始めるグラフ解析

2,239 views

Published on

Ichigaya Geek Night Lightning Talk (2016-12-21)

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Spark GraphX で始めるグラフ解析

  1. 1. Spark GraphX で始めるグラフ解析 市ヶ谷Geek★Night#11 Lightning Talk 2016-12-21 @mogproject
  2. 2. About me http://mogproject.com
  3. 3. Spark GraphX Apache Spark GraphX とは ▸Spark に標準で組み込まれているグラフ処理 API ▸Spark クラスタ = データストア / 計算資源 ▸大規模グラフを扱いやすい
  4. 4. Spark GraphX 今回お話しする「グラフ」は・・・
  5. 5. Why GraphX? 実用例 ▸大規模グラフの分析プロジェクト (論文の共著者関係分析) ▸ノード数: 500万〜 エッジ数: 9000万〜 ▸最初は Python製 の NetworkX を使っていた ▸グラフが大きくなるにつれ、メモリ不足が発生 ▸Scala / Spark に慣れているなら手軽に利用可能
  6. 6. Why GraphX? 実用例: (無向)グラフの中心性計算 ▸次数中心性 ▸次数 = 各ノードに接続しているエッジの数 ▸直接的なつながりの多いノードを発見できる ▸媒介中心性 ▸全てのノードの組に対して最短経路を求める ▸より多くの組の最短経路上に位置するノードを計算
  7. 7. コード例 build.sbt ▸"spark-graphx" への依存を追加 ▸Scala 2.12 には未対応 scalaVersion := "2.11.8" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.0.2", "org.apache.spark" %% "spark-graphx" % "2.0.2" ) build.sbt
  8. 8. コード例 グラフデータの読み込み ▸エッジリストからグラフを生成する例 0 1 0 2 1 2 1 3 3 4 4 5 4 6 5 6 edge_list_1.txt
  9. 9. コード例 グラフデータの読み込み ▸GraphLoader のメソッドを利用する import org.apache.spark._ import org.apache.spark.graphx._ val conf = new SparkConf().setMaster(s"local[2]") val sc = new SparkContext(conf) val g: Graph[Int, Int] = GraphLoader.edgeListFile(sc, "path/to/edge_list.txt") Main.scala
  10. 10. コード例 次数中心性の計算 ▸Graph#degrees を呼び出すだけ g.degrees.sortByKey().collect().foreach { case (n, v) => println(s"Node: ${n} -> Degree: ${v}") } Node: 0 -> Degree: 2 Node: 1 -> Degree: 3 Node: 2 -> Degree: 2 Node: 3 -> Degree: 2 Node: 4 -> Degree: 3 Node: 5 -> Degree: 2 Node: 6 -> Degree: 2 Main.scala 出力例
  11. 11. コード例 次数中心性の計算 3 2 3 2 2 2 2
  12. 12. コード例 媒介中心性の計算 ▸なかなか大変 (説明省略)
  13. 13. コード例 媒介中心性の計算 ▸実行結果 Node: 0 -> Betweenness: 0.0 Node: 1 -> Betweenness: 8.0 Node: 2 -> Betweenness: 0.0 Node: 3 -> Betweenness: 9.0 Node: 4 -> Betweenness: 8.0 Node: 5 -> Betweenness: 0.0 Node: 6 -> Betweenness: 0.0 出力例
  14. 14. コード例 媒介中心性の計算 8 0 8 0 9 0 0
  15. 15. 後日談 性能面の課題 ▸グラフ操作の種類 (アルゴリズム) によって得手不得手があ る ▸Spark ノード間の通信コスト ▸Spark の UI でモニタリングしつつチューニングを繰り返す
  16. 16. 後日談 C++ で実装し直した ▸とある処理の所要時間: 854日 (GraphX) -> 15日 (C++) 60 倍 高速化!
  17. 17. Thanks! ご清聴ありがとうございました ▸コード例はこちら ▸https://github.com/mogproject/example-graphx クリスマスに働く人のイラスト ©︎いらすとや

×