Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Elasticsearchで作る形態素解析サーバ

6,318 views

Published on

ElasticsearchにAnalyze APIプラグインをインストールすることで形態素解析サーバを作成することができます。

Published in: Technology
  • DOWNLOAD THI5 BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Elasticsearchで作る形態素解析サーバ

  1. 1. Elasticsearchで作る形態素解析サーバ 第10回elasticsearch勉強会
  2. 2. 第10回elasticsearch勉強会 ■ 名前: 菅谷信介 ■ 所属: N2SM, Inc. ■ オープンソース活動: • Apache Portals (Jetspeed2) コミッタ • Codehausコミッタ • Seasar Projectコミッタ • CodeLibsプロジェクト運営               などなど・・・ ■ Blog: http://www.chazine.com/ ■ Twitter: https://twitter.com/shinsuke_sugaya/ 自己紹介 2
  3. 3. 第10回elasticsearch勉強会 まずは背景から… 3
  4. 4. 第10回elasticsearch勉強会 今回の話の背景 4 ■ Elasticsearchにいろいろとデータを入れている ■ そのデータを使って、機械学習をいろいろと適用する ■ 日本語のテキストデータだと、形態素解析が必要 ■ Java以外にもPythonを使ってみたり… ■ Mecabとか使うなら環境作らないと… ■ 動かしたいOSもLinuxだったり、Macだったり… ■ Mecabとかチューニングどうするのだろう… ➔ 環境構築とかいろいろと面倒なことが多すぎる…
  5. 5. 第10回elasticsearch勉強会 !? 5
  6. 6. 第10回elasticsearch勉強会 Elasticsearchに Analyze APIが あるではないか! 6
  7. 7. 第10回elasticsearch勉強会 Analyze API https://www.elastic.co/guide/en/elasticsearch/reference/1.5/indices-analyze.html 7 ■ Elasticsearchが提供するRESTなAPI ■ 指定したAnalyzerでテキストを分解してくれる $ curl -XGET 'localhost:9200/_analyze?analyzer=standard' -d 'this is a test' { "tokens" : [ { "token" : "this", "start_offset" : 0, "end_offset" : 4, "type" : "<ALPHANUM>", "position" : 1 }, { "token" : "is", "start_offset" : 5, "end_offset" : 7, …
  8. 8. 第10回elasticsearch勉強会 日本語だと品詞とかも 取得して、 いろいろと調整したいな… 8
  9. 9. 第10回elasticsearch勉強会 Analyze APIで 取得できない… 9
  10. 10. 第10回elasticsearch勉強会 !? 10
  11. 11. 第10回elasticsearch勉強会 そういえば、 Extend _analyze APIなら いろいろと取得できたな… 11
  12. 12. 第10回elasticsearch勉強会 Extend _analyze API https://github.com/johtani/elasticsearch-extended-analyze 12 ■ @johtaniさんが提供するプラグイン ■ RESTなAPIでAnalyze情報を細かく取得できる $ curl -XGET 'localhost:9200/_extended_analyze?analyzer=standard' -d 'this is a test' { "custom_analyzer" : false, "analyzer" : { "standard" : [ { "token" : "this", "start_offset" : 0, "end_offset" : 4, "type" : "<ALPHANUM>", "position" : 1, "extended_attributes" : { "org.apache.lucene.analysis.tokenattributes.PositionLengthAttribute" : { "positionLength" : 1 },
  13. 13. 第10回elasticsearch勉強会 おー、 attributeもいろいろ取れて 便利、便利ー 13
  14. 14. 第10回elasticsearch勉強会 …がしかし、 ガンガン呼び出すと 遅い… パフォーマンス問題が… 14
  15. 15. 第10回elasticsearch勉強会 Extend _analyze APIは 詳細なAnalyze状況を 確認するものだから そもそも用途が 間違っている… 15
  16. 16. 第10回elasticsearch勉強会 !? 16
  17. 17. 第10回elasticsearch勉強会 作るしかない! 17
  18. 18. 第10回elasticsearch勉強会 要件 「最小限の情報だけ返す」 「複数のAnalyzeを1回で実行」 18
  19. 19. 第10回elasticsearch勉強会 Analyze APIプラグイン https://github.com/codelibs/elasticsearch-analyze-api 19 ■ RESTでAnalyzeするElasticsearchプラグイン ■ 指定されたattribute情報だけを返却する ■ 1リクエストで複数のAnalzyeを実行することが可能 ■ Analyze用のインデックスを作成して利用 curl -XPOST "localhost:9200/_analyze_api" -d'{ "{target_name1}":{ "index":"{index_name}", "analyzer":"{analyzer_name}" "text":"{target_text1}" }, "{target_name2}":{ "index":"{index_name}", "analyzer":"{analyzer_name}" "text":"{target_text2}" }, … }'
  20. 20. 第10回elasticsearch勉強会 Analyze APIプラグインの例 https://github.com/codelibs/elasticsearch-analyze-api 20 $ curl -XPOST “localhost:9200/_analyze_api?start_offset=true" -d '{“text1”:{“index”:”.analyzer”,”analyzer”:”standard”,”text”:”This is a Pen...”},...}’ { “text1”: [ { “term”: ”this”, “start_offset”: 0 }, { “term”: ”is”, “start_offset”: 5 }, ... “text2”: [ { “term”: ”that” } ...
  21. 21. 第10回elasticsearch勉強会 まとめ 21
  22. 22. 第10回elasticsearch勉強会 ■ Analyze APIプラグインを作成 ■ Analyze(形態素解析)サーバを構築することが可能 ➜ 構築が容易(プラグインをインストールするだけ) ■ Analyzerの構成をElasticsearchの機能で管理( ➜ 設定の変更もREST APIで可能 ■ 指定したAttributeだけを取得可能 ■ 複数のテキストを1つのリクエストで複数のAnalyzeが 可能 まとめ 22

×