Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

20140905 AWS Night in ITHD LT2

1,218 views

Published on

Twitter - (AWS Kinesis | Apache Kafka) - Apache Storm のサンプル実装についてのスライド

Published in: Technology
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ...................................ALL FOR EBOOKS................................................. Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • -- DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT -- ......................................................................................................................... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... (Unlimited)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THI5 BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

20140905 AWS Night in ITHD LT2

  1. 1. AWS Night in ITHD 2014.09.05 LT② AWS Kinesis & Apache Storm を題材にしたクラウド可搬性 TIS STC 松井暢之
  2. 2. タイトル 堅すぎた orz...
  3. 3. 松井暢之(まついのぶゆき) @n_matsui nbyk.matsui nmatsui TIS株式会社 コーポレート本部 戦略技術センター ~2003 2003~2008 2009 2010~2012 現場PJでアーキテクト兼モデラー兼プログラマ兼…を歴任 基盤技術センター(現戦略技術センター)にて不芳PJの火消しに奔走 全社生産性向上の企画策定に従事 オープンでエッジな技術を活用した事業企画に従事 2013~ の企画開発を開始 http://cloudconductor.org
  4. 4. 某お客様の言葉 Amazon Redshift を使ってみたい けれども
  5. 5. 某お客様の言葉 AWSから別環境へ 乗り換えられなく なると困るので
  6. 6. 某お客様の言葉 導入に躊躇してる んだよ。 ?
  7. 7. 某お客様の言葉 結局今は、最も安 かった某クラウドを 使っているけれど
  8. 8. 某お客様の言葉 より良いクラウドが 出てくればまた乗 り換えるだろうね。
  9. 9. この場で 言うのも アレですが
  10. 10. クラウド乗り換えのニーズ • お客様は本来、必要なサー ビスが予算内で利用できる ならば、クラウドベンダは問 わないはず
  11. 11. マネージドサービスの活用 • 一方各種マネージドサービ スは、適切に使えば運用 負荷やコストを激減できる ため可能であれば活用す べき
  12. 12. • これらの相反するニーズを解 決するのが、SIerの腕のみ せどころ
  13. 13. • SIerであるからには、適切に マネージドサービスを活用し、 必要となれば簡単にクラウド を乗り換えられる柔軟なアー キテクチャを 考案すべき
  14. 14. システムサービスインテグレータ • これからのSIerは、クラウド サービスに精通し、クラウド 可搬性を担保できるシステ ム構築能力が必須となるだ ろう
  15. 15. ということで タイトルを 回収したので
  16. 16. 本題
  17. 17. こんなの作っ た
  18. 18. TweetStreamWordCounter Amazon Kinesis KafkaとKinesisは 任意に選択可能 Bolt 1 Bolt 2 Bolt 3 Bolt 1 : KafkaとKinesisの差異吸収 Bolt 2 : 形態素解析とフィルタリング Bolt 3 : 語句出現頻度カウント Producer Apache Kafka Redis Kinesis Spout Kafka Spout ApacheStorm Twitter
  19. 19. TweetStreamWordCounter • Twitterから取得したTweetを形態素解析し、 語句の出現頻度をリアルタイムにカウントし続 ける • Tweetをリアルタイムに分散処理するエンジン(複 合イベント処理エンジン)として、Apache Stromを 利用
  20. 20. TweetStreamWordCounter • 入力データがバーストし、複合イベント処理エン ジンの処理能力を一時的に超えても 耐えられるように、スケーラブルでPull型の ストリーミングエンジンを置く • Amazon KinesisやApache Kafkaを利用
  21. 21. デモ
  22. 22. 地味 でした
  23. 23. ソースコード(github) • 環境構築(Vagrantfile) • nmatsui/twitter-kinesis-kafka-storm_vagrant • Tweet収集と転送(Producer) • nmatsui/twitter-kinesis-kafka-storm_producer • リアルタイム処理(Storm) • nmatsui/twitter-kinesis-kafka-storm_wordcounter
  24. 24. 実は環境構築に苦労したという • Kinesisはすごい簡単 • しかしKafka+Stromの環境設定で苦労 • kafka serverのエンドポイントはIPアドレスで指定 すると動かないとか • ビルドにもひたすら苦労 • awslabs/kinesis-storm-spoutがバグってるわ pom.xmlも無いわ、apache stormが依存するjar がMaven Centralに無いわ
  25. 25. CloudConductor • システムのインフラ設計や運用手順等を パターン化し、様々なクラウドで自動構築 を可能にすることで、インフラ運用の標準化・省 力化を図りクラウド可搬性を向上させる クラウドオーケストレーションツール を目指して鋭意開発中
  26. 26. CloudConductor • 公式サイト • http://cloudconductor.org/ • ソースコード(github) • cloudconductor/cloud_conductor • 最新はdevelopブランチ
  27. 27. 以上

×