Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

データの民主化のために StackStorm を活用した事例

2,940 views

Published on

Tech Night @ Shiodome #8 https://techsio.connpass.com/event/83273/ 発表資料

Published in: Technology
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

データの民主化のために StackStorm を活用した事例

  1. 1. StackStorm / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • StackStorm • • • 2
  3. 3. : / @laclefyoshi • 2011/04 • 2015/09 • • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) • Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017) • Apache Kafka 
 (Apache Kafka Meetup Japan #4; 2018) 3
  4. 4. StackStorm
  5. 5. StackStorm • https://stackstorm.com/ • OSS 
 + LDAP Auth plugin https://git.io/vxbbJ • • Shell Script • Crontab 5
  6. 6. • JP1: • Digdag: • Jenkins: CI • Airflow: GCP Cloud Composer • Oozie: Hadoop 6
  7. 7. StackStorm • • High Availability • Crontab Scheduler • • • • Runner Action Workflow • • REST API 7
  8. 8. StackStorm API 1. test.test Action 
 
 
 2. API pack: test name: test runner_type: remote-shell-cmd parameters: p: type: string required: true curl -k -X POST -H "Content-type: application/json" -H "X-Auth-Token: $TOKEN" --data-binary '{"action": “test.test", 
 “parameters": {"p": "..."}}' "https://${ST2_ADDR}/api/v1/executions" 8
  9. 9. StackStorm • Design UI Commercial • GUI • OSS • Commercial Role Based Access Control • OSS • Action Workflow • Jinja/YAQL 9
  10. 10. st2XXX, mistral-XXX replication cluster mistral (PostgreSQL 9.6) Runner Action Workflow Rule 10
  11. 11. • • 1 • Timer • st2rulesengine Timer HA • https://docs.stackstorm.com/reference/ha.html#st2rulesengine • MongoDB Primary st2rulesengine Timer • Concurrency policy Zookeeper Timer 11
  12. 12. GCP? • BigQuery 
 • BigQuery • SQL • Tableau • Pandas R • StackStorm BigQuery 12
  13. 13. • • DB • • • → 
 14
  14. 14. 1/2 • 15
  15. 15. 2/2 • StackStorm 16
  16. 16. StackStorm 1. StackStorm 2. 
 3. StackStorm 4. StackStorm 5. 6. 17
  17. 17. StackStorm • Amazon Data Value Chain 
 Subscribe Deliver https://www.slideshare.net/AmazonWebServices/a-look-under-the-hood-how- amazoncom-uses-aws-services-for-analytics-at-massive-scale-abd329-reinvent-2017 18
  18. 18. Action • Action • • • • LDAP • Stackdriver • Action • SQL BigQuery • BigQuery CSV • 19
  19. 19. • StackStorm Action • Action Workflow 
 • YAML • 
 • YAML 
 • JP1 GUI 20
  20. 20. 21
  21. 21. API • YAML • GUI API • Action Workflow • StackStorm Action Workflow API • API • 22
  22. 22. 1/2 23 2. StackStorm Web UI 
 StackStorm API 1. Action Workflow API
  23. 23. 2/2 24 JP1 2. JP1 1 StackStorm API 1. Action Workflow API
  24. 24. • StackStorm • API • StackStorm • • Rule • JP1 • Workflow • 25
  25. 25. 26 • StackStorm • Action Workflow YAML • • Action Workflow

×