Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

データの民主化のために StackStorm を活用した事例

2,090 views

Published on

Tech Night @ Shiodome #8 https://techsio.connpass.com/event/83273/ 発表資料

Published in: Technology
  • Be the first to comment

データの民主化のために StackStorm を活用した事例

  1. 1. StackStorm / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • StackStorm • • • 2
  3. 3. : / @laclefyoshi • 2011/04 • 2015/09 • • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) • Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017) • Apache Kafka 
 (Apache Kafka Meetup Japan #4; 2018) 3
  4. 4. StackStorm
  5. 5. StackStorm • https://stackstorm.com/ • OSS 
 + LDAP Auth plugin https://git.io/vxbbJ • • Shell Script • Crontab 5
  6. 6. • JP1: • Digdag: • Jenkins: CI • Airflow: GCP Cloud Composer • Oozie: Hadoop 6
  7. 7. StackStorm • • High Availability • Crontab Scheduler • • • • Runner Action Workflow • • REST API 7
  8. 8. StackStorm API 1. test.test Action 
 
 
 2. API pack: test name: test runner_type: remote-shell-cmd parameters: p: type: string required: true curl -k -X POST -H "Content-type: application/json" -H "X-Auth-Token: $TOKEN" --data-binary '{"action": “test.test", 
 “parameters": {"p": "..."}}' "https://${ST2_ADDR}/api/v1/executions" 8
  9. 9. StackStorm • Design UI Commercial • GUI • OSS • Commercial Role Based Access Control • OSS • Action Workflow • Jinja/YAQL 9
  10. 10. st2XXX, mistral-XXX replication cluster mistral (PostgreSQL 9.6) Runner Action Workflow Rule 10
  11. 11. • • 1 • Timer • st2rulesengine Timer HA • https://docs.stackstorm.com/reference/ha.html#st2rulesengine • MongoDB Primary st2rulesengine Timer • Concurrency policy Zookeeper Timer 11
  12. 12. GCP? • BigQuery 
 • BigQuery • SQL • Tableau • Pandas R • StackStorm BigQuery 12
  13. 13. • • DB • • • → 
 14
  14. 14. 1/2 • 15
  15. 15. 2/2 • StackStorm 16
  16. 16. StackStorm 1. StackStorm 2. 
 3. StackStorm 4. StackStorm 5. 6. 17
  17. 17. StackStorm • Amazon Data Value Chain 
 Subscribe Deliver https://www.slideshare.net/AmazonWebServices/a-look-under-the-hood-how- amazoncom-uses-aws-services-for-analytics-at-massive-scale-abd329-reinvent-2017 18
  18. 18. Action • Action • • • • LDAP • Stackdriver • Action • SQL BigQuery • BigQuery CSV • 19
  19. 19. • StackStorm Action • Action Workflow 
 • YAML • 
 • YAML 
 • JP1 GUI 20
  20. 20. 21
  21. 21. API • YAML • GUI API • Action Workflow • StackStorm Action Workflow API • API • 22
  22. 22. 1/2 23 2. StackStorm Web UI 
 StackStorm API 1. Action Workflow API
  23. 23. 2/2 24 JP1 2. JP1 1 StackStorm API 1. Action Workflow API
  24. 24. • StackStorm • API • StackStorm • • Rule • JP1 • Workflow • 25
  25. 25. 26 • StackStorm • Action Workflow YAML • • Action Workflow

×