Successfully reported this slideshow.
Your SlideShare is downloading. ×

データの民主化のために StackStorm を活用した事例

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

データの民主化のために StackStorm を活用した事例

  1. 1. StackStorm / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • StackStorm • • • 2
  3. 3. : / @laclefyoshi • 2011/04 • 2015/09 • • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) • Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017) • Apache Kafka 
 (Apache Kafka Meetup Japan #4; 2018) 3
  4. 4. StackStorm
  5. 5. StackStorm • https://stackstorm.com/ • OSS 
 + LDAP Auth plugin https://git.io/vxbbJ • • Shell Script • Crontab 5
  6. 6. • JP1: • Digdag: • Jenkins: CI • Airflow: GCP Cloud Composer • Oozie: Hadoop 6
  7. 7. StackStorm • • High Availability • Crontab Scheduler • • • • Runner Action Workflow • • REST API 7
  8. 8. StackStorm API 1. test.test Action 
 
 
 2. API pack: test name: test runner_type: remote-shell-cmd parameters: p: type: string required: true curl -k -X POST -H "Content-type: application/json" -H "X-Auth-Token: $TOKEN" --data-binary '{"action": “test.test", 
 “parameters": {"p": "..."}}' "https://${ST2_ADDR}/api/v1/executions" 8
  9. 9. StackStorm • Design UI Commercial • GUI • OSS • Commercial Role Based Access Control • OSS • Action Workflow • Jinja/YAQL 9
  10. 10. st2XXX, mistral-XXX replication cluster mistral (PostgreSQL 9.6) Runner Action Workflow Rule 10
  11. 11. • • 1 • Timer • st2rulesengine Timer HA • https://docs.stackstorm.com/reference/ha.html#st2rulesengine • MongoDB Primary st2rulesengine Timer • Concurrency policy Zookeeper Timer 11
  12. 12. GCP? • BigQuery 
 • BigQuery • SQL • Tableau • Pandas R • StackStorm BigQuery 12
  13. 13. • • DB • • • → 
 14
  14. 14. 1/2 • 15
  15. 15. 2/2 • StackStorm 16
  16. 16. StackStorm 1. StackStorm 2. 
 3. StackStorm 4. StackStorm 5. 6. 17
  17. 17. StackStorm • Amazon Data Value Chain 
 Subscribe Deliver https://www.slideshare.net/AmazonWebServices/a-look-under-the-hood-how- amazoncom-uses-aws-services-for-analytics-at-massive-scale-abd329-reinvent-2017 18
  18. 18. Action • Action • • • • LDAP • Stackdriver • Action • SQL BigQuery • BigQuery CSV • 19
  19. 19. • StackStorm Action • Action Workflow 
 • YAML • 
 • YAML 
 • JP1 GUI 20
  20. 20. 21
  21. 21. API • YAML • GUI API • Action Workflow • StackStorm Action Workflow API • API • 22
  22. 22. 1/2 23 2. StackStorm Web UI 
 StackStorm API 1. Action Workflow API
  23. 23. 2/2 24 JP1 2. JP1 1 StackStorm API 1. Action Workflow API
  24. 24. • StackStorm • API • StackStorm • • Rule • JP1 • Workflow • 25
  25. 25. 26 • StackStorm • Action Workflow YAML • • Action Workflow

×