Shu Sugimoto presented a demo of StackStorm running on Amazon EKS. He discussed JPNAP's current use of StackStorm for auto-provisioning customer ports using workflows and actions. The demo deployed StackStorm and its dependencies like MongoDB, RabbitMQ and PostgreSQL to an Amazon EKS cluster. It showed bringing down nodes to demonstrate high availability. Considerations were provided around pod affinity, component high availability, and database and queue choices between AWS and GCP.

1. INTERNET MULTIFEED CO.Copyright ©
StackStorm on AWS EKS demo
Shu Sugimoto
Software Development Manager, JPNAP
2018-06-12(Tue)

2. INTERNET MULTIFEED CO.Copyright ©
Who am I
• Shu Sugimoto
• handles: shu, shusgmt
• Software Development Manager, JPNAP
• Agile, DevOps
• based in Tokyo
• JPNAP
• IXP: Internet Exchange Provider
• one of the largest in APAC
• JPNAP Tokyo
• Peak traffic: 1.09Tbps
2

3. INTERNET MULTIFEED CO.Copyright ©
st2 “facts” at JPNAP
• Use case
• Auto-provisioning of customer ports
• Involves switches, routers, servers
• st2 = Workflow engine
• kicks by executions API call
• no sensors
• Packs/Actions/Workflows development
• Mostly develop our own: nothing from exchange packs
• 99% of actions are `remote-shell-cmd` runner
• ~10 private packs, ~50 actions + workflows
• A very few python code
• 100% Mistral
• Heavily dependent on `core.ask`/inquiries
3

4. INTERNET MULTIFEED CO.Copyright ©
st2 “facts” at JPNAP
• “Current” deployment
• st2-docker
• Multi-process mode
• No HA yet
• Principle: Keep important data out of st2
• No Kubernetes yet
• Community LDAP Auth backend enabled
• Packs CI/CD
• CI: Not yet
• CD: Jenkins
4

6. INTERNET MULTIFEED CO.Copyright ©
Demo scenario
• Run several workflows
• Bring down two nodes in same AZ
• See what happens
6

7. INTERNET MULTIFEED CO.Copyright ©
Demo info
• All in AWS Oregon(us-west-2) region
• Amazon EKS
• k8s version: 1.10
• 6x m5.large instances for nodes
• across 3 AZ by autoscaling group
• Amazon EFS for efs-provisioner
• gives RWX storage backend
• Middlewares
• MongoDB: helm chart: stable/mongodb-replicaset
• 3 replicas (default)
• RabbitMQ: helm chart: stable/rabbitmq-ha
• 3 replicas (default)
• PostgreSQL: Amazon RDS Aurora PostgreSQL
• db.r4.large
• 2 AZ HA Cluster
• Redis: Amazon ElastiCache Redis
• cache.m4.large
• 3 AZ HA Cluster
• StackStorm
• All components are scaled to 6 pods
• …except following components:
• st2sensorcontainer
• st2rulesengine
• st2resultstracker (we don’t need it any more in st2 2.7)

8. INTERNET MULTIFEED CO.Copyright ©
Setup summary
• create k8s cluster with `eksctl`
• edit Launch Configuration for nodes to add `yum -y install nfs-utils` to support efs-
provisioner, and replace all nodes
• configure EBS storage class
• create EFS
• configure efs-provisioner
• setup helm
• install MongoDB with helm chart
• install RabbitMQ with helm chart
• create Aurora PostgreSQL instance
• create ElastiCache Redis instance
• prepare configmap
• kubectl apply -f configmaps.yml
• kubectl apply -f st2.yml
• kubectl apply -f init-st2.yml
• detailed step-by-step guide is here:
• https://qiita.com/shusugmt/items/1cdb382579352c67dc08
• (use Google translate!)
8

9. INTERNET MULTIFEED CO.Copyright ©
Caveats / Considerations
• Pod affinity needs to be set for real HA scenario
• Following components (still) need HA specific care
• st2sensorcontainer
• st2rulesengine
9

10. INTERNET MULTIFEED CO.Copyright ©
Caveats / Considerations
• Building k8s cluster with EKS is much harder than GKE
• `eksctl` solves this
• Running and managing cluster: ?
• GKE provides “1 click upgrade” solution
• RWX PV problem can be solved in AWS with EFS
• GCP/GKE still lacks built-in support for RWX PV
• Need custom built image without RWX PV
https://github.com/shusugmt/st2-docker-pack-prebuild-
example
• EFS = managed NFS
• Performance, Stability: ?
10

11. INTERNET MULTIFEED CO.Copyright ©
Caveats / Considerations
• MongoDB / RabbitMQ
• helm chart seems good choice
• Amazon MQ has no luck since it is ActiveMQ
• st2 can’t support ActiveMQ since underlying library (which
is celery/kombu) can’t talk AMQP1.0
• RDBMS for Mistral
• AWS: Amazon RDS
• GCP: Cloud SQL
• Redis for coordination backend
• AWS: Amazon ElastiCache
• GCP: ?
• helm chart available: stable/redis-ha
11