The document provides an overview of 'Stackmate,' a tool that integrates with AWS CloudFormation to manage cloud resources using a JSON-based templating system. It describes the functionalities including parsing, validation, and execution of stack templates while ensuring proper resource creation order and dependency management. Additionally, future directions for Stackmate and Stacktician are mentioned, focusing on extending support for more resource types and improved orchestration capabilities.

1. StackMate
Your friend in the Cloud Business
@chiradeep

2. Overview
• Introduction to CloudFormation
• What it takes
• Progress
• TODO
• FAQ
• Speculation

3. CloudFormation
• AWS service introduced in 2011
• Application Management using blueprints
• Integrates with provisioning tools within the
instance
• “Infrastructure as code”
– Readable (json)
– Domain specific (cloud resources jargon)
– Reviewable (text document)
– Reusable

4. Example: XenApp Farm in the Cloud

5. Example: XenApp farm in VPC
61 resources
23 resource types
53 ordering
dependencies

6. Complex
Ordering of
Resource
creation

7. CloudFormation Template
Declarative specification of an application ‘stack’
– Cloud resources and services
• Compute, Network, Storage, Object Storage, etc
– Parameters and properties to customize the
resources
– Outputs (metadata) generated by the creation of
the stack
– Implicit or explicit ordering of resource creation
Text file in JSON format.

8. Typed and Validated Parameters
DBName": {
"Default": "MyDatabase",
"Description" : "MySQL database name",
"Type": "String",
"MinLength": "1",
"MaxLength": "64",
"AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription" : "must begin with a letter and contain only alphanumeric
characters."
},
InstanceType" : {
"Description" : "WebServer EC2 instance type",
"Type" : "String",
"Default" : "m1.small",
"AllowedValues" : [
"t1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m2.xlarge","m2.2xlarge","m2.4
xlarge","m3.xlarge","m3.2xlarge","c1.medium","c1.xlarge","cc1.4xlarge","cc2.8xlarge","c
g1.4xlarge"],
"ConstraintDescription" : "must be a valid EC2 instance type."
},
Default Value
Type
Constraints

9. Resources
WebServer: {
"Type": "AWS::EC2::Instance",
"Metadata" : {
#bootstrap script fetched by cfn-init
},
"Properties": {
"ImageId" : { “Ref” : “ImageId”},
"InstanceType" : { "Ref" : "InstanceType" },
"SecurityGroups" : [ {"Ref" : "WebServerSecurityGroup"} ],
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash -vn",
"yum update -y aws-cfn-bootstrapn”,
"# Install LAMP packagesn",
”/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r WebServer ”,
"# Setup MySQL, create a user and a databasen",
"mysqladmin -u root password '", { "Ref" : "DBRootPassword" }, "' || error_exit 'Failed
to initialize root password'n",
"# Configure the PHP application - in this case, fixup the page with the right
references to the databasen",
"sed -i "s/REPLACE_WITH_DATABASE/localhost/g" /var/www/html/index.phpn",
"# All is well so signal successn",
"/opt/aws/bin/cfn-signal -e 0 -r "LAMP setup complete" '", { "Ref" : "WaitHandle"
}, "'n"
]]}}
}
Reference to Parameters
Reference to Other Resources
ec2-initbootstrap
Fetchfrommetadataserver

10. Outputs
"Outputs" : {
"WebsiteURL" : {
"Value" : { "Fn::Join" : ["", ["http://", { "Fn::GetAtt" :
[ "WebServer", "PublicDnsName" ]}]] },
"Description" : "URL for newly created LAMP stack"
}
Reference to Resources created by the stack

11. Wait Conditions
• Special resource for temporal ordering of
application provisioning
– E.g., ensure DB service is up in DB instance before
instantiating App Server instance
• Wait Condition Handle
– URL for application to signal that wait condition
has been satisfied
– Injected into instance via user data

12. Stack Operations
• Create
– cfn-create-stack --stack-name -f, --template-file | -u, --template-url -
d, --disable-rollback -p, --parameters -t, -timeout
– Atomic: all resources are created or none are created
– Unless rollback is disabled
• Delete
– Destroy in proper order
• Update
– Delta from old template: delete removed
resources, create new ones.

13. StackMate
CloudFormation Template Parser / Validator
+
Resource creation / deletion orchestrator
+
Wait condition server
For
CloudStack

14. StackMate Architecture
Application
Template Stacker
• Parse
• Validate
• Resolve
dependencies
• Execution plan
StackExecutor
• Workflow engine
• Execution of plan
• CS API client
CloudStack
Mgmt
Server
Output
Template
Parameters,
CS API key,
Mappings
Ruote is a Workflow engine written in Ruby

15. Stacker
• Parsing
– Simple: JSON.parse(File.read(template))
• Resolve dependencies
– Recurse through json data structure looking for ‘Ref:’
hashes.
– Fill in parameters if possible
– Build dependency graph for each resource
• Execution plan
– Topological sort of resource dependency graph
– Rollback plan: reverse sort

16. Directed Acyclic Graph of dependencies Ruote Process Definition
Sequence of CloudStack API calls

17. Q: Why Ruote (or why use workflow)
• A: it is a (long-running, distributed) workflow
– Persistence
– Recovery
– Rollback
– Possible parallelism
• Topological sort does not need to produce linear plan
• Ruote is mature
– Used by Rightscale, EngineYard

18. • Stacktician is a web application that
embeds StackMate
• Graphical UI
• Wait condition server
• Database persistence enables
• Query of stack execution status
• Single sign-on with CloudStack
credentials
• Stack execution history

20. Stacktician Architecture
StackMate
Stacker StackExecutor
Persistence
Extensions
The Rails logo is a registered trademarks of David Heinemeier Hansson
Stack
Model
CloudStack
Mgmt
Server
Rest API via browser
DB
Stack
Ctrller

21. Stacktician Architecture
• Ruby on Rails application
• StackMate gem is used for parsing, validation, etc
• Bootstrap-based UI
• Ruote worker threads
– One worker thread for persistence
– One thread per resource per stack
• Thread is mostly idle (sleep, waiting for CS API call to finish)
• Can run Ruote workers outside of Rails for scale
• Cloudstack Ruby client
• Session is persisted in db

22. Native CloudStack Resources
• Resources that do not have an AWS relative
– E.g., firewall rule, port forwarding rule
• Resources that are modeled differently
– LB, Autoscale, IAM
• E.g.,:
– CS::Compute::FirewallRule
– CS::Compute::PortForwardingRule
– CS:Compute::IsolatedNetwork
• Needs work

23. StackMate next steps
• Support delete stack
– Add tags to resources created by StackMate
– Delete resources in reverse order
• Support more AWS resources
– Only Instance, SecurityGroup and WaitCondition
today
– Need VPC support
• Support CS resource types in template

24. Stacktician next steps
• Support delete stack, atomic operation
• Support more resources (AWS and CS-specific)
• Integrate with CloudStack authentication
• Email /AMQP notification of stack events
• Support CF Query API in addition to Rails-
based REST API

25. Stacktician next steps
• Allow user to specify URL for template content
• Scaling using Resque to send API jobs to
backend workers.
• Full featured admin interface

26. Stacktician future
• Metadata server
– Instance not limited to 32k of userdata at boot
• Update template semantics
– Tricky with corner cases
• First create new resources in new template
• Delete resources not found in new template

27. FAQ
• Does it work with Chef/Puppet
– Yes, use userdata or metadata (future) to configure chef solo or
chef client or puppet client
• Why not use Chef/Puppet
– Workflow
– Atomic operation
– Support for more CS resource types
• Did you look at <xyz> project that does something similar?
– Nope
• What about TOSCA ?
– Similar, but at a meta-level. That is, TOSCA does not specify the
resources, but a way to specify resources and dependencies.
Still early

28. FAQ
• License
– MIT
• Where
– https://github.com/chiradeep/stackmate
– https://github.com/chiradeep/stacktician
• Contribution
– Pull requests accepted happily

29. Stacktician Futures
StackMate
Stacker StackExecutor
Persistence
Extensions
The Rails logo is a registered trademarks of David Heinemeier Hansson
Stack
Model
CloudStack
Mgmt
Server
Rest API via browser
Stack
Ctrller
Google
Compute
Engine
S3
(e.g., Riak
CS, Ceph,
Cloudian)
XYZ PAAS
on
CloudStackxAAS on
CloudStack
xAAS on
xyzCloud

30. Stacktician Futures
• Complex multi-service orchestration
– E.g., deploy my app on CS in zone 1 and then
register the public IP in a availability monitoring
service
– Deploy my app in CS, then call a load testing
service in the Google cloud
– Deploy my app in CS, using storage from
S3, Google Storage and Azure Storage
– etc
The Rails logo is a registered trademarks of David Heinemeier Hansson