StackMateYour friend in the Cloud Business@chiradeep
Overview• Introduction to CloudFormation• What it takes• Progress• TODO• FAQ• Speculation
CloudFormation• AWS service introduced in 2011• Application Management using blueprints• Integrates with provisioning tool...
Example: XenApp Farm in the Cloud
Example: XenApp farm in VPC61 resources23 resource types53 orderingdependencies
ComplexOrdering ofResourcecreation
CloudFormation TemplateDeclarative specification of an application ‘stack’– Cloud resources and services• Compute, Network...
Typed and Validated ParametersDBName": {"Default": "MyDatabase","Description" : "MySQL database name","Type": "String","Mi...
ResourcesWebServer: {"Type": "AWS::EC2::Instance","Metadata" : {#bootstrap script fetched by cfn-init},"Properties": {"Ima...
Outputs"Outputs" : {"WebsiteURL" : {"Value" : { "Fn::Join" : ["", ["http://", { "Fn::GetAtt" :[ "WebServer", "PublicDnsNam...
Wait Conditions• Special resource for temporal ordering ofapplication provisioning– E.g., ensure DB service is up in DB in...
Stack Operations• Create– cfn-create-stack --stack-name -f, --template-file | -u, --template-url -d, --disable-rollback -p...
StackMateCloudFormation Template Parser / Validator+Resource creation / deletion orchestrator+Wait condition serverForClou...
StackMate ArchitectureApplicationTemplate Stacker• Parse• Validate• Resolvedependencies• Execution planStackExecutor• Work...
Stacker• Parsing– Simple: JSON.parse(File.read(template))• Resolve dependencies– Recurse through json data structure looki...
Directed Acyclic Graph of dependencies Ruote Process DefinitionSequence of CloudStack API calls
Q: Why Ruote (or why use workflow)• A: it is a (long-running, distributed) workflow– Persistence– Recovery– Rollback– Poss...
• Stacktician is a web application thatembeds StackMate• Graphical UI• Wait condition server• Database persistence enables...
Stacktician ArchitectureStackMateStacker StackExecutorPersistenceExtensionsThe Rails logo is a registered trademarks of Da...
Stacktician Architecture• Ruby on Rails application• StackMate gem is used for parsing, validation, etc• Bootstrap-based U...
Native CloudStack Resources• Resources that do not have an AWS relative– E.g., firewall rule, port forwarding rule• Resour...
StackMate next steps• Support delete stack– Add tags to resources created by StackMate– Delete resources in reverse order•...
Stacktician next steps• Support delete stack, atomic operation• Support more resources (AWS and CS-specific)• Integrate wi...
Stacktician next steps• Allow user to specify URL for template content• Scaling using Resque to send API jobs tobackend wo...
Stacktician future• Metadata server– Instance not limited to 32k of userdata at boot• Update template semantics– Tricky wi...
FAQ• Does it work with Chef/Puppet– Yes, use userdata or metadata (future) to configure chef solo orchef client or puppet ...
FAQ• License– MIT• Where– https://github.com/chiradeep/stackmate– https://github.com/chiradeep/stacktician• Contribution– ...
Stacktician FuturesStackMateStacker StackExecutorPersistenceExtensionsThe Rails logo is a registered trademarks of David H...
Stacktician Futures• Complex multi-service orchestration– E.g., deploy my app on CS in zone 1 and thenregister the public ...
StackMate - CloudFormation for CloudStack
Upcoming SlideShare
Loading in …5
×

StackMate - CloudFormation for CloudStack

3,280 views

Published on

Stackmate and Stacktician are orchestrators that allow CloudFormation templates to be deployed on CloudStack.

Published in: Technology, Business
  • Be the first to comment

StackMate - CloudFormation for CloudStack

  1. 1. StackMateYour friend in the Cloud Business@chiradeep
  2. 2. Overview• Introduction to CloudFormation• What it takes• Progress• TODO• FAQ• Speculation
  3. 3. CloudFormation• AWS service introduced in 2011• Application Management using blueprints• Integrates with provisioning tools within theinstance• “Infrastructure as code”– Readable (json)– Domain specific (cloud resources jargon)– Reviewable (text document)– Reusable
  4. 4. Example: XenApp Farm in the Cloud
  5. 5. Example: XenApp farm in VPC61 resources23 resource types53 orderingdependencies
  6. 6. ComplexOrdering ofResourcecreation
  7. 7. CloudFormation TemplateDeclarative specification of an application ‘stack’– Cloud resources and services• Compute, Network, Storage, Object Storage, etc– Parameters and properties to customize theresources– Outputs (metadata) generated by the creation ofthe stack– Implicit or explicit ordering of resource creationText file in JSON format.
  8. 8. Typed and Validated ParametersDBName": {"Default": "MyDatabase","Description" : "MySQL database name","Type": "String","MinLength": "1","MaxLength": "64","AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*","ConstraintDescription" : "must begin with a letter and contain only alphanumericcharacters."},InstanceType" : {"Description" : "WebServer EC2 instance type","Type" : "String","Default" : "m1.small","AllowedValues" : ["t1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m2.xlarge","m2.2xlarge","m2.4xlarge","m3.xlarge","m3.2xlarge","c1.medium","c1.xlarge","cc1.4xlarge","cc2.8xlarge","cg1.4xlarge"],"ConstraintDescription" : "must be a valid EC2 instance type."},Default ValueTypeConstraints
  9. 9. ResourcesWebServer: {"Type": "AWS::EC2::Instance","Metadata" : {#bootstrap script fetched by cfn-init},"Properties": {"ImageId" : { “Ref” : “ImageId”},"InstanceType" : { "Ref" : "InstanceType" },"SecurityGroups" : [ {"Ref" : "WebServerSecurityGroup"} ],"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", ["#!/bin/bash -vn","yum update -y aws-cfn-bootstrapn”,"# Install LAMP packagesn",”/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r WebServer ”,"# Setup MySQL, create a user and a databasen","mysqladmin -u root password ", { "Ref" : "DBRootPassword" }, " || error_exit Failedto initialize root passwordn","# Configure the PHP application - in this case, fixup the page with the rightreferences to the databasen","sed -i "s/REPLACE_WITH_DATABASE/localhost/g" /var/www/html/index.phpn","# All is well so signal successn","/opt/aws/bin/cfn-signal -e 0 -r "LAMP setup complete" ", { "Ref" : "WaitHandle"}, "n"]]}}}Reference to ParametersReference to Other Resourcesec2-initbootstrapFetchfrommetadataserver
  10. 10. Outputs"Outputs" : {"WebsiteURL" : {"Value" : { "Fn::Join" : ["", ["http://", { "Fn::GetAtt" :[ "WebServer", "PublicDnsName" ]}]] },"Description" : "URL for newly created LAMP stack"}Reference to Resources created by the stack
  11. 11. Wait Conditions• Special resource for temporal ordering ofapplication provisioning– E.g., ensure DB service is up in DB instance beforeinstantiating App Server instance• Wait Condition Handle– URL for application to signal that wait conditionhas been satisfied– Injected into instance via user data
  12. 12. Stack Operations• Create– cfn-create-stack --stack-name -f, --template-file | -u, --template-url -d, --disable-rollback -p, --parameters -t, -timeout– Atomic: all resources are created or none are created– Unless rollback is disabled• Delete– Destroy in proper order• Update– Delta from old template: delete removedresources, create new ones.
  13. 13. StackMateCloudFormation Template Parser / Validator+Resource creation / deletion orchestrator+Wait condition serverForCloudStack
  14. 14. StackMate ArchitectureApplicationTemplate Stacker• Parse• Validate• Resolvedependencies• Execution planStackExecutor• Workflow engine• Execution of plan• CS API clientCloudStackMgmtServerOutputTemplateParameters,CS API key,MappingsRuote is a Workflow engine written in Ruby
  15. 15. Stacker• Parsing– Simple: JSON.parse(File.read(template))• Resolve dependencies– Recurse through json data structure looking for ‘Ref:’hashes.– Fill in parameters if possible– Build dependency graph for each resource• Execution plan– Topological sort of resource dependency graph– Rollback plan: reverse sort
  16. 16. Directed Acyclic Graph of dependencies Ruote Process DefinitionSequence of CloudStack API calls
  17. 17. Q: Why Ruote (or why use workflow)• A: it is a (long-running, distributed) workflow– Persistence– Recovery– Rollback– Possible parallelism• Topological sort does not need to produce linear plan• Ruote is mature– Used by Rightscale, EngineYard
  18. 18. • Stacktician is a web application thatembeds StackMate• Graphical UI• Wait condition server• Database persistence enables• Query of stack execution status• Single sign-on with CloudStackcredentials• Stack execution history
  19. 19. Stacktician ArchitectureStackMateStacker StackExecutorPersistenceExtensionsThe Rails logo is a registered trademarks of David Heinemeier HanssonStackModelCloudStackMgmtServerRest API via browserDBStackCtrller
  20. 20. Stacktician Architecture• Ruby on Rails application• StackMate gem is used for parsing, validation, etc• Bootstrap-based UI• Ruote worker threads– One worker thread for persistence– One thread per resource per stack• Thread is mostly idle (sleep, waiting for CS API call to finish)• Can run Ruote workers outside of Rails for scale• Cloudstack Ruby client• Session is persisted in db
  21. 21. Native CloudStack Resources• Resources that do not have an AWS relative– E.g., firewall rule, port forwarding rule• Resources that are modeled differently– LB, Autoscale, IAM• E.g.,:– CS::Compute::FirewallRule– CS::Compute::PortForwardingRule– CS:Compute::IsolatedNetwork• Needs work
  22. 22. StackMate next steps• Support delete stack– Add tags to resources created by StackMate– Delete resources in reverse order• Support more AWS resources– Only Instance, SecurityGroup and WaitConditiontoday– Need VPC support• Support CS resource types in template
  23. 23. Stacktician next steps• Support delete stack, atomic operation• Support more resources (AWS and CS-specific)• Integrate with CloudStack authentication• Email /AMQP notification of stack events• Support CF Query API in addition to Rails-based REST API
  24. 24. Stacktician next steps• Allow user to specify URL for template content• Scaling using Resque to send API jobs tobackend workers.• Full featured admin interface
  25. 25. Stacktician future• Metadata server– Instance not limited to 32k of userdata at boot• Update template semantics– Tricky with corner cases• First create new resources in new template• Delete resources not found in new template
  26. 26. FAQ• Does it work with Chef/Puppet– Yes, use userdata or metadata (future) to configure chef solo orchef client or puppet client• Why not use Chef/Puppet– Workflow– Atomic operation– Support for more CS resource types• Did you look at <xyz> project that does something similar?– Nope• What about TOSCA ?– Similar, but at a meta-level. That is, TOSCA does not specify theresources, but a way to specify resources and dependencies.Still early
  27. 27. FAQ• License– MIT• Where– https://github.com/chiradeep/stackmate– https://github.com/chiradeep/stacktician• Contribution– Pull requests accepted happily
  28. 28. Stacktician FuturesStackMateStacker StackExecutorPersistenceExtensionsThe Rails logo is a registered trademarks of David Heinemeier HanssonStackModelCloudStackMgmtServerRest API via browserStackCtrllerGoogleComputeEngineS3(e.g., RiakCS, Ceph,Cloudian)XYZ PAASonCloudStackxAAS onCloudStackxAAS onxyzCloud
  29. 29. Stacktician Futures• Complex multi-service orchestration– E.g., deploy my app on CS in zone 1 and thenregister the public IP in a availability monitoringservice– Deploy my app in CS, then call a load testingservice in the Google cloud– Deploy my app in CS, using storage fromS3, Google Storage and Azure Storage– etcThe Rails logo is a registered trademarks of David Heinemeier Hansson

×