SlideShare a Scribd company logo
1 of 31
Download to read offline
Moving up the stack with
Stacktician
Amogh Vasekar
Agenda
• Introduction
• History
• Architecture and Implementation
• Current State
• Enhancements
Agenda
• Introduction
• History
• Architecture and Implementation
• Current State
• Enhancements
Introduction
What is Amazon CloudFormation?
•Create and manage collection of related AWS resources
•Typically, the resources will form your whole application
“stack”
In essence, “Infrastructure-as-code”
•JSON file
•Describe resources
•Describe dependencies
•Readable, reusable, reviewable
Sample JSON template
Usefulness
• Single click application deployment for multiple users
• Transparent, reusable
• Ease of resource management and dependencies
• Atomic deployments
• Thus, migrate new business applications to “cloud”
Simply put, CloudFormation for CloudStack 
Web interface for template and stack management
Agenda
• Introduction
• History
• Architecture and Implementation
• Current State
• Enhancements
History
• Started by @chiradeep
• Introduced at CCC-13 as a prototype
• Execute AWS CloudFormation templates on CloudStack
*AS IS*
• Problem – low fidelity between AWS and
CloudStack resource types (more on this later)
Agenda
• Introduction
• History
• Architecture and
Implementation
• Current State
• Enhancements
Architecture
Two components
StackMate
•Execution engine for running the templates and interfacing with CloudStack
Stacktician
•Web interface
•Embeds StackMate underneath
StackMate in 2 mins
• Treat template = stack definition as workflow
• Execute workflow by making appropriate CloudStack API
calls
• Core engine for executing templates, does the heavy lifting
• Command line tool
Stacktician in 2 mins
• Web UI for StackMate
• Backed by database for persistence
• Uses Ruby meta-programming to interface with StackMate
Where we were last
time…
• Resolve complex resource dependencies
• Intrinsic functions
• Wait Conditions
• Create simple resources like AWS::Instance and
AWS::SecurityGroup using StackMate
AWS templates on
CloudStack• Simple resources work well
• Too many differences in more complex resources like VPC and networks
 No route tables
• Semantic differences between APIs for same resources
 Different types of values for parameters
• Dependency resolution for CloudStack using AWS templates became too
complex
 Sometimes, two AWS APIs act as a single one in CloudStack
 CloudStack networking renders AWS options like DhcpOptions
unnecessary
 Too many NoOp resources used as work-around for above
CloudStack Namespace
• Instead create CloudStack specific namespace!
• No more mapping of resources from AWS namespace to
CloudStack space
• Adds the ability to validate API parameters
• Every CloudStack resource becomes a “participant” in the
workflow execution
• Essentially need to call create and delete APIs
CloudStack Namespace
• Problems
 New resources added frequently (e.g tags, UCS, Nicira)
 APIs evolve to support new parameters etc.
 Too many CloudStack resources to maintain (~51)
• Solution
 Use API discovery
 Auto generate code
 Use API information to validate all required parameters exist / format
of parameters
StackMate participants
• Bootstrap StackMate
• Publish CloudStack version specific gem
Errors and rollback
• Want stack creation to be atomic
• But, provide an option to user to disable rollback (e.g. for
debugging)
 Catch resource creation errors, and initiate rollback
 Execute deletion in reverse order of dependencies
 Skip any resources not yet created
 If error while deleting, notify user (log for StackMate, UI status for
Stacktician)
 Stacktician injects additional code in StackMate for DB management
Metadata server
• Resources, like VMs, can launch additional initialization scripts upon
configuration
 E.g. – Launch a script via “userdata” parameter
• Typically, script needs additional metadata to work with
• Metadata is specific to a resource
Request metadata
Metadata server
And expose retrieval as API
Metadata Server
• StackMate
In memory metadata, embeds a Sinatra server for
waitcondition and metadata
• Stacktician
Rails serves metadata, persisted in DB
(Special thanks to Simon Waterhouse for integrating cfn-init scripts and creating scripts for
Windows)
Stacktician Scaling
Improvements• Ruby MRI has global interpreter lock. Eventually, allowed for
only one stack execution thread to run at a time
 Use JRuby container, run in JRuby compatible server
• Async wait for stack : one thread per stack launch would
overwhelm server
 GetBack gem for managing thread pool
Delete Stack
Stacktician
•Resource IDs in database
•Execute delete APIs in reverse order of dependency creation
•Not fully tested yet
StackMate
•No DB, no IDs
•Use resource tags to figure out resources, and delete
•In progress
Improvements
StackMate
•Better logging, log level control
•Light-weight CloudStack client for making API calls
Stacktician
•Supports REST-based API for stack operations, querying and manipulation
•Command line tools for all the above operations
•sm-create-stack, sm-create-template etc
Agenda
• Introduction
• History
• Architecture and Implementation
• Current State
• Enhancements
Demo
Agenda
• Introduction
• History
• Architecture and Implementation
• Current State
• Enhancements
Future enhancements
Better plugin architecture
•Allow creation of custom type of resources
•Interface with other types of Clouds – GCE, AWS etc.
•Allow for scenarios like resource A on CloudStack, B on AWS within single
stack
•Currently only in StackMate, and very rudimentary. Enforces some basic
constraints, uses tag “stackmate_participant = true” for discovery
 Bad plugins rejected at run-time
•Needs more work to make robust
Future enhancements
• Nested stacks
 Treat stack as first class object. Template can contain another stack
definition as a resource
 Works in StackMate, disabled for lack of good implementation
• Support conditional functions in template (e.g Fn:And, Fn:Or,
Fn:If)
• Support update stack
 Semantics to be well defined
Q & A
Questions? Thoughts? Comments? Suggestions?
https://github.com/stackmate
Feel free to log issues – more the better!
Big thanks to Chiradeep for all the help

More Related Content

What's hot

Greetings from AWS User Group Taiwan
Greetings from AWS User Group TaiwanGreetings from AWS User Group Taiwan
Greetings from AWS User Group TaiwanCliff Chao-kuan Lu
 
Hosting Drupal on Amazon EC2
Hosting Drupal on Amazon EC2Hosting Drupal on Amazon EC2
Hosting Drupal on Amazon EC2Kornel Lugosi
 
Batch Processing with Amazon EC2 Container Service
Batch Processing with Amazon EC2 Container ServiceBatch Processing with Amazon EC2 Container Service
Batch Processing with Amazon EC2 Container ServiceAmazon Web Services
 
Grails in the Cloud (2013)
Grails in the Cloud (2013)Grails in the Cloud (2013)
Grails in the Cloud (2013)Meni Lubetkin
 
Utah Codecamp Cloud Computing
Utah Codecamp Cloud ComputingUtah Codecamp Cloud Computing
Utah Codecamp Cloud ComputingTom Creighton
 
Feedback on AWS re:invent 2016
Feedback on AWS re:invent 2016Feedback on AWS re:invent 2016
Feedback on AWS re:invent 2016Laurent Bernaille
 
AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...
AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...
AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...AwsReinventSlides
 
Serverless Framework Intro
Serverless Framework IntroServerless Framework Intro
Serverless Framework IntroNikolaus Graf
 
Meetup #3: Migrate a fast scale system to AWS
Meetup #3: Migrate a fast scale system to AWSMeetup #3: Migrate a fast scale system to AWS
Meetup #3: Migrate a fast scale system to AWSAWS Vietnam Community
 
Serverless Framework (2018)
Serverless Framework (2018)Serverless Framework (2018)
Serverless Framework (2018)Rowell Belen
 
Running High Availability Websites with Acquia and AWS
Running High Availability Websites with Acquia and AWSRunning High Availability Websites with Acquia and AWS
Running High Availability Websites with Acquia and AWSAcquia
 
Cloudsolutionday 2016: Getting Started with Severless Architecture
Cloudsolutionday 2016: Getting Started with Severless ArchitectureCloudsolutionday 2016: Getting Started with Severless Architecture
Cloudsolutionday 2016: Getting Started with Severless ArchitectureAWS Vietnam Community
 
Aws atlanta march_2015
Aws atlanta march_2015Aws atlanta march_2015
Aws atlanta march_2015Adam Book
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent
 
StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)Simba Khadder
 

What's hot (20)

Intro to Serverless
Intro to ServerlessIntro to Serverless
Intro to Serverless
 
Greetings from AWS User Group Taiwan
Greetings from AWS User Group TaiwanGreetings from AWS User Group Taiwan
Greetings from AWS User Group Taiwan
 
[Jun AWS 201] Technical Workshop
[Jun AWS 201] Technical Workshop[Jun AWS 201] Technical Workshop
[Jun AWS 201] Technical Workshop
 
Hosting Drupal on Amazon EC2
Hosting Drupal on Amazon EC2Hosting Drupal on Amazon EC2
Hosting Drupal on Amazon EC2
 
Batch Processing with Amazon EC2 Container Service
Batch Processing with Amazon EC2 Container ServiceBatch Processing with Amazon EC2 Container Service
Batch Processing with Amazon EC2 Container Service
 
104 meets cloud
104 meets cloud104 meets cloud
104 meets cloud
 
Grails in the Cloud (2013)
Grails in the Cloud (2013)Grails in the Cloud (2013)
Grails in the Cloud (2013)
 
Utah Codecamp Cloud Computing
Utah Codecamp Cloud ComputingUtah Codecamp Cloud Computing
Utah Codecamp Cloud Computing
 
Serverless Patterns
Serverless PatternsServerless Patterns
Serverless Patterns
 
Feedback on AWS re:invent 2016
Feedback on AWS re:invent 2016Feedback on AWS re:invent 2016
Feedback on AWS re:invent 2016
 
AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...
AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...
AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...
 
Wikipedia Cloud Search Webinar
Wikipedia Cloud Search WebinarWikipedia Cloud Search Webinar
Wikipedia Cloud Search Webinar
 
Serverless Framework Intro
Serverless Framework IntroServerless Framework Intro
Serverless Framework Intro
 
Meetup #3: Migrate a fast scale system to AWS
Meetup #3: Migrate a fast scale system to AWSMeetup #3: Migrate a fast scale system to AWS
Meetup #3: Migrate a fast scale system to AWS
 
Serverless Framework (2018)
Serverless Framework (2018)Serverless Framework (2018)
Serverless Framework (2018)
 
Running High Availability Websites with Acquia and AWS
Running High Availability Websites with Acquia and AWSRunning High Availability Websites with Acquia and AWS
Running High Availability Websites with Acquia and AWS
 
Cloudsolutionday 2016: Getting Started with Severless Architecture
Cloudsolutionday 2016: Getting Started with Severless ArchitectureCloudsolutionday 2016: Getting Started with Severless Architecture
Cloudsolutionday 2016: Getting Started with Severless Architecture
 
Aws atlanta march_2015
Aws atlanta march_2015Aws atlanta march_2015
Aws atlanta march_2015
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
 
StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)
 

Viewers also liked

In this inquiry we introduce Denmark 10
In this inquiry we introduce Denmark 10In this inquiry we introduce Denmark 10
In this inquiry we introduce Denmark 10Bill Berg
 
Towards automated testing - CloudStack Collab Conference
Towards automated testing - CloudStack Collab ConferenceTowards automated testing - CloudStack Collab Conference
Towards automated testing - CloudStack Collab Conferenceamoghvk
 
Real Time Realitites
Real Time RealititesReal Time Realitites
Real Time Realititesmarkisuak
 
Stay healthy, stay fresh in nature
Stay healthy, stay fresh in natureStay healthy, stay fresh in nature
Stay healthy, stay fresh in natureShima Samuel
 
Meetup - DevCloud and CloudStack 4.2
Meetup - DevCloud and CloudStack 4.2Meetup - DevCloud and CloudStack 4.2
Meetup - DevCloud and CloudStack 4.2amoghvk
 

Viewers also liked (6)

In this inquiry we introduce Denmark 10
In this inquiry we introduce Denmark 10In this inquiry we introduce Denmark 10
In this inquiry we introduce Denmark 10
 
Towards automated testing - CloudStack Collab Conference
Towards automated testing - CloudStack Collab ConferenceTowards automated testing - CloudStack Collab Conference
Towards automated testing - CloudStack Collab Conference
 
Courier website
Courier websiteCourier website
Courier website
 
Real Time Realitites
Real Time RealititesReal Time Realitites
Real Time Realitites
 
Stay healthy, stay fresh in nature
Stay healthy, stay fresh in natureStay healthy, stay fresh in nature
Stay healthy, stay fresh in nature
 
Meetup - DevCloud and CloudStack 4.2
Meetup - DevCloud and CloudStack 4.2Meetup - DevCloud and CloudStack 4.2
Meetup - DevCloud and CloudStack 4.2
 

Similar to Stacktician - CloudStack Collab Conference 2014

StackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStackStackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStackChiradeep Vittal
 
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxBetter, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxJohn Burwell
 
Sebastien goasguen cloud stack the next year
Sebastien goasguen   cloud stack the next yearSebastien goasguen   cloud stack the next year
Sebastien goasguen cloud stack the next yearShapeBlue
 
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
Streaming with Spring Cloud Stream and Apache Kafka - Soby ChackoStreaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
Streaming with Spring Cloud Stream and Apache Kafka - Soby ChackoVMware Tanzu
 
NetflixOSS for Triangle Devops Oct 2013
NetflixOSS for Triangle Devops Oct 2013NetflixOSS for Triangle Devops Oct 2013
NetflixOSS for Triangle Devops Oct 2013aspyker
 
Innovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXCInnovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXCkscaldef
 
Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWSMigrating enterprise workloads to AWS
Migrating enterprise workloads to AWSTom Laszewski
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)Amazon Web Services
 
How IT at Getty Images Brokers Cloud Services
How IT at Getty Images Brokers Cloud ServicesHow IT at Getty Images Brokers Cloud Services
How IT at Getty Images Brokers Cloud ServicesRightScale
 
Cloud Architecture: Patterns and Best Practices
Cloud Architecture: Patterns and Best PracticesCloud Architecture: Patterns and Best Practices
Cloud Architecture: Patterns and Best PracticesSascha Möllering
 
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech TalksDeep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech TalksAmazon Web Services
 
Container Orchestration
Container OrchestrationContainer Orchestration
Container Orchestrationdfilppi
 
Serverless Web Apps using API Gateway, Lambda and DynamoDB
Serverless Web Apps using API Gateway, Lambda and DynamoDBServerless Web Apps using API Gateway, Lambda and DynamoDB
Serverless Web Apps using API Gateway, Lambda and DynamoDBAmazon Web Services
 
Cloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: OpenstackCloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: OpenstackMicrosoft
 
Architectures, Frameworks and Infrastructure
Architectures, Frameworks and InfrastructureArchitectures, Frameworks and Infrastructure
Architectures, Frameworks and Infrastructureharendra_pathak
 
OpenStack: Toward a More Resilient Cloud
OpenStack: Toward a More Resilient CloudOpenStack: Toward a More Resilient Cloud
OpenStack: Toward a More Resilient CloudMark Voelker
 
Velocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ NetflixVelocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ Netflixaspyker
 

Similar to Stacktician - CloudStack Collab Conference 2014 (20)

StackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStackStackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStack
 
Cloud Formation
Cloud FormationCloud Formation
Cloud Formation
 
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxBetter, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
 
Sebastien goasguen cloud stack the next year
Sebastien goasguen   cloud stack the next yearSebastien goasguen   cloud stack the next year
Sebastien goasguen cloud stack the next year
 
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
Streaming with Spring Cloud Stream and Apache Kafka - Soby ChackoStreaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
 
NetflixOSS for Triangle Devops Oct 2013
NetflixOSS for Triangle Devops Oct 2013NetflixOSS for Triangle Devops Oct 2013
NetflixOSS for Triangle Devops Oct 2013
 
Innovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXCInnovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXC
 
Migrating enterprise workloads to AWS
Migrating enterprise workloads to AWSMigrating enterprise workloads to AWS
Migrating enterprise workloads to AWS
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
 
How IT at Getty Images Brokers Cloud Services
How IT at Getty Images Brokers Cloud ServicesHow IT at Getty Images Brokers Cloud Services
How IT at Getty Images Brokers Cloud Services
 
Cloud Architecture: Patterns and Best Practices
Cloud Architecture: Patterns and Best PracticesCloud Architecture: Patterns and Best Practices
Cloud Architecture: Patterns and Best Practices
 
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech TalksDeep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
Deep Dive on AWS Lambda - January 2017 AWS Online Tech Talks
 
Container Orchestration
Container OrchestrationContainer Orchestration
Container Orchestration
 
Serverless Web Apps using API Gateway, Lambda and DynamoDB
Serverless Web Apps using API Gateway, Lambda and DynamoDBServerless Web Apps using API Gateway, Lambda and DynamoDB
Serverless Web Apps using API Gateway, Lambda and DynamoDB
 
Cloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: OpenstackCloud Architect Alliance #15: Openstack
Cloud Architect Alliance #15: Openstack
 
Intro to CloudStack
Intro to CloudStackIntro to CloudStack
Intro to CloudStack
 
Architectures, Frameworks and Infrastructure
Architectures, Frameworks and InfrastructureArchitectures, Frameworks and Infrastructure
Architectures, Frameworks and Infrastructure
 
OpenStack: Toward a More Resilient Cloud
OpenStack: Toward a More Resilient CloudOpenStack: Toward a More Resilient Cloud
OpenStack: Toward a More Resilient Cloud
 
Velocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ NetflixVelocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ Netflix
 
Managing Your Cloud Assets
Managing Your Cloud AssetsManaging Your Cloud Assets
Managing Your Cloud Assets
 

Recently uploaded

Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmComputer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmDeepika Walanjkar
 
CS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfCS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfBalamuruganV28
 
priority interrupt computer organization
priority interrupt computer organizationpriority interrupt computer organization
priority interrupt computer organizationchnrketan
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfalene1
 
Levelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodLevelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodManicka Mamallan Andavar
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSsandhya757531
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.elesangwon
 
multiple access in wireless communication
multiple access in wireless communicationmultiple access in wireless communication
multiple access in wireless communicationpanditadesh123
 
70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical trainingGladiatorsKasper
 
Secure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech LabsSecure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech Labsamber724300
 
Turn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxTurn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxStephen Sitton
 
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONTHE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONjhunlian
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxRomil Mishra
 
Prach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism CommunityPrach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism Communityprachaibot
 
List of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfList of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfisabel213075
 
Robotics Group 10 (Control Schemes) cse.pdf
Robotics Group 10  (Control Schemes) cse.pdfRobotics Group 10  (Control Schemes) cse.pdf
Robotics Group 10 (Control Schemes) cse.pdfsahilsajad201
 
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork
 
Theory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfTheory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfShreyas Pandit
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Romil Mishra
 

Recently uploaded (20)

Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmComputer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
 
CS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfCS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdf
 
priority interrupt computer organization
priority interrupt computer organizationpriority interrupt computer organization
priority interrupt computer organization
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
 
Levelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument methodLevelling - Rise and fall - Height of instrument method
Levelling - Rise and fall - Height of instrument method
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
 
Designing pile caps according to ACI 318-19.pptx
Designing pile caps according to ACI 318-19.pptxDesigning pile caps according to ACI 318-19.pptx
Designing pile caps according to ACI 318-19.pptx
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
 
multiple access in wireless communication
multiple access in wireless communicationmultiple access in wireless communication
multiple access in wireless communication
 
70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training
 
Secure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech LabsSecure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech Labs
 
Turn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxTurn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptx
 
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTIONTHE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
THE SENDAI FRAMEWORK FOR DISASTER RISK REDUCTION
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptx
 
Prach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism CommunityPrach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism Community
 
List of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfList of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdf
 
Robotics Group 10 (Control Schemes) cse.pdf
Robotics Group 10  (Control Schemes) cse.pdfRobotics Group 10  (Control Schemes) cse.pdf
Robotics Group 10 (Control Schemes) cse.pdf
 
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
Stork Webinar | APM Transformational planning, Tool Selection & Performance T...
 
Theory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfTheory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdf
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________
 

Stacktician - CloudStack Collab Conference 2014

  • 1. Moving up the stack with Stacktician Amogh Vasekar
  • 2. Agenda • Introduction • History • Architecture and Implementation • Current State • Enhancements
  • 3. Agenda • Introduction • History • Architecture and Implementation • Current State • Enhancements
  • 4. Introduction What is Amazon CloudFormation? •Create and manage collection of related AWS resources •Typically, the resources will form your whole application “stack” In essence, “Infrastructure-as-code” •JSON file •Describe resources •Describe dependencies •Readable, reusable, reviewable
  • 6. Usefulness • Single click application deployment for multiple users • Transparent, reusable • Ease of resource management and dependencies • Atomic deployments • Thus, migrate new business applications to “cloud”
  • 7. Simply put, CloudFormation for CloudStack  Web interface for template and stack management
  • 8. Agenda • Introduction • History • Architecture and Implementation • Current State • Enhancements
  • 9. History • Started by @chiradeep • Introduced at CCC-13 as a prototype • Execute AWS CloudFormation templates on CloudStack *AS IS* • Problem – low fidelity between AWS and CloudStack resource types (more on this later)
  • 10. Agenda • Introduction • History • Architecture and Implementation • Current State • Enhancements
  • 11. Architecture Two components StackMate •Execution engine for running the templates and interfacing with CloudStack Stacktician •Web interface •Embeds StackMate underneath
  • 12. StackMate in 2 mins • Treat template = stack definition as workflow • Execute workflow by making appropriate CloudStack API calls • Core engine for executing templates, does the heavy lifting • Command line tool
  • 13. Stacktician in 2 mins • Web UI for StackMate • Backed by database for persistence • Uses Ruby meta-programming to interface with StackMate
  • 14. Where we were last time… • Resolve complex resource dependencies • Intrinsic functions • Wait Conditions • Create simple resources like AWS::Instance and AWS::SecurityGroup using StackMate
  • 15. AWS templates on CloudStack• Simple resources work well • Too many differences in more complex resources like VPC and networks  No route tables • Semantic differences between APIs for same resources  Different types of values for parameters • Dependency resolution for CloudStack using AWS templates became too complex  Sometimes, two AWS APIs act as a single one in CloudStack  CloudStack networking renders AWS options like DhcpOptions unnecessary  Too many NoOp resources used as work-around for above
  • 16. CloudStack Namespace • Instead create CloudStack specific namespace! • No more mapping of resources from AWS namespace to CloudStack space • Adds the ability to validate API parameters • Every CloudStack resource becomes a “participant” in the workflow execution • Essentially need to call create and delete APIs
  • 17. CloudStack Namespace • Problems  New resources added frequently (e.g tags, UCS, Nicira)  APIs evolve to support new parameters etc.  Too many CloudStack resources to maintain (~51) • Solution  Use API discovery  Auto generate code  Use API information to validate all required parameters exist / format of parameters
  • 18. StackMate participants • Bootstrap StackMate • Publish CloudStack version specific gem
  • 19. Errors and rollback • Want stack creation to be atomic • But, provide an option to user to disable rollback (e.g. for debugging)  Catch resource creation errors, and initiate rollback  Execute deletion in reverse order of dependencies  Skip any resources not yet created  If error while deleting, notify user (log for StackMate, UI status for Stacktician)  Stacktician injects additional code in StackMate for DB management
  • 20. Metadata server • Resources, like VMs, can launch additional initialization scripts upon configuration  E.g. – Launch a script via “userdata” parameter • Typically, script needs additional metadata to work with • Metadata is specific to a resource Request metadata
  • 21. Metadata server And expose retrieval as API
  • 22. Metadata Server • StackMate In memory metadata, embeds a Sinatra server for waitcondition and metadata • Stacktician Rails serves metadata, persisted in DB (Special thanks to Simon Waterhouse for integrating cfn-init scripts and creating scripts for Windows)
  • 23. Stacktician Scaling Improvements• Ruby MRI has global interpreter lock. Eventually, allowed for only one stack execution thread to run at a time  Use JRuby container, run in JRuby compatible server • Async wait for stack : one thread per stack launch would overwhelm server  GetBack gem for managing thread pool
  • 24. Delete Stack Stacktician •Resource IDs in database •Execute delete APIs in reverse order of dependency creation •Not fully tested yet StackMate •No DB, no IDs •Use resource tags to figure out resources, and delete •In progress
  • 25. Improvements StackMate •Better logging, log level control •Light-weight CloudStack client for making API calls Stacktician •Supports REST-based API for stack operations, querying and manipulation •Command line tools for all the above operations •sm-create-stack, sm-create-template etc
  • 26. Agenda • Introduction • History • Architecture and Implementation • Current State • Enhancements
  • 27. Demo
  • 28. Agenda • Introduction • History • Architecture and Implementation • Current State • Enhancements
  • 29. Future enhancements Better plugin architecture •Allow creation of custom type of resources •Interface with other types of Clouds – GCE, AWS etc. •Allow for scenarios like resource A on CloudStack, B on AWS within single stack •Currently only in StackMate, and very rudimentary. Enforces some basic constraints, uses tag “stackmate_participant = true” for discovery  Bad plugins rejected at run-time •Needs more work to make robust
  • 30. Future enhancements • Nested stacks  Treat stack as first class object. Template can contain another stack definition as a resource  Works in StackMate, disabled for lack of good implementation • Support conditional functions in template (e.g Fn:And, Fn:Or, Fn:If) • Support update stack  Semantics to be well defined
  • 31. Q & A Questions? Thoughts? Comments? Suggestions? https://github.com/stackmate Feel free to log issues – more the better! Big thanks to Chiradeep for all the help