CouchConf-SF-AWS-Cloudformation-to-Automate-Couchbase-Membase-Clusters

AWS CloudFormation to Automate Couchbase Membase Clusters Usman Shakeel 7/29/2011

AWS Computing Platform

Couchbase Membase on AWSCloud Formation  Ease of deployment  One template – reuse with multiple configurationsAuto-Scale  Automatically add more nodes to your clusterElastic Beanstalk  Automatically handle the deployment details, capacity provisioning, load-balancing and auto-scaling of your Couchbase/Membase based application )Only pay for the AWSresources used

CloudFormation Automates your Cloud creation and management in a few clicks A wide range of AWS resources supported Declarative, Flexible and Customizable via Parameters Many reusable sample templates available (including WordPress, Tracks, Gollum, Drupal, Joomla, Insoshi, Redmine) Transparent and Open Templates Integrated with Amazon Simple Notification Service (Amazon SNS) No Extra Charge CloudFormer

CloudFormation – Stack Stack is a collection of resources instantiated using a template and parameter values Order of instantiation: determined based on the dependencies specified by the template author Resources are tagged by the stack name A stack is rolled back if it cannot be fully instantiated CloudFormation Stack: API, CLI and Console  CreateStack, DeleteStack  DescribeStackResources, DescribeStackEvents  GetTemplate, ValidateTemplate

CloudFormation – Template A Template is a JSON formatted text file Describes the AWS resources along with the inter-connection Captures the resource relationships Can be parameterized – One template different setups Provides output back to the user Resources are created using logical names to avoid namespace conflicts

CloudFormation Automation with Cloud-init Cloud-init uses the Amazon EC2 UserData field to pass configuration information Stack indicates CREATE_COMPLETE only at the successful installation of packages Easy to upgrade to new OS releases and patches over creating a custom AMI Attribute values dependent on other CloudFormation resources can be passed in to the custom scripts at runtime

CloudFormation Template Anatomy{ "Description" : "A text description for the template usage", "Parameters": { // A set of inputs used to customize the template per deployment }, "Resources" : { // The set of AWS resources and relationships between them }, "Outputs" : { // A set of values to be made visible to the stack creator }, "AWSTemplateFormatVersion" : "2010-09-09"}

CloudFormation Template – Intrinsic Functions "Fn::Base64" : {valueToEncode} "Fn::FindInMap" : ["MapName", "Key", "Value"] "Fn::GetAtt" : ["logicalNameOfResource", "attributeName"] "Fn::GetAZs" : "region" "Fn::Join" : ["delimiter"[comma-delimited list of values]] "Ref" : "logicalName"

Auto Scaling Launch or Terminate EC2 instances based on your policies, schedules or health-checks Scale up to a minimum number of secondary nodes in a Couchbase Membase Cluster Scale based on CPU, network, memory usage or other CloudWatch metrics Scale based on Custom Metrics (data captured from the cluster itself)  CloudWatch can capture custom metrics from you application enabling you to define custom auto scaling policies Auto Scaling groups, policies and configurations can be created in a CloudFormation template

Auto Scaling Example3000 - 3000 CPU’s for one firm’s risk management processesNumber of EC2 Instances 300 CPU’s on weekends300 - Wednesday Thursday Friday Saturday Sunday Monday Tuesday 4/22/2009 4/23/2009 4/24/2009 4/25/2009 4/26/2009 4/27/2009 4/28/2009

DemoCouchbase Membase 6-Pack Cluster creation withCloudFormation Walk through Membase 6-Pack Cluster Template Run CloudFormation Template via the Console Show the newly created Membase clusterAutoscaling Membase 6-Pack Cluster Create Autoscaling Group for the Membase Cluster Create Autoscaling policies Implement Autoscaling Demo the autoscaled Membase Cluster

Membase CloudFormation Template - Parameters

Membase CloudFormation Template - Mappings

Membase CloudFormation Template - Resources Membase Cluster Coordinator Node Use an AMI based on the mapping Create an Instance based on Type parameter Parameter for Membase Package Installation

Membase CloudFormation Template - Resources User Data defines the script that will be run at instance creation Trigger for Wait Handle

Membase CloudFormation Template - Resources Create Auto Scaling group for secondary nodes Secondary node configurations defined separately Wait on Coordinator node creation before creating the Auto Scaling group

Cluster Coordinator NodeSecondary nodes created by Auto Scaling Group

Secondary node 2 Secondary nodesconfigurations defined terminated separately

Auto Scaling starts 2 new nodes to maintain a minimum of 6 nodes

What else? Auto scale Membase Cluster Coordinator Node  Recycle AWS Elastic IP  Membase Cluster end-point stays the same Attach AWS SNS notifications to Auto Scaling events Create Custom Auto Scaling policies  Push custom Membase cluster metrics to Cloud Watch and scale Automate Membase application deployment  Elastic Bean Stalk  Chef and Puppet Fail-over down Membase Cluster nodes appropriately Parameterize the Cluster size and Auto Scale policies

More Information CloudFormation http://aws.amazon.com/cloudformation Auto Scaling http://aws.amazon.com/autoscaling

CouchConf-SF-AWS-Cloudformation-to-Automate-Couchbase-Membase-Clusters

by Couchbase

Accessibility

Categories

Upload Details

Usage Rights

Statistics

CouchConf-SF-AWS-Cloudformation-to-Automate-Couchbase-Membase-Clusters Presentation Transcript