AWS CloudFormation and Puppet at PuppetConf - Jinesh Varia

Architecting for the Cloud: AWS CloudFormation and Puppet
Jinesh Varia

Scale,
Pace of Innovation,
Expansion and
Ecosystem

Cloud Customers in 190 Countries

» Amazon EC2 with Windows Server
2008,
Spot Instances,
Boot from Amazon EBS
» Amazon CloudFront Streaming
» Amazon VPC enters Unlimited Beta
» AWS Region in Northern California
» International Support for AWS
Import/Export
» AWS Multi-Factor Authentication
» Virtual Private Cloud
» Lower Reserved Instance Pricing
» Reserved Instances in EU Region
» Elastic MapReduce
» SQS in EU Region
» Amazon RDS
» High-Memory Instances
» Lower EC2 Pricing
» New SimpleDB Features
» FPS General Availability
» Amazon SNS
» AWS Security Center
2009
Jan
2010
Jan
Jul
Sep
Oct
Dec
Aug
Nov
Feb
Mar
Apr
Jun
May
Feb
Mar
» Amazon EC2 with Windows
» Amazon EC2 in EU Region
» AWS Toolkit for Eclipse
» Amazon EC2 Reserved
Instances
» Amazon CloudFront
Private Content
» SAS70 Type II Audit
» AWS SDK for .NET
» Amazon Elastic MapReduce
in Europe
» Amazon EC2 Reserved Instances
with Windows, Extra Large High
Memory Instances
» Amazon S3 Versioning Feature
» Consolidated Billing for AWS
» Lower pricing for Outbound Data
Transfer
» AWS Import/Export
» New CloudFront Feature
» Monitoring, Auto Scaling & Elastic Load Balancing
» EBS Shared Snapshots
» SimpleDB in EU Region
» Monitoring, Auto Scaling &
Elastic Load Balancing in EU
» Lower pricing tiers for
Amazon CloudFront
» AWS Management Console
The pace of innovation in 2009

» Free Monitoring EC2
» Amazon Route 53
» PCI DSS Level 1 Certification
» Mobile SDKs (Android, iPhone)
» Large Object S3 Support
» Florida POP
» Import/Export APAC
» Amazon SNS
» Combined AWS Data Transfer Savings
» Amazon EMR Bootstrap Actions
» Amazon ELB Session Stickiness
» Amazon RDS in EU
» New Singapore Region
» RDS Reserved
» CloudFront Default Root
» Startup Challenge 2010
» CloudFront Invalidation
» AWS Elastic Beanstalk
» Amazon Simple Email Service
» Improved AWS Support “Bronze”
» Amazon CloudWatch Console
» CloudFront HTTPS
» NYC Edge Location
» Lowers Pricing HTTP
» AWS Import Export GA
» Amazon SNS
» Amazon S3 Console
» Amazon EBS CloudWatch
» Amazon RDS Read Replicas
» Suse EC2 Linux
» Amazon SNS Console
» Amazon ELB HTTPS
» AWS Free Tier
» EMR Resizing Cluster
» EMR JobFlow Debugging
» Simple DB Consistent Reads
» Simple DB Conditional Puts
» VM Connector
» Tokyo Region
» AWS Support JP
2010
Jan
2011
Jan
Jul
Sep
Oct
Dec
Aug
Nov
Feb
Mar
Apr
Jun
May
Feb
Mar
» New VPC
» Dedicated Instances
» Windows
2008 R2
» Amazon S3 Lowered Pricing
» CloudFront GA, SLA
» S3 Multipart
» GPGPU Instance Types
» ISO27001/2 Certification
» Amazon SQS Longer retention, Free Tier
Amazon S3 Bucket Policies
» Amazon VPC IP Address
» Cluster Compute Instances
» Amazon S3 RRS Notifications
» AWS Java SDK
» Windows BYOL
» Singapore Pop
» CloudFront Private Streaming
» Lowered Pricing EC2
» AWS IAM
» Amazon VPC Console
» Micro Instances
» Amazon Linux AMI
» Amazon EC2 Tagging, Filtering, Idempotency,
» Oracle Certified AWS
» AWS PHP SDK
» AWS CloudFormation
» Amazon S3 Static Websites
» AWS IAM Website Login
» Paris Edge Location
» Amazon EC2 Reserved Instances
with Windows, Extra Large High
Memory Instances
» Amazon S3 Versioning Feature
» Consolidated Billing for AWS
» Lower pricing for Outbound Data
Transfer
» VPC in EU
» Amazon RDS in US-west
» Amazon CloudFront Access Logs
» Amazon RDS Multi-AZ
» Amazon S3 RRS
» Amazon RDS Console
And pace accelerates in 2010….

“Every day is a launch day”
» On-demand Red Hat
» Stockholm Edge Location
» AWS Elastic Beanstalk new enhancements
» New Data Transfer pricing
» Free Inbound Data Transfer
» Spot Integration with HPC instances
» Amazon EMR in APAC
» AWS Mobile SDKs
» Live Streaming with CloudFront
» AWS IAM GA
» AWS IAM Web Console
» AWS Import/Export for EBS
» AWS CloudFormation new features
» AWS SDK for Ruby
» Attachment support for Amazon SES
» AWS Startup Challenge goes global
» AWS DirectConnect
» Amazon VPC Everywhere
» Mulit-AZ VPC
» AWS IAM Identity Federation
» AWS toolkit of eclipse 2.0
» AWS GovCloud US
» Spot in Amazon EMR
» Amazon ElastiCache
» Amazon VM import Win2k3
» VM Connector
» Tokyo Region
» AWS Support JP
» AWS IAM for CloudFront
» VPC Virtual Networking
» VPC Internet Access
2011
Jan
2012
Jan
Jul
Sep
Oct
Dec
Aug
Nov
Feb
Mar
Apr
Jun
May
Feb
Mar
» AWS CloudFormation
» Amazon S3 Static Websites
» AWS IAM Website Login
» Paris Edge Location
» Amazon Route53
» New VPC
» Dedicated Instances in VPC
» Windows 2008 R2
» New AZ in JP
» AWS IAM GA
» AWS IAM Web Console
» AWS Beanstalk Tomcat 7 Support
» Amazon CloudWatch Custom Metrics
» Amazon CloudWatch lower pricing
» AWS SAP Certification
» Amazon RDS for Oracle
» Amazon ELB ipv6 support, Zone Apex
» Amazon ELB Security Group integration
» Amazon Route53 GA, ELB integration
» Amazon Route 53 Weighted RR
» New pricing control for Spot
» AWS CloudFormation new enhancements
» AWS Mobile SDK GA
» AWS Toolkit of Visual Studio
» AWS DirectConnect USWest Location
» AWS Elastic Beanstalk
» Amazon Simple Email Service
» Improved AWS Support “Bronze”
» Amazon CloudWatch Console

Each day, AWS adds the equivalent server capacity to power Amazon when it was a global, $2.76B enterprise (circa 2000)

GovCloud-US
US West
(Northern California)
US East
(Northern Virginia)
Europe West
(Dublin)
Asia Pacific Region
(Singapore)
Asia Pacific Region
(Japan)
Ashburn, Dallas, Los Angeles, Miami, Newark, Palo Alto, Seattle, St. Louis, Amsterdam, Dublin, Frankfurt, London, Hong Kong, Singapore, Tokyo, New York, Paris
Amazon CloudFront
Edge Locations

The AWS Cloud
Your Application
Tools to access services
Libraries and SDKs
.NET/Java etc.
Web Interface
Management Console
Tools
AWS Toolkit Eclipse, VS
Command Line Interface
Cross Service features
Auth, Authorization, FederationAWS IAM, MFA
Monitoring
Amazon CloudWatch
Deployment and Automation
AWS Elastic BeanstalkAWS CloudFormation
High-level building blocks
Content Delivery
Amazon CloudFront
Email
Amazon SES
Payments
Amazon DevPay
Amazon FPS
Parallel Processing
Amazon Elastic MapReduce
Messaging
Amazon SNS
Amazon SQS
Workforce
Amazon Mechanical Turk
Low-level building blocks
Compute
Amazon EC2
Auto Scaling
Network
Amazon VPC,
ELB, DirectConnect
Amazon Route 53
Storage
Amazon S3
Amazon EBS
Database
Amazon RDS
Amazon SimpleDB
Amazon ElastiCache
Amazon Global Physical Infrastructure
(Geographical Regions, Availability Zones, Edge Locations)

www.yourApp.com
media.yourApp.com
(Static data)
Amazon CloudFront
Amazon Route 53
Elastic Load
Balancer
Amazon
CloudWatch
Amazon S3
Bucket
Amazon SNS
Notifications
Auto Scaling Group
Amazon SimpleDB
App Tier
Email
ElastiCache Tier
Amazon RDS
AZ-1
AZ-1
Region

Corporate data center
Availability Zone 1
DirectConnect
Location
10G
VPC Subnet
Router
VPN Gateway
Customer Gateway
Corporate Headquarters
VPC Subnet
Internet Gateway
Amazon VPC
Availability Zone 2
Branch Offices
The New Cloud-Ready Enterprise IT
Amazon S3
Amazon SES
Amazon SimpleDB
Amazon SQS
AWS Region

The “Living” AWS Cloud
Your Application
Tools to access services
Libraries and SDKs
.NET/Java etc.
Web Interface
Management Console
Tools
AWS Toolkit Eclipse, VS
Command Line Interface
Cross Service features
Auth, Authorization, FederationAWS IAM, MFA
Monitoring
Amazon CloudWatch
Deployment and Automation
AWS Elastic BeanstalkAWS CloudFormation
High-level building blocks
Content Delivery
Amazon CloudFront
Email
Amazon SES
Payments
Amazon DevPay
Amazon FPS
Parallel Processing
Amazon Elastic MapReduce
Messaging
Amazon SNS
Amazon SQS
Workforce
Amazon Mechanical Turk
Low-level building blocks
Compute
Amazon EC2
Auto Scaling
Network
Amazon VPC
Elastic LB
Amazon Route 53
Storage
Amazon S3
Amazon EBS
Database
Amazon RDS
Amazon SimpleDB
Amazon ElastiCache
Amazon Global Physical Infrastructure
(Geographical Regions, Availability Zones, Edge Locations)

The Need for Speed

DevOps

Ops = businessOps != businessOps ? business

Ops Ξ business

Elasticity
is the
fundamental
property
of the cloud

Implement Elasticity
Elasticity during the day
25% Savings

Elasticity during the year
50% Savings

Optimize during the month
75% Savings

Mr. Automate
Development
Automate
Using
Cloud APIs
Management
Logistics
Monitoring
Deployment

The Automation You Always Meant to Build
Provision and attach 1TB of storage in 2 minutes (from the back of an auto-rickshaw in India).
10 new Linux servers in 2 minutes (while sitting by the pool on a nice day).
Monitoring server resources from an iPhone (in a bar).
Source: Autodesk

AWS CloudFormation“Provision your infrastructure stack using one script”

www.yourApp.com
media.yourApp.com
(Static data)
Input Parameters
Resources
Outputs
JSON
Plain Text
Perfect for Version Control
Validate-able
Mappings
Custom Metadata
Amazon CloudFront
Amazon Route 53
Elastic Load
Balancer
Amazon
CloudWatch
JSON Template
AWS CloudFormation
Service
Amazon S3
Bucket
Amazon SNS
Notifications
Auto Scaling Group
Atomically creates and
destroys groups of
AWS Cloud Resources
Amazon SimpleDB
App Tier
Configures the resources
Multi-Tier or Multi-AZ stacks
Manages the ordering
of provisioning
Email
ElastiCache Tier
Amazon RDS
AZ-1
Rolls back in case of failure
Or issues
AZ-1
Region

AWS CloudFormation “Stacks”
JSON Template

Declarative language

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Create an EC2 instances",
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",
"Type" : "String"
}
},
"Mappings" : {
"RegionMap" : {
"us-east-1" : {
"AMI" : "ami-76f0061f"
},
"us-west-1" : {
"AMI" : "ami-655a0a20"
},
"eu-west-1" : {
"AMI" : "ami-7fd4e10b"
},
"ap-southeast-1" : {
"AMI" : "ami-72621c20"
},
"ap-northeast-1" : {
"AMI" : "ami-8e08a38f"
}
}
},
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : { "Ref" : "KeyName" },
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},
"UserData" : { "Fn::Base64" : "80" }
}
}
},
"Outputs" : {
"InstanceId" : {
"Description" : "InstanceId of the newly created EC2 instance",
"Value" : { "Ref" : "Ec2Instance" }
},
"AZ" : {
"Description" : "Availability Zone of the newly created EC2 instance",
"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "AvailabilityZone" ] }
},
"PublicIP" : {
"Description" : "Public IP address of the newly created EC2 instance",
"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicIp" ] }
}
}
}

Headers
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Create an EC2 instances",
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",
"Type" : "String"
}
},
"Mappings" : {
"RegionMap" : {
"us-east-1" : {
"AMI" : "ami-76f0061f"
},
"us-west-1" : {
"AMI" : "ami-655a0a20"
},
"eu-west-1" : {
},
"AMI" : "ami-72621c20"
},
}
}
},
"Resources" : {
"Ec2Instance" : {
"Properties" : {
"UserData" : { "Fn::Base64" : "80" }
}
}
},
"Outputs" : {
"InstanceId" : {
},
"AZ" : {
},
"PublicIP" : {
}
}
}
Parameters
Mappings
Resources
Outputs

Parameters
Provision-time specification
Command line options

"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing
EC2 KeyPair to enable SSH access to
the instance",
"Type" : "String"
}
},

Mappings
Conditionals
Case statements

"Mappings" : {
"RegionMap" : {
"us-east-1" : {
"AMI" : "ami-76f0061f"
},
"us-west-1" : {
"AMI" : "ami-655a0a20"
},
"eu-west-1" : {
},
"AMI" : "ami-72621c20"
},
}
}
},

"Mappings": {
"AWSInstanceType2Arch" : {
"t1.micro" : { "Arch" : "64" },
"m1.large" : { "Arch" : "64" },
"m1.xlarge" : { "Arch" : "64" },
"m2.xlarge" : { "Arch" : "64" },
"m2.2xlarge" : { "Arch" : "64" },
"m2.4xlarge" : { "Arch" : "64" },
"c1.xlarge" : { "Arch" : "64" },
"cc1.4xlarge" : { "Arch" : "64" }
},

Dereference this mappings
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI",
{
"Ref": "AWS::Region"
},

Resources

"Resources" : {
"Ec2Instance" : {
"Properties" : {
"UserData" : { "Fn::Base64" : "80" }
}
}
}

Parameter reference

"ImageId" : {
"Fn::FindInMap" :
[ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]
},

Map conditional
"ImageId" : {
"Fn::FindInMap" :
},

"ImageId" : {
"Fn::FindInMap" :
},
Name of map

"ImageId" : {
"Fn::FindInMap" :
},
Intrinsic property reference

Outputs
Returned values

"Outputs" : {
"InstanceId" : {
},
"AZ" : {
},
"PublicIP" : {
}
}
}

"AppDatabase": {"Type": "AWS::CloudFormation::Stack",
"Metadata": { … },
"Properties": {
"TemplateURL": {
"Fn::Join": [
"/",
[
{ … },
"RDS_MySQL_55.template"
]
]
},
Embedded Stacks

Custom Metadata
"Resources": {"Ec2Instance": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"Comment": "This metadata is available via the cfn-describe-stack-resource command line tool or the DescribeStackResource API call",
"MyAMI": { "Fn::FindInMap": [ "RegionMap", { "Ref": "AWS::Region“ }, "AMI"]},
"MyRegion": {"Ref": AWS::Region"},
"MyStack": {"Ref": "AWS::StackName"}
},
"Properties": {
"ImageId": {"Fn::FindInMap": ["RegionMap",{"Ref": "AWS::Region"}, AMI"]},
"UserData": { "Fn::Base64": "80"}
}
}
},

Standardized Application Stacks
Apache
Apache
IIS
Apache
Mongrel
Tomcat
ASP.NET
Mongrel
Web Server
Rails
Struts
ASP.NET MVC
Rails
App Server
Your Code
Your Code
Your Code
Your Code
MVC
logger
Log4J
Log4Net
logger
Your Code
RubyGems
Spring
Spring.NET
RubyGems
Libraries
memcached
Hibernate
nHibernate
memcached
Packages
Ruby Runtime
JEE
.NET
Ruby Runtime
DB Caching
Centos
Linux
Windows
Centos
Framework
OS
Java Stack
.NET Stack
RoR stack

Bootstrapping Applications with AWS CloudFormation

1. Frozen Pizza Model
IIS
IIS
IIS
IIS
Apache
Apache
IIS
IIS
IIS
IIS
Tomcat
Tomcat
ASP.NET MVC
ASP.NET MVC
ASP.NET MVC
ASP.NET MVC
Struts
Struts
Your Code
Your Code
Your Code
Your Code
Your Code
Your Code
Log4Net
Log4Net
Log4Net
Log4Net
Log4J
Log4J
Spring.NET
Spring.NET
Spring.NET
Spring.NET
Spring
Spring
nHibernate
nHibernate
nHibernate
nHibernate
Hibernate
Hibernate
.NET
.NET
.NET
.NET
JEE
JEE
Amazon EC2
Windows
Windows
Windows
Windows
Linux
Linux
Java AMI
Java Stack

Build Job does the following:
build the artifact,
publish it to Artifactory,
build the package,
publish the package to the repo .
Then there is a follow on job that mounts a base OS image, installs the packages and then creates the final AMI.
Source: http://techblog.netflix.com/2011/08/building-with-legos.html

2. Take N Bake Pizza Model
Apache
Your Code
Amazon S3
Tomcat
Struts
Log4J
Spring
Fetch on boot time
Apache
Struts
Tomcat
Source Control
Hibernate
Your Code
JEE
Linux
Log4J
Spring
IIS
IIS
IIS
IIS
Hibernate
IIS
IIS
IIS
IIS
JEE
.NET
.NET
.NET
.NET
Linux
Amazon EC2
Windows
Windows
Windows
Windows
Golden AMI
Java Stack

Cloud-Init and EC2 Instance User Data
Cloud-init supports several different mechanisms for passing data to the instance including ways to pass larger, more structured data and a way to provide a script that is executed at instance launch time.
Amazon Linux AMIs
Ubuntu Distributions

"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -exn",
"yum -y install git-coren",
"yum -y install php-pearn",
"pear install Crypt_HMAC2-1.0.0n",
"pear install HTTP_Request-1.4.4n",
"pear channel-discover pear.amazonwebservices.comn",
"pear install aws/sdkn",
Bootstrap using User Data

3. Made to Order Pizza Model
Amazon S3
Apache
Apache
Struts
Tomcat
Log4J
Hibernate
Your Code
Spring
Tomcat
Struts
Cookbooks
Recipes
Source Control
Your Code
PuppetMaster
Log4J
Spring
Hibernate
JEE
PuppetClient
Agent
Linux
Linux
Windows
Amazon EC2
AMI (JeOS)
Java Stack

Instances ask you a question “Who am I and what is my role?”

Mcollective + CloudFormation
Great for small or large clusters of servers
Simple naming conventions
Parallel job execution
Consistent servers
Great EC2 Demo : http://docs.puppetlabs.com/mcollective/ec2demo.html

Best Practices
Puppet is great for incremental implementation!
All modules and manifests should be kept under version control.
Manage users and groups from the outset.
Puppet Environments are your friend
Skinny classes, fat modules.
Use 'notify' for logging. Make it easy to check logs.
‘The Trifecta‘- Use the Package, file, service.

3 approaches to designing your AMIs
Easier to Setup
Inventory of fully baked AMIs
(Frozen/Ready made)
“Golden AMIs” with fetch on boot
(Take N’ Bake)
AMIs with JeOSand PuppetMaster (Made to Order)
More Control
Easier to maintain

More Tools: CloudFormer
Create a template from the running resources in your account
Select the resources that should be included
Customize the logical names
Define the template output section
Creates a starting point template for your to edit
Add parameters
Abstract properties and flow properties
One-click launch in your account
CloudFormer is an appliance that runs in your account

One more thing….

Optimizing = Cost Savings
Free Memory
Free CPU
Free HDD
At 1-min intervals
PUT
2 weeks
Alarm
Amazon CloudWatch
Instance
Custom Metrics
“You could save a bunch of money by switching
to a small instance, Click on CloudFormation Script to
Save”

In Summary,
Bridge the gap : Ops = business
Elasticity is the fundamental property of the cloud and implement elasticity
AWS CloudFormation gives you an easy way to create the set of cloud resources
3 Pizza Models 
Bootstrapping applications using CloudFormation and Puppet removes the muck

Thank you!
Jinesh Varia
jvaria@amazon.com Twitter:@jinman

Still working……

Let go of (physical) control
but retain your ownership

Enterprise Security Features
Amazon VPC
AWS Identity And Access Management
User management
Policy-based granular access control
Web login to individual users
Identity Federation (New!)
Multi-Factor Authentication
Services Security features
Amazon S3 ACL and Bucket policies
Amazon EC2 Security Groups, iptables
HTTPS API Endpoints

SAS 70 Type II Audit
ISO 27001/2 Certification
PCI DSS 2.0 Level 1-5
HIPAA/SOX Compliance
FISMA A&A Low
Encrypt data in transit
Encrypt data at rest
Protect your AWS Credentials
Rotate your keys
Secure your application
Enforce IAM policies
Use MFA, VPC, Leverage S3 bucket policies, EC2 Security groups, EFS in EC2 Etc..
In the Cloud, Security is a Shared Responsibility
How we secure our
infrastructure
How can you secure your application and what is your responsibility?
What security options and features are available to you?

New World
Old World
Build security in every layer

mcollective

http://puppetlabs.com	2015
http://understeer.hatenablog.com	1382
http://java.dzone.com	985
http://blog.rajatpandit.com	108
http://lanyrd.com	74
http://feeds.feedburner.com	68
https://puppetlabs.com	47
http://paper.li	43
http://cloud.dzone.com	16
https://confluence.skunk-works.no	13
http://127.0.0.1	11
http://puppetlabs.iron-point.com	9
http://architects.dzone.com	7
http://us-w1.rockmelt.com	5
http://webcache.googleusercontent.com	5
http://col.xtend.int	5
http://twitter.com	4
http://a0.twimg.com	3
http://confluence.matomyrnd.com	3
http://www.techgig.com	2
http://www.dzone.com	2
http://planet.mysql.com	2
http://translate.googleusercontent.com	2
http://www.puppetlabs.com	1
https://twitter.com	1
http://dev.puppetlabs.com	1
http://confluence.skunk-works.no	1
http://www.diffbot.com&_=1360869206159 HTTP	1
http://confluence.xtend.int	1

AWS CloudFormation and Puppet at PuppetConf - Jinesh Varia

by Amazon Web Services

Accessibility

Categories

Upload Details

Usage Rights

29 Embeds 4,817

Statistics

AWS CloudFormation and Puppet at PuppetConf - Jinesh Varia Presentation Transcript