Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
(and AWS Lambda)
& aoepeople!
E-Commerce:
Magento
CMS:
TYPO3
Portals:
ZF, Symfony,…
Mobile Searchperience:
ElasticSearch
250+ people
world-wide
(in 8 lo...
CloudFormation Lambda “Immutable”
/i(m)ˈmyo͞ odəb(ə)l/
adjective
unchanging over time or unable to be changed.
“disposable”
“ephemeral”
Pet
Cattle
not disposable disposable
disposable
not disposable
disposable
Static Resources BuildVPC Build
Build Bucket
IAM Setup
VPC
Static Resources
Build
Manual Setup
separate
CloudFormation
Stacks
use resources
from underlyin...
Build Bucket
IAM Setup
Environment A (e.g. prod) Environment B (e.g. stage)
VPC VPC
Static Resources Static Resources
Buil...
Build Bucket
IAM Setup
Environment A (e.g. prod) Environment B (e.g. stage) Environment C (e.g. qa) Environment D (e.g. de...
Private subnets
Public subnets
ElastiCache (Redis)
with replication groups
for cache and sessions
RDS (multi-az) with
DB s...
Build
Auto Scaling group
Auto Scaling group
Elastic Load
Balancer
Auto-
Scaling
Group
Launch
Configurati
on
Scaling
Policy
Build
Auto Scaling group
Auto Scaling group
Elastic Load
Balancer
Auto-
Scaling
Group
Launch
Configurati
on
Scaling
Policy...
+1
Baking AMIs
“Chef vs. Puppet?”
http://fbrnc.net/blog/2015/11/
how-to-provision-an-ec2-instance
“…Ansible!”
“BASH!”
Keep it simple…!
✔ ✔ (✔)
the “last mile”
most
underestimated
CFN feature!
var r = require('cfn-response');
exports.handler = function (event, context) {
[…]
var res = {};
if (event.RequestType == ...
"IndexerDb": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
[...]
"MasterUserPassword": {"Fn::GetAtt": ["GenerateDbPass...
• launch some ASGs (set DesiredCapacity)
• create database passwords
• tag all resources (incl. ElastiCache!)
• restore da...
DesiredCapacity =
Number of Instances in current ASG x 1.2
better safe
than sorry…
DesiredCapacity =
Number of Instances in current ASG x 1.2
DesiredCapacity =
Number of Instances in current ASG x 1.2
ScalingPolicy
takes care…
"CountInstances": {
"Type": "Custom::InstanceCounter",
"Properties": {
"ServiceToken": {"Ref": "InstanceCounter"},
"AutoSc...
Update Route53 record set
Update Route53 record set
Delete old stacks
"UpdateR53": {
"DependsOn": [ "MagentoWaitCondition", "MagentoWorkerWaitCondition", "Elb"],
"Type": "Custom::Route53Update...
+1
Baking AMIs
Tools+CI
Have fun
filling out
47 form
fields! :)
aoepeople/stackformation
command-line tool
(Symfony console, uses
AWS SDK for PHP)
integrates nicely into
your CI (Jenknis...
pre-process
Userdata
Lambda JS
“Template”
superset of CFN
template (JSON)
Parameterslookup from other
stacks, env vars,…
T...
Stack
magento-stage-build-5
Stack
magento-stage-build-6
Stack
magento-prod-build-6
Stack
magento-prod-build-5
CloudFormati...
blueprints:
- stackname: 'magento-{env:BUILD}'
template: 'magento.template'
stackPolicy: 'policy.json'
OnFailure: 'DO_NOTH...
aoepeople/cfn-vpc
aoepeople/cfn-lambdahelper
aoepeople/cfn-amibaker
via composer
so we can
integrate this into
our CI pipeline…
•
aoepeople/awsinspector
command-line tool
(Symfony console, uses
AWS SDK for PHP)
Domain models for PHP
$repository = new AwsInspectorModelElbRepository();
$dns = $repository->findElbsByTags([
'Environment' => 'deploy',
'Build...
> bin/awsinspector.php ec2:ssh -t Environment:prod –c Type –c Build
filter by tag
Please select an instance
[0] i-1033ed9b...
logins,
orders,…
deployments,
scaling activity,…
JMeter
response time,
error rate,…
CPU, load,
network I/O,…
correlate
metrics from
various sources
GrafanaElasticSearch
(Service)
time-series
database
leftovers from
“disposed” resources
…
“Ready for testing”
(WaitCondition)
setup
terminate
instances
and wait
• Create Security
allowing access
to all IPs and
...
CloudFormation Stack
• Install JMeter
• Download
testcase from S3
• Signal “Ready”
and wait
• Create SG
allowing access
to...
https://blogs.aws.amazon.com/application-
management/post/Tx38Z5CAM5WWRXW/Faster-Auto-Scaling-in-AWS-
CloudFormation-Stack...
Provisioning
“Ready for baking”
(WaitCondition)
EC2
AMI Baker
(Lambda) Node.js SDK
AWS CLI
“Baking Completed”
(WaitConditi...
•
Follow me on twitter!
My blog
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
Upcoming SlideShare
Loading in …5
×

of

Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 1 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 2 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 3 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 4 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 5 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 6 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 7 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 8 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 9 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 10 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 11 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 12 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 13 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 14 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 15 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 16 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 17 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 18 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 19 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 20 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 21 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 22 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 23 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 24 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 25 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 26 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 27 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 28 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 29 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 30 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 31 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 32 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 33 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 34 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 35 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 36 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 37 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 38 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 39 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 40 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 41 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 42 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 43 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 44 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 45 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 46 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 47 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 48 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 49 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 50 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 51 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 52 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 53 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 54 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 55 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 56 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 57 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 58 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 59 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 60 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 61 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 62 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 63 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 64 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 65 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 66 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 67 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 68 Immutable Deployments with AWS CloudFormation and AWS Lambda Slide 69
Upcoming SlideShare
Magento Imagine 2013: Fabrizio Branca - Learning To Fly: How Angry Birds Reached the Heights of Store Performance
Next
Download to read offline and view in fullscreen.

27 Likes

Share

Download to read offline

Immutable Deployments with AWS CloudFormation and AWS Lambda

Download to read offline

San Francisco AdvancedAWS Meetup, 2016-03-17

Related Books

Free with a 30 day trial from Scribd

See all

Immutable Deployments with AWS CloudFormation and AWS Lambda

  1. 1. (and AWS Lambda)
  2. 2. & aoepeople!
  3. 3. E-Commerce: Magento CMS: TYPO3 Portals: ZF, Symfony,… Mobile Searchperience: ElasticSearch 250+ people world-wide (in 8 locations) Global Enterprise Projects Infrastructure: AWS
  4. 4. CloudFormation Lambda “Immutable”
  5. 5. /i(m)ˈmyo͞ odəb(ə)l/ adjective unchanging over time or unable to be changed. “disposable” “ephemeral”
  6. 6. Pet Cattle
  7. 7. not disposable disposable
  8. 8. disposable not disposable disposable
  9. 9. Static Resources BuildVPC Build
  10. 10. Build Bucket IAM Setup VPC Static Resources Build Manual Setup separate CloudFormation Stacks use resources from underlying stacks as input parameters Lambda functions, Monitoring,…
  11. 11. Build Bucket IAM Setup Environment A (e.g. prod) Environment B (e.g. stage) VPC VPC Static Resources Static Resources Build BuildBuild Build Build X Build X+1 Build X Build X+1 Manual Setup different scopes (build, environment, account, global…)
  12. 12. Build Bucket IAM Setup Environment A (e.g. prod) Environment B (e.g. stage) Environment C (e.g. qa) Environment D (e.g. dev) IAM Setup VPC VPC VPC VPC Static Resources Static Resources Static Resources Static Resources Build BuildBuild Build Build Build Build Build Build X Build X+1 Build X Build X+1 Build X Build X+1 Build X Build X+1 Manual Setup Manual Setup Build Bucket Access
  13. 13. Private subnets Public subnets ElastiCache (Redis) with replication groups for cache and sessions RDS (multi-az) with DB subnet group Bastion server s3: media storage* Route 53: DNS configuration CloudFront distribution SSL Certificates Security group for Varnish servers Security group for Magento servers Security group for Load Balancer Static Resources
  14. 14. Build Auto Scaling group Auto Scaling group Elastic Load Balancer Auto- Scaling Group Launch Configurati on Scaling Policy
  15. 15. Build Auto Scaling group Auto Scaling group Elastic Load Balancer Auto- Scaling Group Launch Configurati on Scaling Policy Auto Scaling group Auto Scaling group
  16. 16. +1 Baking AMIs
  17. 17. “Chef vs. Puppet?” http://fbrnc.net/blog/2015/11/ how-to-provision-an-ec2-instance “…Ansible!” “BASH!”
  18. 18. Keep it simple…!
  19. 19. ✔ ✔ (✔)
  20. 20. the “last mile” most underestimated CFN feature!
  21. 21. var r = require('cfn-response'); exports.handler = function (event, context) { […] var res = {}; if (event.RequestType == 'Create') { res.Password = randomPassword(20); } r.send(event, context, r.SUCCESS, res); };
  22. 22. "IndexerDb": { "Type": "AWS::RDS::DBInstance", "Properties": { [...] "MasterUserPassword": {"Fn::GetAtt": ["GenerateDbPassword", "Password"]}, [...] } }, "GenerateDbPassword": { "Type": "Custom::PasswordGenerator", "Properties": { "ServiceToken": {"Ref": "PasswordGeneratorArn"} } },
  23. 23. • launch some ASGs (set DesiredCapacity) • create database passwords • tag all resources (incl. ElastiCache!) • restore database and media files • (one-time) install scripts (db migrations,…) • detect Varnish backends • wait for healthy backends in the ELBs • run infrastructure tests • cache warming • update Route53 records sets • delete old stacks • …
  24. 24. DesiredCapacity = Number of Instances in current ASG x 1.2 better safe than sorry…
  25. 25. DesiredCapacity = Number of Instances in current ASG x 1.2
  26. 26. DesiredCapacity = Number of Instances in current ASG x 1.2 ScalingPolicy takes care…
  27. 27. "CountInstances": { "Type": "Custom::InstanceCounter", "Properties": { "ServiceToken": {"Ref": "InstanceCounter"}, "AutoScalingGroupTags": [ {"Key": "Environment", "Value": "prod"}, {"Key": "Type", "Value": "Frontend"} ], "Min": 1, "Max": 10, "Factor": "1.5" } }, "FrontendAsg": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { [...] "DesiredCapacity": {"Fn::GetAtt": ["CountInstances", "Count"]}, "Tags": [ {"Key": "Environment", "Value": "prod", "PropagateAtLaunch": true}, {"Key": "Type", "Value": “Frontend", "PropagateAtLaunch": true} ] } },
  28. 28. Update Route53 record set
  29. 29. Update Route53 record set Delete old stacks
  30. 30. "UpdateR53": { "DependsOn": [ "MagentoWaitCondition", "MagentoWorkerWaitCondition", "Elb"], "Type": "Custom::Route53Update", "Properties": { "ServiceToken": {"Ref": "R53Updater"}, "Name": {"Fn::Join": ["", [ "www-", {"Ref": "EnvironmentName"} ]]}, "HostedZoneId": {"Ref": "HostedZoneId"}, "AliasTargetDNSName": {"Fn::GetAtt": ["Elb", "DNSName"]}, "AliasTargetHostedZoneId": {"Fn::GetAtt": ["Elb", "CanonicalHostedZoneNameID"]}, "Comment": "Updated via CloudFormation/Lambda" } }, "DeleteStacks": { "Condition": "DeleteOldStacks", "DependsOn": ["UpdateR53"], "Type": "Custom::StackDeleter", "Properties": { "ServiceToken": {"Ref": "StackDeleter"}, "TagFilter": { "Environment": {"Ref": "EnvironmentName"}, "Type": "Deployment"}, "ExceptStackName": {"Ref": "AWS::StackName"} } }
  31. 31. +1 Baking AMIs Tools+CI
  32. 32. Have fun filling out 47 form fields! :)
  33. 33. aoepeople/stackformation command-line tool (Symfony console, uses AWS SDK for PHP) integrates nicely into your CI (Jenknis,…)
  34. 34. pre-process Userdata Lambda JS “Template” superset of CFN template (JSON) Parameterslookup from other stacks, env vars,… Template Parameter Values create/ update Stack policies, Tags,…
  35. 35. Stack magento-stage-build-5 Stack magento-stage-build-6 Stack magento-prod-build-6 Stack magento-prod-build-5 CloudFormation Template merge & pre-process “CloudFormation+X” Template(s) + Dynamic Parameters + Stack Policies + Behavior + Tags … Blueprint magento-{env:ENVIRONMENT}-build-{env:BUILD} Blueprint magento-{env:ENVIRONMENT}-setup Stack magento-stage-setup Stack magento-prod-setup
  36. 36. blueprints: - stackname: 'magento-{env:BUILD}' template: 'magento.template' stackPolicy: 'policy.json' OnFailure: 'DO_NOTHING' parameters: Build: '{env:BUILD}' KeyPair: '{var:KeyPair}' VPC: '{resource:setupstack:VPC}' Subnet: '{resource:setupstack:Subnet}' InstanceSg: '{resource:setupstack:InstanceSg}' InstanceProfile: '{output:setupstack:InstanceProfile}' BootAmi: 'ami-06116566' tags: Environment: 'prod' Build: '{env:BUILD}' enforce “immutability” by denying updates!
  37. 37. aoepeople/cfn-vpc aoepeople/cfn-lambdahelper aoepeople/cfn-amibaker via composer
  38. 38. so we can integrate this into our CI pipeline…
  39. 39.
  40. 40. aoepeople/awsinspector command-line tool (Symfony console, uses AWS SDK for PHP) Domain models for PHP
  41. 41. $repository = new AwsInspectorModelElbRepository(); $dns = $repository->findElbsByTags([ 'Environment' => 'deploy', 'Build' => 554, 'Type' => 'Frontend’ ])->getFirst()->getDNSName();
  42. 42. > bin/awsinspector.php ec2:ssh -t Environment:prod –c Type –c Build filter by tag Please select an instance [0] i-1033ed9b (Type: Frontend; Environment: prod; Build: 477) [1] i-4ff36ec8 (Type: Backend; Environment: prod ; Build: 477) [2] i-5ab4322b (Type: Worker; Environment: prod; Build: 477) [3] i-705ad42f (Type: Worker; Environment: prod; Build: 476) > • will take jump hosts into account (ProxyCommand) • auto-detects your local (encrypted) private keys • multiplexed ssh connections • run commands directly
  43. 43. logins, orders,… deployments, scaling activity,… JMeter response time, error rate,… CPU, load, network I/O,…
  44. 44. correlate metrics from various sources
  45. 45. GrafanaElasticSearch (Service) time-series database
  46. 46. leftovers from “disposed” resources
  47. 47. … “Ready for testing” (WaitCondition) setup terminate instances and wait • Create Security allowing access to all IPs and attach to UAT’s ELB environment (incl. restoring db+media snapshot from prod) environment* Delete self* *via Lambda custom resource Pushing samples in near real-time via custom JMeter backend listener “Testing Completed” (WaitCondition) JMeter test CloudFormation Stack GrafanaElasticSearch (Service) CloudWatch Lambda find relevant resources for the current deployment (via tags), collect metrics, and push them to ElasticSearch
  48. 48. CloudFormation Stack • Install JMeter • Download testcase from S3 • Signal “Ready” and wait • Create SG allowing access to all IPs and attach to UAT’s ELB Auto-Scaling Group of Load Generator Instances Run Stress Test2. Time timeout 3600 jmeter –n –t testcase.jmx Delete test environment* Delete self* 3. 4. *via Lambda custom resource … Spin up test environment (incl. restoring db+media snapshot from prod) 1. CloudFormation Stack setup terminate instances JMeter test Pushing samples in near real-time via custom JMeter backend listener “Ready for testing” (WaitCondition) “Testing Completed” (WaitCondition)
  49. 49. https://blogs.aws.amazon.com/application- management/post/Tx38Z5CAM5WWRXW/Faster-Auto-Scaling-in-AWS- CloudFormation-Stacks-with-Lambda-backed-Custom-Resou
  50. 50. Provisioning “Ready for baking” (WaitCondition) EC2 AMI Baker (Lambda) Node.js SDK AWS CLI “Baking Completed” (WaitCondition) shutdown (-> terminate) ec2.createImage aws ec2 wait image-available AMI Baker (Lambda) Node.js SDK ec2.deleteImage Custom CloudFormation Resource Time
  51. 51.
  52. 52. Follow me on twitter! My blog
  • AnnParker22

    Nov. 25, 2021
  • AlejandroRey15

    Jun. 20, 2018
  • young.yang

    Mar. 13, 2018
  • HideakiSuzuki2

    May. 31, 2017
  • DubaiEscortBunniesUAE

    May. 30, 2017
  • kuniakiyoda3

    Feb. 26, 2017
  • alexanderpervak

    Jan. 16, 2017
  • derickdsouza

    Oct. 11, 2016
  • JonathanDay1

    Jul. 26, 2016
  • shashisatya

    Jul. 14, 2016
  • chao2suke

    Jun. 30, 2016
  • gregorynicholas

    Jun. 23, 2016
  • erthad

    Jun. 16, 2016
  • DonnaAndert

    May. 16, 2016
  • HirokazuNishi1

    Apr. 13, 2016
  • guna_r

    Mar. 29, 2016
  • beejhuff

    Mar. 28, 2016
  • MichaelLarkin7

    Mar. 25, 2016
  • jeremypruitt

    Mar. 24, 2016
  • benjamincheng

    Mar. 23, 2016

San Francisco AdvancedAWS Meetup, 2016-03-17

Views

Total views

44,087

On Slideshare

0

From embeds

0

Number of embeds

29,149

Actions

Downloads

122

Shares

0

Comments

0

Likes

27

×