Automating the application lifecycle
19 mars 2015 . #TIAD . @tiadparis
# TIAD@ tiadparis
Who am I?
2
Laurent Bernaille @d2si
• Linux background
• Cloud enthousiast
• Opensource advocate
• Love discovering, building (and breaking…) new things
• Passionate about the ongoing IT transformations
@lbernail
# TIAD@ tiadparis
Story behind this talk/demo
3
A classic company with a « standard » on-premise IT
New business objectives in a competitive space: IOT
Creation of small, independant start-up teams
Use of Amazon Web Services for agility and scalability
An opportunity to use a new application delivery process
# TIAD@ tiadparis
Automating the application lifecyle: Objectives
4
Integrate a new application in hours instead of days
Create a new environment in minutes instead of month
Deploy a new version of the application in minutes instead of hours
# TIAD@ tiadparis
What it was (is?) like
5
DEV
Version
Control
System Continuous
Integration
Analyse code
Build Repository
Push code to VCS
Traditional Continuous integration
• Build a binary application artifact
• WAR, JAR, RPM, DEB, ZIP
Artifact
Traditional Deployment
• Build environments
• Deploy the artifact in environments
OPS
Configured server(s)
Staging
Configured server(s)
Production
We can do better!
# TIAD@ tiadparis
Demo outline
6
1. Infrastructure build
1. Creation of application back-ends
2. Deployment of the application
3. New release
# TIAD@ tiadparis
Step 1: Build infrastructure automatically
7
Build a full environment from scratch
• Subnets
• Routing tables
• Technical services
# TIAD@ tiadparis
Step 1: Build infrastructure automatically
7
CloudFormation
Build a full environment from scratch
• Subnets
• Routing tables
• Technical services
Cloudformation
• Describe infrastructure components
• Build them
• « puppet » for infrastructure
• AWS only: see also Terraform / Heat
# TIAD@ tiadparis
Step 1: Build infrastructure automatically
7
CloudFormation
Build a full environment from scratch
• Subnets
• Routing tables
• Technical services
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
Cloudformation
• Describe infrastructure components
• Build them
• « puppet » for infrastructure
• AWS only: see also Terraform / Heat
# TIAD@ tiadparis
Step 1: Build infrastructure automatically
7
CloudFormation
NAT NATBastion
Build a full environment from scratch
• Subnets
• Routing tables
• Technical services
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
Cloudformation
• Describe infrastructure components
• Build them
• « puppet » for infrastructure
• AWS only: see also Terraform / Heat
# TIAD@ tiadparis
Step 1: Build infrastructure automatically
7
CloudFormation
NAT NATBastion
Build a full environment from scratch
• Subnets
• Routing tables
• Technical services
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
Why automate these rare actions?
• Avoid errors
• Create identical environments (Production, Staging) easily
• Ability to deploy somewhere else very quickly
• Not so rare
• Easier to update
Cloudformation
• Describe infrastructure components
• Build them
• « puppet » for infrastructure
• AWS only: see also Terraform / Heat
# TIAD@ tiadparis
Under the hood
8
NAT NATBastion
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
"VPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : "10.100.0.0/16"
}
”PublicSubnet1" : {
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"AvailabilityZone" : "eu-west-1a"
"CidrBlock” : "10.100.1.0/25”
}
}
"NatInstance1" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"ImageId" : "ami-6e7bd919",
"InstanceType" : "t2.micro"
"SourceDestCheck" : "False",
"UserData” :
{ "Fn::Base64" : { "Fn::Join" : [”n", [
"#!/bin/bash",
"echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf",
"sysctl -p,
"iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE",
"iptables-save > /etc/sysconfig/iptables"
]]}}
}
}
# TIAD@ tiadparis
Step 2: Build application components
9
Build all the backends
• Databases
• Buckets
• Cache servers
• Queues & Topics
NAT NATBastion
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
# TIAD@ tiadparis
Step 2: Build application components
9
CloudFormation
Build all the backends
• Databases
• Buckets
• Cache servers
• Queues & Topics
NAT NATBastion
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
DynamoDB
DynamoDB
• AWS NoSQL database
• « Cassandra as a service »
# TIAD@ tiadparis
Step 2: Build application components
9
CloudFormation
Build all the backends
• Databases
• Buckets
• Cache servers
• Queues & Topics
NAT NATBastion
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
Why separate from servers where the application run?
• Different lifecycle
• Can be shared between releases
DynamoDB
DynamoDB
• AWS NoSQL database
• « Cassandra as a service »
# TIAD@ tiadparis
Difficulty: managing variables
10
VPC Addresses
Subnets
Instance types
Infra template
# TIAD@ tiadparis
Difficulty: managing variables
10
VPC Addresses
Subnets
Instance types
Infra template
Parameters
# TIAD@ tiadparis
Difficulty: managing variables
10
VPC Addresses
Subnets
Instance types
Infra template
Parameters
Backend template
VPC Id
Subnet Ids
DB Name
# TIAD@ tiadparis
Difficulty: managing variables
10
VPC Addresses
Subnets
Instance types
Infra template
Parameters
Backend template
VPC Id
Subnet Ids
DB Name
# TIAD@ tiadparis
Difficulty: managing variables
10
VPC Addresses
Subnets
Instance types
Infra template
Parameters
Backend template
VPC Id
Subnet Ids
DB Name
?
?
# TIAD@ tiadparis
Difficulty: managing variables
10
VPC Addresses
Subnets
Instance types
Infra template
Parameters
Backend template
VPC Id
Subnet Ids
DB Name
?
?
Wrapper to manage inputs/outputs
Outputs
# TIAD@ tiadparis
Step 3: Deploy application
11
Deploy application
• Load-balancers
• Servers
• DNS Alias
NAT NATBastion
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
DynamoDB
# TIAD@ tiadparis
Step 3: Deploy application
11
CloudFormation
Deploy application
• Load-balancers
• Servers
• DNS Alias
NAT NATBastion
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
DynamoDB
WEB WEB
Load-balancer included to allow deployment of several versions
Web server is built from an image with application pre-installed
# TIAD@ tiadparis
Difficulty: managing variables, again
12
VPC Addresses
Subnets
Instance types
Infra template Backend template
VPC Id
Subnet Ids
DB Name
Outputs
Application template
Instance Types
DB Name
# TIAD@ tiadparis
Difficulty: managing variables, again
12
VPC Addresses
Subnets
Instance types
Infra template Backend template
VPC Id
Subnet Ids
DB Name
Outputs
Application template
Instance Types
DB Name
WEB
DynamoDB
?
# TIAD@ tiadparis
Difficulty: managing variables, again
12
VPC Addresses
Subnets
Instance types
Infra template Backend template
VPC Id
Subnet Ids
DB Name
Outputs
Application template
Instance Types
DB Name
WEB
DynamoDB
?
"UserData” :
{ "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bashn",
"echo ", { "Ref" : "Properties" } ," >> /var/www/html/application.propertiesn”
]]}}
Properties
Properties
# TIAD@ tiadparis
Updating the application: the old-fashion way
13
DEV
Version
Control
System Continuous
Integration
Analyse code
Build Repository
Push code to VCS
Traditional Continuous integration
• Build a binary application artifact
• WAR, JAR, RPM, DEB, ZIP
Artifact
Traditional Deployment
• Use previously built environments
• Deploy the artifact
OPS
Configured server(s)
Staging
Configured server(s)
Production
Environments will drift
# TIAD@ tiadparis
Updating the application: with config management
14
Environments can still drift
Not suited for autoscaling
DEV
Version
Control
System Continuous
Integration
Analyse code
Build Repository
Push code to VCS
Traditional Continuous integration
• Build a binary application artifact
• WAR, JAR, RPM, DEB, ZIP
Artifact
Use config management tools
• Same recipes across environments
• Same artefact across environments
• Manual application deployment
OPS
Configured server(s)
Staging
Configured server(s)
Production
# TIAD@ tiadparis
Updating the application: in the cloud
15
Building can take a while
Some drift still possible
DEV
Version
Control
System Continuous
Integration
Analyse code
Build Repository
Push code to VCS
Traditional Continuous integration
• Build a binary application artifact
• WAR, JAR, RPM, DEB, ZIP
Artifact
Use config management tools
• Same recipes across environments
• Same artefact across environments
• Deploy application at server start
OPS
Configured server(s)
Staging
Configured server(s)
Production
# TIAD@ tiadparis
Updating the application
16
DEV
Version
Control
System Continuous
Integration
Analyse code
Build
Repository
Push code to VCS
New Continuous integration
• Build an application artifact
• Build a server image
Artifact
OPS
Staging Production
# TIAD@ tiadparis
Updating the application
16
DEV
Version
Control
System Continuous
Integration
Analyse code
Build
Repository
Push code to VCS
New Continuous integration
• Build an application artifact
• Build a server image
Artifact
OPS
Staging Production
Provision,
Config, deploy
Reference
templatesBuild
Application
templates
# TIAD@ tiadparis
Updating the application
16
DEV
Version
Control
System Continuous
Integration
Analyse code
Build
Repository
Push code to VCS
New Continuous integration
• Build an application artifact
• Build a server image
Artifact
OPS
Configured server(s)
Staging
Configured server(s)
Production
Provision,
Config, deploy
Reference
templatesBuild
Application
templates
# TIAD@ tiadparis
Demo
17
Push code
Integration
AWS images
AWS
Application
templates
Web hook
Packer
• Automate the creation of templates
• Developped by @mitchellh / @hashicorp
# TIAD@ tiadparis
Demo
17
Push code
Integration
AWS images
AWS
Application
templates
Web hook
Build
WEB WEB
Packer
• Automate the creation of templates
• Developped by @mitchellh / @hashicorp
# TIAD@ tiadparis
Packer
18
"builders": [{
"type": "amazon-ebs",
"region": ”eu-west-1",
"source_ami": "ami-f0b11187",
"instance_type": "t2.small",
"ssh_username": "ubuntu",
"ami_name": "demo-{{isotime "2006-01-02T15-04-05"}}"
}],
"provisioners": [
{
"type": "file",
"source": "/jenkins_workspace/site",
"destination": "/tmp/www"
},
{
"type": "shell",
"inline": [
"sudo apt-get update",
"sudo apt-get install -y apache2 php5 libapache2-mod-php5 php5-curl php5-mysql",
"sudo rm /var/www/html/*",
"sudo mv /tmp/www/* /var/www/html",
"sudo service apache2 restart"
]
}]
# TIAD@ tiadparis 19
Push code
Continuous
Integration
Application
templates
IntegrationPerspectives
# TIAD@ tiadparis 19
Production
Push code
Continuous
Integration
Application
templates
Integration
Prod DNS
Perspectives
# TIAD@ tiadparis 19
Production
Push code
Continuous
Integration
Application
templates
Integration
Prod DNS
Perspectives
# TIAD@ tiadparis 19
Production
Test
Push code
Continuous
Integration
Application
templates
Integration
Prod DNS
Perspectives
# TIAD@ tiadparis 19
Production
Test
Push code
Continuous
Integration
Application
templates
Integration
Prod DNS
Perspectives
# TIAD@ tiadparis 19
Production
Test
Push code
Continuous
Integration
Application
templates
Integration
Prod DNS
Perspectives
# TIAD@ tiadparis
Key take-aways
20
Everything can be automated
Very important change: Immutable servers
• New application artifact: Images / Containers
• Very challenging for organizations
# TIAD@ tiadparis
Thank you
@lbernail
Fork the code of this demo on github
https://github.com/lbernail/demo

TIAD : Automating the aplication lifecycle

  • 1.
    Automating the applicationlifecycle 19 mars 2015 . #TIAD . @tiadparis
  • 2.
    # TIAD@ tiadparis Whoam I? 2 Laurent Bernaille @d2si • Linux background • Cloud enthousiast • Opensource advocate • Love discovering, building (and breaking…) new things • Passionate about the ongoing IT transformations @lbernail
  • 3.
    # TIAD@ tiadparis Storybehind this talk/demo 3 A classic company with a « standard » on-premise IT New business objectives in a competitive space: IOT Creation of small, independant start-up teams Use of Amazon Web Services for agility and scalability An opportunity to use a new application delivery process
  • 4.
    # TIAD@ tiadparis Automatingthe application lifecyle: Objectives 4 Integrate a new application in hours instead of days Create a new environment in minutes instead of month Deploy a new version of the application in minutes instead of hours
  • 5.
    # TIAD@ tiadparis Whatit was (is?) like 5 DEV Version Control System Continuous Integration Analyse code Build Repository Push code to VCS Traditional Continuous integration • Build a binary application artifact • WAR, JAR, RPM, DEB, ZIP Artifact Traditional Deployment • Build environments • Deploy the artifact in environments OPS Configured server(s) Staging Configured server(s) Production We can do better!
  • 6.
    # TIAD@ tiadparis Demooutline 6 1. Infrastructure build 1. Creation of application back-ends 2. Deployment of the application 3. New release
  • 7.
    # TIAD@ tiadparis Step1: Build infrastructure automatically 7 Build a full environment from scratch • Subnets • Routing tables • Technical services
  • 8.
    # TIAD@ tiadparis Step1: Build infrastructure automatically 7 CloudFormation Build a full environment from scratch • Subnets • Routing tables • Technical services Cloudformation • Describe infrastructure components • Build them • « puppet » for infrastructure • AWS only: see also Terraform / Heat
  • 9.
    # TIAD@ tiadparis Step1: Build infrastructure automatically 7 CloudFormation Build a full environment from scratch • Subnets • Routing tables • Technical services eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets Cloudformation • Describe infrastructure components • Build them • « puppet » for infrastructure • AWS only: see also Terraform / Heat
  • 10.
    # TIAD@ tiadparis Step1: Build infrastructure automatically 7 CloudFormation NAT NATBastion Build a full environment from scratch • Subnets • Routing tables • Technical services eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets Cloudformation • Describe infrastructure components • Build them • « puppet » for infrastructure • AWS only: see also Terraform / Heat
  • 11.
    # TIAD@ tiadparis Step1: Build infrastructure automatically 7 CloudFormation NAT NATBastion Build a full environment from scratch • Subnets • Routing tables • Technical services eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets Why automate these rare actions? • Avoid errors • Create identical environments (Production, Staging) easily • Ability to deploy somewhere else very quickly • Not so rare • Easier to update Cloudformation • Describe infrastructure components • Build them • « puppet » for infrastructure • AWS only: see also Terraform / Heat
  • 12.
    # TIAD@ tiadparis Underthe hood 8 NAT NATBastion eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets "VPC" : { "Type" : "AWS::EC2::VPC", "Properties" : { "CidrBlock" : "10.100.0.0/16" } ”PublicSubnet1" : { "Type" : "AWS::EC2::Subnet", "Properties" : { "VpcId" : { "Ref" : "VPC" }, "AvailabilityZone" : "eu-west-1a" "CidrBlock” : "10.100.1.0/25” } } "NatInstance1" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : "ami-6e7bd919", "InstanceType" : "t2.micro" "SourceDestCheck" : "False", "UserData” : { "Fn::Base64" : { "Fn::Join" : [”n", [ "#!/bin/bash", "echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf", "sysctl -p, "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE", "iptables-save > /etc/sysconfig/iptables" ]]}} } }
  • 13.
    # TIAD@ tiadparis Step2: Build application components 9 Build all the backends • Databases • Buckets • Cache servers • Queues & Topics NAT NATBastion eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets
  • 14.
    # TIAD@ tiadparis Step2: Build application components 9 CloudFormation Build all the backends • Databases • Buckets • Cache servers • Queues & Topics NAT NATBastion eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets DynamoDB DynamoDB • AWS NoSQL database • « Cassandra as a service »
  • 15.
    # TIAD@ tiadparis Step2: Build application components 9 CloudFormation Build all the backends • Databases • Buckets • Cache servers • Queues & Topics NAT NATBastion eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets Why separate from servers where the application run? • Different lifecycle • Can be shared between releases DynamoDB DynamoDB • AWS NoSQL database • « Cassandra as a service »
  • 16.
    # TIAD@ tiadparis Difficulty:managing variables 10 VPC Addresses Subnets Instance types Infra template
  • 17.
    # TIAD@ tiadparis Difficulty:managing variables 10 VPC Addresses Subnets Instance types Infra template Parameters
  • 18.
    # TIAD@ tiadparis Difficulty:managing variables 10 VPC Addresses Subnets Instance types Infra template Parameters Backend template VPC Id Subnet Ids DB Name
  • 19.
    # TIAD@ tiadparis Difficulty:managing variables 10 VPC Addresses Subnets Instance types Infra template Parameters Backend template VPC Id Subnet Ids DB Name
  • 20.
    # TIAD@ tiadparis Difficulty:managing variables 10 VPC Addresses Subnets Instance types Infra template Parameters Backend template VPC Id Subnet Ids DB Name ? ?
  • 21.
    # TIAD@ tiadparis Difficulty:managing variables 10 VPC Addresses Subnets Instance types Infra template Parameters Backend template VPC Id Subnet Ids DB Name ? ? Wrapper to manage inputs/outputs Outputs
  • 22.
    # TIAD@ tiadparis Step3: Deploy application 11 Deploy application • Load-balancers • Servers • DNS Alias NAT NATBastion eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets DynamoDB
  • 23.
    # TIAD@ tiadparis Step3: Deploy application 11 CloudFormation Deploy application • Load-balancers • Servers • DNS Alias NAT NATBastion eu-west-1a Public subnets Private subnets eu-west-1b Public subnets Private subnets DynamoDB WEB WEB Load-balancer included to allow deployment of several versions Web server is built from an image with application pre-installed
  • 24.
    # TIAD@ tiadparis Difficulty:managing variables, again 12 VPC Addresses Subnets Instance types Infra template Backend template VPC Id Subnet Ids DB Name Outputs Application template Instance Types DB Name
  • 25.
    # TIAD@ tiadparis Difficulty:managing variables, again 12 VPC Addresses Subnets Instance types Infra template Backend template VPC Id Subnet Ids DB Name Outputs Application template Instance Types DB Name WEB DynamoDB ?
  • 26.
    # TIAD@ tiadparis Difficulty:managing variables, again 12 VPC Addresses Subnets Instance types Infra template Backend template VPC Id Subnet Ids DB Name Outputs Application template Instance Types DB Name WEB DynamoDB ? "UserData” : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bashn", "echo ", { "Ref" : "Properties" } ," >> /var/www/html/application.propertiesn” ]]}} Properties Properties
  • 27.
    # TIAD@ tiadparis Updatingthe application: the old-fashion way 13 DEV Version Control System Continuous Integration Analyse code Build Repository Push code to VCS Traditional Continuous integration • Build a binary application artifact • WAR, JAR, RPM, DEB, ZIP Artifact Traditional Deployment • Use previously built environments • Deploy the artifact OPS Configured server(s) Staging Configured server(s) Production Environments will drift
  • 28.
    # TIAD@ tiadparis Updatingthe application: with config management 14 Environments can still drift Not suited for autoscaling DEV Version Control System Continuous Integration Analyse code Build Repository Push code to VCS Traditional Continuous integration • Build a binary application artifact • WAR, JAR, RPM, DEB, ZIP Artifact Use config management tools • Same recipes across environments • Same artefact across environments • Manual application deployment OPS Configured server(s) Staging Configured server(s) Production
  • 29.
    # TIAD@ tiadparis Updatingthe application: in the cloud 15 Building can take a while Some drift still possible DEV Version Control System Continuous Integration Analyse code Build Repository Push code to VCS Traditional Continuous integration • Build a binary application artifact • WAR, JAR, RPM, DEB, ZIP Artifact Use config management tools • Same recipes across environments • Same artefact across environments • Deploy application at server start OPS Configured server(s) Staging Configured server(s) Production
  • 30.
    # TIAD@ tiadparis Updatingthe application 16 DEV Version Control System Continuous Integration Analyse code Build Repository Push code to VCS New Continuous integration • Build an application artifact • Build a server image Artifact OPS Staging Production
  • 31.
    # TIAD@ tiadparis Updatingthe application 16 DEV Version Control System Continuous Integration Analyse code Build Repository Push code to VCS New Continuous integration • Build an application artifact • Build a server image Artifact OPS Staging Production Provision, Config, deploy Reference templatesBuild Application templates
  • 32.
    # TIAD@ tiadparis Updatingthe application 16 DEV Version Control System Continuous Integration Analyse code Build Repository Push code to VCS New Continuous integration • Build an application artifact • Build a server image Artifact OPS Configured server(s) Staging Configured server(s) Production Provision, Config, deploy Reference templatesBuild Application templates
  • 33.
    # TIAD@ tiadparis Demo 17 Pushcode Integration AWS images AWS Application templates Web hook Packer • Automate the creation of templates • Developped by @mitchellh / @hashicorp
  • 34.
    # TIAD@ tiadparis Demo 17 Pushcode Integration AWS images AWS Application templates Web hook Build WEB WEB Packer • Automate the creation of templates • Developped by @mitchellh / @hashicorp
  • 35.
    # TIAD@ tiadparis Packer 18 "builders":[{ "type": "amazon-ebs", "region": ”eu-west-1", "source_ami": "ami-f0b11187", "instance_type": "t2.small", "ssh_username": "ubuntu", "ami_name": "demo-{{isotime "2006-01-02T15-04-05"}}" }], "provisioners": [ { "type": "file", "source": "/jenkins_workspace/site", "destination": "/tmp/www" }, { "type": "shell", "inline": [ "sudo apt-get update", "sudo apt-get install -y apache2 php5 libapache2-mod-php5 php5-curl php5-mysql", "sudo rm /var/www/html/*", "sudo mv /tmp/www/* /var/www/html", "sudo service apache2 restart" ] }]
  • 36.
    # TIAD@ tiadparis19 Push code Continuous Integration Application templates IntegrationPerspectives
  • 37.
    # TIAD@ tiadparis19 Production Push code Continuous Integration Application templates Integration Prod DNS Perspectives
  • 38.
    # TIAD@ tiadparis19 Production Push code Continuous Integration Application templates Integration Prod DNS Perspectives
  • 39.
    # TIAD@ tiadparis19 Production Test Push code Continuous Integration Application templates Integration Prod DNS Perspectives
  • 40.
    # TIAD@ tiadparis19 Production Test Push code Continuous Integration Application templates Integration Prod DNS Perspectives
  • 41.
    # TIAD@ tiadparis19 Production Test Push code Continuous Integration Application templates Integration Prod DNS Perspectives
  • 42.
    # TIAD@ tiadparis Keytake-aways 20 Everything can be automated Very important change: Immutable servers • New application artifact: Images / Containers • Very challenging for organizations
  • 43.
    # TIAD@ tiadparis Thankyou @lbernail Fork the code of this demo on github https://github.com/lbernail/demo