1© Copyright 2015 EMC Corporation. All rights reserved.
INFRASTRUCTURE AS CODE
BAN SNOWFLAKE SERVERS
1© Copyright 2015 EMC Corporation. All rights reserved.
2© Copyright 2015 EMC Corporation. All rights reserved. 2© Copyright 2015 EMC Corporation. All rights reserved.
Jonas Rosland
Developer Advocate
EMC {code}
@jonasrosland
Loves retro gaming
Talks a lot
3© Copyright 2015 EMC Corporation. All rights reserved. 3© Copyright 2015 EMC Corporation. All rights reserved.
Reid Vandewiele
Solutions Engineer
Puppet Labs®
@reidmv
Biking, Hiking
Code
4© Copyright 2015 EMC Corporation. All rights reserved.
5© Copyright 2015 EMC Corporation. All rights reserved.
WHAT IS DEVOPS?
6© Copyright 2015 EMC Corporation. All rights reserved.
AN IDEA OR COLLABORATIVE
CULTURE/PHILOSOPHY BETWEEN
TECHNICAL TEAMS
7© Copyright 2015 EMC Corporation. All rights reserved.
UNIFIED PROCESSES AND TOOLS
FOR FASTER END-TO-END DELIVERY
OF QUALITY SOFTWARE
8© Copyright 2015 EMC Corporation. All rights reserved.
NOT A JOB DESCRIPTION!
9© Copyright 2015 EMC Corporation. All rights reserved.
SO WHY SHOULD YOU
CARE?
10© Copyright 2015 EMC Corporation. All rights reserved.
YOU CAN'T WAIT 6 MONTHS FOR A
FEATURE UPGRADE/BUG FIX/ETC
11© Copyright 2015 EMC Corporation. All rights reserved.
SOLVE CONFIGURATION DRIFT
ISSUES QUICKLY
12© Copyright 2015 EMC Corporation. All rights reserved.
KNOW EXACTLY WHAT'S WRONG,
WHERE AND WHY
13© Copyright 2015 EMC Corporation. All rights reserved.
http://46zwyrvli634e39iq2l9mv8g.wpengine.netdna-cdn.com/wp-content/uploads/2013/06/DevOps-infinity-loop2.png
14© Copyright 2015 EMC Corporation. All rights reserved.
LAST BUT NOT LEAST, IMPROVE
TEAM COMMUNICATION
15© Copyright 2015 EMC Corporation. All rights reserved.
SO WHAT'S THE RESULT OF
DEVOPS?
16© Copyright 2015 EMC Corporation. All rights reserved.
IT'S REALLY BIG
ACTUALLY...
17© Copyright 2015 EMC Corporation. All rights reserved.
DEVOPS PRACTICES AND IT
PERFORMANCE IMPACT
ORGANIZATIONAL PERFORMANCE
http://www.slideshare.net/realgenekim/2014-state-of-devops-findings-velocity-conference
18© Copyright 2015 EMC Corporation. All rights reserved.
LET'S START WITH
INFRASTRUCTURE AS
CODE
19© Copyright 2015 EMC Corporation. All rights reserved.
WHAT DO WE DEFINE AS
INFRASTRUCTURE?
20© Copyright 2015 EMC Corporation. All rights reserved.
21© Copyright 2015 EMC Corporation. All rights reserved.
22© Copyright 2015 EMC Corporation. All rights reserved.
23© Copyright 2015 EMC Corporation. All rights reserved.
• Deploy, monitor and connect together all the pieces
needed to run services for the organization
• Processes and tools for faster end-to-end delivery of
quality services
• Automation comes built-in
WHY DEFINE INFRASTRUCTURE AS CODE?
24© Copyright 2015 EMC Corporation. All rights reserved.
• Desired state specified in text files
• Autonomic (self-corrects to desired state)
• State should be known through monitoring
• Remove snowflake servers
HOW DO WE DO THIS?
25© Copyright 2015 EMC Corporation. All rights reserved.
• Easy to read and edit
• Shareable
• Can use standard version control like Git or SVN
• Becomes executable documentation
WHY STORE THEM IN TEXT FILES?
26© Copyright 2015 EMC Corporation. All rights reserved.
node 'www2' {
class { 'apache': } # use apache module
apache::vhost { 'awesomewebsite.com': # define vhost
port => '80',
docroot => '/var/www/html'
}
}
SIMPLE EXAMPLE
27© Copyright 2015 EMC Corporation. All rights reserved.
user { root:
ensure => present,
password => '$ecretP@ssw0rd',
}
CHANGE ROOT PASSWORD
28© Copyright 2015 EMC Corporation. All rights reserved.
MANUALLY CONFIGURED
ENVIRONMENTS ARE LIKE A HOUSE
OF CARDS IN A CHINA SHOP
- NEAL FORD
29© Copyright 2015 EMC Corporation. All rights reserved.
• Deploying, provisioning and scaling automatically is
virtually impossible if every server is unique
• Adds friction between the requestor and the deployer
• Mistakes happen
• We're all human
SNOWFLAKE SERVERS
30© Copyright 2015 EMC Corporation. All rights reserved.
WHAT CAN HAPPEN IF YOU
DON'T TREAT
INFRASTRUCTURE AS CODE?
31© Copyright 2015 EMC Corporation. All rights reserved.
UP TO 60% OF FAILURES ARE
CAUSED BY HUMAN ERROR, NOT
HARDWARE FAILURE
32© Copyright 2015 EMC Corporation. All rights reserved.
EXAMPLE - KNIGHT CAPITAL GROUP
https://infocus.emc.com/wp-content/uploads/2012/08/KCG.jpg
33© Copyright 2015 EMC Corporation. All rights reserved.
• Manual deployment of new trading software
• 7 of 8 servers correctly updated
• Old function still alive on the 8th server led to…
KNIGHT CAPITAL GROUP - WHAT HAPPENED?
34© Copyright 2015 EMC Corporation. All rights reserved.
$440 MILLION LOSS
IN 45 MINUTES
35© Copyright 2015 EMC Corporation. All rights reserved.
SO
36© Copyright 2015 EMC Corporation. All rights reserved.
TREAT YOUR
INFRASTRUCTURE AS
CODE
37© Copyright 2015 EMC Corporation. All rights reserved.
BARE METAL  APP STACK
INFRASTRUCTURE AS CODE
38© Copyright 2015 EMC Corporation. All rights reserved.
• Razor
– Text / JSON configuration
– Bare-metal provisioning
– Policy based
• Puppet
– Text configuration manifests
– Software configuration management
INFRASTRUCTURE AS CODE
39© Copyright 2015 EMC Corporation. All rights reserved.
OS – DNS – DHCP - DB
Storage – Network - Compute
INFRASTRUCTURE AS CODE
RAZOR
40© Copyright 2015 EMC Corporation. All rights reserved.
OS – DNS – DHCP - DB
Storage – Network - Compute
App Queue
Business
Logic
PUPPET
INFRASTRUCTURE AS CODE
41© Copyright 2015 EMC Corporation. All rights reserved.
STARTING ENVIRONMENT:
INFRASTRUCTURE AS CODE
One Razor Server
One Puppet Server
42© Copyright 2015 EMC Corporation. All rights reserved.
LIVE DEMO
INFRASTRUCTURE AS CODE
43© Copyright 2015 EMC Corporation. All rights reserved.
INFRASTRUCTURE AS CODE
Boot
PXE To
Razor
OS
Installed
Puppetize! Decommission
One-time Provisioning
Lifetime Management
44© Copyright 2015 EMC Corporation. All rights reserved.
Boot
PXE To
Razor
OS
Installed
Puppetize! Decommission
One-time Provisioning
Lifetime Management
INFRASTRUCTURE AS CODE
45© Copyright 2015 EMC Corporation. All rights reserved.
• Self documenting infrastructure
• You now have source code for how anything in your
datacenter is setup
• Executable documentation
RECAP OF THE BENEFITS
46© Copyright 2015 EMC Corporation. All rights reserved.
ALRIGHT
47© Copyright 2015 EMC Corporation. All rights reserved.
THEN WHAT?
48© Copyright 2015 EMC Corporation. All rights reserved.
HOW ABOUT
AUTOMATING SOME
MORE?
49© Copyright 2015 EMC Corporation. All rights reserved.
• Jenkins CI
• Travis CI
• Codeship
• CloudFoundry
• OpenShift
DEPLOYMENT AUTOMATION
50© Copyright 2015 EMC Corporation. All rights reserved.
• Docker
• Framework for application container management
• HUGE community
CONTAINERS
51© Copyright 2015 EMC Corporation. All rights reserved.
• Splunk
• Loggly
• LogInsight
• ELK Stack
– ElasticSearch
– LogStash
– Kibana
LOG MANAGEMENT
52© Copyright 2015 EMC Corporation. All rights reserved.
• vCenter Operations
• New Relic
• AppDynamics
• DataDog
PERFORMANCE MANAGEMENT
53© Copyright 2015 EMC Corporation. All rights reserved.
• Nagios
• Sensu
• VictorOps
• PagerDuty
MONITORING
54© Copyright 2015 EMC Corporation. All rights reserved.
• Slack
• HipChat
• Trello
• Asana
COMMUNICATION
55© Copyright 2015 EMC Corporation. All rights reserved.
• Github
• Quip
COLLABORATION
56© Copyright 2015 EMC Corporation. All rights reserved.
• And by smartly we mean cooperatively between
Devs and Ops
• Will help operators understand what's running and
happening in the infrastructure
• Will give developers insight into the underlying
infrastructure and the way it behaves under load
USE THESE TOOLS SMARTLY
57© Copyright 2015 EMC Corporation. All rights reserved.
• Start talking about it
• Choose low-hanging fruit
• Learn, make wrong decisions, learn again
• Involve the broader team
– Give books, presentations, blog posts etc.
SO HOW DO WE GET STARTED?
58© Copyright 2015 EMC Corporation. All rights reserved.
QUESTIONS?
Jonas Rosland
Developer Advocate @ EMC {code}
@jonasrosland
VMworld 2015 San Francisco - INF5432 - Infrastructure as Code - Ban Snowflake Deployments

VMworld 2015 San Francisco - INF5432 - Infrastructure as Code - Ban Snowflake Deployments

  • 1.
    1© Copyright 2015EMC Corporation. All rights reserved. INFRASTRUCTURE AS CODE BAN SNOWFLAKE SERVERS 1© Copyright 2015 EMC Corporation. All rights reserved.
  • 2.
    2© Copyright 2015EMC Corporation. All rights reserved. 2© Copyright 2015 EMC Corporation. All rights reserved. Jonas Rosland Developer Advocate EMC {code} @jonasrosland Loves retro gaming Talks a lot
  • 3.
    3© Copyright 2015EMC Corporation. All rights reserved. 3© Copyright 2015 EMC Corporation. All rights reserved. Reid Vandewiele Solutions Engineer Puppet Labs® @reidmv Biking, Hiking Code
  • 4.
    4© Copyright 2015EMC Corporation. All rights reserved.
  • 5.
    5© Copyright 2015EMC Corporation. All rights reserved. WHAT IS DEVOPS?
  • 6.
    6© Copyright 2015EMC Corporation. All rights reserved. AN IDEA OR COLLABORATIVE CULTURE/PHILOSOPHY BETWEEN TECHNICAL TEAMS
  • 7.
    7© Copyright 2015EMC Corporation. All rights reserved. UNIFIED PROCESSES AND TOOLS FOR FASTER END-TO-END DELIVERY OF QUALITY SOFTWARE
  • 8.
    8© Copyright 2015EMC Corporation. All rights reserved. NOT A JOB DESCRIPTION!
  • 9.
    9© Copyright 2015EMC Corporation. All rights reserved. SO WHY SHOULD YOU CARE?
  • 10.
    10© Copyright 2015EMC Corporation. All rights reserved. YOU CAN'T WAIT 6 MONTHS FOR A FEATURE UPGRADE/BUG FIX/ETC
  • 11.
    11© Copyright 2015EMC Corporation. All rights reserved. SOLVE CONFIGURATION DRIFT ISSUES QUICKLY
  • 12.
    12© Copyright 2015EMC Corporation. All rights reserved. KNOW EXACTLY WHAT'S WRONG, WHERE AND WHY
  • 13.
    13© Copyright 2015EMC Corporation. All rights reserved. http://46zwyrvli634e39iq2l9mv8g.wpengine.netdna-cdn.com/wp-content/uploads/2013/06/DevOps-infinity-loop2.png
  • 14.
    14© Copyright 2015EMC Corporation. All rights reserved. LAST BUT NOT LEAST, IMPROVE TEAM COMMUNICATION
  • 15.
    15© Copyright 2015EMC Corporation. All rights reserved. SO WHAT'S THE RESULT OF DEVOPS?
  • 16.
    16© Copyright 2015EMC Corporation. All rights reserved. IT'S REALLY BIG ACTUALLY...
  • 17.
    17© Copyright 2015EMC Corporation. All rights reserved. DEVOPS PRACTICES AND IT PERFORMANCE IMPACT ORGANIZATIONAL PERFORMANCE http://www.slideshare.net/realgenekim/2014-state-of-devops-findings-velocity-conference
  • 18.
    18© Copyright 2015EMC Corporation. All rights reserved. LET'S START WITH INFRASTRUCTURE AS CODE
  • 19.
    19© Copyright 2015EMC Corporation. All rights reserved. WHAT DO WE DEFINE AS INFRASTRUCTURE?
  • 20.
    20© Copyright 2015EMC Corporation. All rights reserved.
  • 21.
    21© Copyright 2015EMC Corporation. All rights reserved.
  • 22.
    22© Copyright 2015EMC Corporation. All rights reserved.
  • 23.
    23© Copyright 2015EMC Corporation. All rights reserved. • Deploy, monitor and connect together all the pieces needed to run services for the organization • Processes and tools for faster end-to-end delivery of quality services • Automation comes built-in WHY DEFINE INFRASTRUCTURE AS CODE?
  • 24.
    24© Copyright 2015EMC Corporation. All rights reserved. • Desired state specified in text files • Autonomic (self-corrects to desired state) • State should be known through monitoring • Remove snowflake servers HOW DO WE DO THIS?
  • 25.
    25© Copyright 2015EMC Corporation. All rights reserved. • Easy to read and edit • Shareable • Can use standard version control like Git or SVN • Becomes executable documentation WHY STORE THEM IN TEXT FILES?
  • 26.
    26© Copyright 2015EMC Corporation. All rights reserved. node 'www2' { class { 'apache': } # use apache module apache::vhost { 'awesomewebsite.com': # define vhost port => '80', docroot => '/var/www/html' } } SIMPLE EXAMPLE
  • 27.
    27© Copyright 2015EMC Corporation. All rights reserved. user { root: ensure => present, password => '$ecretP@ssw0rd', } CHANGE ROOT PASSWORD
  • 28.
    28© Copyright 2015EMC Corporation. All rights reserved. MANUALLY CONFIGURED ENVIRONMENTS ARE LIKE A HOUSE OF CARDS IN A CHINA SHOP - NEAL FORD
  • 29.
    29© Copyright 2015EMC Corporation. All rights reserved. • Deploying, provisioning and scaling automatically is virtually impossible if every server is unique • Adds friction between the requestor and the deployer • Mistakes happen • We're all human SNOWFLAKE SERVERS
  • 30.
    30© Copyright 2015EMC Corporation. All rights reserved. WHAT CAN HAPPEN IF YOU DON'T TREAT INFRASTRUCTURE AS CODE?
  • 31.
    31© Copyright 2015EMC Corporation. All rights reserved. UP TO 60% OF FAILURES ARE CAUSED BY HUMAN ERROR, NOT HARDWARE FAILURE
  • 32.
    32© Copyright 2015EMC Corporation. All rights reserved. EXAMPLE - KNIGHT CAPITAL GROUP https://infocus.emc.com/wp-content/uploads/2012/08/KCG.jpg
  • 33.
    33© Copyright 2015EMC Corporation. All rights reserved. • Manual deployment of new trading software • 7 of 8 servers correctly updated • Old function still alive on the 8th server led to… KNIGHT CAPITAL GROUP - WHAT HAPPENED?
  • 34.
    34© Copyright 2015EMC Corporation. All rights reserved. $440 MILLION LOSS IN 45 MINUTES
  • 35.
    35© Copyright 2015EMC Corporation. All rights reserved. SO
  • 36.
    36© Copyright 2015EMC Corporation. All rights reserved. TREAT YOUR INFRASTRUCTURE AS CODE
  • 37.
    37© Copyright 2015EMC Corporation. All rights reserved. BARE METAL  APP STACK INFRASTRUCTURE AS CODE
  • 38.
    38© Copyright 2015EMC Corporation. All rights reserved. • Razor – Text / JSON configuration – Bare-metal provisioning – Policy based • Puppet – Text configuration manifests – Software configuration management INFRASTRUCTURE AS CODE
  • 39.
    39© Copyright 2015EMC Corporation. All rights reserved. OS – DNS – DHCP - DB Storage – Network - Compute INFRASTRUCTURE AS CODE RAZOR
  • 40.
    40© Copyright 2015EMC Corporation. All rights reserved. OS – DNS – DHCP - DB Storage – Network - Compute App Queue Business Logic PUPPET INFRASTRUCTURE AS CODE
  • 41.
    41© Copyright 2015EMC Corporation. All rights reserved. STARTING ENVIRONMENT: INFRASTRUCTURE AS CODE One Razor Server One Puppet Server
  • 42.
    42© Copyright 2015EMC Corporation. All rights reserved. LIVE DEMO INFRASTRUCTURE AS CODE
  • 43.
    43© Copyright 2015EMC Corporation. All rights reserved. INFRASTRUCTURE AS CODE Boot PXE To Razor OS Installed Puppetize! Decommission One-time Provisioning Lifetime Management
  • 44.
    44© Copyright 2015EMC Corporation. All rights reserved. Boot PXE To Razor OS Installed Puppetize! Decommission One-time Provisioning Lifetime Management INFRASTRUCTURE AS CODE
  • 45.
    45© Copyright 2015EMC Corporation. All rights reserved. • Self documenting infrastructure • You now have source code for how anything in your datacenter is setup • Executable documentation RECAP OF THE BENEFITS
  • 46.
    46© Copyright 2015EMC Corporation. All rights reserved. ALRIGHT
  • 47.
    47© Copyright 2015EMC Corporation. All rights reserved. THEN WHAT?
  • 48.
    48© Copyright 2015EMC Corporation. All rights reserved. HOW ABOUT AUTOMATING SOME MORE?
  • 49.
    49© Copyright 2015EMC Corporation. All rights reserved. • Jenkins CI • Travis CI • Codeship • CloudFoundry • OpenShift DEPLOYMENT AUTOMATION
  • 50.
    50© Copyright 2015EMC Corporation. All rights reserved. • Docker • Framework for application container management • HUGE community CONTAINERS
  • 51.
    51© Copyright 2015EMC Corporation. All rights reserved. • Splunk • Loggly • LogInsight • ELK Stack – ElasticSearch – LogStash – Kibana LOG MANAGEMENT
  • 52.
    52© Copyright 2015EMC Corporation. All rights reserved. • vCenter Operations • New Relic • AppDynamics • DataDog PERFORMANCE MANAGEMENT
  • 53.
    53© Copyright 2015EMC Corporation. All rights reserved. • Nagios • Sensu • VictorOps • PagerDuty MONITORING
  • 54.
    54© Copyright 2015EMC Corporation. All rights reserved. • Slack • HipChat • Trello • Asana COMMUNICATION
  • 55.
    55© Copyright 2015EMC Corporation. All rights reserved. • Github • Quip COLLABORATION
  • 56.
    56© Copyright 2015EMC Corporation. All rights reserved. • And by smartly we mean cooperatively between Devs and Ops • Will help operators understand what's running and happening in the infrastructure • Will give developers insight into the underlying infrastructure and the way it behaves under load USE THESE TOOLS SMARTLY
  • 57.
    57© Copyright 2015EMC Corporation. All rights reserved. • Start talking about it • Choose low-hanging fruit • Learn, make wrong decisions, learn again • Involve the broader team – Give books, presentations, blog posts etc. SO HOW DO WE GET STARTED?
  • 58.
    58© Copyright 2015EMC Corporation. All rights reserved. QUESTIONS? Jonas Rosland Developer Advocate @ EMC {code} @jonasrosland

Editor's Notes

  • #43 Kick off the new VM to be provisioned. It’ll take awhile for the OS to be installed and for Puppet to configure the app, so we kick it off now before spending a little more time talking about what will happen. Demo: - Start VM - Show Razor policies - Show Puppet code - Show Puppet GUI - Show VM progress