Puppet Camp Sydney 2014 - A Build Engineering Team’s Journey of Infrastructure as Code
Upcoming SlideShare
Loading in...5
×
 

Puppet Camp Sydney 2014 - A Build Engineering Team’s Journey of Infrastructure as Code

on

  • 1,381 views

A Build Engineering Team’s Journey of Infrastructure as Code - the challenges that we’ve faced and the practices that we implemented as we went along our journey. 

A Build Engineering Team’s Journey of Infrastructure as Code - the challenges that we’ve faced and the practices that we implemented as we went along our journey. 

Statistics

Views

Total Views
1,381
Slideshare-icon Views on SlideShare
998
Embed Views
383

Actions

Likes
3
Downloads
11
Comments
2

7 Embeds 383

http://devopsbox.es 260
https://twitter.com 77
http://localhost 40
http://www.linkedin.com 3
http://feedly.com 1
https://www.linkedin.com 1
http://www.slideee.com 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Great presentation Peter.
    Are you sure you want to
    Your message goes here
    Processing…
  • Really enjoyed this presentation Peter. Thanks for sharing your experiences at Atlassian. I'll be taking this to the team I work in to show what others are doing with Puppet.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Puppet Camp Sydney 2014 - A Build Engineering Team’s Journey of Infrastructure as Code Puppet Camp Sydney 2014 - A Build Engineering Team’s Journey of Infrastructure as Code Presentation Transcript

    • Monday, 10 February 14
    • Peter Leschev @peterleschev Husband, Father of 3 & Atlassian Build Engineering Team Lead Monday, 10 February 14
    • A Build Engineering Team’s Journey of Infrastructure as Code Monday, 10 February 14
    • Build Engineering today @ Atlassian • Build platform & services used internally within the company • 60k builds per month • 35k automated tests for JIRA Monday, 10 February 14
    • Build Engineering today @ Atlassian • 600 build agents (own hardware + EC2 instances) • include SCM clients, JDKs, JVM build tools, databases, headless browser testing, python builds, NodeJS, installers & more • Maintain 20 AMIs of various build configurations • 6 Bamboo Servers • maven.atlassian.com / 6 Nexus instances • Monitoring - opsview / graphite / statsd Monday, 10 February 14
    • Infrastructure as Code = Puppet + SCM ? Monday, 10 February 14
    • 3 years ago... • Manually maintained snowflakes • Started using puppet Monday, 10 February 14
    • Production rollout puppetmaster build agents Monday, 10 February 14
    • Production rollout failure puppetmaster build agents Monday, 10 February 14
    • Confidence of Change HIGH NONE Dev Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • Monday, 10 February 14
    • https://bitbucket.org/ http://atlassian.com/git Monday, 10 February 14
    • Style in Pull Requests Monday, 10 February 14
    • Puppet Lint https://github.com/rodjek/puppet-lint Tim Sharpe @rodjek • Automated style checking • Setup automated build that runs checks & posts results • Still need to implement a ratchet build Monday, 10 February 14
    • Confidence of Change initial + Code review HIGH NONE Dev Code review Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • Using Staging for Development puppetmaster • Coding on Puppet Master • Culture of manually modifying production - Configuration Drift • Impact on Builds staging puppet environment build agents Monday, 10 February 14
    • Vagrant http://www.vagrantup.com/ Mitchell Hashimoto @mitchellh • Easily spin up Infrastructure locally on your laptop • Disposable / reproducible environments • Machine provisioning via Virtual Box / VMWare / AWS • Configuration applied via Shell Scripts / Puppet / Chef • Develop and test infrastructure changes locally Monday, 10 February 14
    • Vagrant Vagrantfile vagrant basebox Monday, 10 February 14 http://www.vagrantup.com/ Mitchell Hashimoto @mitchellh
    • Vagrant Mitchell Hashimoto @mitchellh http://www.vagrantup.com/ Spins up a local VM to a known state Make some puppet changes and then run: to apply your changes SSH into your VM using: to check your changes Destroy the VM when done Monday, 10 February 14
    • Confidence of Change initial + Code review + Vagrant HIGH NONE Dev Code review Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • Vagrant != Production • Vagrant basebox differences with production machines • Originally using publicly available vagrant baseboxes • Installed packages biggest differences • Generating a basebox manually was a painful process Monday, 10 February 14
    • Veewee Automated Vagrant basebox generation https://github.com/jedi4ever/veewee Patrick Debois @patrickdebois + Ubuntu installation iso Monday, 10 February 14 Veewee definitions.rb preseed.cfg postinstall.sh vagrant basebox
    • Veewee Automated Vagrant basebox generation Monday, 10 February 14 https://github.com/jedi4ever/veewee Patrick Debois @patrickdebois
    • Basebox generation via CI • Latest basebox generated in CI & published to fileshare • No need to generate baseboxes locally Monday, 10 February 14
    • There are still differences! • VirtualBox Guest additions • Reduced to a minimal Monday, 10 February 14
    • Common Preseed / Postinstall + preseed.cfg vagrant basebox Monday, 10 February 14 postinstall.sh PXEBoot custom ISOs
    • Packer Monday, 10 February 14 http://packer.io Mitchell Hashimoto @mitchellh
    • Confidence in Change initial + Code review + Vagrant + Veewee HIGH NONE Dev Code review Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • Developing locally Rolling out to staging Rolling out to production Broken build agents! Monday, 10 February 14
    • Cucumber • Behaviour Driven Development Monday, 10 February 14
    • Cucumber & Vagrant Vagrant Virtual Box VM Custom Provisioner via ssh puppet apply cucumber *.features Monday, 10 February 14
    • Disadvantages • Requires cucumber dependencies to be installed on tested VM • Tests run within the VM making testing firewall rules harder Monday, 10 February 14
    • Confidence in Change initial + Code review + Vagrant + Veewee + Cukes HIGH NONE Dev Code review Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • “ Monday, 10 February 14 But it works on my machine! – Every Developer ”
    • Continuous Integration • ‘From scratch’ provisioning • Confidence that you can rebuild in disaster Monday, 10 February 14
    • “ The Pets: you give nice names, you stroke them, and when they get ill, you nurse them back to health, taking a long time over it The Cattle: you give them numbers. When they get ill, you shoot them ” – Tim Bell, CERN Monday, 10 February 14
    • Confidence in Change initial + Code review + Vagrant + Veewee + Cukes + CI HIGH NONE Dev Code review CI & Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • Provisioning from scratch is slow Monday, 10 February 14
    • Spread out CI provision VM1 provision VM2 provision VM3 provision VM4 Monday, 10 February 14 provision VM1 provision VM2 Moved from sequential to parallel provisioning provision VM3 provision VM4
    • There are so many MacPros you can steal Monday, 10 February 14
    • The ones I have my eye on.... Monday, 10 February 14
    • Profiling Puppet Runs Add “--evaltrace” to puppet apply + Collect and show the longest occurrences of: “Evaluated in ([d.]+) seconds” Monday, 10 February 14 =
    • Profiling Cucumber runs http://itshouldbeuseful.wordpress.com/2010/11/10/find-your-slowest-running-cucumber-features/ Monday, 10 February 14
    • Delta Provisioning • Provision locally & for CI • Faster & different class of problems found • Matches production state ‘from scratch’ provision delta provision provision VM1 import VM1 box on success export VM1 Monday, 10 February 14 fileshare provision VM1
    • Confidence in Change initial + CI + Code review + Delta CI + Vagrant + Veewee + Cukes HIGH NONE Dev Code review CI & Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • Infrequent Releases Monday, 10 February 14
    • Painful Puppet Rollouts • Puppet runs impacted running builds • Disabling all the build agents • Performing the roll out • • git clone / librarian-puppet / symlink update on puppetmaster Manually kick off puppet on all the build agents • Enabling all the build agents • Set of Puppet environments for every bamboo server Monday, 10 February 14
    • Graceful Service restarts + Bamboo Agent JVM process watches for touch file & shutdowns when Idle (written as a Bamboo Plugin) Monday, 10 February 14
    • Puppet Environments • BEFORE - Multiple puppet envs for each Bamboo Server • jbac_staging • jbac_production • cbac_staging • cbac_production • etc • AFTER - Changed to use ‘staging’ & ‘production’ only Monday, 10 February 14
    • Updates on Puppetmaster • BEFORE: Manually on puppetmaster • git clone the puppet tree • run librarian-puppet to pull external modules • Update staging / production symlink • AFTER: Bamboo build which performs the above steps automatically Monday, 10 February 14
    • Less Human interaction + More automation = Higher Confidence Monday, 10 February 14
    • Less Human Effort = Increased frequency of releases Monday, 10 February 14
    • Confidence in Change initial + Cukes + Code review + CI + Vagrant + Delta CI + Veewee + Frequent releases HIGH NONE Dev Code review CI & Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • “ I’m scared! “ ” – Peter Leschev, 3 years ago ” Should I be scared? Monday, 10 February 14 – Peter Leschev, 3 months ago
    • Hipchat integration Monday, 10 February 14
    • Confidence in Change initial + Cukes + Notification + Code review + CI + Vagrant + Delta CI + Veewee + Frequent releases HIGH NONE Dev Code review CI & Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • Confidence in Change before after HIGH NONE Dev Code review CI & Rollout Lifecycle of an infra change Monday, 10 February 14 Soak in Prod
    • Confidence in Change or Finding & fixing problems sooner rather than later Monday, 10 February 14
    • Commit Graph Monday, 10 February 14
    • Snowflakes Pets Cattle Stateless Machines Monday, 10 February 14
    • We’re still on the Journey Come join us! atlassian.com/jobs Monday, 10 February 14
    • Questions? Monday, 10 February 14
    • Thank you! Monday, 10 February 14
    • Monday, 10 February 14