Deploying Software at
Scale
Kris Buytaert
@krisbuytaert

Kris Buytaert
● I used to be a Dev,
● Then Became an Op
● Chief Trolling Officer and Open Source
Consultant @inuits.eu
● Everything is an effing DNS Problem
● Building Clouds since before the bookstore
● Some books, some papers, some blogs
● Evangelizing devops

Todays Goals
● A reproducable way to deploy and upgrade
software
● Automatically
● Fast
● Consistent

What's this devops thing
anyhow ?

What's the problem ?
The community of developers whose work you
see on the Web, who probably don’t know what
ADO or UML or JPA even stand for, deploy better
systems at less cost in less time at lower risk
than we see in the Enterprise. This is true even
when you factor in the greater flexibility and
velocity of startups.
Tim Bray , on his blog January 2010

The Old Days
● “Put this Code Live, here's a tarball” NOW!
● What dependencies ?
● No machines available ?
● What database ?
● Security ?
● High Availability ?
● Scalability ?
● My computer can't install this ?

devops
● Culture
● (Lean)
● Automation
● Measurement
● Sharing
Damon Edwards and John Willis
Gene Kim

devops (<)> continuous delilvery

Nirvana
An “ecosystem” that supports continuous delivery, from
infrastructure, data and configuration management to
business.
Through automation of the build, deployment, and testing
process, and improved collaboration between developers,
testers, and operations, delivery teams can get changes
released in a matter of hours — sometimes even minutes–no
matter what the size of a project or the complexity of its code
base.
Continuous Delivery , Jez Humble

How many times a day ?
● 10 @ Flickr
● Deployments used to be pain
● Nobody dared to deploy a site
● Practice makes perfect
● Knowing you can vs constantly doing it

" Our job as engineers (and ops, dev-ops, QA,
support, everyone in the company actually) is to
enable the business goals. We strongly feel that
in order to do that you must have the ability to
deploy code quickly and safely. Even if the
business goals are to deploy strongly QA’d code
once a month at 3am (it’s not for us, we push all
the time), having a reliable and easy
deployment should be non-negotiable."
Etsy Blog upon releasing Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

How do we get there ?

OS Baseline
● Automated Deployments
● Reproducable
● Kickstart, FAI, Preseeding,
● JeOS

Infrastructure as Code
● Treat configuration automation as code
● Development best practices
• Model your infrastructure
• Version your cookbooks / manifests
• Test your cookbooks/ manifests
• Dev/ test /uat / prod for your infra
● Model your infrastructure
● A working service = automated ( Application Code +
Infrastructure Code + Security + Monitoring )
● Think Puppet, Chef, Cfengine, Ansible , ....

Version Control
● Git !
● Version ALL the things:
• Source code Application
• Source code Infrastructure
• Builds
• Tests
• Pipelines
• Scripts
• Documentation
• Monitoring scripts

Continuous Integration
● Builds
● Nightly Builds
● Builds with tests
● Nightly Builds with tests
● Frequent integration
● Continuous Integration

Jenkins
● Open Source Continuous Integration Server
● A zillion plugins (400)
● Have developers build stable and deployable
code
● Test Infra code

Jenkins Pipeline

What's in your Pipeline ?

A pipeline
● Checkout code
● Syntax
● Style
● Code Coverage
● Tests
● Build
● More Tests
● Package

App Requirements
● Testable
● Configuration isolated
● Automated Deployments
● “If my computer can't install it , the installer is
borken” Luke Kanies at Fosdem (2007)
● Bulk provisioning of data
● http://www.krisbuytaert.be/blog/how-i-my-java

Why ops like to package
● Packages give you features
•Consistency, security, dependencies
● Uniquely identify where files come from
•Package or cfg-mgmt
● Source repo not always available
•Firewall / Cloud etc ..
● Weird deployment locations , no easy access
● Little overhead when you automate
● CONFIG does not belong in a package

#packagingsucks
• Really ..
● Missing upstream
● Ancient upstream
● Unneeded dependencies
● Broken upstream
● Distro Policies
● “Maximum RPM” ?
● Packaging is needed
● Dependencies, tooling, repositories
● Anger driven development

Not all packages are equal

#packaginlove

fpm
fpm -t rpm -s dir -n hornetq -v 2.2.5 hornetq
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.nNkVwh
+ umask 022
+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV
+ umask 022
+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
+ tar -zxf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA
+ umask 022
+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/brp-strip
+ /usr/lib/rpm/brp-strip-static-archive
+ /usr/lib/rpm/brp-strip-comment-note
Processing files: hornetq-2.2.5-1.x86_64
Checking for unpackaged file(s): /usr/lib/rpm/check-files /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
Wrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpm
Wrote: /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.z2UL3B
+ umask 022
+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
+ rm -rf /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD
+ exit 0
Created /usr/local/hornetq-2.2.5.x86_64.rpm

fpm in action
● https://github.com/Inuits/build-gems
● Fork, pull
● Jenkins pulls , builds , pushes to repo
● (variants for Nagios Plugins / Jenkins Plugins
available)

A pipeline
● Checkout code ● Upload to Repo
● Syntax
● Style
● Code Coverage
● Tests
● Build
● More Tests
● Package

Pulp
Pulp is a Python application for managing
software repositories and their associated
content, such as packages, errata, and
distributions. It can replicate software
repositories from a variety of supported sources,
such as http/https, file system, ISO, and RHN, to
a local on-site repository. It provides mechanisms
for systems to gain access to these repositories,
providing centralized software installation.

Pulp
● Redhat Community
● Redhat Emerging Technology
● Part of Katello

Pulp
● “manages” its own apache instance
● Symlinks , no copies
● Queues
•Syncing in the background
•No more screens ;)
● Actions are not instantly
•e.g. Add / sync / delete
● Hello mongodb :(
● v1 vs v2
● Only use repo functionality , cfgmgmt is in charge of packages

Repository Management

Version vs Latest
● Version your repos ?
ensure => latests
● Latest your environments ?
● Strict versioning in config ?
ensure => '0.98.4'

A pipeline
● Checkout code ● Upload to Repo
● Syntax ● Deploy on Test
● Style
● Code Coverage
● Tests
● Build
● More Tests
● Package

The Marionette Collective
● Distributed ssh ++
● What version of ssh do I have installed on my
servers ?
● On what servers is XYZ running ?
● Clean all my ssl certs ?
● Restart apache on all servers with fact X

mc-package
mc-package -W /dev/ status jdk
* [ ============================================================> ] 33 / 33
servicesdb01.dev.com version = -absent
services.dev.google.com version = jdk-1.6.0_13-fcs
drbdtest02.dev.google.com version = -absent
services3.dev.google.com version = jdk-1.6.0_20-fcs
um.dev.google.com version = jdk-1.5.0_19-fcs
devtools03.uat.com version = jdk-1.6.0_29-fcs
alexandria02.dev.google.com version = -absent
weblink01.dev.com version = -absent
wikitest.dev.google.com version = jdk-1.6.0_24-fcs
payment.dev.google.com version = jdk-1.5.0_17-fcs
tiff2pdf01.dev.com version = -absent
devdoos.dev.com version = jdk-1.6.0_30-fcs
wiki.dev.google.com version = jdk-1.6.0_24-fcs
reporting01.dev.com version = -absent
devtools01-dev.uat.com version = jdk-1.6.0_23-fcs
devtools02.uat.com version = jdk-1.6.0_29-fcs
drbdtest01.dev.google.com version = -absent
---- package agent summary ----
Nodes: 33/33
Versions: 1 * 1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs, 1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0
Elapsed Time: 1.73 s

What to Trigger ?
● Update Package
• Only updates package
● Trigger Puppet Run
• Updates config + package

A pipeline
● Checkout code ● Upload to Repo
● Syntax ● Deploy on Test
● Style ● More Tests
● Code Coverage ● Promote
● Tests ● Deploy on UAT
● Build ● More Tests
● More Tests ● Promote
● Package ● Deploy on Prod

Done ?
● Close the feedback loop,
● Send metric on deployment
echo "deployed.$package_name 1 `date +
%s`" > /dev/tcp/<%= graphite_host %>/2003

Done ?
A Software project is not done until your last
enduser is in his grave !

But remember
Everything is a Fscking DNS Problem
No really, Everything is a Fscking DNS Problem
If it's not a fucking DNS Problem ..
It's an arp problem
If it's not an arp problem...
It's a Full Filesystem Problem
If your filesystem isn't full
It's a Spanning Tree problem
If it's not a spanning Tree problem...
It's a USB problem
If it's not a USB Problem
It might be an ntp problem
If it's not an ntp problem
It's a sharing IRQ Problem
If it's not a sharing IRQ Problem
But most often .. its a Freaking Dns Problem !
Or someone playing tricks on you
Jan 2006

Contact
Kris Buytaert
Kris.Buytaert@inuits.be
Further Reading
@krisbuytaert
http://www.krisbuytaert.be/b
log/
http://www.inuits.be/
Inuits
Duboistraat 50
2060 Antwerpen
Belgium
891.514.231
+32 475 961221