Operations: a
developer’s guide
Anna Shipman
@annashipman

@annashipman
Technical architect at the
Government Digital Service

@annashipman
My background

@annashipman
Previous career in
publishing

@annashipman
Self taught programmer:
HTML, CSS, terrible JS

@annashipman
First employed gig, 2005:
backend, Java

@annashipman
First job at GDS, 2012:
redirecting URLs

@annashipman

@annashipman
Wrote perl to generate
nginx config

@annashipman
Didn’t understand the
emails from the
infrastructure team

@annashipman
So joined the
infrastructure team

@annashipman
Which led to where I
am now…

@annashipman
Technical architect on a
large infrastructure project

@annashipman
What I learned has made
me a better developer

@annashipman
Knowing these things might
help you

@annashipman
Wrangling servers
Virtualisation
Containerisation
Some tools to make you a better developer

@annashipman
Each section will have a
take-home

@annashipman
1. Wrangling servers

@annashipman
How the internet works
Wrangling servers

@annashipman
“It’s a series of tubes”
—Sen. Ted Stevens
Wrangling servers

@annashipman
Servers
Wrangling servers

@annashipman
Where are your servers?
Wrangling servers

@annashipman
Where are your servers?
You own them (e.g. in the office)
Wrangling servers

@annashipman
Where are your servers?
You own them (e.g. in the office)
Shared hosting (e.g. Dreamhost)
Wrangling servers

@annashipman
Where are your servers?
You own them (e.g. in the office)
Shared hosting (e.g. Dreamhost)
The cloud (e.g. AWS)
Wrangling servers

@annashipman
Where are your servers?
You own them (e.g. in the office)
Shared hosting (e.g. Dreamhost)
The cloud (e.g. AWS)
PaaS/application hosting (e.g. Heroku)
Wrangling servers

@annashipman
Where are your servers?
You own them (e.g. in the office)
Shared hosting (e.g. Dreamhost)
The cloud (e.g. AWS)
PaaS/application hosting (e.g. Heroku)
Something else/don’t know
Wrangling servers

@annashipman
You need to make sure the
server has the software
you need
Wrangling servers

@annashipman
Handcrafting servers
Wrangling servers

@annashipman
But what happens if your
server dies?
Wrangling servers

@annashipman
It’s also easy to make a
mistake
Wrangling servers

@annashipman
Configuration management
Wrangling servers

@annashipman
Tools that use configuration
you’ve written to build
servers
Wrangling servers

https://www.getfilecloud.com/blog/2014/08/top-8-configuration-management-tools-for-sys-admins/

@annashipman
GOV.UK uses Puppet, my
current project uses Ansible
Wrangling servers

@annashipman
Config management tools
automate building your
servers
Wrangling servers

@annashipman
So it is reliable and
repeatable
Wrangling servers

@annashipman
Some getting started
guides at the end
Wrangling servers

@annashipman
If nothing else, just write a
script
Wrangling servers

@annashipman
“Cattle not pets”
Wrangling servers

@annashipman
You should not be afraid to
lose your servers
Wrangling servers

@annashipman
2. Virtualisation

@annashipman
Creating logical computing
resources from available
physical resources
Virtualisation

@annashipman
Virtual machines
Virtualisation

@annashipman
What’s a hypervisor?
Virtualisation

@annashipman
Hypervisor is the software
that runs the VMs
Virtualisation

@annashipman

@annashipman
Slight digression into
cloud computing
Virtualisation

@annashipman
“Someone else’s
computers”
Virtualisation

@annashipman
Your servers are in a data
centre
Virtualisation

@annashipman
A layer of abstraction that
pools the resources
Virtualisation

@annashipman

@annashipman
Some advantages of cloud computing
Virtualisation

@annashipman
Some advantages of cloud computing
Increased uptime & disaster recovery
Virtualisation

@annashipman
Some advantages of cloud computing
Increased uptime & disaster recovery
Can get a VM straight away
Virtualisation

@annashipman
Some advantages of cloud computing
Increased uptime & disaster recovery
Can get a VM straight away
Charged for what you use
Virtualisation

@annashipman
That’s the cloud – now
your computer
Virtualisation

@annashipman
How is virtualisation useful
to you?
Virtualisation

@annashipman
Used to be software = slow,
resource-intensive
Virtualisation

@annashipman
From 2005 Intel & AMD
started doing hardware-
accelerated virtualisation
Virtualisation

@annashipman
So can run lots of VMs on
your computer
Virtualisation

@annashipman
Vagrant is a lightweight way
to create VMs
Virtualisation

@annashipman
Vagrant.configure(2) do |config|
config.vm.box = "puppetlabs/centos-7.0-64-puppet"
config.vm.provider :virtualbox |vb|
vb.customize ["modifyvm", :id, "--memory", "2048"]
end
end
Virtualisation

@annashipman
Vagrant.configure(2) do |config|
config.vm.box = "ubuntu/trusty64"
end
Virtualisation

@annashipman
$ vagrant up
Virtualisation

@annashipman
$ vagrant ssh
Virtualisation

@annashipman
$ cd /vagrant
Virtualisation

@annashipman
Links to documentation and
simple examples at the end
Virtualisation

@annashipman
You can develop locally
using the same software as
is deployed remotely
Virtualisation

@annashipman
You can use configuration
management to build
your VM
Virtualisation

@annashipman
It makes it easy
to collaborate
Virtualisation

@annashipman
Vagrant makes it easy to
get started/pick up where
you left off
Virtualisation

@annashipman
Currently Otto uses Vagrant
Virtualisation

@annashipman
Vagrant can improve your
project tomorrow
Virtualisation

@annashipman
3. Containerisation

@annashipman
(Docker is not the only
containerisation tech)
Containerisation

@annashipman
Docker is based on
Linux Containers 1. Put netting on garlic and put back outside
Containerisation

@annashipman
2008: LXC
2004: Solaris zones
2000: BSD jails
1982: chroot (tech based on)
Containerisation

@annashipman
You can run
multiple containers
Containerisation

@annashipman
Containers are isolated from
each other
Containerisation

@annashipman
Why would you use
containers?
Containerisation

@annashipman
Configuring a VM can be
slow
Containerisation

@annashipman
You might instead decide to
take a snapshot
Containerisation

@annashipman
And that is what you deploy
to production
Containerisation

@annashipman
Containers are immutable
Containerisation

@annashipman
So you can be sure what
you’ve tested is what is
running in production
Containerisation

@annashipman
With all the
same dependencies 1. Put netting on garlic and put back outside
Containerisation

@annashipman
However…
Containerisation

@annashipman
The immutability can make
developing locally difficult
Containerisation

@annashipman
You can’t run
them everywhere
Containerisation

@annashipman
Possibly a harder
concept to grasp
Containerisation

@annashipman
Right now, it’s not essential
that you understand Docker
unless you are interested
Containerisation

@annashipman
2 + 3: Virtualisation +
Containerisation
Turtles

@annashipman
Locally, it’s Vagrant
or Docker
Turtles

@annashipman
But if you deploy to
the cloud…
Turtles

@annashipman

@annashipman
How do you know where the
problem is?
Turtles

@annashipman
It’s turtles all the way down
Turtles

@annashipman
BREAK

@annashipman
Wrangling servers
“Cattle not pets”

@annashipman
Virtualisation
Use Vagrant

@annashipman
Containerisation
You don’t need to learn
Docker right now

@annashipman
4. Some tools to make
you a better developer

@annashipman
“Should all my tools be written in
JavaScript (warning: if the
answer is ‘no’, there might be
a revolution!!!)”
—Remy
Tools

@annashipman
Build systems –
Grunt, Gulp…
Tools

@annashipman
Don’t just work
out of the box
Tools

@annashipman
Dependencies and plug-ins
Tools

@annashipman
You could use NPM
Tools

@annashipman
Make
Tools

@annashipman
Can create tasks to do
anything: clean,
compile, build…
Tools

@annashipman
Advantages of Make
Tools

@annashipman
Advantages of Make
Dependency tracking and resolution
Tools

@annashipman
Advantages of Make
Dependency tracking and resolution
Only does what it needs to
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
Compiles your SCSS to CSS
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
This is what Grunt, Gulp etc are doing
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
$ make styles.css
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
Only runs if SCSS is newer than CSS
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
$ make print.css
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
$ make all
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
$ make all
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
$ make
Tools

@annashipman
all: styles.css print.css
styles.css: styles.scss
sass styles.scss:styles.css
print.css: print.scss
sass print.scss:print.css
If print.css doesn’t need updating, only
runs styles.css
Tools

@annashipman
Advantages of Make
Dependency tracking and resolution
Only does what it needs to
Tools

@annashipman
Advantages of Make
Dependency tracking and resolution
Only does what it needs to
Included in your OS*
*Linux, Unix, MacOS
Tools

@annashipman
Has everything you need
Tools

@annashipman
It just isn’t written in JS
Tools

@annashipman
Unix tools
Tools

@annashipman
grep
Tools

@annashipman
grep
Search given input files for patterns
Tools

@annashipman
cat
Tools

@annashipman
cat
Concatenate and print files
Tools

@annashipman
awk
Tools

@annashipman
awk
Scan input files for patterns; perform action
Tools

@annashipman
And the most useful:
man
Tools

@annashipman
man
format and display manual pages
Tools

@annashipman
$ man grep
Tools

@annashipman
$ man man
Tools

@annashipman
Being unixy
Tools

@annashipman
Each tool does one thing
very well
Tools

@annashipman
Composable
Tools

@annashipman
|
Tools

@annashipman
| (Pipe)
Tools

@annashipman
Read a file of text, determine the n most
frequently used words, and print out a
sorted list of those words along with their
frequencies.
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
1 tr -cs A-Za-z 'n' |
2 tr A-Z a-z |
3 sort |
4 uniq -c |
5 sort -rn |
6 sed ${1}q
Tools

@annashipman
Read a file of text, determine the n most
frequently used words, and print out a
sorted list of those words along with their
frequencies.
Tools

@annashipman
You might not want to write
all your programs in Unix
Tools

@annashipman
It can be really useful for
doing complex tasks
Tools

@annashipman
It’s worth getting familiar
with Unix tools locally
Tools

@annashipman
So that the first time isn’t
when you’re debugging on
the server
Tools

@annashipman
$ rake -T | grep "some detail"
Tools

@annashipman
Read the output
Tools

@annashipman
Git instructions
Tools

@annashipman
$ git add accidental_new_file.txt
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: accidental_new_file.txt
Tools

@annashipman
$ git add accidental_new_file.txt
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: accidental_new_file.txt
Tools

@annashipman
Unix error messages
Tools

@annashipman
$ grep -r anna
Tools

@annashipman
$ grep -r anna
grep: warning: recursive search of stdin
Tools

@annashipman
$ grep -r anna .
Tools

@annashipman
Unix tools can help you
Tools

@annashipman
The main three things you
should take home:

@annashipman
Use Vagrant

@annashipman
Unix tools are your friends

@annashipman
Read the output

Thank you!
Anna Shipman
@annashipman

Configuration management
https://www.scriptrock.com/articles/the-7-configuration-management-tools-you-need-to-know
http://gettingstartedwithchef.com/first-steps-with-chef.html
https://docs.vagrantup.com/v2/getting-started/provisioning.html
Virtualisation
http://searchvirtualdatacentre.techtarget.co.uk/definition/Virtualisation
http://searchservervirtualization.techtarget.com/definition/server-virtualization
http://www.infoworld.com/article/2621446/server-virtualization/server-virtualization-top-10-benefits-of-server-
virtualization.html
Using Vagrant
https://www.vagrantup.com/
http://blog.bennycornelissen.nl/otto-a-modern-developers-new-best-friend/
https://github.com/patrickdlee/vagrant-examples (Useful getting started examples)
Docker
http://patg.net/containers,virtualization,docker/2014/06/05/docker-intro/
https://zeltser.com/security-risks-and-benefits-of-docker-application/
Containerisation vs Virtualisation
http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production
https://www.scriptrock.com/articles/docker-vs-vagrant
Make instead of Grunt/Gulp
http://blog.keithcirkel.co.uk/why-we-should-stop-using-grunt/
https://www.youtube.com/watch?v=0RYETb9YVrk (Talk on using NPM as a build tool)
https://blog.jcoglan.com/2014/02/05/building-javascript-projects-with-make/
Tools for better dev
http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/ (More detail on the 6-line Unix program)