Power Of Zero

The Power Of Zero
Small teams, Lofty dreams, to do nothing?
Andrew Forward
andrew@crossfit.com 
afoward@gmail.com 
@a4word

Automation vs Autonomy
Immutable(ish)
Deployment Pipeline
The key to zero-click is understanding how to script
your actions (automation) and your triggers (autonomy)
Trigger some actions, the results trigger more actions, 
which create more triggers which call more actions.
Context aware deployments so you can use
Erlang's hot code swapping AND rebuild servers from scratch

Thanks to Forgetting Sarah Marshall, for helping teach me to power of doing nothing (http://www.imdb.com/title/tt0800039/)

http://es.memegenerator.net/instance/55689787/futurama-fry-not-quite-sure-if-nothing-is-actually-something

Just because something is free does not
imply there are it has no cost
Managing automation takes resources, but  
the beneﬁts can far outweigh those costs

BATNA!
Best alternative to
no action (kinda)
Do you really
prefer to “click”
Or maybe, maintaining
that 38 step deployment
guide
Strong analogies to the
benefits and challenges of TDD

Automation vs Autonomy
Immutable(ish)
Deployment Pipeline

A set of related functions performed
automatically by equipment. Assumes that
the operator performs any requirements
before and / or after the automated
sequence in order to complete the task.
Automation
A developer kicks off 
a script to create a
new version of the
software
After it completes, she
kicks off a deploy to
production
The scripts are
the related
functions are done
automatically.

A state of equipment in which it can
perform the programmed operations under
deﬁned conditions without human input or
guidance.
Autonomy Instead, delegated to
other automation to
figure that out
It’s still about doing
things automatically
But without humans
making judgement
about when and where

Multiple automated sequences a machine
can perform without human input that
result in a task being completed.
Semi-Autonomous
There are degrees of
autonomy

Path to Zero
Manual
(by hand)
Automation 
(script)
Autonomy 
(meta scripts)
Can’t do the task manually? 
Then how could  
you possibly automate it?
Once you know how, 
then teach the task 
to your computer
Finally, teach your computer
when to run the task

Trigger Action
API postback
ﬁle change
Pull-Request 
merged
compile code
package code
git push
git hook
git merge
run unit tests
test pass
version
change
create database
unit tests
migrate databasecreate server
destroy server
smoke tests
UAT tests
build release
test fails
update DNS
backup database
generate code
time
email received

Ensure file exists
touch /tmp/smoke.passed &&
while inotifywait -q -e open /tmp/smoke.passed; do
echo "Smoke tests passed"
./bin/remote/deploy
done
Watch for changes
Do some action
inotify-wait

while fswatch -1 . > /dev/null; do
mix compile &&
mix test
done
Watch for directory changes in project
Do some action
fswatchmix test.watch
For `mix test`, use this library instead

while true; do
if [[ -e /tmp/smoke.passed ]]; then
echo "Smoke tests passed"
./bin/remote/deploy
rm /tmp/smoke.passed;
sleep 300
fi
done
Does the file exist?
Do some action
Remove the trigger
pollCheck again in 5 minutes

Check every 5 minutes
Do some action
cron
*/5 * * * * /src/nameui/bin/create/backup.sh

config :your_app, YourApp.Scheduler,
jobs: [
# Every minute
{"* * * * *", {Heartbeat, :send, []}},
# Every 15 minutes
{"*/15 * * * *", fn -> System.cmd("rm", ["/tmp/tmp_"]) end},
# Runs every midnight:
{"@daily", {Backup, :backup, []}}
]
Direct integration with your application
Easily dispatch to shell
Easily call other processes
quantum (ex)Cron like syntax with syntax sugar

#!/bin/bash
MESSAGE=$(git log --format=%B -n 1 HEAD)
echo "Take actions based on the message."
post-commit
Stored in .git/hooks, and  
several hooks available
Grab the last message
Take some action

Push Poll
Triggered event delivered 
as soon as it’s observed
inotify reacts as soon 
as a file is changed
The target polls the source
to ask “have you changed”
Setting up a cron job to check
the liveness of a service

Explicit Implicit
The primary source 
triggered the change
You ask your CircleCI build  
if finished successfully
A secondary source allows 
us to infer an event
Your smoke test touches a
“smoke.passed” on success

So what  
should  
you  
automate?

Let’s automate your $5 idea
An idea that you must be willing to allocate at least but ideally not
much more than $5/month

https://mikediamondservices.com/blog/home-plumbing-the-hidden-system-within-your-walls/

Continuous
Integration
Delivery
Deployment
Testing
Monitoring
Integration
Delivery
Deployment
Testing
Monitoring

Pipeline
Deployment Automation
Test Automation
Jenkins
CircleCI Workﬂows
BitBucket Pipeline
JUnit
ExUnit
Orchestration
Build Automation
Checks & Balances
PhantomJS
Feedback Loop
Selenium
Visibility
Monitoring Automation
Code Deploy
Release Management

Event triggered
(pre-conditions met) 
Actions performed
Continue on  
Success
Failures are pushed 
back to user
Artefacts
Stored

Development Production
v1.2.3
Test 
Tag Launch
Prepare 
Release 
Retain
Release 
v1.2.3

v 1 . 5 . 4
}
Major
}
minor
}
patch
v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain

v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain

#!/bin/bash
if [ "$(MIX_QUIET=true mix version.is_release)" != "" ]; then
else
echo "Continue making the app awesome."
fi
v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
.git/hooks/post-commit
Look for vX.Y.Z trigger
Automation triggered
Sprinkle breadcrumbs to 
help “prove” things are setup as expected

TARGET_SYSTEM=$(cat .upstream)
mix test &&
mix version.tag &&
./bin/remote/deploy $TARGET_SYSTEM
v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
Locate the  
production server
Run tests
Tag release
Deploy to target (aka production)

./bin/package/checkout v$VERSION
v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
git fetch &&
git fetch --tags &&
git reset --hard $TAG
Get your code on the server
Checkout the appropriate
tag, branch, or master

mix deps.get &&
mix compile &&
(cd assets &&
npm install &&
brunch build --production) &&
mix phx.digest
./bin/package/prerelease
v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
Update dependencies
Compile
Oh, JavaScript
Prepare all assets 
(images, CSS, JS)

mix release --upgrade
./bin/package/release
v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
Thank you Paul Schoenfelder 
https://hex.pm/packages/distillery

v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
Build
Release
Elixir (Mix) Project Erlang Release

v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
Usage: nameui <task>
Service Control
=======================
start # start nameui as a daemon
foreground # start nameui in the foreground
console # start nameui with a console attached
stop # stop the nameui daemon
Upgrades
=======================
upgrade <version> # upgrade nameui to <version>
downgrade <version> # downgrade nameui to <version>
Many other commands available,
Hot code swaps are totally possible

./bin/package/retain
v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
Storing in DropBox, S3, etc would  
be more scalable (but BitBucket is free!)
Let’s keep our releases 
so we can launch on other servers 
without having to compile

VERSION=$(mix version.current)
APP=nameui
./_build/prod/rel/${APP}/releases/${VERSION}/${APP}.tar.gz
./bin/package/retain
v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
That’s where distillery 
stores the ERTS release

v1.2.3
Test 
Tag
Launch
Prepare 
Release 
Retain
#!/bin/bash
VERISON=$1
APP_BIN=/src/namerel/bin
RUNNING=$($APP_BIN/nameui ping)
if [[ "$RUNNING" == "pong" ]]; then
$APP_BIN/nameui upgrade $VERSION
else
$APP_BIN/nameui start
fi
Am I already running
If yes, upgrade
If no, then start
./bin/run/launch $VERSION

Wraps distillery to
Help with deployment
A newer library also helping
with deployment

Development ProductionBuild
Additional testing 
(Browser based, perf, etc)
Build, Tag, Test,
Prepare, Release,
Launch
“Deploy” the release and launch

Development
Production
Build
“Build” using bootleg
“Deploy” using bootleg

Development Build AND production
“Build” using bootleg
“Deploy” using bootleg

Development
Build ProductionStage

image: elixir:1.5.1
pipelines:
default:
- step:
script:
- mix local.hex --force
- mix deps.get
- mix test
Looks like scripting to me

version: 2.0
jobs:
build:
docker:
- image: circleci/elixir:1.4.2
- image: circleci/postgres:9.4.12-alpine
env:
- POSTGRES_USER=root
working_directory: ~/circleci-demo-elixir-phoenix
steps:
- checkout
- run: mix local.rebar --force
- run: mix local.hex --force
- run: mix deps.get
- run: mix test
Looks familiar

version: 0.0
os: linux
hooks:
ApplicationStop:
- location: scripts/stop.sh
runas: owner
AfterInstall:
- location: scripts/compile.sh
runas: root
ApplicationStart:
- location: scripts/start.sh
runas: owner
ValidateService:
- location: scripts/verify.sh
runas: owner
More generic, but under the hood 
a similar set of scripts required

Even with 3rd party (SAAS) tools you still
need to know what you are automating
Especially for newer technologies like Elixir.

Infrastructure 
as Code
Configuration
Synchronization
Docker
Definition Files
Snowflake Server
Chef
Immutable ServersIdempotent
Images / Snapshots 
AMIs
Vagrant
Habitat
Phoenix Server

Recoverable over Scaleable
Instead of handling all errors,
Make it easy to restore from sratch
(Improve MTTR versus MTTF)
Easy to “turn off”  
to reduce costs
Easy to “upgrade”
And replace (immutable infra)
You are not Facebook
Facebook wasn’t  
originally Facebook
Facebook has 15k 
employeesAnd yeah,
If you can “recover” well,
You can probably scale well 
(when needed)
My $5 idea can’t 
afford a cluster

Recoverable?
Rebuild a production 
server
Rebuild a release Update DNS
Restore (Rebuild) SSL Certs
Restore (Rebuild) Database
Switch cloud  
providers?

Let’s Deploy Some Changes
What change means in the
context of your app.

New Improve
Migrate to 
Alphine OS
Add  
Stripe API
Remove  
Docker
Create Stripe  
Invoice
Add Account 
Summary
New Supervised 
Worker
Install 
Postgres
10
Send an 
Email
First 
Docker Release
Migrate to 
Elixir 1.6
Admin 
Page
Payment Page
Next Docker 
Release
Add umbrella 
App
Improve UX
Hot code  
swap
Fix SQL query
Update  
Email Text
Ubuntu  
16.04
Fix a bug
Update Terms
Adding 
Mailchimp API
Create 
Postgres DB
Start  
application
Add  
Logging

New infrastructure ﬁrst and improving
them second promotes easy upgraded
All infrastructure must eventually be replaced

Zero Down Time
Zero Down Time?
Zero Down Time!

https://signalvnoise.com/posts/1244-deﬁning-the-problem-of-elevator-waiting-times

Think about zero-downtime more as
never 0% down versus alway 100% up
Turning off features
Read-only access
Or even a 
“Be Right Back”

Leave bread crumbs when deploying 
(gives users, and YOU insight into what’s up)
Exposing the deployed version is also very helpful

You can `stop` the full site (but leave it up)

defmodule Nameui.Web.Router do
use Nameui.Web, :router
pipeline :browser do
plug :accepts, ["html"]
plug :fetch_session
plug :fetch_flash
plug :protect_from_forgery
plug :put_secure_browser_headers
plug Nameui.Web.Plugs.BeRightBack
end
end For every request, apply the BeRightBack check

defmodule Nameui.Web.Plugs.BeRightBack do
@moduledoc """
A `Plug` to redirect to the be right back page if
that's the current state of the system
"""
import Plug.Conn
import Phoenix.Controller, only: [redirect: 2]
use FnExpr
alias Nameui.FeatureFlags, as: FF
def init(options), do: options
def call(conn, _opts) do
conn.request_path
|> invoke(cond do
&1 != "/brb" && FF.enabled?(:brb) -> :brb
&1 == "/brb" && FF.disabled?(:brb) -> :index
true -> :asis
end)
|> invoke(case &1 do
:asis -> conn
_ -> redirect_to(conn, &1)
end)
end
def redirect_to(conn, page) do
conn
|> redirect(to: Nameui.Web.Router.Helpers.page_path(conn, page))
|> halt()
end
end
Site wide features using Plug
FeatureFlags as a Key / Value process
If “Be Right Back”, then redirect all traffic 
to an “please wait” page
`mix version.bin.ff`to install

Be Right Back?
• New supervision processes
• Incompatible database changes
• New SSL certiﬁcates
• Deploying a new server
• Mucked up deploy
You could write your own upgrade 
scripts; but probably overkill
Avoidable, no really, totally avoidable 
with some smart rollouts
Unverified (sorry!) but in theory certs 
only loaded once, so a re-start would
be needed
Immutable infrastructure in the small
Most likely, and most embarrassing reason

Just last night, my “build” stopped working 
because Node was consuming too much memory

autonomous in action
MANUAL: 
Installed swap  
space
AUTOMATION: 
fallocate 
mkswap 
swapon
AUTONOMY: 
New image,  
with new swap 
back to GREEN
MANUAL: 
Looked at the
Interwebs to avoid
“Scaling” to a $10 idea
AUTOMATION: 
Codified the
Script
AUTONOMY: 
Trigger a major  
release

Would a server by 
any other name
compute as well?
Let’s build a server

Code
Data
Conﬁg
Tools / OS
nameui (phoenix) 
namedb (ecto)
namedb database
MIX_ENV
POSTGRES_PASSWORD
Ubuntu 16.04 
Elixir 1.5, Erlang 19 
Postgres 9.6

Yes, that’s the one we are using

https://github.com/capbash/bits

$ mix doex
doex v0.7.4
doex is a API client for Digital Ocean's API v2.
Available tasks:
mix doex.delete # Execute a Digital Ocean API DELETE request
mix doex.get # Execute a Digital Ocean API GET request
mix doex.post # Execute a Digital Ocean API POST request
mix doex.put # Execute a Digital Ocean API PUT request
All major REST verbs supported, 
to help future proof API

doex v0.7.4
doex is a API client for Digital Ocean's API v2.
Available tasks:
mix doex.config # Reads, updates or deletes Doex config
mix doex.ls # List your resources.
mix doex.droplets.create # Create a droplet on Digital Ocean
mix doex.imagelets.create # Create a DitigalOcean snapshot based on available templates
mix doex.snapshots.create # Creates a snapshot of an existing Digital Ocean droplet
mix doex.block # Block the command line until a condition is met
mix doex.ip # Get the IP of a droplet
mix doex.scp # Secure copy a file from <src> to your droplet's <target>
mix doex.ssh # Execute a command on your droplet
Config / query your account
Create servers
Manipulate servers
Imagelets is
“my” term

mix doex.imagelets.create phoenix
--region tor1
--size c-2
--image ubuntu-16-04-x64
--erlang 19.3-1
--elixir 1.5.1
--phoenix 1.3.0
--postgres 9.6
--sleep 10
Digital Ocean configs
Elixir configs
“Warm up” time on server

$ mix doex.get /snapshots
{:ok,
%{"links" => %{}, "meta" => %{"total" => 4},
"snapshots" => [%{"created_at" => "2017-04-30T19:01:32Z", "id" => "24531019",
"min_disk_size" => 20,
"name" => "erlang1931elixir141phoenix121postgres96",
"regions" => ["tor1"], "resource_id" => "47461113",
"resource_type" => "droplet", "size_gigabytes" => 2.54},
%{"created_at" => "2017-06-27T15:22:54Z", "id" => "25900884",
%{"created_at" => "2017-08-09T22:54:54Z", "id" => "26938800",
%{"created_at" => "2017-08-10T12:12:52Z", "id" => "26954823",
"resource_type" => "droplet", "size_gigabytes" => 3.13}]}}
Digital Ocean endpoint
Used for scripting

mix doex.droplets.create namedb01
--region tor1
--size 512mb
--ipv6
--private_networking
--snapshot erlang1931elixir151phoenix130postgres96
--sleep 30
--block
Smallest offering from Digital Ocean
Our newly created snapshot
“Warm up” and wait until ready

v 1 . 5 . 4
}
Major
}
minor
}
patch

v 1 . 6 . 0
Server maintenance

v 2 . 0 . 0
Rebuild infrastructure

#!/bin/bash
...
VERSION_NAME=$(mix version.name)
case "$VERSION_NAME" in
"major")
echo "Big changes!”
;;
"minor")
echo "Medium changes!"
;;
"patch")
echo "Small changes!"
;;
esac
Build new server(s) 
and start from scratch
Start / Stop the server 
(shhh, not really zero-downtime)
Hot code update 
Leveraging erlang’s  
out-of-the-box feature

Development V1.x
V2.x
4) Be Right Back
1) Build new server
2) Deploy latest version
3) Do other stuff… 
(Floating IPs, SSL Certs)
The more often you 
do it, to more likely you
will automate it
5) Update Load Balancer /  
Floating IP

It’s over! Now a bit about me.

That’s when I started
with elixir, maybe v0.3
First release in GitHub 
v0.5.0
PR submitted PR merged

http://quotesgram.com/phd-graduation-quotes/
Motivation for doing a PhD
Keen interest  
in subject
Getting to call 
yourself Doctor
Love studying
Unable to ﬁnd a job

Triggers
• https://linux.die.net/man/1/inotifywait
• https://github.com/emcrisostomo/fswatch
• https://github.com/c-rack/quantum-elixir

Config
• http://michal.muskala.eu/2017/07/30/configuring-
elixir-libraries.html
• https://medium.com/nebo-15/runtime-configuration-
migrations-and-deployment-for-elixir-
applications-6295b892fa6a
• https://hex.pm/packages/deferred_config
• https://atom.io/packages/remote-sync
• https://atom.io/

Infrastructure as Code
• https://martinfowler.com/bliki/SnowﬂakeServer.html
• https://info.thoughtworks.com/continuous-delivery-
from-months-to-weeks-ebook.html
• https://github.com/capbash/bits
• https://github.com/capbash/dio
• https://hex.pm/packages/doex

TDD
• https://pragprog.com/screencast/v-kbtdd/test-
driven-development
• https://books.google.ca/books/about/
Test_driven_Development.html?id=gFgnde_vwMAC
• https://medium.com/planet-arkency/testing-is-a-
separate-skill-and-thats-why-you-are-
frustrated-7239b1500a6c
• https://hex.pm/packages/mix_test_watch

Deployment
• http://erlang.org/doc/design_principles/
release_handling.html
• http://erlang.org/doc/apps/erts/
• https://github.com/edeliver/edeliver
• https://github.com/bitwalker/distillery
• https://hex.pm/packages/version_tasks

Deployment
• https://www.youtube.com/watch?v=2FX4Dg5XBWs
• https://hexdocs.pm/distillery/use-with-phoenix.html
• https://blog.polyscribe.io/a-complete-guide-to-
deploying-elixir-phoenix-applications-on-
kubernetes-part-2-docker-and-81e934c3fceb
• http://michal.muskala.eu/2017/07/30/conﬁguring-
elixir-libraries.html

Deployment
• http://crypt.codemancers.com/posts/2016-10-06-
elixir-phoenix-distillery/
• https://medium.com/@pentacent/automating-elixir-
tests-continuous-integration-with-bitbucket-
pipelines-8b432017237a
• https://medium.com/mint-digital/elixir-deployments-
on-aws-ee787aa02a9d

Versionning
• http://semver.org/
• https://www.youtube.com/watch?v=oyLBGkS5ICk
• https://www.linkedin.com/pulse/semantic-
versioning-eduardo-estrella

|>
• https://pragdave.me/blog/2017/05/02/put-this-in-
your-pipe.html
• https://vimeo.com/216107561
• http://benalman.com/news/2010/11/immediately-
invoked-function-expression/
• https://hex.pm/packages/fn_expr

mix gen
• https://pragdave.me/blog/2017/04/18/elixir-project-
generator.html
• https://hex.pm/packages/gen_template_project
• https://hex.pm/packages/gen_template_api_client
• https://hex.pm/packages/
gen_template_ecto_service

SSL (Let’s Encrypt)
• https://medium.com/@a4word/phoenix-app-
secured-with-let-s-encrypt-469ac0995775
• https://github.com/capbash/sslcerts

Naming Things
• https://www.slideshare.net/pirhilton/how-to-name-
things-the-hardest-problem-in-programming

@a4word aforward@uottawa.ca
aforward@gmail.comaforward
andrew@crossﬁt.com@a4word
Andrew Forward

Power Of Zero

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Power Of Zero

Similar to Power Of Zero (20)

Recently uploaded

Recently uploaded (20)

Power Of Zero