How we build containers at Shopify

1. Building
containers
Go-Toronto
Nov/2016
@lxfontes

2. #bfcm

3. Continuous Delivery Pipeline 1-1
BuildGit Push Test Deploy

4. Your company keeps growing

5. Growing

6. Growing MOAR...

8. Clone repository
docker build
- Install RubyGems
- Install NPM Packages
- Compile static assets
- Seal Docker image
docker push
- Upload to Docker Registry
Building a Rails Container

9. Inertia
You get used to it
Your build time only goes up
Number of workers only goes up
“Our build is complex”
“We already use Docker, the Cloud. This is as fast as it gets!”
“It is automated, we don’t care it takes 30 minutes”
“The XYZ team should fix it”

10. it was a reality check
20 minutes per build
Flakiness
Resource Starvation
Really expensive
* where do you put secrets??????

11. Why is it slow?
push
webhook
worker bootstrap
docker build cache not
reused between builds
cold asset precompilation
2nd build

12. peed

13. Where are we: Daily stats
+200
Devs
+700
Builds
+42k
Tests
per build
~3
Images
per build

14. Commitment
Container
Build
Git Push
Automated
Tests
Deploy
5s 90s 200s 300s
From Git Push to Deployed in ~10 minutes
Every developer can deploy to production
Goals:
Testing Rails at
Scale by Emil
How We Deploy
Shopify by Kat
This right here

15. push 30 sec POLL
Pull / Build
Coordinator
+ API
zookeeper
workers
registry
branch affinity

16. ohai bootstrap

17. Clone repository
docker build
- Install RubyGems
- Install NPM Packages
- Compile static assets
- Seal Docker image
docker push
- Upload to Docker Registry
Building a Rails Container

18. Dockerfile

19. mo layers, mo problems
* apparently docker 1.13
can squash these

20. Look ma, no Dockerfiles
prepare
Install OS packages
precompile
Load (ejson) secrets
Populate artifact cache
compile
Combine app + artifact cache
Seal image
Daily seed 20min
Builds 2min
Inherit
Cache
Inherit
Cache

21. Docker Layers
Base
Ubuntu-ish
Prepare
OS packages
Intermediate (Daily seed)
App + Libraries
Final
Delta between Intermediate … Current

22. Locutus trade offs
FAST
Local caches
Secure
Stable
But needs love
We need to maintain:
Infrastructure
Orchestration
Web UI
Scripts Copy & Pasta
Dockerfiles / Buildpacks :(
Troubleshooting :(

23. Pipa
* Kite in Portuguese
(and sounds cool)

24. Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: Requirements
Build whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus

25. Is there an app fo dat?
Orchestration / UI
Docker first
On-premise
Worker fully sandboxed
Concurrency Primitives
“Only 3 builds for app X at same time”
Parallelism
“Send job to 30 workers”

26. Buildkite
On-premise
Orchestration *only*
Github Hooks
UI / Reporting
API
Agent is a single binary (Go)
Also distributed as docker container
Concurrency Primitives
Parallelism
Per Build artifacts
Per Build shared key-value store
Switch workers!!
No batteries included … on purpose

27. Buildkite Pipeline Sample - buildkite/sample-pipelines
Waits for previous step
to complete
Requires Interaction
UI/API/slack
Branch filtering

28. Can it scale?

29. BuildkiteGitHub BK Pipa
Docker
Registry
BK Tests
Webhook Build Container
Switch Worker
Start Tests
Fetch Image
Tests Done
Ship It!
Tests
Upload Image
100 cores 8820 cores
Pipa: Goal

30. Pipa: Architecture goals
Everything exposed via environment variables
PIPA_SSH_KEY / PIPA_APP_COMMIT_ID
Chainable (simplify IF/ELSE ops)
pipa build --unless v1 -- docker build -t myimage:v1 .
Configurable via environment, config file, argument switches
PIPA_APP_NAME=myapp pipa build ...
pipa build --app-name myapp …
Isolated Docker Daemon (docker-in-docker)
Reset after each build
Kubernetes: external SSD / Local: give it a directory

31. Environment / Process Tree
pipa
wrapper
AWS / GCS / SSH Keys
Kubernetes secrets
env vars with paths
APP Name
Buildkite / Jenkins / etc
GIT (local)
APP SHA
Buildkite / Jenkins / etc
GIT (local)
Your code
Signals

32. Prototype: I’m going to write this all in bash

33. Single Binary
Shell files as Assets
using go-bindata
docker / aws-cli / rsync

35. LOL

36. BUT, normalized environment is solid!

37. Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: Requirements
Build whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus

38. Toolbox: spf13/cobra
CLI handling
docker, kubernetes, rkt, etcd
ABUSE: func init()

39. Toolbox: kelseyhightower/envconfig
*spf13/cobra deficiency
Fills struct with env vars

40. Why both?
spf13/cobra
kelseyhightower/envconfig

41. Result
Google Cloud Storage
Amazon S3
JSON Asymmetric Encryption
Shopify/ejson
Registry Interaction
push / pull / exists
mmap diff 2 dirs
burke/treediff
Environment normalizer (entrypoint)

42. Using it
Everything is namespaced!
pipa cache pull /tmp/artifacts
pipa build --if base -- docker build -t $PIPA_IMAGE_FULL_NAME .
pipa image exists --local --tag base
pipa image exists --remote --tag base
Downloads s3://artifacts/app_group/app_env/app_name/cache.tar to /tmp/artifacts
Only runs command if registry/app_group/app_env/app_name:base exists
Checks if registry/app_group/app_env/app_name:base exists (locally/remotely)

43. Pipeline Inference
How should pipa build your app?
Customize it in your repo; or
Select a template; or
Let pipa figure it out!

44. Built-in pipelines

45. Pipeline Output

46. Pipeline
Parallelization
3 parallel builds

47. Pipeline Selection
Custom script in repo
Custom pipeline in repo
Default pipeline in repo
User requested specific pipeline
Check for common file locations
repo/Dockerfile? -> docker
repo/borg? -> borg
Default to Heroku Buildpack
PIPA_PIPELINE_CMD
PIPA_PIPELINE_FILE
PIPA_PIPELINE_TEMPLATE
repo/.buildkite/pipeline.yml

48. High Level
push webhook
Agent poll
SSL
docker
images
cache
artifacts
Queue
API
User Interface

49. ssh-key
gcloud-key
buildkite-token
docker-auth
Sec: Kubernetes Layout
EJSON
priv/pub keypairs
/builder-secrets
/app-secrets
/var/lib/docker (dind)
/builds (git clone)
/cache (artifacts)
6x nodes
24 pods
/mnt/disks/ssd0/builder1/docker

50. Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: Requirements
Build whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus?

51. Gems

53. Tar + Gzip

54. Go + Docker = fsouza/go-dockerclient
Respect environment variables
Reads $HOME/.docker/config.json or $HOME/.dockercfg

55. Rocker - grammarly/rocker
Dockerfile on steroids

56. Herokuish - gliderlabs/herokuish
If you like Heroku, you will want this
Packaged as docker container or single binary

57. alpinelinux.org
based on musl libc and busybox
Lots of *up-to-date* packages (yes debian, looking at you)

58. Demo Time