The document outlines Shopify's journey to implement Docker for their production environment, detailing their timeline from initial investigations in 2013 to a successful containerization strategy by 2014. It explains the stages of preparation, containerization, and feature development, including image builds and deployment processes. Challenges faced and solutions found during this transition are discussed, emphasizing the shift from monolithic architecture to a container-based approach.

1. Docker at Shopify
From This-Looks-Fun to Production

2. Simon Eskildsen
Infrastructure @ Shopify
@Sirupsen

3. Shopiscale
• 200-300K RPM during day, > 500K during large sales
• ~100 large metal servers running containers
• 100+ product developers, ~10 infrastructure, ~10 core ops
• 10 year old monolith
• Ruby on Rails, MySQL, Redis, memcached, ..

4. Make it easier to
solve problems at
Shopify

5. Container
-------
Platform

6. Timeline to production
• Mid 2013. Investigated Mesos and Docker
• Winter 2014 Legwork for containerization, failed attempt to
deploy Mesos
• Spring 2014 Containerization work and no success with
CoreOS either
• Summer 2014 Pivot to Chef, runit and Capistrano. !'ed it

7. Ship Docker as the only
change

8. Successfully shipping Docker
1. Play
2. Prepare
3. Containerize
4. Feature

9. 1. Play stage
• Develop feel for Docker
• Study namespaces, companies, cgroups, ..
• Prototyping
• Don't get stuck or starstruck

10. 2. Preparation stage
• Secrets (github.com/Shopify/ejson)
• Assets
• Logs
• Graceful shutdown
• Monitoring
• Roles/Procfile
• Containerization patterns

11. 3. Containerization stage
• Don't add new features to your infrastructure
• Building images/CI/dev
• Deployment
• Static orchestration
• Union filesystems
• init in container, Linux namespaces, cgroups, routing,
monitoring, security updates, registry..

12. 4. Feature stage !
• Buttons
• Distributed orchestration
• Service discovery
• Routing
• CoreOS
• PaaS
• ..

13. Verschlimmbesserung

14. What we took to
production

15. Image Builds

16. Image builder
1. Commit <sha> lands in master
2. Build server builds image for new SHA (~2m)
3. Pushed to internal registry: registry.shopify.com/
shopify:<sha>

17. We don't use Dockerfiles

18. Builder API
• script/prepare System-level packages or other infrequent
changes on top of base image.
• script/build Frequently changed dependencies, run on
every build.
• config/roles.json Roles for the application, our
Procfile.

19. Final image for Shopify
Ubuntu
----------------
Baseimage (750 MiB)
================
`script/prepare` (100MiB)
----------------
`script/build`: Start of week (800MiB)
----------------
`script/build`: Start of day (100MiB)
-----------------
`script/build`: Current commit (80MiB)

20. container "shopify" do
role :web
instances 5
health_check true
http_health_check "/ping/ponies"
revision "file"
env({
RAILS_ENV: "production",
UNICORN_WORKERS: node[:cores] / 5
})
end
Turns into supervisor scripts:
docker run -e CONTAINER=1 -e HOST_IP=172.16.1.227
-e ENV=production -e PORT=20003
-e APP_HEALTH_CHECK=/ping/ponies
-e ENDPOINT=172.16.1.227:20003 -e UNICORN_WORKERS=5
-e NODENAME=borg-shopify-unicorn-4.omgweb.dc.shopify.com
--volume=/tmp/borg-run/keys/11607:/key
--cidfile=/tmp/borg-run/keys/11607/cidfile
--name="borg-shopify-unicorn-4-$$"
--net=host internal.registry.com/shopify:$(cat /etc/borg/shopify/current) web

21. Deploys
1. Deploy two stacks at once.
2. Capistrano updates revision in /etc/shopify/revision.
3. ls -d /etc/sv/shopify-web* | xargs -L1 sudo sv
restart
• Restart ~20% of containers at a time.

22. Containers are the future,
but it's still hard in
production

24. Simon Eskildsen, Infrastructure @ Shopify
@Sirupsen

25. Thank You.

26. Staircase Tutorial, Andreas Leversa (https://creativecommons.org/licenses/by/2.0/)
Sandbox, Kate Mereand-Sinha (https://creativecommons.org/licenses/by/2.0/)
Panama canal construction photo, World Bank Photo Collection (https://creativecommons.org/licenses/by-nc-nd/2.0/)
Other pictures used by permissions of Florian Weingarten
Flight of the Unicorns, Eric Wagner (https://creativecommons.org/licenses/by-nc-sa/2.0/)
Container, Izabela Reimers (https://creativecommons.org/licenses/by-nc-sa/2.0/)
Sam Satch, Chinese Ornamental Chilli Sprouting (https://creativecommons.org/licenses/by-nd/2.0/)
Anvil & Hammer, Brad Knabel (https://creativecommons.org/licenses/by-nc-nd/2.0/)
Matt Kowal, DIY Darkroom - Alternative Process (https://creativecommons.org/licenses/by-sa/2.0/)
Over-stuffed Walrus picture kindly provided with permission by the Horniman Museum and Gardens, U.K.
Video of me in walrus costume, Lily Rogers and Emma Craig