Spotify uses Docker and Helios to deploy over 100 backend services across 5000 servers. Docker provides repeatable deployments by running the same tested image in production. Helios ensures Docker containers are deployed and running correctly across servers. Spotify is moving more services to Docker, starting with their first Docker-based service going live that week.

1. Docker at Spotify
!
Rohan Singh
@rohansingh

2. Spotify
“Music for every moment”
!
40+ million active users
!
20+ million songs available
(we’re in 55+ countries, differs by market)
!
Live in the US for 2 years
2

3. What are we working with?
!
100+ distinct backend services
!
5000+ production servers in 4 sites worldwide
!
Almost 300 servers per ops engineer
3

4. Deployment at Spotify
4

5. Build Debian packages
!
Requisition new hardware
!
Use combination of Puppet
and internal tooling (deployify)
5

6. old way:
SSH to a bunch of machines,
apt-get install || upgrade
!
deployify:
Uses fabric, which just SSH’s,
runs apt-get install || upgrade
6

7. What’s wrong with that?
7

8. Requisitioning new hardware is slow
!
Hardware utilization is low
!
We’ve automated and improved,
but still too slow
8

9. Deployments often fail…
!
…though usually only partially
9

10. !
Machines get denormalized easily
10

11. Debian packaging is its own special hell
So many damn files!
!
debhelper
!
Distro & version specific
!
apt-get update
11

12. What do we want?
12

13. Deployments that are:
!
Repeatable
!
Straightforward
!
Fault-tolerant
13

14. Why Docker?
!
It gives us “repeatable”,
right out of the box
!
!
And it’s fairly straightforward, too!
14

15. Repeatability?
!
!
The code you test isn’t necessarily
the code that runs in prod
15

16. Repeatability — Docker
!
Build an image
!
Run tests against the image
!
Run the tested image in prod
16

17. Straightforward
!
Just one Dockerfile
!
Pretty simple syntax
!
Immutable images — easy to reason about
17

18. Fault-tolerance?
Even if your code is perfect,
deployments might fail
!
Puppet or Debian repos are down
!
Network issues
!
The machine dies
18

19. Fault-tolerance — Docker
Just one thing has to work —
pulling & running an image
!
If it fails, just try again
!
No machines left in weird states*
!
!
*(in theory)
19

20. What more do we need?
20

21. Deploying Docker containers across
our entire fleet of servers
!
Some way to deal with Docker or container failures
!
That’s pretty much it
21

22. Looking around for a Docker platform
Several really feature-full offerings
!
Pretty opinionated and all-encompassing
!
All we need is reliably deploying containers
to a fleet of Docker hosts
22

23. Helios
23

24. Makes sure your containers are deployed & running
exactly where you want them…
!
…and that’s pretty much it
!
!
github.com/spotify/helios
24

25. What’s in the box?
Helios master
frontend that you talk to
!
Helios agent
runs alongside each Docker instance
!
Zookeeper
!
Straightforward CLI
25

26. How does it work?
Create a job —
tell Helios which Docker image to deploy
!
Deploy the job —
tell Helios where to run the Docker containers
26

27. See it in action
27

28. What’s next?
28

29. Helios is just the beginning
!
Brought up plenty of Helios+Docker agents
in our private cloud for our devs
!
Integrated Helios with JUnit —
write JUnit tests run against Docker containers
29

30. Dockerized integration tests
30

31. This week —
first Helios+Docker-based service in production
!
!
In the future —
Containerizing more of our infrastructure
31

32. Give it a go
Open source:
github.com/spotify/helios
!
Try it out:
git clone https://github.com/spotify/helios.git
&& cd helios && vagrant up
!
!
!
Come play with us (spotify.com/jobs)
32