Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Docker at Spotify

10,528 views

Published on

Why we're using Docker at Spotify and what we're doing with it.

Published in: Software, Technology, Business
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Docker at Spotify

  1. 1. Docker at Spotify ! Rohan Singh @rohansingh
  2. 2. Spotify “Music for every moment” ! 40+ million active users ! 20+ million songs available (we’re in 55+ countries, differs by market) ! Live in the US for 2 years 2
  3. 3. What are we working with? ! 100+ distinct backend services ! 5000+ production servers in 4 sites worldwide ! Almost 300 servers per ops engineer 3
  4. 4. Deployment at Spotify 4
  5. 5. Build Debian packages ! Requisition new hardware ! Use combination of Puppet and internal tooling (deployify) 5
  6. 6. old way: SSH to a bunch of machines, apt-get install || upgrade ! deployify: Uses fabric, which just SSH’s, runs apt-get install || upgrade 6
  7. 7. What’s wrong with that? 7
  8. 8. Requisitioning new hardware is slow ! Hardware utilization is low ! We’ve automated and improved, but still too slow 8
  9. 9. Deployments often fail… ! …though usually only partially 9
  10. 10. ! Machines get denormalized easily 10
  11. 11. Debian packaging is its own special hell So many damn files! ! debhelper ! Distro & version specific ! apt-get update 11
  12. 12. What do we want? 12
  13. 13. Deployments that are: ! Repeatable ! Straightforward ! Fault-tolerant 13
  14. 14. Why Docker? ! It gives us “repeatable”, right out of the box ! ! And it’s fairly straightforward, too! 14
  15. 15. Repeatability? ! ! The code you test isn’t necessarily the code that runs in prod 15
  16. 16. Repeatability — Docker ! Build an image ! Run tests against the image ! Run the tested image in prod 16
  17. 17. Straightforward ! Just one Dockerfile ! Pretty simple syntax ! Immutable images — easy to reason about 17
  18. 18. Fault-tolerance? Even if your code is perfect, deployments might fail ! Puppet or Debian repos are down ! Network issues ! The machine dies 18
  19. 19. Fault-tolerance — Docker Just one thing has to work — pulling & running an image ! If it fails, just try again ! No machines left in weird states* ! ! *(in theory) 19
  20. 20. What more do we need? 20
  21. 21. Deploying Docker containers across our entire fleet of servers ! Some way to deal with Docker or container failures ! That’s pretty much it 21
  22. 22. Looking around for a Docker platform Several really feature-full offerings ! Pretty opinionated and all-encompassing ! All we need is reliably deploying containers to a fleet of Docker hosts 22
  23. 23. Helios 23
  24. 24. Makes sure your containers are deployed & running exactly where you want them… ! …and that’s pretty much it ! ! github.com/spotify/helios 24
  25. 25. What’s in the box? Helios master frontend that you talk to ! Helios agent runs alongside each Docker instance ! Zookeeper ! Straightforward CLI 25
  26. 26. How does it work? Create a job — tell Helios which Docker image to deploy ! Deploy the job — tell Helios where to run the Docker containers 26
  27. 27. See it in action 27
  28. 28. What’s next? 28
  29. 29. Helios is just the beginning ! Brought up plenty of Helios+Docker agents in our private cloud for our devs ! Integrated Helios with JUnit — write JUnit tests run against Docker containers 29
  30. 30. Dockerized integration tests 30
  31. 31. This week — first Helios+Docker-based service in production ! ! In the future — Containerizing more of our infrastructure 31
  32. 32. Give it a go Open source: github.com/spotify/helios ! Try it out: git clone https://github.com/spotify/helios.git && cd helios && vagrant up ! ! ! Come play with us (spotify.com/jobs) 32

×