Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Three Years of Microservices at SoundCloud - Distributed Matters Berlin 2015

1,925 views

Published on

SoundCloud is the largest repository of audio on the web, used by more than 200 million people every month, who upload more than 11 hours of audio every minute. Like so many others, we have migrated from a typical monolithic architecture to microservices. While the benefits brought by this style of SOA to our productivity and reliability are clear, the architecture required some non-obvious changes in the way we operate systems, and a way to tackle the overhead associated with having hundreds of small moving parts to serve every request. In this talk we’ll share the toolkit and strategy SoundCloud uses to keep its microservices explosion manageable. What do we do about the operations overhead? How to spread devops skills across teams to support the “you build it, you run it” vision? How to deal with breaking changes and asynchronous behaviours? How to deal with chatty interactions? Which protocol? How do I even get a diagram telling me how all this stuff is put together?

Published in: Software

Three Years of Microservices at SoundCloud - Distributed Matters Berlin 2015

  1. 1. No free lunch, indeed: Three years Phil Calçado SoundCloud of microservices at SoundCloud
  2. 2. No free lunch, indeed: Three years Phil Calçado SoundCloud of microservices at SoundCloud
  3. 3. > 11 hours of audio uploaded every minute ~ 300 million people every month
  4. 4. heaps have been written about microservices in the past year-ish
  5. 5. tl;dr • Rapid provisioning • Basic Monitoring • Rapid application deployment
  6. 6. These make sense • Rapid provisioning • Basic Monitoring • Rapid application deployment this makes sense why does it make me so nervous?
  7. 7. the SoundCloud story you might know
  8. 8. we moved to microservices because $reasons http://bit.ly/how_we_ended_up_with_microservices
  9. 9. the pre-history
  10. 10. SoundCloud, circa 2011
  11. 11. let’s prepare for the “microservices explosion"
  12. 12. #1 provisioning
  13. 13. what was cool in 2010-11
  14. 14. what was cool in 2010-11 doozer lxc 12factor.net
  15. 15. much better than anything else at the time
  16. 16. a problem no resource limits (i.e. cgroups) + naïve scheduling = loud neighbour in your own datacentre
  17. 17. a problem made for most of our services migrated to
  18. 18. the problem time start work on v1 v1 100% deployed start work on v2
  19. 19. attempt #1 - before we go sophisticated, let’s simplify what we have
  20. 20. warmed up pool machine intake
  21. 21. just a bit of glue code
  22. 22. just a bit of glue code a LOT
  23. 23. machine intake
  24. 24. #2 telemetry
  25. 25. state of telemetry tools circa 2011-12 wasn’t great
  26. 26. StatsD
  27. 27. let’s build our own!
  28. 28. but that’s not what broke…
  29. 29. obvious with a monolith monolith
  30. 30. not so much now ? ? ?
  31. 31. standardise dashboards
  32. 32. standardise operations https://twitter.github.io/twitter-server/Features.html#http-admin-interface
  33. 33. visualise
  34. 34. #3 deployment
  35. 35. > git SquashFS > make unit tests integratio n tests acceptanc e tests perf tests > make /dev/null
  36. 36. we ended up with 7 different deployment scripts
  37. 37. > make > gitunit tests integratio n tests unit tests integratio n tests acceptanc e tests perf tests
  38. 38. containers let you spawn your mini- SoundCloud
  39. 39. but why was I so nervous?
  40. 40. because we messed up
  41. 41. there are simple and incremental ways to address these • Rapid provisioning • Basic Monitoring • Rapid application deployment
  42. 42. “wtf? do you think Netflix got it right the first time?” — @adrianco
  43. 43. some of the good things
  44. 44. Q&A

×