Slides from a presentation given by Alison Gilles and Josh Baer during StrataNYC 2017. Covers the decision, challenge and strategy (technical, organizational, people) for converting Spotify's 2500 node Hadoop cluster's worth of data and processing to Google Cloud. Finally, touches on Spotify's resulting infrastructure on GCP.

1. An evolution of data infrastructure
Spotify in the Cloud

2. Alison Gilles
Director of Engineering,
Data Infrastructure
Josh Baer
Technical Product Owner,
Data Infrastructure

3. Moving to the Cloud

4. Music Streaming Service
Launched in 2008
Premium and Free Tiers
Available in 60 Countries

5. Over 140M Active Users

6. More than 30M Songs

7. Over 1 billion plays per day

8. We have lots of data

9. • Spotify tribe (department) responsible for data
processing platform
• Organized into 10 squads (engineering teams),
split between Stockholm and NYC
• Squads each own a problem space —
e.g. event delivery, real time processing
Data Infrastructure

10. Data Processing Backend Services

11. The Decision

12. • Spotify was completely on-premise/bare metal
• Previously had experimented with cloud, but
nothing stuck
• By 2014 we had launched in the US, growth was
rapid and we had trouble keeping up
Some history…

13. Owning and operating physical machines is not a
competitive advantage for Spotify.
Why move to the Cloud?

14. • Google’s Big Data toolset is best in class
• Collaborating with Google engineers was a great
fit for us
Why Google?

15. [Imagine a gif of Captain Picard
saying “Make it so!” in this blank
space]

16. The Challenge

17. • ~ 2,500 Nodes (50K CPU Cores)
• > 100 PB Capacity
• > 100 TB Memory accessible by jobs
• 20K Jobs/Day from 2K unique workflows from 100
different teams
Hadoop at Spotify

19. • We can’t move it ourselves
• People hate to be blocked
• Everyone had quite a bit of other work to do
But wait, there’s more!

20. • Everything’s going to change
• All new technology and we’re the experts
• PLEASE no one quit!
And…

21. The Strategy

22. Everybody stop everything

23. Everybody stop everything

24. Keep it simple for Spotify
engineers through tooling

25. • Unblock teams by ensuring they have the data
they need where they expect it
• Last month:
• 80,000 jobs: on-premise -> cloud
• 30,000 jobs: cloud -> on-premise
Copy all the things!

26. Everybody stop everything new

27. Two paths to choose between
And a strategy of not blocking teams

28. The Forklifting Path

29. • We want to treat our machines more like cattle,
less like pets
• Our previous data set-up relied on a very custom
“pet-like” approach
• How could we replicate the function of our
previous setup and adopt our new approach?
Forklifting Path Challenges

30. • A batch job scheduler for Kubernetes
• Containerize your workflows, then define a schedule
• https://github.com/spotify/styx
Styx
Forklifting Path Technology

31. • “Hadoop Cluster as a Service” utilizing Google
Cloud Dataproc and GCS
• Spin up clusters to run workflows or point
workflows at pre-existing clusters
• https://github.com/spotify/spydra
Spydra
Forklifting Path Technology

32. • Forklifting still requires time from teams
• Forklifting means not cleaning up tech debt
Forklifting Path Learnings

33. The Rewrite Path

34. • Update your workflow using
BigQuery or Scio (Dataflow)
• Uses the latest and greatest
• Fully managed
The Rewrite Path

35. • Scala API around Apache Beam
• Currently used by six other
companies
• https://github.com/spotify/scio
Scio
The Rewrite Path

36. It always takes longer than you expect,
even when you take into account
Hofstadter's Law.
Hofstadter's Law

37. The Task Force

38. • Some teams started rewriting immediately
• Some teams forklift themselves
• Others needed a “push”
We need a push
The Task Force

39. • A couple dedicated infrastructure engineers
• A focused 1-2 weeks sprint
• Move ALL THE THINGS
What were they?
The Task Force

40. Teams with a
mission…
What about people stuff?
Life gets even better

41. The Stack

43. BigQuery
• MAU: 25% of total Spotify
employees
• Over 3mm queries +
scheduled jobs per month
Ad-Hoc / Interactive Analysis

44. PubSub
• 1 trillion requests per day
• P99 latencies less than 400 ms
Event Delivery

45. Cloud Dataflow and Dataproc
• 5000 Dataflow jobs run per day
• Most via Scio
• Dataproc spins up Hadoop
clusters per workflows
Big Data Processing

46. • An area of development
• Currently users of Spark,
Tensorflow, Keras, SciKit Learn
• Evaluating ML Engine
Machine Learning

47. • Data quality tooling
• Data management
• User privacy infrastructure
Other future endeavors…

48. Key Takeaways

49. Lesson #1: Know Thy Org
Understand how your organization operates and
tailor your migration path towards that

50. Lesson #2: Embrace Change
Be honest with teams, challenge people

51. Lesson #3: Open Source
Blend of Google primitives and Spotify specific
things. Open source when possible!
spotify.github.io

52. Engineers, Managers, Product
Owners needed in NYC and
Stockholm
https://www.spotifyjobs.com/
We’re Hiring!

53. [email redacted]
@l_phant
Q&A
Alison Gilles
Director of Engineering,
Data Infrastructure
Josh Baer
Technical Product Owner,
Data Infrastructure
[email redacted]
@agilles