Spotify: behind the scenes
•
12 likes•4,020 views
Slides from my guest lecture at Uppsala University, class of Distributed Systems, on the 10th of February 2011
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Spotify: behind the scenes
Similar to Spotify: behind the scenes (20)
Recently uploaded
Recently uploaded (20)
Spotify: behind the scenes
- 1. behind the scenes Spotify Ricardo Santos @ricardovice
- 3. Main goals • A big catalogue, tons of music • Available everywhere • Great user experience • More convenient than piracy • Fast • Reliable, high availability • Scalable to many, many users
- 4. Business idea • Free ad-funded version • Paid subscription where users get: • No advertisements • Mobile access • Offline playback • API access
- 5. “music itself is going to become like running water or electricity” David Bowie, 2002
- 6. Accessibility • People should always be able to access music • Whenever they want • Wherever they are
- 10. The catalogue • All content is delivered by labels • Currently over 10 million tracks • Growing every day, around 10k per day • 96-320 kbps audio streams, most are Ogg Vorbis q5, 160kbps
- 11. that all sounds cool, but let’s talk engineering!
- 12. “It’s Easy, Really.” Blaine Cook, 2007
- 13. Handling Growth • Scaling is not an exact science • There is no such thing as a magic formula • Usage patterns differ • There is always a limit to what you can handle • Fail gracefully • Continuous evolution process
- 14. Usage patterns Typically, some services are more demanding than others, this can be due to: • Higher popularity • Higher complexity • Both combined
- 15. Decoupling • Divide and conquer! • Resources assigned individually • Using the right tools to address each problem • Organization and delegation • Problems are isolated • Easier to handle growth
- 16. Decoupling Spotify’s internal services include: • Access Point • User • Playlist • Search • Browse Can you guess which one is the most complex?
- 17. Playlist!
- 18. Playlist! Though it may sound simple, by far the most demanding: • For each user there are several playlists • Push notifications • Offline writing • Conflict resolution without user interaction
- 19. Metadata services Search and Browse allow users to find music • Both handle read requests • But their usage and responses differ • Data sources should be optimized for each of these, called indices • These are hard to maintain, easier to regenerate
- 21. Speed thrills
- 22. Latency matters • High latency is a problem, not only in First Person Shooters • Increased latency of Google searches by 100 – 400ms decreased usage by 0.2 – 0.6% (Jake Brutlag, 2009) • Slow performance is one of the major reasons users abandon services • Users don't come back
- 23. Focus on low latency • Our SLA is maintained by monitoring latency on the client side • On average, the human notion of “instantly” is 200ms • The current median latency to begin to play a track in Spotify is 265ms • Due to disk lookup, at times it's actually faster to start playing a track from network than from disk
- 24. Playing a track • Check local cache • Request first piece from Spotify servers • Meanwhile, search P2P for remainder • Switch between servers P2P as needed • Towards the end of a track, start pre- fetching the next one via P2P rather than our servers
- 25. When to start playing? • Trade off between stutter latency • Look at last 15 min of transfer rates • Model as Markov chain and simulate • Coupled with some heuristics
- 27. Production storage • Production storage is a cache with fast drives lots of RAM • Serves the most popular content • A cache miss will generate a request to master storage, slightly higher latency • Production storage is available in several data centers to ensure closeness to the user (latency wise)
- 28. Master storage • Works as a DHT, with some redundancy • Contains all available tracks but has slower drives and access • Tracks are kept in several formats, adding up to around 290TB
- 30. P2P helps • Easier to scale • Less servers • Less bandwidth • Better uptime • Less costs • Fun!
- 31. P2P overview • Not a piracy network, all tracks are added by Spotify • Used on all desktop clients (no mobile) • Each client connected to = 60 others • All nodes are equals (no super nodes) • A track is downloaded from several peers
- 32. P2P custom protocol • Ask for most urgent pieces first • If a peer is slow, re-request from new peers • When buffers run low, download from central servers • If loading from servers, estimate at what point P2P will catch up • If buffers are very low, stop uploading
- 33. P2P finding peers • Partial central tracker (BitTorrent-style) • Broadcast query in small neighborhood (Gnutella-style) • Two mechanisms results in higher availability • Limited broadcast for local (LAN) peer discovery (cherry on top...)
- 34. P2P security • The P2P network needs to be a safe and trusted one • All exchanged files have to come originally from Spotify • All peers should be trusted Spotify clients
- 35. Security trough obscurity • Our client needs to be able to read metadata and play music • At the same time we have to prevent reverse engineering from doing the same • Therefor, we can't openly discuss the details
- 36. but… • Closed environment • Integrity of downloaded files is checked • Data transfers are encrypted • Usernames are not exposed in P2P network, all peers assigned pseudonym • Software obfuscation, makes life difficult for reverse engineers
- 38. So, what's the outcome? • At over 10 million users the responses are • 55.4% from client cache • 35.8% from the P2P network • 8.8% from the servers
- 40. Oh, and we have cake as well! :D spotify.com/jobs jobs@spotify.com
- 41. I'd like to know more... • Get in touch with us • Checkout Gunnar Kreitz's slides and academic papers on the subject: http://www.csc.kth.se/~gkreitz/spotify-p2p10/