Your SlideShare is downloading. ×

Storm at Spotify

6,229

Published on

Slides for the NYC Storm user group meetup @spotify, Mar 25, 2014

Slides for the NYC Storm user group meetup @spotify, Mar 25, 2014

Published in: Technology
1 Comment
22 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,229
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
114
Comments
1
Likes
22
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. March 25, 2014 Neville Li neville@spotify.com @sinisa_lyh Storm at Spotify
  • 2. •@Spotify since 2011 •Recommendation Team •Data & Backend •Storm, Scalding, Spark, Scala… About Me
  • 3. March 25, 2014 Spotify in numbers Started in 2006, available in 55 markets 20+ million songs, 20,000 added per day 24+ million active users, 6+ million subscribers 1.5 billion playlists !
  • 4. Big Data @spotify 600 node cluster Every day •400GB service logs •4.5TB user data •5,000 Hadoop jobs •61TB generated
  • 5. March 25, 2014 What is Storm? In data-layman’s terms • Real time stream processing • Like Hadoop without HDFS • Like Map/Reduce with many reducer steps • Fault tolerant & guaranteed message processing Photo © Blaine Courts http://www.flickr.com/photos/blainecourts/8417266909/
  • 6. Storm @spotify •storm-0.8.0 •22 node cluster •15+ topologies •200,000+ tuples per second •recommendation, ads, monitoring, analytics, etc.
  • 7. “Never Gonna Give You Up” Rick Astley Map ! First Storm Application @Spotify 7
  • 8. RT Market Launch Stats
  • 9. Other Uses •Trending tracks •Email campaign •App performance tracking •UX tracking
  • 10. Anatomy of A Storm Topology From play to recommendation
  • 11. Social Listening Take 1 •PUB/SUB •Almost real-time •Spammy •Hard to scale All characters appearing in this work are fictitious. Any resemblance to real persons, living or dead, is purely coincidental. this

×