Your SlideShare is downloading. ×
0
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Storm at Spotify
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Storm at Spotify

6,431

Published on

Slides for the NYC Storm user group meetup @spotify, Mar 25, 2014

Slides for the NYC Storm user group meetup @spotify, Mar 25, 2014

Published in: Technology
1 Comment
22 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,431
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
117
Comments
1
Likes
22
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. March 25, 2014 Neville Li neville@spotify.com @sinisa_lyh Storm at Spotify
  • 2. •@Spotify since 2011 •Recommendation Team •Data & Backend •Storm, Scalding, Spark, Scala… About Me
  • 3. March 25, 2014 Spotify in numbers Started in 2006, available in 55 markets 20+ million songs, 20,000 added per day 24+ million active users, 6+ million subscribers 1.5 billion playlists !
  • 4. Big Data @spotify 600 node cluster Every day •400GB service logs •4.5TB user data •5,000 Hadoop jobs •61TB generated
  • 5. March 25, 2014 What is Storm? In data-layman’s terms • Real time stream processing • Like Hadoop without HDFS • Like Map/Reduce with many reducer steps • Fault tolerant & guaranteed message processing Photo © Blaine Courts http://www.flickr.com/photos/blainecourts/8417266909/
  • 6. Storm @spotify •storm-0.8.0 •22 node cluster •15+ topologies •200,000+ tuples per second •recommendation, ads, monitoring, analytics, etc.
  • 7. “Never Gonna Give You Up” Rick Astley Map ! First Storm Application @Spotify 7
  • 8. RT Market Launch Stats
  • 9. Other Uses •Trending tracks •Email campaign •App performance tracking •UX tracking
  • 10. Anatomy of A Storm Topology From play to recommendation
  • 11. Social Listening Take 1 •PUB/SUB •Almost real-time •Spammy •Hard to scale All characters appearing in this work are fictitious. Any resemblance to real persons, living or dead, is purely coincidental. this

×