Your SlideShare is downloading. ×
Last.fm vs Xbox
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Last.fm vs Xbox

1,139
views

Published on

Given at DIBI, Newcastle Apr 2010. …

Given at DIBI, Newcastle Apr 2010.

http://dibiconference.com/

Published in: Technology

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,139
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Last.fm vs. Xbox David Singleton last.fm/user/underpangs twitter.com/dsingleton
  • 2. Music discovery powered by Scrobbling Personalised radio, social network, events, a “wikipedia of music” High traffic Monthly visitors: 40 million Monthly page views: 500,000 million
  • 3. Last.fm Architecture Load balancer HTTP Cache Web Server Object Cache Database
  • 4. Xbox Live Platform, millions of users Last.fm Radio App Built by Microsoft Powered by our API Launched along side Facebook & Twitter
  • 5. Last.fm vs. Xbox
  • 6. So, what’s there to talk about? Good Things ™ New users. It’s really cool. Bad Things™ Lots of new users, traffic spikes A very important, high profile, launch How did Last.fm approach this?
  • 7. Xbox Live: 15 million users assuming a 10% take-up rate = 1,500,000 users startup: 5 requests + starting radio: 5 requests + 15 minutes of radio: 60 requests 1 hour of radio = 250 requests per user an hour of radio per user is a rough averaged guess 1,500,000 users = 375,000,000 requests over 24 hours assuming an even distribution = 4,500 requests / second Likely peaking at more than triple = 15,000 requests / second Last.fm: 2,000 requests/sec based on number of servers and apache configuration estimated max capacity of 3,500 requests per second
  • 8. Oh fuck
  • 9. What next? Picked a metric: requests per second Estimated traffic increase vs capacity Selected our goals; Serve requests faster Reduce number requests
  • 10. Profiling traffic Used traffic generated beta testing Web server request logs Common format, widely supported format Hundreds of existing tools We generated some stats using AWK...
  • 11. Which API requests were made? Method 71638 track.getInfo 53941 artist.getImages 15150 radio.getPlaylist 7308 library.getArtists 5020 user.getRecentStations 4979 ads.getVideos 4205 radio.tune 3155 track.love 1507 artist.getInfo 1258 user.getRecommendedArtists 1135 user.getInfo 1130 geo.getTopArtists 1128 radio.gamerStations 1102 tag.getTopArtists 1021 track.ban 1006 user.getLovedTracks 340 library.addArtist 206 auth.getMobileSession
  • 12. Which API requests were made?
  • 13. Raw data from beta Calls Method Total Average 53941 artist.getImages 19647 0.36 71638 track.getInfo 15789 0.22 15150 radio.getPlaylist 6962 0.46 7308 library.getArtists 2402 0.33 4979 ads.getVideos 1810 0.36 5020 user.getRecentStations 1674 0.33 1102 tag.getTopArtists 1488 1.35 1258 user.getRecommendedArtists 1457 1.16 4205 radio.tune 923 0.22 1130 geo.getTopArtists 575 0.51 1507 artist.getInfo 440 0.29 1128 radio.gamerStations 298 0.26 1006 user.getLovedTracks 271 0.27 1135 user.getInfo 171 0.15 206 auth.getMobileSession 38 0.19 136 user.signUp 32 0.24 123 user.terms 16 0.13 3155 track.love 0 0.00
  • 14. How long did each method take?
  • 15. Why so many track.getInfo calls? A tiny UI tweak... ...responsible for 25% of calls. Arrggghhhhhh Added that information to a sensible API call Microsoft kindly updated the app
  • 16. What next?
  • 17. What about the getImages calls? Powers an artist slideshow visualisation Results of this call won’t change often Set a HTTP cache timeout Set caching on a few other calls too
  • 18. Cached Requests 4
  • 19. Request generation Calls Method Total Average 53941 artist.getImages 19647 0.36 71638 track.getInfo 15789 0.22 15150 radio.getPlaylist 6962 0.46 7308 library.getArtists 2402 0.33 4979 ads.getVideos 1810 0.36 5020 user.getRecentStations 1674 0.33 1102 tag.getTopArtists 1488 1.35 1258 user.getRecommendedArtists 1457 1.16 4205 radio.tune 923 0.22 1130 geo.getTopArtists 575 0.51 1507 artist.getInfo 440 0.29 1128 radio.gamerStations 298 0.26 1006 user.getLovedTracks 271 0.27 1135 user.getInfo 171 0.15 206 auth.getMobileSession 38 0.19 136 user.signUp 32 0.24 123 user.terms 16 0.13 3155 track.love 0 0.00
  • 20. kcachegrind http://kcachegrind.sourceforge.net webgrind http://code.google.com/p/webgrind/
  • 21. What happens if things break? Simulated failing calls Highlighted essential calls Acted as a dry-run for launch day failures Informed our backup plans
  • 22. Only essential requests
  • 23. Prepare for the worst Unexpected problems we’ve had: Servers overheating (twice) Hardware (almost) stolen from data-centers Power outage in the office
  • 24. Backup plans, AKA The “Kill List” Plan Effect Severity Disable radio DB- Faster calls Minor backing Disable Flash Player Save 200 req/sec Major Drop non essential Reduce Xbox traffic Extreme Xbox API calls by 0 - 50% Drop X% of radio Reduce Xbox traffic Nuclear tune calls by X%
  • 25. Communication
  • 26. Last.fm: Launch Day (When traffic attacks)
  • 27. How did it go? Our estimate was about 50% over Didn’t exceed capacity (but got quite close) Profiling and caching was essential Or we would have gone down
  • 28. What did we learn? Use timezones to rollout slowly Traffic will follow daily trends Live monitoring is essential Backup plans are comforting Pre-fill caches before launch
  • 29. So, how does this help me?
  • 30. 1. Estimate Choose your metric Estimate launch traffic Compare against capacity Make performance targets Know your limitations
  • 31. 2. Profile requests Start with a sample of traffic Extract data for your metric Visualise the results Identify expensive requests for your metric Use profiling tools on individual requests
  • 32. 3. Optimise Reduce number of requests Set the right HTTP caching headers Combine with reverse web proxy Prime caches for common calls Use an object cache Avoid language level optimisation
  • 33. Web Request Load balancer HTTP Cache Web Server Object Cache Database
  • 34. Web Request Load balancer HTTP Cache Web Server Object Cache Database
  • 35. Web Request Load balancer HTTP Cache Web Server Object Cache Database
  • 36. 4. Plan for failure Simulate failures Know your weak spots Prepare backups plans Communicate with users and partners
  • 37. 5. Launch it! Roll out slowly, if you can Setup live monitoring If something goes wrong; Don’t panic Keep people updated Have some champagne on ice
  • 38. 1. Start with an estimate 2. Profile your traffic 3. Make optimisations 4. Prepare for the worst 5. Launch it!
  • 39. Last.fm vs. Xbox Questions? David Singleton last.fm/user/underpangs twitter.com/dsingleton