Riak at ideeli

418 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
418
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
3
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Riak at ideeli

  1. 1. riak@ideeli Aaron Brown, Lead Systems Engineer abrown@ideeli.com @aaronbbrown777 Thursday, February 14, 13
  2. 2. What is an ideeli? Thursday, February 14, 13
  3. 3. Flash Sales http://www.flickr.com/photos/moonlightbulb/4064060773/ Thursday, February 14, 13
  4. 4. Flash Sales Thursday, February 14, 13
  5. 5. Flash Sales Thursday, February 14, 13
  6. 6. Flash Sales Thursday, February 14, 13
  7. 7. Flash Sales Thursday, February 14, 13
  8. 8. Obligatory Stats • 2,000,000 visitors/week • 80,000 front end requests/minute • 120 production servers • 300 deploys a year • 40ms front end response time Thursday, February 14, 13
  9. 9. Obligatory Riak Stats • 1.2MM keys • 4 virtualized nodes • 4ms response • 40,000 GETs/min • 2,000 PUTs/min Thursday, February 14, 13
  10. 10. The Spike Thursday, February 14, 13
  11. 11. The Stack Thursday, February 14, 13
  12. 12. New Toys Are Fun (9AM - 5PM) Thursday, February 14, 13
  13. 13. What Counts 1AM - 7AM Thursday, February 14, 13
  14. 14. K.I.S.S. Keep It Simple Stupid Thursday, February 14, 13
  15. 15. K.I.S.S. Keep It Simple Stupid Kan It Sell Shoes? Thursday, February 14, 13
  16. 16. 3 Uses for Riak • Cache Warming Thursday, February 14, 13
  17. 17. 3 Uses for Riak • Cache Warming • Feature Testing (LABS) Thursday, February 14, 13
  18. 18. 3 Uses for Riak • Cache Warming • Feature Testing (LABS) • User Favorites Thursday, February 14, 13
  19. 19. Cache Warming Thursday, February 14, 13
  20. 20. The Spike Thursday, February 14, 13
  21. 21. Cache Warming Pseudo-code: every.30.minutes do   all_urls.each do |url|     curl "#{url}?force_cache=true"   end end Thursday, February 14, 13
  22. 22. Cache Warming Thursday, February 14, 13
  23. 23. Response times Thursday, February 14, 13
  24. 24. History Thursday, February 14, 13
  25. 25. memcached Thursday, February 14, 13
  26. 26. memcached ~40ms Thursday, February 14, 13
  27. 27. memcached 5/6 requests ~ 40ms 1/6 requests ~ 3000ms = ~500ms average Thursday, February 14, 13
  28. 28. Domino Effect http://www.flickr.com/photos/soham_pablo/309379628/ Thursday, February 14, 13
  29. 29. Manual Intervention Thursday, February 14, 13
  30. 30. membase (couchbase) Thursday, February 14, 13
  31. 31. Riak All requests ~40ms Thursday, February 14, 13
  32. 32. Riak All requests ~40ms Thursday, February 14, 13
  33. 33. LABS • Feature Testing Framework • Key based on user id • ValueYAML string w/ list of experiments Thursday, February 14, 13
  34. 34. Favorites Thursday, February 14, 13
  35. 35. Favorites Thursday, February 14, 13
  36. 36. Favorites • Partially normalized • user_#{id} → array of products product_#{id} → metadata about product Thursday, February 14, 13
  37. 37. Rails Integration • Modified riak-client gem for Rails 2.3 • Load balancing built in • Multiple primary and secondary servers • Custom ActiveSupport::Cache store Thursday, February 14, 13
  38. 38. Rails Integration module ActiveSupport   module Cache     class RailsStore < Store       def read       end       def write       end       def delete       end   end end config.after_initialize do   ActionController::Base.cache_store = Rails.rails_store.fragments end Thursday, February 14, 13
  39. 39. Monitoring Thursday, February 14, 13
  40. 40. Graphite Thursday, February 14, 13
  41. 41. New Relic Thursday, February 14, 13
  42. 42. Alerting • Compute random string • PUT value into Riak • GET value out of Riak • alert if value_in != value_out • alert if operation took too long Thursday, February 14, 13
  43. 43. Tuning Thursday, February 14, 13
  44. 44. Bitcask Merges Thursday, February 14, 13
  45. 45. Disk Usage Thursday, February 14, 13
  46. 46. Heavy I/O Thursday, February 14, 13
  47. 47. expiry_grace_time Thursday, February 14, 13
  48. 48. Offset Merge Windows Thursday, February 14, 13
  49. 49. Puppet class ideeli_riak ( $merge_window_start, $merge_window_end ) { ... file { '/etc/riak/app.config' : content => template("ideeli_riak/app.config.erb"), } ... } class riak-node { $merge_window_start = inline_template( ‘<%= (scope.function_fqdn_rand([6]).to_i + 20) % 23 %>' ) class { 'ideeli_riak' : merge_window_start => $merge_window_start, merge_window_end => 6 } } Thursday, February 14, 13
  50. 50. Riak Loves Disk Thursday, February 14, 13
  51. 51. New Disk Array Thursday, February 14, 13
  52. 52. New Disk Array Thursday, February 14, 13
  53. 53. Improvement! Thursday, February 14, 13
  54. 54. We’re Hiring http://ideelicareers.com/ Thursday, February 14, 13
  55. 55. Questions? Thursday, February 14, 13

×