riak@ideeli
Aaron Brown, Lead Systems Engineer
abrown@ideeli.com
@aaronbbrown777
Thursday, February 14, 13
What is an ideeli?
Thursday, February 14, 13
Flash Sales
http://www.flickr.com/photos/moonlightbulb/4064060773/
Thursday, February 14, 13
Flash Sales
Thursday, February 14, 13
Flash Sales
Thursday, February 14, 13
Flash Sales
Thursday, February 14, 13
Flash Sales
Thursday, February 14, 13
Obligatory Stats
• 2,000,000 visitors/week
• 80,000 front end requests/minute
• 120 production servers
• 300 deploys a yea...
Obligatory Riak Stats
• 1.2MM keys
• 4 virtualized nodes
• 4ms response
• 40,000 GETs/min
• 2,000 PUTs/min
Thursday, Febru...
The Spike
Thursday, February 14, 13
The Stack
Thursday, February 14, 13
New Toys Are Fun
(9AM - 5PM)
Thursday, February 14, 13
What Counts
1AM - 7AM
Thursday, February 14, 13
K.I.S.S.
Keep It Simple Stupid
Thursday, February 14, 13
K.I.S.S.
Keep It Simple Stupid
Kan It Sell Shoes?
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
• Feature Testing (LABS)
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
• Feature Testing (LABS)
• User Favorites
Thursday, February 14, 13
Cache Warming
Thursday, February 14, 13
The Spike
Thursday, February 14, 13
Cache Warming
Pseudo-code:
every.30.minutes do
  all_urls.each do |url|
    curl "#{url}?force_cache=true"
  end
end
Thurs...
Cache Warming
Thursday, February 14, 13
Response times
Thursday, February 14, 13
History
Thursday, February 14, 13
memcached
Thursday, February 14, 13
memcached
~40ms
Thursday, February 14, 13
memcached
5/6 requests ~ 40ms
1/6 requests ~ 3000ms
=
~500ms average
Thursday, February 14, 13
Domino Effect
http://www.flickr.com/photos/soham_pablo/309379628/
Thursday, February 14, 13
Manual Intervention
Thursday, February 14, 13
membase
(couchbase)
Thursday, February 14, 13
Riak
All requests ~40ms
Thursday, February 14, 13
Riak
All requests ~40ms
Thursday, February 14, 13
LABS
• Feature Testing Framework
• Key based on user id
• ValueYAML string w/ list of experiments
Thursday, February 14, 13
Favorites
Thursday, February 14, 13
Favorites
Thursday, February 14, 13
Favorites
• Partially normalized
• user_#{id} → array of products
product_#{id} → metadata about product
Thursday, Februar...
Rails Integration
• Modified riak-client gem for Rails 2.3
• Load balancing built in
• Multiple primary and secondary serve...
Rails Integration
module ActiveSupport
  module Cache
    class RailsStore < Store
      def read
      end
      def writ...
Monitoring
Thursday, February 14, 13
Graphite
Thursday, February 14, 13
New Relic
Thursday, February 14, 13
Alerting
• Compute random string
• PUT value into Riak
• GET value out of Riak
• alert if value_in != value_out
• alert if...
Tuning
Thursday, February 14, 13
Bitcask Merges
Thursday, February 14, 13
Disk Usage
Thursday, February 14, 13
Heavy I/O
Thursday, February 14, 13
expiry_grace_time
Thursday, February 14, 13
Offset Merge Windows
Thursday, February 14, 13
Puppet
class ideeli_riak ( $merge_window_start, $merge_window_end ) {
...
file { '/etc/riak/app.config' :
content => templ...
Riak Loves Disk
Thursday, February 14, 13
New Disk Array
Thursday, February 14, 13
New Disk Array
Thursday, February 14, 13
Improvement!
Thursday, February 14, 13
We’re Hiring
http://ideelicareers.com/
Thursday, February 14, 13
Questions?
Thursday, February 14, 13
Upcoming SlideShare
Loading in …5
×

Riak at ideeli

367 views
281 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
367
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Riak at ideeli

  1. 1. riak@ideeli Aaron Brown, Lead Systems Engineer abrown@ideeli.com @aaronbbrown777 Thursday, February 14, 13
  2. 2. What is an ideeli? Thursday, February 14, 13
  3. 3. Flash Sales http://www.flickr.com/photos/moonlightbulb/4064060773/ Thursday, February 14, 13
  4. 4. Flash Sales Thursday, February 14, 13
  5. 5. Flash Sales Thursday, February 14, 13
  6. 6. Flash Sales Thursday, February 14, 13
  7. 7. Flash Sales Thursday, February 14, 13
  8. 8. Obligatory Stats • 2,000,000 visitors/week • 80,000 front end requests/minute • 120 production servers • 300 deploys a year • 40ms front end response time Thursday, February 14, 13
  9. 9. Obligatory Riak Stats • 1.2MM keys • 4 virtualized nodes • 4ms response • 40,000 GETs/min • 2,000 PUTs/min Thursday, February 14, 13
  10. 10. The Spike Thursday, February 14, 13
  11. 11. The Stack Thursday, February 14, 13
  12. 12. New Toys Are Fun (9AM - 5PM) Thursday, February 14, 13
  13. 13. What Counts 1AM - 7AM Thursday, February 14, 13
  14. 14. K.I.S.S. Keep It Simple Stupid Thursday, February 14, 13
  15. 15. K.I.S.S. Keep It Simple Stupid Kan It Sell Shoes? Thursday, February 14, 13
  16. 16. 3 Uses for Riak • Cache Warming Thursday, February 14, 13
  17. 17. 3 Uses for Riak • Cache Warming • Feature Testing (LABS) Thursday, February 14, 13
  18. 18. 3 Uses for Riak • Cache Warming • Feature Testing (LABS) • User Favorites Thursday, February 14, 13
  19. 19. Cache Warming Thursday, February 14, 13
  20. 20. The Spike Thursday, February 14, 13
  21. 21. Cache Warming Pseudo-code: every.30.minutes do   all_urls.each do |url|     curl "#{url}?force_cache=true"   end end Thursday, February 14, 13
  22. 22. Cache Warming Thursday, February 14, 13
  23. 23. Response times Thursday, February 14, 13
  24. 24. History Thursday, February 14, 13
  25. 25. memcached Thursday, February 14, 13
  26. 26. memcached ~40ms Thursday, February 14, 13
  27. 27. memcached 5/6 requests ~ 40ms 1/6 requests ~ 3000ms = ~500ms average Thursday, February 14, 13
  28. 28. Domino Effect http://www.flickr.com/photos/soham_pablo/309379628/ Thursday, February 14, 13
  29. 29. Manual Intervention Thursday, February 14, 13
  30. 30. membase (couchbase) Thursday, February 14, 13
  31. 31. Riak All requests ~40ms Thursday, February 14, 13
  32. 32. Riak All requests ~40ms Thursday, February 14, 13
  33. 33. LABS • Feature Testing Framework • Key based on user id • ValueYAML string w/ list of experiments Thursday, February 14, 13
  34. 34. Favorites Thursday, February 14, 13
  35. 35. Favorites Thursday, February 14, 13
  36. 36. Favorites • Partially normalized • user_#{id} → array of products product_#{id} → metadata about product Thursday, February 14, 13
  37. 37. Rails Integration • Modified riak-client gem for Rails 2.3 • Load balancing built in • Multiple primary and secondary servers • Custom ActiveSupport::Cache store Thursday, February 14, 13
  38. 38. Rails Integration module ActiveSupport   module Cache     class RailsStore < Store       def read       end       def write       end       def delete       end   end end config.after_initialize do   ActionController::Base.cache_store = Rails.rails_store.fragments end Thursday, February 14, 13
  39. 39. Monitoring Thursday, February 14, 13
  40. 40. Graphite Thursday, February 14, 13
  41. 41. New Relic Thursday, February 14, 13
  42. 42. Alerting • Compute random string • PUT value into Riak • GET value out of Riak • alert if value_in != value_out • alert if operation took too long Thursday, February 14, 13
  43. 43. Tuning Thursday, February 14, 13
  44. 44. Bitcask Merges Thursday, February 14, 13
  45. 45. Disk Usage Thursday, February 14, 13
  46. 46. Heavy I/O Thursday, February 14, 13
  47. 47. expiry_grace_time Thursday, February 14, 13
  48. 48. Offset Merge Windows Thursday, February 14, 13
  49. 49. Puppet class ideeli_riak ( $merge_window_start, $merge_window_end ) { ... file { '/etc/riak/app.config' : content => template("ideeli_riak/app.config.erb"), } ... } class riak-node { $merge_window_start = inline_template( ‘<%= (scope.function_fqdn_rand([6]).to_i + 20) % 23 %>' ) class { 'ideeli_riak' : merge_window_start => $merge_window_start, merge_window_end => 6 } } Thursday, February 14, 13
  50. 50. Riak Loves Disk Thursday, February 14, 13
  51. 51. New Disk Array Thursday, February 14, 13
  52. 52. New Disk Array Thursday, February 14, 13
  53. 53. Improvement! Thursday, February 14, 13
  54. 54. We’re Hiring http://ideelicareers.com/ Thursday, February 14, 13
  55. 55. Questions? Thursday, February 14, 13

×