Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Case studies session 2

2,758 views

Published on

Published in: Software, Technology
  • Be the first to comment

Case studies session 2

  1. 1. Blackbird Billions of rows, couple of milliseconds away Ishan Chhabra Shrijeet Paliwal Abhijit Pol
  2. 2. $0.56 $2.42 1.25 $2.11 $1.26 $2.78 $0.756 $0.809 $2.42 1.25 $2.11 $1.26 $2.78 $1.256 $1.809 $2.42 1.25 $2.11 $1.26 Site/PageGeo/WeatherTime of DayBrand AffinityUser [ + ][ + ]
  3. 3. User Segments 3. Bid Request 5. Rocket Fuel Winning Ad 2. Ad Request 6. Ad Served 1. Page Request 4. Bid & Ad Browser User Engagements Publishers Data Partners Exchange Partners Optimize Simple View of Rocket Fuel Platform Real-time Bidder User Engagement User Data Store Model Scoring
  4. 4. So what is Blackbird?
  5. 5. 5 B 6 B 45 B Facebook likes Searches on Google Bid Requests Considered by Rocketfuel Requests per day
  6. 6. 400 100 20 2 Blink of an eye SF to Tokyo network round trip One beat of a hummindbird's wing Look up in Blackbird Time (ms)
  7. 7. Powered by
  8. 8. HBase, we have a problem..
  9. 9. Object NoSQL Mapper
  10. 10. List<KeyValue>
  11. 11. High Performance Collections
  12. 12. » Data loss on concurrent modification » Read per write » High amount of data per write » O(n)
  13. 13. » Significantly reduced flushes, compaction, network usage, GC. » O(1)
  14. 14. Combined Column: 100 entries c1:combined 1 entry c1:rand1 2 entries c1:rand2 1 entry c1:rand3 Logical Collection Append Only, the HBase view
  15. 15. Optimizing reads using normalization Combined Column: 100 entries c1:combined 1 entry c1:rand1 2 entries c1:rand2 1 entry c1:rand3 Combined Column: 103 entries c1:combined
  16. 16. .filter(⋋), .transform(⋋), ⨍
  17. 17. Secondary Indexes
  18. 18. High Throughput Low Latency Lookups
  19. 19. Not so easy! HBase is designed for high throughput writes
  20. 20. Key Ideas Read as little as possible Stay stable, uniform, data local Don’t go to disk Even if you have to go to disk, make it fast
  21. 21. Protobufs, Protobufs, everywhere
  22. 22. Stay stable, uniform, data local at all times Good quality hardware Properly designed row keys Off peak daily major compaction
  23. 23. Give me all your Cache! 128 Gb machines with 50% block Cache High Cache hit ratio (90% +) by effective utilization
  24. 24. It’s time to disk(o) 15K SAS drives Local & Short circuit reads (20-30% improvement)
  25. 25. High throughput writes are supported too!
  26. 26. Small Writes • Append Only • Protobufs Large Memstores • 4 Gb • Avoids flushes, memory churn, compaction • Maintains read performance by avoiding multiple seeks Tuned Compaction • Avoid Minor compactions • Off Peak Major compaction
  27. 27. Reliability & Availability
  28. 28. Organize the chaos or pay the cost..
  29. 29. » Blind writes can grow rows & table too big » Newbie clients 'guess' a lot » Simple queries such as row count can be hard on the fly Be aware…
  30. 30. Web app Bid Serving Ad servingData augmentation Batch data pipelines Ops Housekeeping Real time data pipelines Multitenant Blackbird
  31. 31. Multi tenancy makes it hard to find the defaulter Use ACLs & client side metrics in all access paths
  32. 32. Draft guidelines for new clients, help them estimate the growth Keep track of growth, row count, row size, column size etc.
  33. 33. Maintaining SLA Guarantees
  34. 34. It’s a delicate equilibrium that is hard to maintain Shield it with aggressive alerting, dashboards & canary monitoring
  35. 35. 1st region server dies after several hours of clogged RPC queue Bad region moves to another region server & soon kills it too! 2jmj7l5rSw0yVb_vlWAYkK_Ybwk stgLVlK_SsLMn4HoG82ymp-QlRtA Clients can go rouge, it can get as bad as a DoS attack Protection via dynamic blacklists & size limit filters
  36. 36. Surviving the failures
  37. 37. » In absence of proxy: ‘The client is part of the cluster’ [1] » Client must report availability error to calling application thread in short time span » Follow circuit breaker pattern for read calls (Anecdote) » ‘pseudo’ puts (local file) for write calls [1] Blog post from Lars Hofhansl http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
  38. 38. Shoutouts!
  39. 39. Obligatory “we are hiring” slide! http://rocketfuel.com/careers ishan@rocketfuel.com shrijeet@rocketfuel.com apol@rocketfuel.com

×