SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
3.
User
Segments
3. Bid Request
5. Rocket Fuel
Winning Ad
2. Ad
Request
6. Ad Served
1. Page Request
4. Bid & Ad
Browser
User
Engagements
Publishers
Data Partners
Exchange Partners
Optimize
Simple View of Rocket Fuel
Platform
Real-time Bidder
User
Engagement
User Data
Store
Model
Scoring
34.
It’s a delicate equilibrium that is hard to maintain
Shield it with aggressive alerting, dashboards & canary monitoring
35.
1st region server dies after
several hours of clogged RPC
queue
Bad region moves to another region
server & soon kills it too!
2jmj7l5rSw0yVb_vlWAYkK_Ybwk
stgLVlK_SsLMn4HoG82ymp-QlRtA
Clients can go rouge, it can get as bad as a DoS attack
Protection via dynamic blacklists & size limit filters
37.
» In absence of proxy: ‘The client is part of the cluster’ [1]
» Client must report availability error to calling application thread in short time span
» Follow circuit breaker pattern for read calls (Anecdote)
» ‘pseudo’ puts (local file) for write calls
[1] Blog post from Lars Hofhansl http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html