5. User
Segments
3. Bid Request
5. Rocket Fuel
Winning Ad
2. Ad
Request
6. Ad Served
1. Page Request
4. Bid & Ad
Browser
User
Engagements
Publishers
Data Partners
Exchange Partners
Optimize
Simple View of Rocket Fuel
Platform
Real-time Bidder
User
Engagement
User Data
Store
Model
Scoring
37. It’s a delicate equilibrium that is hard to maintain
Shield it with aggressive alerting, dashboards & canary monitoring
38. 1st region server dies after
several hours of clogged RPC
queue
Bad region moves to another region
server & soon kills it too!
2jmj7l5rSw0yVb_vlWAYkK_Ybwk
stgLVlK_SsLMn4HoG82ymp-QlRtA
Clients can go rouge, it can get as bad as a DoS attack
Protection via dynamic blacklists & size limit filters
40. » In absence of proxy: ‘The client is part of the cluster’ [1]
» Client must report availability error to calling application thread in short time span
» Follow circuit breaker pattern for read calls (Anecdote)
» ‘pseudo’ puts (local file) for write calls
[1] Blog post from Lars Hofhansl http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html