Distributed Quota Management in an Ad Ad Serving Environment OR How we manage ad spends across data centres without over spending
Distributed Quota Management
in an Ad Serving Environment
How Reduce Data Manages its budgets without exceeding ad spends
Managing Quotas / Budgets in An Ad Serving
Environment is not a trivial task.
Ad Campaign Budgets Across a
Distributed infrastructure when not managed
properly leads to excess spends.
Sync Every 5 seconds
Simple Syncing of data across distributed data structure is easy but sticking
within limits is tough.
5 second refresh not guaranteed to be upto date. Reporting delays caused
spends to exceed.
Server speciﬁc budget was not proportionately allocated, based on historical
availability of trafﬁc on that server.
No clear mechanism of allocation of budget spends or reallocation (when
servers go down).
Our Solution was based on this white paper
Quota enforcement for high-performance
distributed storage systems
Epoc : A day is divided into epocs.
At each epoc DSPServers request for quota from Quota
Ad Server doesn't decide how much quota it needs instead
a new entity the Quota Management Cluster decides how
much quota to assign to it.
Traffic for a Campaign maybe available at multiple
endpoints(US East, US West, Asia)
How Quotas are Decided
Quota is distributed to end points (different data centre locations) and to
the individual servers depending on the historical traffic / consumption
Two factors that decide how much is allocated in each epoc:
Pacing variable as decided by the advertiser
Historical consumption data (for example has the campaign been
spending lesser than planned?, if so, increase EPOC allocation)
Quota Management Cluster
Quota management cluster has multiple servers and needs
a data store which supports transactions.
The Quota Management Cluster is Synchronized using
Sync at every EPOC 2 ways
Ad servers can sync with the Quota management server at EPOC which could
be say every hour.
Ad servers refund unspent balance to Quota Management Cluster.
Quota management can be in one location initially but can be made redundant
to each data centre.
Quota Management Cluster uses a shared data store on cassandra to sync its
other nodes for allocation.
Quota data based
Quota Management Server
Sync at every EPOC
Push budgets, get unspent amounts
Sync allocation data to cassandra cluster
Check Server status
Get list of available ad servers
Push available budgets on a restart.
Calculate available spends
Handling Ad Server Shut Downs
Zookeeper tracks server failures.
Quota management system reallocates remaining budget
(last budget allocated - amount spent) to a new or the same
server when it restarts.
The Quota Management Servers talk to Ad Servers using
REST Calls. This required lowest development time and was
built on top of the existing stack.
Alternatives we considered but decided against.
3. RPC over protobuf
Get in Touch