Developing Real Time Bidding Solutions with RTBkit Webinar On Demand

Open-Source RTB for Everyone
Developing Real Time
Bidding Solutions with
© Datacratic Inc. 2014 All Rights Reserved.

Overview
● The Project
● The Problems With RTB
o System
o Selection
o Value
● RTB Competitive
Landscape
● How RTBkit Addresses
the Problems with RTB
● Demo

A Little History
● Created by machine-learning and digital
marketing company Datacratic
● Code base evolved from running RTB in
production from 2011-2013
● Open sourced in Feb. 2013, with ongoing
support from Datacratic
● Apache-style governance started Jan. 2014

Participation and Governance
Participation and Governance
● Apache-style governance
o BDNFL - Benevolent
Dictator Not for Life
o Councillors
o Committers
● Outside contributions welcome
● Github pull request workflow --
committers review and merge
● Contributor guidelines
● Users can become
Contributors
● Contributors can become
Committers -- currently two
outside Committers

Support, Community, Adoption
Support and Community
● Free support from the
community and Datacratic
● 230 active developers
● 35 committers, 11 outside of
Datacratic
● Users in 25+ countries
● ~15 installations in prod: N.
America, Germany, France,
Russia, Argentina, China
● Datacratic supports
development, code review,
governance and evolution
● Participation and contributions
from Rubicon Project

Development Support
Development Support
● Getting Started Guide
● Developer Tutorials
● Working integration test system
o mock Exchange and Ad Server
o fixed-price Bidding Agent
● Example Code
● Documentation
● Ubuntu AMI
● Google Group support
● Pull request review and support

The Problems With RTB
SYSTEM VALUESELECTION
Provided by RTBkit Customized by User
General/Technical Specific/Business

Solves the RTB System Problem
SYSTEM SELECTION VALUE
Scale
Speed
Distribution
Reliability

Addresses the RTB Selection Problem
Scale
Speed
Show user an ad?
What ad?
Distribution
Reliability

Addresses the RTB Value Problem
Scale
Speed
Distribution
Show user an ad?
What ad?
What is it worth?
What should I pay?Reliability

RTB Technology Landscape
System Pros Cons Degree of
Difficulty
Exchange /
DSP UI
Easy to get started ● Manual, hard to scale
● Lack of control over bidding
strategy and data
Low
Intermediate
Hosted
Bidding
● More control over bidding
strategy and use of data
● Don't have to do Ops
Strategy and use of data mediated by
vendor and product features
Medium
Roll Your Own
Bidder
Full control of all aspects of the
system
Solely responsible for everything Hardest
● Benefit from core problems
being solved
● Benefit from
● community
● Flexible customization
● Full control (optionally) but
requires digging in
● Responsible for ops
Hard

Addresses the System ProblemHow

Architectural Overview
RTBkit Core
● Router
● Banker
● Post Auction Service
● Service Monitor
● Agent Configuration Service
Plugins
● Exchange Connectors
● AdServer Connector
● Bidding Agent
● Augmenter
● Logger

Bidder Core Advantages
● Core working bidder system
● High-performance real-time components
● Multiple data center support
● Reliable global banker
● Guaranteed response time to exchanges
● High-performance parsing, routing, filtering,
logging and monitoring
● Arbitrary use of custom business logic and
your data

Inventory Integration
● Ships with 9 Exchange Connectors
o Rubicon Project
o AdX
o FBX
o AppNexus
o Nexage
o MoPub
o GumGum
o BidSwitch
● OpenRTB 2.1 compatible

Router Responsibilities
Router Responsibilities
● Gets bid requests from Exchange
Connector
● Uses Filters to filter eligible campaigns
● Passes bid requests through Augmenter
● Passes bid requests to Bidding Agents to
generate bid responses
● Communicates with Banker to guarantee no
overspend
● Guarantees timely response

Router Components
RTBkit
Exchange
Connector
Static
Filters
Augmentation
Loop
Dynamic
Filters
Auction
Loop
Slave
Banker
Master
Banker
Bidding
Agents
Augmenter
Post
Auction
Service
Agent
Config
RTBkitRouter
Exchanges

Router Data Flows
RTBkitRouter
Exchanges
Exchange
Connector
Static
Filters
Augmentation
Loop
Dynamic
Filters
Auction
Loop
Slave
Banker
Master
Banker
Bidding
Agents
Augmenter
Post
Auction
Service
Agent
Config

Bid Request Lifecycle
RTBkit
Router
Exchange
Connector
Static
Filters
Dynamic
Filters
Augmenter
Post
Auction
Service
Ad Server
Conversion
Source
Bidding
Agents
Ad Server
Connector
Exchanges

RTBkit Data Flows
Five asynchronous data
flows flow through the
Router:
● Bid request processing
● Banking updates
● Event Matching
● Notifying Bidding Agents
of Events
● Filtering and Bidding
Agent configuration
RTBkitRouter
Exchange
Connector
Static
Filters
Augmentation
Loop
Dynamic
Filters
Auction
Loop
Slave
Banker
Master
Banker
Bidding
Agents
Augmenter
Post
Auction
Service
Agent
Config
Exchanges

Ad Server and Conversion Integration
● Standard HTTP JSON connector for
receiving Wins, Clicks and Conversions
● Passes all events to PAS for matching
● Logs events and matched events

Post Auction Service
● Clearinghouse for matching all bids to Wins,
Clicks and Conversions
● Current bottleneck, can process events in
the hundreds / sec
● Recently improved: sharded hash tables,
one thread per core
● More improvements on the near roadmap
Post
Auction
Service
Router
Bids
Ad Server
Connector
Events
Shadow
Account
Bidding
Agents
Logger
Matched
Events
Matched
Events
Wins and
Inferred Losses

Banker Responsibilities and Design
Budget Management
● Single source of truth for budget
● Authorizes and caps spending
Designed for Reliability
● Designed for high-latency, low-bandwidth
● Insulates banker state by using "shadow
account" bookkeeping
● Totals always go up -- you can always
reason about the relative timing of entries
● Updates global state once per minute
● Atomic, idempotent persistence
Currency Support
● Strongly typed 64-bit int Currency, won't
allow cross-currency conversions
● Automatic scaling conversions (e.g. CPM to
micro-dollars)

Banker Account Types
● Budget Account
o All budget for Account tree set in
Master Banker Budget Account
o Cannot bid from this account
o Can transfer budget to child Spend
Accounts
o Only exists in the Master Banker
● Spend Account
o Must have a parent Budget Account
o Can bid from this account
o Can track spend directly
o Cannot have children
o Shadowed into separate processes
Budget Account
Spend
Account
Master Banker
Spend
Account

Account Hierarchies and Spend Tracking
● Spend accumulates from Children to Parents
● Temporary bookkeeping for bids happens in
shadow accounts in separate processes
● Shadows sync once per second
o Router shadow - tracks budget
committed to pending bids
o PAS shadow - tracks budget to debit on
Wins and to credit on Losses
Budget
A
Spend
A:B
Master Banker
Spend
A:C
Shadow
Spend
A:B
Router Slave Banker
Shadow
Spend
A:C
Shadow
Spend
A:C
PAS Slave
Banker
Shadow
Spend
A:B

Banker APIs
● REST API suitable for human reader and
outside tool integration
● Also used by Router, Post Auction Service
and Bidding Agents
● API wraps Account Type, Account
Hierarchy and Currency Pool concepts

Monitoring and Operations Tools
● Extensive instrumentation
● Lock-free, high-performance Carbon
logging library, with tunable sampling
rate, one-second granularity and useful
aggregate functions
● Can add any custom metrics
● Operational dashboard included

Monitoring and Operations Tools
● All standard and custom metrics
charted in graphite

Addresses the Selection ProblemHow

Filter Design and Features
● Router passes bid requests through
Filtering pipeline
● Bids must pass all filters to reach Agents
● Thread safe
● Useful primitives for custom filters
● Predefined Agent and Creative filters
● Designed to guarantee performance first,
be flexible and powerful second

Filter Levels
● Agent Filters
o Control whether Agent bids on bid
request
● Creative Filters
o Control whether Creative eligible to be
the one returned in bid response
Agent
Format
Location
Exchange
Language
Creative
Exchange
Location
Language
Host
URL
Segments
Hour of Week
Fold Position
User Partition

Filter Priorities and Performance
Filter Priorities and Performance
● Execution order optimized for performance
● Broad, inexpensive filters run first
● Expensive filters run later (or not at all!)
● Only fast filters on real-time thread
● Static filters support batch processing
● Filter matching tests match and retrieves
eligible creatives in one pass
Format
Location
Exchange
Language
Creative
Agent
Exchange
Location
Language
Host
URL
Segments
Hour of Week
Fold Position
User Partition
Agent
Bid
Request
Bid
RequestBid
RequestBid
Request

Augmenters: Your Logic and Data
Augmenters: Filter Based on Your Logic
and Data
● Bid Requests pass through Augmenter after
Filtering, before Bidding
● Custom filtering based on combinations of
bid request fields, your data, etc.

Augmenter Implementation
● Provides thread pool
● Enforces 5ms timeout
● Sync and async support
● Ships with Redis Augmenter
● Separate config for each Bidding Agent
● Can subscribe to other RTBkit data streams
to write data
o e.g. - frequency cap Augmenter
subscribes to PAS MATCHEDWINs
Router
Bid
Request
Wicked
Fast DB TM
Thread Pool
Augmenter
Impl.
Augmenter
Post
Auction
Service
Data Sink
Callback
Augmented
Bid
Request

Addresses the Value ProblemHow

Bidding Agent Configuration
● Bidding Agents configure the Router, PAS
and Augmenter
o creatives per campaign
o dynamic filters
o augmenters and augmenter filters
o ad markup

Custom Bidding Agent Implementation
● C++ or JavaScript
● Programmatic configuration
● Custom bidding logic based on
o bid attributes
o a custom Win Cost Model to adjust for
desired margin and data costs
● Custom Pacing logic supportedCustom Bidding
Agent
Configuration
Bidding Logic
Win Cost Model
Pacing
Currency Helpers
Router
Bid Request1Bid
Response
2
onWin
onLoss
onNoBudget
onTooLate
onDroppedBid
onInvalidBid
3
Post
Auction
Service
onImpression
onClick
onVisit 4

Augmenters: Your Logic and Data
Augmenters: Modify Bid Requests Based
on Your Logic and Data
● Allows you to augment the bid request,
adding any fields you want
● Augmented fields are then available to the
Bidding Agents

Future Directions
Already Delivered in 2014
● PAS horizontal scalability
Improved Packaging
● HTTP Bidding Agent API
This Year
● “RTBkit 2.0” support for HTTP
plugin APIs for
● Augmenter Data
● Loggers
● Analytic Tools
● Support third-party marketplace
for data, analytics, bid strategy,
etc.

RTBkit Resources
Links
● http://github.com/rtbkit
● http://rtbkit.org
● https://groups.google.com/a/rtbkit.
org/forum/#!forum/discuss
Developers Getting Started Guide
● https://github.com/rtbkit/rtbkit/wiki/
Getting-Started

About Datacratic
• Datacratic is an enterprise software company
that develops machine learning and artificial
intelligence technology, which enables real-
time machine based decisioning to be
deployed into a wide range of applications.
• Datacratic’s products are currently optimized for
Data Management Platforms (DMPs), Demand
Side Platforms (DSPs), Agency Trading Desks
(ATDs), E-Commerce web sites and others in
the real-time marketing ecosystem.
Mark Weiss @marksweiss
● Head of Customer Solutions at Datacratic
www.datacratic.com
opensource.datacratic.com/
We're hiring! datacratic.com/site/careers

You are Invited!
RTBkit Meetup September 4th, London, UK
Who should attend?
Developers or engineers with interested in Ad Technology & Real-Time bidding.
Hosted by:
Presenters:
James Prudhomme, CEO Datacratic & Miguel Sempere, Data Engineer Affectv

Developing Real Time Bidding Solutions with RTBkit Webinar On Demand

Recommended

Recommended

More Related Content

Similar to Developing Real Time Bidding Solutions with RTBkit Webinar On Demand

Similar to Developing Real Time Bidding Solutions with RTBkit Webinar On Demand (20)

Recently uploaded

Recently uploaded (20)

Developing Real Time Bidding Solutions with RTBkit Webinar On Demand