RTBkit 2.0 Roadmap Preview

2.0 Roadmap Preview
Open-Source RTB for Everyone
Copyright © 2014 Datacratic Inc. All rights reserved.

Overview
● About the RTBkit Project
● How you can participate
● RTBkit Today – 1.0
● What’s New in 2.0
● 2.0 Roadmap
● Resources

A Little History
• RTBKit was created by machine-learning and digital
marketing company Datacratic
• Code base evolved from running RTB in production
from 2011-present
• Open sourced in Feb. 2013, with ongoing support from
Datacratic
• Apache-style governance started Jan. 2014

Participation and Governance
• Apache-style governance
• BDNFL - Benevolent Dictator Not
for Life
• Councillors
• Committers
• Outside contributions welcome
• Github pull request workflow --
committers review and merge
• Contributor guidelines
• Users can become Contributors
• Contributors can become
Committers -- currently two
outside Committers

Development Support
Development Support
● Getting Started Guide
● Developer Tutorials
● Working integration test system
o mock Exchange and Ad Server
o fixed-price Bidding Agent
● Example Code
● Documentation
● Ubuntu AMI
● Google Group support
● Pull request review and support
© Datacratic Inc. 2014 All Rights Reserved.

Support, Community, Adoption
• Free support from the
community and Datacratic
• 250+ active developers
• 35 committers, 11 outside of
Datacratic
• Users in 25+ countries
• ~25 installations in prod: N.
America, Germany, France,
Russia, Argentina, China,
India
• Datacratic supports
development, code review,
governance and evolution
• Participation and contributions
from Rubicon Project

Addresses the RTB Value Problem
General/Technical Specific/Business
Scale
Speed
SYSTEM SELECTION VALUE
Distribution
Show user an ad?
What ad?
What is it worth?
Reliability What should I pay?
Provided by RTBkit Customized by User

RTB Technology Landscape
System Pros Cons Degree of
Difficulty
Exchange /
DSP UI
Easy to get started ● Manual, hard to scale
● Lack of control over bidding
strategy and data
Low
Intermediate
Hosted
Bidding
● More control over bidding
strategy and use of data
● Don't have to do Ops
Strategy and use of data mediated by
vendor and product features
Medium
● Core systems and
integration problems solved
● Benefit from community
● Flexible customization
● Flexible use of data
● Requires significant development
● Responsible for operations of a
high-volume distributed system
High
Roll Your Own
Bidder
Full control of all aspects of the
system
Solve all the hard systems problems
yourself first, then build the rest of the
system you need.
Highest

1.0 Overview

Architectural Overview
RTBkit Core
● Router
● Banker
● Post Auction Service
● Service Monitor
● Agent Configuration Service
Plugins
● Exchange Connectors
● AdServer Connector
● Bidding Agent
● Augmenter
● Logger

Bidder Core Advantages
● Core working bidder system
● High-performance real-time components
● Multiple data center support
● Reliable global banker
● Guaranteed response time to exchanges
● High-performance parsing, routing, filtering,
logging and monitoring
● Arbitrary use of custom business logic and
your data

Inventory Integration
● Ships with 9 Exchange Connectors
o Rubicon Project
o AdX
o FBX
o AppNexus
o Nexage
o MoPub
o GumGum
o BidSwitch
● OpenRTB 2.1 compatible

Bid Request Lifecycle
RTBkit
Router
Exchange
Connector
Static
Filters
Dynamic
Filters
Augmenter
Post
Auction
Service
Ad Server
Conversion
Source
Bidding
Agents
Ad Server
Connector
Exchanges

RTBkit Data Flows
Five asynchronous data
flows flow through the
Router:
● Bid request processing
● Banking updates
● Event Matching
● Notifying Bidding Agents
of Events
● Filtering and Bidding
Agent configuration
Router RTBkit
Exchange
Connector
Static
Filters
Augmentation
Loop
Dynamic
Filters
Auction
Loop
Slave
Banker
Augmenter
Bidding
Agents
Master
Banker
Agent
Config
Post
Auction
Service
Exchanges

Banker Responsibilities and Design
Budget Management
● Single source of truth for budget
● Authorizes and caps spending
Designed for Reliability
● Designed for high-latency, low-bandwidth
● Insulates banker state by using "shadow
account" bookkeeping
● Totals always go up -- you can always
reason about the relative timing of entries
● Updates global state once per minute
● Atomic, idempotent persistence
Currency Support
● Strongly typed 64-bit int Currency, won't
allow cross-currency conversions
● Automatic scaling conversions (e.g. CPM to
micro-dollars)

Account Hierarchies & Spend Tracking
● Spend accumulates from Children to Parents
● Temporary bookkeeping for bids happens in
shadow accounts in separate processes
● Shadows sync once per second
o Router shadow - tracks budget
committed to pending bids
o PAS shadow - tracks budget to debit on
Wins and to credit on Losses
Master Banker
Budget
A
Spend
A:B
Spend
A:C
Shadow
Spend
A:B
Shadow
Spend
A:C
Router Slave Banker
PAS Slave
Banker
Shadow
Spend
A:C
Shadow
Spend
A:B

Monitoring and Operations Tools
● Extensive instrumentation
● Lock-free, high-performance Carbon
logging library, with tunable sampling
rate, one-second granularity and useful
aggregate functions
● Can add any custom metrics
● Operational dashboard included

Filter Priorities and Performance
Filter Priorities and Performance
● Execution order optimized for performance
● Broad, inexpensive filters run first
● Expensive filters run later (or not at all!)
● Only fast filters on real-time thread
● Static filters support batch processing
● Filter matching tests match and retrieves
eligible creatives in one pass
Creative
Format
Location
Exchange
Language
Agent
Exchange
Location
Language
Host
URL
Agent
Segments
Hour of Week
Fold Position
User Partition
Bid
Request
Bid
RequBeisdt
RequBeisdt
Request

Augmenter Implementation
● Provides thread pool
● Enforces 5ms timeout
● Sync and async support
● Ships with Redis Augmenter
● Separate config for each Bidding Agent
● Can subscribe to other RTBkit data streams
to write data
o e.g. - frequency cap Augmenter
subscribes to PAS MATCHEDWINs
Router
Bid
Request
Wicked
Fast DB TM
Augmented
Thread Pool
Augmenter
Impl.
Augmenter
Post
Auction
Service
Data Sink
Callback
Bid
Request

Custom Bidding Agent Implementation
● C++ or JavaScript
● Programmatic configuration
● Custom bidding logic based on
o bid attributes
o a custom Win Cost Model to adjust for
desired margin and data costs
Router
Bid 1 Bid Request
Response
Custom Bidding ● Custom Pacing logic supported
Agent
Configuration
Bidding Logic
Win Cost Model
Pacing
Currency Helpers
2
3
onWin
onLoss
onNoBudget
onTooLate
onDroppedBid
onInvalidBid
onImpression
onClick
onVisit 4
Post
Auction
Service

2.0 Roadmap Preview

The Big Story in 2.0 … Plugins!
● Why Plugins? We want to support
an ecosystem of revenue
opportunities on RTBkit.
● Design supports this goal
● Plugins identified by URL
● Can have multiple versions
● Support multiple RTBkit versions
● Support multiple RTBkit
installations
● Requests can be authenticated
● Vendors of public plugins can set
up direct billing

Public Plugin Ecosystem
● Major goal of RTBkit 2.0
● RTBkit users can leverage
third-party Plugins
● Public Plugin requirements:
● Multitenant – one plugin can
serve many RTBkit instances
● Multi-location – one logical plugin
can be deployed in multiple
locations
● Authentication and Security
● Per-call metering for billing
● Support multiple RTBkit versions
Plugin
RTBkit
Plugin
Plugin
Plugin
Plugin
Plugin
Plugin
Plugin Plugin
Plugin
Plugin
Plugin
Plugin
Plugin Plugin
RTBkit
RTBkit
Plugin
Plugin

Plugins – Ecosystem Business Use Cases
● RTBkit users benefit and Vendors
benefit:
● Bidder Plugins – campaign
management and bidding logic
● Bid Optimization
● First-party Data in bidding logic
● Per-bid bid price modifier
● First-party Data for attribution
● Third-party Data
● Custom Attribution
● Offline Analytics
● Real-time Analytics
● Real-time Audience Modeling

Private Plugins
● Live in same network as
RTBkit or reachable across
firewall
● Simple Private Plugins
● Single HTTP endpoint responding to
Configuration and Plugin API calls
● Configuration calls hardcoded
● Scalable Private Plugins
● Multiple Plugin Workers behind load
balancer respond to Plugin API calls
● Configuration calls hardcoded

Public Plugins
● Plugin providers can make
Plugins available to multiple
RTBkit installations
● Provider gives clients
authentication token and
configures firewall
● Configuration API in the public
Internet is zone-aware and
provides access to Plugin
endpoints
● Supports Plugin marketplace

RTBkit 2.0 Plugin-based Architecture
● All extension points redefined as
passive HTTP plugins:
● OO  Services
● C++  Any Language You
Want™
● Standard approaches to:
● Configuration / Discovery
● Deployment
● Security
● Compatibility
● Public and Private
● Even RTBkit core redefined as a
family of APIs

Plugin Endpoints – Config & Worker
● Plugins expose /config endpoint
● Configuration calls are HTTPS
● Configuration brokers keys and
accepts IPs to allow traffic from
● Configuration uniquely identifies
and versions a Plugin
● RTBkit Core uses own
Configuration Plugin to call
available Plugins /config, retrieve
their URLs
● Plugins mark themselves as dead
Plugin
/conf /do_work
Configuration Plugin
RTBkit

Plugin APIs
● Configuration
● Returns list of Live and Dead Plugins
● Bidder
● Receives stream of Bid Requests
● Returns stream of Bid Responses
● Multiple logical bidders behind one
endpoint disambiguated with ext field
● Analytics
● Receives stream of data for use in
offline reporting systems
● Event Id
● “Channel” for the type of Event
● list of subscribed Plugins
● Data
● Plugin API implementation of
Augmenter
Bidder
RTBkit 2.0
Analytics Config
Data

Plugin Configurations
● Plugins configure RTBkit 2.0 Core
using Configurations
● Id, Version, Type
● URLs
● Rate
● Bidder Plugins
● Is Banking
● Pre-processing Subscriptions – such
as Filtering
● Data Plugin Bidder is using
● Banking Plugins
● Budget
● Bid Rate
● Analytics
● Channels
Bidder
RTBkit 2.0
Analytics Plugin X
Banker

Multi-Data Center Support & Routing
● RTBkit instances know their
location – tied to a region
● By default only connect to
Plugins in the same region
● RTBkit ignores Plugins not
responding to heartbeat calls
● Standard list of locations used
by RTBkit and Plugins
● Timeouts and failures handled
by type of Plugin
West East

RTBkit and Plugin Security
● HTTPS only used when calling
the Configuration API
● All HTTP requests signed with
a HMAC key returned by the
Configuration API
● Supports rejecting forged requests
● RTBkit periodically calls
Plugin Configuration endpoint
to update IPs that will call it
● Plugins should reverse proxy
to reduce attack surface

Packaging and Compatibility
● Test suite to ensure:
● Plugins compatible with RTBkit 2.0
● Changes to RTBkit 2.0 compatible with
connected Plugins
● Plugins speak OpenRTB 2.2
● RTBkit 2.0 package options:
● Docker
● Binary distribution of executables,
headers, libs
● Backwards compatible with
RTBkit 1.0 – Router will
support 1.0 or 2.0 interfaces
via dependency injection
Plugin 2
Plugin 3 Plugin 1
RTBkit 2.0
Plugin 4

RTBkit 2.0 Core APIs
● Control Blocks
● Pre-packaged sets of RTBkit services
required to control an RTBkit
installation
● Lower scaling requirements
● One required per data center
● REST API
● Worker Blocks
● Pre-packaged set of RTBkit services
that require high throughput and may
need to be horizontally scaled
● High scaling requirements
● At least one per data center, but may
deploy many for scaling or redundancy

2.0 Roadmap and Progress
Already Delivered in 2014
● HTTP Bidding Agent APIs –
Precursor to full 2.0 API
Up Next – Q4 2014
● First API open source code
releases -- written in Go not C++
● RTBkit updates to be compatible
with new APIs
Up Next – Q1 2015
● Data and Logger HTTP APIs for
Data and Analytics Plugins
© Datacratic Inc. 2014 All Rights Reserved.

RTBkit Resources
Links
● http://github.com/rtbkit
● http://rtbkit.org
● https://groups.google.com/a/rtbkit.
org/forum/#!forum/discuss
Developers Getting Started Guide
● https://github.com/rtbkit/rtbkit/wiki/
Getting-Started

RTBkit & Rubicon Project Deliver Scalable Ad
Tech Infrastructure & Extensible Bidding Stack
• Rubicon Project is a leading technology company automating the buying and selling of
advertising.
• Rubicon Project & RTBkit have combined forces to deliver a scalable ad tech infrastructure, and
makes it easy for media buyers to deploy their own proprietary bidding stack with the open and
extensible architecture of RTBkit.
• The OpenDSP team is a contributor to the RTBkit open source project and active member of the
RTBkit community.
Join the Alpha Program
There are limited seats available in the alpha program. Contact the OpenDSP team at
opendsp@rubiconproject.com to get started.

About Datacratic
Datacratic is an enterprise software
company that develops machine learning
and artificial intelligence technology,
which enables real-time machine based
decisioning to be deployed into
a wide range of applications.
Datacratic’s products are currently optimized for:
• Data Management Platforms (DMPs)
• Demand Side Platforms (DSPs)
• Agency Trading Desks (ATDs)
• E-Commerce web sites
• Others in the real-time marketing ecosystem.
Mark Weiss @marksweiss
Head of Customer Solutions at Datacratic
www.datacratic.com
opensource.datacratic.com/
We're hiring! datacratic.com/site/careers

RTBkit 2.0 Roadmap Preview

Recommended

Recommended

More Related Content

Similar to RTBkit 2.0 Roadmap Preview

Similar to RTBkit 2.0 Roadmap Preview (20)

Recently uploaded

Recently uploaded (20)

RTBkit 2.0 Roadmap Preview

Editor's Notes