RTBkit Introduction & Best Practices
•Download as PPTX, PDF•
3 likes•2,357 views
The document discusses the RTBkit project, an open-source real-time bidding software. It provides an overview of the project's history and governance model. It also outlines several use cases for RTBkit, best practices for deployment, and a preview of new plugin and API features in the upcoming 2.0 version. Resources and contact information are provided at the end.
Recommended
More Related Content
Viewers also liked
Viewers also liked (8)
Similar to RTBkit Introduction & Best Practices
Similar to RTBkit Introduction & Best Practices (20)
Recently uploaded
Recently uploaded (20)
RTBkit Introduction & Best Practices
- 1. Introduction & Best Practices Open-Source RTB for Everyone Copyright © 2014 Datacratic Inc. All rights reserved
- 2. Overview • About the RTBkit Project • Use Cases • Best Practices • What’s New in 2.0 • Resources • Q&A Copyright © 2014 Datacratic Inc. All rights reserved
- 3. About the project Copyright © 2014 Datacratic Inc. All rights reserved
- 4. A Little History • RTBKit was created by machine-learning and digital marketing company Datacratic • Code base evolved from running RTB in production from 2011-present • Open sourced in Feb. 2013, with ongoing support from Datacratic • Apache-style governance started Jan. 2014 Copyright © 2014 Datacratic Inc. All rights reserved
- 5. Participation and Governance • Apache-style governance • BDNFL - Benevolent Dictator Not for Life • Councillors • Committers • Outside contributions welcome • Github pull request workflow -- committers review and merge • Contributor guidelines • Users can become Contributors • Contributors can become Committers -- currently two outside Committers Copyright © 2014 Datacratic Inc. All rights reserved
- 6. Development Support Development Support • Getting Started Guide • Developer Tutorials • Working integration test system – mock Exchange and Ad Server – fixed-price Bidding Agent • Example Code • Documentation • Ubuntu AMI • Google Group support • Pull request review and support Copyright © 2014 Datacratic Inc. All rights reserved
- 7. Addresses the RTB Value Problem Copyright © 2014 Datacratic Inc. All rights reserved GENERAL/TECHNICA L SYSTEM SELECTIO N VALUE Scale Speed Distribution Show user an ad? What ad? What is it worth? What should I pay? Reliability SPECIFIC/BUSINESS PROVIDED BY RTBKIT CUSTOMIZED BY USER
- 8. RTB Technology Landscape System Pros Cons Degree of Difficulty Exchange / DSP UI Easy to get started ● Manual, hard to scale ● Lack of control over bidding strategy and data Low Intermediate Hosted Bidding ● More control over bidding strategy and use of data ● Don't have to do Ops Strategy and use of data mediated by vendor and product features Medium ● Core systems and integration problems solved ● Benefit from community ● Flexible customization ● Flexible use of data ● Requires significant development ● Responsible for operations of a high-volume distributed system High Roll Your Own Bidder Full control of all aspects of the system Solve all the hard systems problems yourself first, then build the rest of the system you need. Highest Copyright © 2014 Datacratic Inc. All rights reserved
- 9. Use Cases & Best Practices Copyright © 2014 Datacratic Inc. All rights reserved
- 10. Use Cases Use Case RTBkit Benefits Startup DSP / Targeted Media Company ● Move directly to having your own bidder in first iteration of company technology ● Full flexible use of proprietary data to pursue specialized strategy, differentiate and create IP Established Small to Midsize DSP / Targeted Media Company ● Move from (perhaps multiple) external buying platforms to unified internal platform. Save on fees and ad ops personnel. ● Improved margins ● Full flexible use of proprietary data to create additional value for customers Larger Demand Side Platform / Hosted Service / Integrated Platform ● Real-world example: Rubicon Project ● Get reliable, high-performance core bidder functionality without having to build it ● Benefit from open source, open architecture and coming plugin ecosystem with RTBkit 2.0 Copyright © 2014 Datacratic Inc. All rights reserved
- 11. Best Practices: Single Node Deployment • Recommended single-node installation on EC2 • Singletons and Real-Time nodes: – C3.8xlarge: – 32 cores, 60GB RAM – 200GB ESB – 1000 provisioned IOPS • Supports about $100,000 monthly media spend • About $1200 / month non-reserved, $700 reserved Copyright © 2014 Datacratic Inc. All rights reserved
- 12. Best Practices: Multi Node Deployment • Recommended multi-node installation on EC2 • Singletons and Real-Time nodes: – C3.8xlarge: – 32 cores, 60GB RAM – 200GB ESB – 1000 provisioned IOPS • 1..N Real-Time Nodes can scale horizontally • HAProxy load balancing for real-time nodes • Aggregating Carbon and Graphite node: – M3.xlarge – 2 cores, 3.69GB RAM Copyright © 2014 Datacratic Inc. All rights reserved
- 13. Best Practices: Multi Node Multi-DC Deployment • Singleton and Real-Time nodes same as Multi-Node setup, per Data Center • One HAProxy load balancer for real-time nodes per DC • Local Carbon and Graphite services on each host and one global aggregating service for all DCs Copyright © 2014 Datacratic Inc. All rights reserved
- 14. Additional RTBkit Best Practices • Graphite – {MAX_UPDATES_PER_SECOND} = 20 – Configure with memcached – Recommended metrics to monitor: • <installation>.<location>.*.dataLogger.* • <installation>.<location>.*.postAuction.bidResult.WIN.messagesReceived • <installation>.<location>.*.postAuction.bidResult.WIN.notInSubmitted • <installation>.<location>.*.postAuction.delivery.CLICK.delivered • <installation>.<location>.*.router.bid • <installation>.<location>.*.router.*.auctionNetworkLatencyMs.mean • <installation>.<location>.*.router.*.auctionReceived • <installation>.<location>.*.router.*.auctionStart • <installation>.<location>.*.router.*.auctionTotalTimeMs.mean • <installation>.<location>.*.router.monitor.systemInSlowMode • Redis – Configure to run as a service – Configure to dump every 60 seconds to near-term in memory store and set up a cron to persist that to DB or long-term storage every N minutes • Delete old logs periodically Copyright © 2014 Datacratic Inc. All rights reserved
- 15. Additional AWS Best Practices • Provisioned IOPS for real-time endpoints and Carbon / Graphite hosts • SSD-backed EBS volumes • HVM (hardware virtual machine) instance types, not PV (paravirtual) for real-time endpoints Copyright © 2014 Datacratic Inc. All rights reserved
- 16. 2.0 Roadmap Preview Copyright © 2014 Datacratic Inc. All rights reserved
- 17. The Big Story in 2.0… Plugins! • Why Plugins? We want to support an ecosystem of revenue opportunities on RTBkit. • Design supports this goal – Plugins identified by URL – Can have multiple versions – Support multiple RTBkit versions – Support multiple RTBkit installations – Requests can be authenticated – Vendors of public plugins can set up direct billing Copyright © 2014 Datacratic Inc. All rights reserved
- 18. Public Plugin Ecosystem • Major goal of RTBkit 2.0 • RTBkit users can leverage third-party Plugins • Public Plugin requirements: – Multitenant – one plugin can serve many RTBkit instances – Multi-location – one logical plugin can be deployed in multiple locations – Authentication and Security – Per-call metering for billing – Support multiple RTBkit versions Plugin RTBkit Plugin Plugin Plugin Plugin Plugin Plugin Plugin Plugin Plugin Plugin Plugin Plugin Plugin Plugin RTBkit RTBkit Plugin Plugin Copyright © 2014 Datacratic Inc. All rights reserved
- 19. Plugins – Ecosystem Business Use Cases • RTBkit users benefit and Vendors benefit: – Bidder Plugins – campaign management and bidding logic – Bid Optimization – First-party Data in bidding logic – Per-bid bid price modifier – First-party Data for attribution – Third-party Data – Custom Attribution – Offline Analytics – Real-time Analytics – Real-time Audience Modeling Copyright © 2014 Datacratic Inc. All rights reserved
- 20. 2.0 Roadmap and Progress Already Delivered in 2014 • HTTP Bidding Agent APIs – Precursor to full 2.0 API Up Next – Q4 2014 • First API open source code releases – written in Go not C++ • RTBkit updates to be compatible with new APIs Up Next – Q1 2015 • Data and Logger HTTP APIs for Data and Analytics Plugins Copyright © 2014 Datacratic Inc. All rights reserved
- 21. RTBkit Resources RTBkit Links • http://github.com/rtbkit • http://rtbkit.org • https://groups.google.com/a/rtbkit.org/forum/#!forum/d iscuss • https://github.com/rtbkit/rtbkit/wiki/Getting-Started Introduction to RTBkit • http://www.slideshare.net/Datacratic/a-technical-introduction- to-rtbkit • http://www.slideshare.net/Datacratic/a-technical-introduction- to-rt-bkit-edit-45minute?related=1 RTBkit 2.0 Presentation • http://www.slideshare.net/Datacratic/rtbkit-20- roadmap-preiew Copyright © 2014 Datacratic Inc. All rights reserved
- 22. RTBkit & Rubicon Project Deliver Scalable Ad Tech Infrastructure & Extensible Bidding Stack • Rubicon Project is a leading technology company automating the buying and selling of advertising. • Rubicon Project & RTBkit have combined forces to deliver a scalable ad tech infrastructure, and makes it easy for media buyers to deploy their own proprietary bidding stack with the open and extensible architecture of RTBkit. • The OpenDSP team is a contributor to the RTBkit open source project and active member of the RTBkit community. Join the Alpha Program • There are limited seats available in the alpha program. Contact the OpenDSP team at opendsp@rubiconproject.com to get started. Copyright © 2014 Datacratic Inc. All rights reserved
- 23. About Datacratic Datacratic is an enterprise software company that develops machine learning and artificial intelligence technology, which enables real-time machine based decisioning to be deployed into a wide range of applications. Datacratic’s products are currently optimized for: • Data Management Platforms (DMPs) • Demand Side Platforms (DSPs) • Agency Trading Desks (ATDs) • E-Commerce web sites • Others in the real-time marketing ecosystem. Copyright © 2014 Datacratic Inc. All rights reserved Mark Weiss@marksweiss Head of Customer Solutions at Datacratic www.datacratic.com opensource.datacratic.com/ We're hiring! datacratic.com/site/careers
- 24. Host an RTBkit Meetup! Copyright © 2014 Datacratic Inc. All rights reserved
- 25. Copyright © 2014 Datacratic Inc. All rights reserved
- 26. Q & A Copyright © 2014 Datacratic Inc. All rights reserved
Editor's Notes
- RTBkit was created by Datacratic, whom I work for, a company based in Montreal that provides machine-learning based products and services, with a focus right now on RTB and lookalike modeling for digital marketing. Datacratic sells optimization for real-time bidding, and starting about three years ago, began building a bidder to serve its customers. The company made the decision to open source RTBkit early last year and the project has been a public open source project since February 2013.
- Finally, we get to the core problem, value. After all, you are bidding, this is an auction, so the fundamental question you need to answer after you have decided that yes, indeed, you want to try and win the auction for the right to show this ad to this person, is how much is it worth to you to show this ad to this person. Again, this computation is the output of the calculation by your Bidding Agent. Notice in this diagram that System is orthogonal to Selection and Value. Again, this is an excellent frame for understanding the philosophy of RTBkit. Commoditize the system problem and provide you tools to address Selection and Value. Notice also that Selection and Value overlap. That's because the value of a particular ad is not independent of the context. Each bidder cares about all the campaigns they have running, the goals of those campaigns, how much budget they have left, how much time they have left to run. Each bidder has different data about the user being shown the ad. And each bidder might also have different business logic interacting with all the metadata of the ad, such as where the ad is running, the size of the ad slot etc. All of this plays into answering both the "What Ad?" and both Value questions. Each ad is worth something different to each bidder, which is why RTBkit provides Selection tools and a Bidding Agent framework but allows system builders to customize their Agent logic. As we will see later, this is also why Augmenters are a key part of the RTBkit design.
- Now let's start drilling into the design and architecture of RTBkit to show how it implements a solution to these problems. We're going to start at a high level and take a tour of each service and component in the system. You'll understand the responsibilities of each piece of the system, the data flows through the system and learn the rationale for some technology choices made. [COVER BULLETS ON CORE] [COVER BULLETS ON PLUGINS] [DISCUSS RATIONALE FOR CORE VS. PLUGINS]