Our platform, which purchases and runs advertisements in the Real-Time Bidding model, processes 250K bid requests and generates 20K events per every second which gives 3TB data every day. Because of machine learning, system monitoring and financial settlements we need to filter, store, aggregate and join these events together. As a result processed events and aggregated statistics are available in Hadoop, Google BigQuery and Postgres. The most demanding are business requirements such as: events that should be joined together can appear 30 days after each other, we are not allowed to create any duplicates, we have to minimalize possible data losses as well as there could not be any differences between generated data outputs. We have designed and implemented the solution which has reduced delay of availability of this data from 1 day to 15 seconds. We will preent: Our first approach to the problem (end-of-day batch jobs) and final solution (real-time stream processing) 2. detailed description of the current architecture 3. how we had tested new data flow before it was deployed and in which way it is being monitored now 4. our one-click deployment process 5. decisions which we made with its advantages and disadvantages and our future plans to improve our current solution. We would like to share our experience connected with scaling solution over clusters of computers in several data centers. We will focus on the current architecture but also on testing and monitoring issues with our deployment process. Finally, we would like to provide an overview of engaged projects like Kafka, Mirrormaker, Storm, Aerospike, Flume, Docker etc. We will describe what we have achieved from given open source and some problems we have come across.

1. ARCHITECTURE & LESSONS LEARNED
BARTOSZ ŁOŚ
REAL-TIME DATA PROCESSING AT RTB HOUSEREAL-TIME DATA PROCESSING AT RTB HOUSE
BIG DATA TECHNOLOGY WARSAW SUMMIT 2017
FEBRUARY 9, 2017

2. TABLE OF CONTENTS
Agenda:
- real-time bidding
- the first iteration: mutable structures
- the second iteration: data-flow
- the third iteration: immutable streams of events
02/23

3. REAL-TIME BIDDING

4. REAL-TIME BIDDING: RTB PLATFORM
Processing bid requests
(350K/s, ~30 SSP networks, <50-100ms)
04/23

5. REAL-TIME BIDDING: DATA & MACHINE LEARNING
Impressions:
~ 150M events / day
~ 4TB data / day
Clicks:
~ 1M events / day
~ 35GB data / day
Conversions:
~ 450K events / day
~ 25GB data / day
05/23

6. THE FIRST ITERATION

7. THE 1ST ITERATION: MUTABLE IMPRESSIONS 07/23

8. THE 1ST ITERATION: DRAWBACKS
Issues:
- long, overloading data migrations (30 days back)
- complex servlets' logic, inability to reprocess
- inflexible, various schemas
- single-DC
- inconsistencies
08/23

9. THE SECOND ITERATION:
DATA-FLOW

10. THE 2ND ITERATION: THE 1ST DATA-FLOW ARCHITECTURE 10/23

11. THE 2ND ITERATION: DISTRIBUTED LOG
Why Apache Kafka:
- distributed log
- topics partitioning
- partition replication
- log retention
- stateless
- efficient data consuming
11/23

12. THE 2ND ITERATION: BATCH LOADING
Why Apache Camus:
- "Kafka to HDFS" pipeline
- map-reduce jobs, batches
- storing offsets in log files
- data partitioning
12/23

13. THE 2ND ITERATION: AVRO & SCHEMA VERSIONING
Why Apache Avro:
- data serialization framework
- rich data structures
- self-describing container files
- reader & writer schemas
- binary data format
- schema registry
13/23

14. THE 2ND ITERATION: ACCURATE STATISTICS
Why Apache Storm:
- real-time processing
- streams of tuples, topologies
- fault-tolerance
Why Trident:
- transactions, exactly-once processing
- microbatches (latency & throughput)
14/23

15. THE 2ND ITERATION: STATS-COUNTER TOPOLOGY 15/23

16. THE 2ND ITERATION: DRAWBACKS
Hybrid architecture:
- aggregates (real-time)
- raw events (2-hour batches)
- joined events (end-of-day batch jobs)
Other issues:
- Hive joins
- mutable events
- servlets' complex logic
16/23

17. THE THIRD ITERATION:
NEW APPROACH

18. THE 3RD ITERATION: NEW APPROACH
{ "IMPRESSION”:
"URL”,
"TIME”,
"CREATIVE”,
...
"CLICKS”,
"CONVERSIONS”
}
{ "CLICK”:
"TIME”,
"IMPRESSION_ID”,
...
"IMPRESSION”
}
{ "CONVERSION”:
"TIME”,
"CLICK_ID”,
...
"CLICK”
}
New approach:
- real-time processing
- publishing light events
- immutable streams of events
18/23

19. THE 3RD ITERATION: HIGH-LEVEL ARCHITECTURE 19/23

20. THE 3RD ITERATION: DATA-FLOW TOPOLOGY 20/23

21. THE 3RD ITERATION: EVENTS MERGE 21/23

22. SUMMARY
What we have achieved:
- multi-DC architecture
- HDFS & BigQuery streaming
- platform monitoring
- much more stable platform
- higher quality of data processing
- better data-flow monitoring, deployment & maintenance
22/23

23. THANK YOU
FOR YOUR ATTENTION