The document discusses a presentation by Lubos Kosco and Michal Senkyr on processing 50 billion user profiles in real-time for Sizmek's advertising platform. Key topics include a data model for user profiles, server configurations, caching strategies, and optimizations that enhance throughput and reduce hardware requirements. The presentation concludes with comparisons showing significant gains in efficiency and performance due to these improvements.

1. Adventures in AdTech
(processing 50B user profiles in real time)
Lubos Kosco, Michal Senkyr
Senior Software Engineers at Sizmek

2. INTRODUCTION

3. Presenters Bio
Lubos is working as a software engineer in Sizmek taking care of 7 datacenters
full of bidding infrastructure to deliver real time bids and impressions for
marketing campaigns of Sizmeks clients. Previously he used to work in Oracle
and Sun Microsystems on products that managed datacenters and monitored
them. He is also one of the leaders of {OpenGrok the source code search
engine.
Michal is a Senior Software Engineer at Sizmek with background in both real-
time and batch distributed systems. Previously employed at Seznam.cz, a
local Czech alternative of Google. A strong proponent of Scala and an
occasional Apache Spark contributor.

4. Sizmek
▪ Largest independent buy-side advertising platform
▪ AI-driven decisioning engine
▪ Platform in more than 70 countries used by more than 20,000
advertisers and 3,600 agencies
▪ Delivering billions of impressions per day
▪ Processing hundreds of billions of bid requests per day

5. Topics
▪ Use case
▪ Data model
▪ Server and driver configuration
▪ Short-lived caching
▪ Spark interoperability
▪ Optimizations

6. USE CASE

7. Real Time Bidding
Record impression
5
4b
Notification
Bid request
Bid + HTML
2
3
Request ad
Serve ad
1
4a
Render ad
Ad HTML
6
7
RTB EXCHANGE
USER
ADSERVER

8. User Profile
▪ Merged from multiple sources
▪ Tracking all active users/devices
• Multiple devices for one user/household (cluster)
▪ Very short SLA
▪ Queried for every bid request
▪ Drives bidding decision
• Leveraging AI models

9. Environment
▪ Volume: Around 50 billion profiles
▪ Throughput: Several millions of requests per second
• Single request can initiate multiple profile lookups
▪ Latency: Tens of milliseconds for each request
• Useless if exceeded
▪ Distribution: Multiple DCs serving hundreds of clients each

10. DATA MODEL

11. Attributes
Two types of user profile attributes:
▪ Primitive
• Single value, overwrite semantics
▪ Collection
• Multiple values, append semantics
Complex structured values

12. Lookups
▪ Single profile
▪ Profile cluster
• Each profile can reference other profiles (graph)
• Merge referenced profiles into main profile
Can specify attribute subset

13. Mutations
▪ Profile merging
• Primitive overwrite & collection append
▪ Attribute deletion
• Remove whole attribute
▪ Value deletion
• Remove single collection element

14. Housekeeping
▪ TTL enforcement
• Each attribute can have a different TTL
▪ Collection attribute pruning
• Keep only N newest values
• Custom processing (deduplication)

15. Data Model
Primitive attributes
user_id: blob
attribute1: blob
attribute2: blob
...
Collection attributes
user_id: blob
attribute_id: byte
value: blob
▪ Co-partitioning
• All attributes of a single profile share the
same shard set
▪ Grouping by collection attribute
• Efficient lookup by attribute id
▪ Efficient value serialization
• Google Protocol Buffers

16. SERVER & DRIVER
CONFIGURATION

17. Server Configuration
▪ Enterprise 2018.1.5 used, repairs automated by manager
▪ Global cluster - 7 DCs (sister sites in every region)
▪ Keyspace per region (Americas, EMEA, APAC) - replication
▪ HW resources divided between Scylla and Spark
▪ Read latency based scylla.yaml:
• background_writer_scheduling_quota: 0.5
• auto_adjust_flush_quota: true

18. Java Driver Configuration
▪ Custom 3.5.1 cassandra based shard aware driver
▪ Increase connections in pooling-options to match number of cores
• core/max, local/remote
▪ Local region keyspace for read, all region keyspaces for write
▪ Custom retry policy
▪ LZ4 compression enabled

19. SHORT-LIVED CACHING

20. Motivation
User profile cluster lookup
▪ Lookup of main profile and all related profiles + merging
▪ Large read amplification
▪ Related profiles provide supplementary information
• Not as time sensitive, minor staleness acceptable

21. Concept
Bidder
Bid Request
ProfileStore
Get
Main Profile + Cached Combined Profile
Get All Profiles in the Cluster
Write Combined Profile into the Cache
Executed for each bid request
Executed for bid request without cached profile

22. Replication Vs. Co-location
▪ Same replication
• Higher load
• Contact only one node on profile lookup
▪ No replication
• Token range replica locations different
• Potentially contact multiple nodes on profile lookup

23. Minimizing Disk Usage
▪ Time window compaction strategy (TWCS)
• Compaction window = TTL
• Min. threshold = 2 (compacts SSTables as soon as possible)
▪ 2 SSTables exist, 1 is actively read, compaction “cycles” them

24. Cache Efficiency
85.3% cache hit ratio
on bid requests

25. SPARK INTEROPERABILITY

26. Spark Cassandra Connector
RDD implementation from Datastax
▪ Shared database sessions
▪ Token range partitioning
• Limited shard awareness
▪ Location awareness
• No traffic between hosts if not needed

27. Custom RDD Implementation
▪ Taking advantage of co-partitioned tables
▪ Aggressive batching
• Accumulation of (unlogged) batches per shard set
• Size-based flushing
▪ Element deletion semantics

28. Custom DataFrame Implementation
▪ Protocol buffer awareness
▪ Taking advantage of Catalyst & Tungsten
▪ Full HiveQL support
• Statistics jobs

29. OPTIMIZATIONS

30. Java Driver Optimization
ScyllaDB team is preparing an optimized Cassandra driver which
introduces shard awareness
Using shard aware driver in PoC run:
1800 profiles / second x core => 2300 profiles / second x
core
== 28% increase in throughput
*https://github.com/haaawk/java-driver/tree/3.5.1-scylla/

31. Conclusion

32. Comparison
■ High availability
■ Better replication
■ Lowered HW requirements
● Nodes: 77 ⇒ 16 (79%)
● Cores: 924 ⇒ 320 (65%)
● RAM: 10TB ⇒ 6TB (38%)
● Disk: 240TB ⇒ 270TB (13%)

33. Thank You
Any Questions ?
Please stay in touch
Lubos.Kosco@Sizmek.com, Michal.Senkyr@Sizmek.com
@LubosKosco, @MichalSenkyr