AdTech requires high speed at massive scale. Sizmek serves millions of requests every second. Requests need to be processed in tens of milliseconds, while involving 10 simultaneous lookups into a database that contains tens of billions of profiles. In this presentation, you will discover how Scylla enables Sizmek’s real-time bidders to query a gigantic user profile store quickly and reliably with only a few nodes. We’ll discuss data modeling, server and driver configuration, techniques to minimize disk access, as well as considerations for leveraging Spark while migrating from HBase.
3. Presenters Bio
Lubos is working as a software engineer in Sizmek taking care of 7 datacenters
full of bidding infrastructure to deliver real time bids and impressions for
marketing campaigns of Sizmeks clients. Previously he used to work in Oracle
and Sun Microsystems on products that managed datacenters and monitored
them. He is also one of the leaders of {OpenGrok the source code search
engine.
Michal is a Senior Software Engineer at Sizmek with background in both real-
time and batch distributed systems. Previously employed at Seznam.cz, a
local Czech alternative of Google. A strong proponent of Scala and an
occasional Apache Spark contributor.
4. Sizmek
▪ Largest independent buy-side advertising platform
▪ AI-driven decisioning engine
▪ Platform in more than 70 countries used by more than 20,000
advertisers and 3,600 agencies
▪ Delivering billions of impressions per day
▪ Processing hundreds of billions of bid requests per day
5. Topics
▪ Use case
▪ Data model
▪ Server and driver configuration
▪ Short-lived caching
▪ Spark interoperability
▪ Optimizations
7. Real Time Bidding
Record impression
5
4b
Notification
Bid request
Bid + HTML
2
3
Request ad
Serve ad
1
4a
Render ad
Ad HTML
6
7
RTB EXCHANGE
USER
ADSERVER
8. User Profile
▪ Merged from multiple sources
▪ Tracking all active users/devices
• Multiple devices for one user/household (cluster)
▪ Very short SLA
▪ Queried for every bid request
▪ Drives bidding decision
• Leveraging AI models
9. Environment
▪ Volume: Around 50 billion profiles
▪ Throughput: Several millions of requests per second
• Single request can initiate multiple profile lookups
▪ Latency: Tens of milliseconds for each request
• Useless if exceeded
▪ Distribution: Multiple DCs serving hundreds of clients each
11. Attributes
Two types of user profile attributes:
▪ Primitive
• Single value, overwrite semantics
▪ Collection
• Multiple values, append semantics
Complex structured values
12. Lookups
▪ Single profile
▪ Profile cluster
• Each profile can reference other profiles (graph)
• Merge referenced profiles into main profile
Can specify attribute subset
13. Mutations
▪ Profile merging
• Primitive overwrite & collection append
▪ Attribute deletion
• Remove whole attribute
▪ Value deletion
• Remove single collection element
14. Housekeeping
▪ TTL enforcement
• Each attribute can have a different TTL
▪ Collection attribute pruning
• Keep only N newest values
• Custom processing (deduplication)
15. Data Model
Primitive attributes
user_id: blob
attribute1: blob
attribute2: blob
...
Collection attributes
user_id: blob
attribute_id: byte
value: blob
▪ Co-partitioning
• All attributes of a single profile share the
same shard set
▪ Grouping by collection attribute
• Efficient lookup by attribute id
▪ Efficient value serialization
• Google Protocol Buffers
17. Server Configuration
▪ Enterprise 2018.1.5 used, repairs automated by manager
▪ Global cluster - 7 DCs (sister sites in every region)
▪ Keyspace per region (Americas, EMEA, APAC) - replication
▪ HW resources divided between Scylla and Spark
▪ Read latency based scylla.yaml:
• background_writer_scheduling_quota: 0.5
• auto_adjust_flush_quota: true
18. Java Driver Configuration
▪ Custom 3.5.1 cassandra based shard aware driver
▪ Increase connections in pooling-options to match number of cores
• core/max, local/remote
▪ Local region keyspace for read, all region keyspaces for write
▪ Custom retry policy
▪ LZ4 compression enabled
20. Motivation
User profile cluster lookup
▪ Lookup of main profile and all related profiles + merging
▪ Large read amplification
▪ Related profiles provide supplementary information
• Not as time sensitive, minor staleness acceptable
21. Concept
Bidder
Bid Request
ProfileStore
Get
Main Profile + Cached Combined Profile
Get All Profiles in the Cluster
Write Combined Profile into the Cache
Executed for each bid request
Executed for bid request without cached profile
22. Replication Vs. Co-location
▪ Same replication
• Higher load
• Contact only one node on profile lookup
▪ No replication
• Token range replica locations different
• Potentially contact multiple nodes on profile lookup
23. Minimizing Disk Usage
▪ Time window compaction strategy (TWCS)
• Compaction window = TTL
• Min. threshold = 2 (compacts SSTables as soon as possible)
▪ 2 SSTables exist, 1 is actively read, compaction “cycles” them
30. Java Driver Optimization
ScyllaDB team is preparing an optimized Cassandra driver which
introduces shard awareness
Using shard aware driver in PoC run:
1800 profiles / second x core => 2300 profiles / second x
core
== 28% increase in throughput
*https://github.com/haaawk/java-driver/tree/3.5.1-scylla/