The document discusses the development of Pinterest's real-time ads platform using Kafka Streams, highlighting its architecture for managing a vast amount of data and ensuring low latency in ad serving. It addresses challenges like state management, budgeting, and the balance between exploring new ads and exploiting successful ones. Contributions to Kafka and various optimizations for system performance are also outlined.

1. Building Pinterest Realtime Ads
Platform Using Kafka Streams
Liquan Pei, Boyang Chen
Kafka Summit SF 2018

2. Liquan Pei
liquanpei@pinterest.com
Boyang Chen
bychen@pinterest.com

3. Visual Discovery Engine
250M MAU
100B Pins created by people
10B Recommendations per day
A great platform for ads

5. Ads Platform

6. Ads Platform
● A recommendation system
○ Machine learning models
● More than a recommendation system
○ Budgeting
○ New Ads Exploration

7. Ads Platform

8. Ads Platform

9. Budgeting

10. Budgeting

11. Stream-Stream Windowed Join

12. Joiner Topology
Insertion
store
Action
store
ad_insertion user_action
Join status

13. Joiner Algorithm
Join
Status
1
Join
Status
Action
Store
4
Join
Status
2
Join
Status
Action
Store
3
Join
Join
Status
Action
Store
5
Insertion
Store

14. Joiner Algorithm
Join
Status 1
Join
Status
2
Join
Status
3
Insertion
Store
Join
4
Join
Status
Insertion
Store
Join
Status
Action
Store
5
Insertion
Store

15. Realtime Spend Joiner
● Large state: TB data.
○ 24-hour join window
● Window store operations
○ Put/Get
○ Commit
● Requirements
○ Sub second latency
○ Fast recovery
○ Fast scale up/down

16. ● num.rolling.segments = number of RocksDB instances.
● A RocksDB is dropped when expired.
Window Store Internal

17. Window Store Operations

18. ● Read/write performance
○ Use point query for fast lookup
■ fetch(key, timeFrom, timeTo);
■ fetch(key, windowStartTime); [ >=Kafka 2.0.0 ]
○ Increase block cache size
○ Reduce action state store size
How to achieve sub-second latency?

19. Window Store Operations

20. ● Each commit triggers RocksDB flush to ensure data is persistent on disk.
● Each RocksDB flush creates SST.
● Accumulated number of SST files will trigger compaction.
● Tune commit.interval.ms.
Kafka Streams Commit

21. Fast recovery
● State rebalance

22. Fast recovery
● Rolling restart could trigger multiple rebalances.
● State shuffling is expensive.
Approaches:
● Recover faster:
○ increase max.poll.records for restore consumer (KIP-276)
○ RocksDB window store batch recovery (KAFKA-7023)
● Single rebalance:
○ Wait for all members to be ready = increase session.timeout.ms.
○ Restore faster: static membership (KIP-345)

23. Fast scale down/up
● Save state in remote storage.
○ S3
○ HDFS

24. Budgeting

25. Windowed Aggregation

26. Aggregator
● Utilize Stream DSL API
● Requirements
○ End to end sub second latency. User action to ads serving.
○ Thousands of ads serving machines needs to consume this data.

27. Output to a compacted topic
Cons:
○ High fanout, broker saturation.
○ Replay could be long.
Pros:
○ Fast correction.
○ Logic simplicity.

28. Cons:
○ Event based: no way to reset.
○ Time based: expensive batch
operation.
Pros:
○ Very small volume.
○ Logic consolidation.
Output to a signal topic

29. Streaming in budget change
Pros:
○ Unblock signal reset without
batch update.
Cons:
○ Consistency guarantee.
○ Strong ordering guarantee.

30. Budgeting Summary
● Low level metrics are critical, especially storage layer.
● Large state shuffling is bad.
● Compacted topic as partitioned key-value store.
● Unified solution for serving stream output.

31. New Ads Exploration
● A new ad is created, however, the Ads Platform doesn’t know about the user
engagement with this ad on different surfaces.
● The faster the Ads Platform knows about the performance of the newly
created ad, the better value we provide to the user.
● Balance between exploiting good ads and exploring new ads.
● Solution: Add a boosting factor to new ads to increase the probability of
winning auction.

32. New Ads Exploration

33. New Ads Exploration
● Need to compute <ad id, past X day impressions>.
● The result published to S3 for serving.
● Backfilling is needed.
○ Exactly same logic as the normal processing.

34. Backfilling

35. Backfilling

36. Backfilling

37. Stream Processing Patterns

38. Streaming Processing Patterns

39. Stream Processing Patterns

40. Stream Platform
● Usability
○ User should only focus on business logic.
○ Support for more state store backends.
○ Type system for easier code sharing.
● Scalability
○ Applications should be able to handle more QPS with more machines.
● Fault Tolerance
○ Application should recover within X minutes.
○ Application should support code and state rollback.
● Developer Velocity
○ The platform should provide standard ways of backfilling.
● Debuggability
○ The platform should provide standard ways of exposing debug information to be queryable.

41. Contributions
● KIP-91 Adding delivery.timeout.ms to Kafka producer.
● KIP-245 Replace StreamsConfig with Properties
● KIP-276 Add config prefix for different consumers
● KIP-300 (ongoing) Add windowed KTable API
● KIP-345 (ongoing) Reduce consumer rebalances through static membership
● KAFKA-6896 Export producer and consumer metrics in Kafka Streams
● KAFKA-7023 Move prepareForBulkLoad() call after customized RocksDBConfigSetter
● KAFKA-7103 Use bulkloading for RocksDBSegmentedBytesStore during init
● RocksDB Metrics Lib

42. Acknowledgements
Guozhang and Matthias from Confluent
Yu Yang, Zack Drach and Shawn Nguyen from Pinterest
The Ads Realtime Team

43. Citations
● Search ads on Pinterest: https://business.pinterest.com/en/blog/introducing-
search-ads-on-pinterest
● Ads demo: https://bn.co/pinterest-promoted-pin-campaign/
● Strencils source: https://stenciltown.omnigroup.com/categories/all/
● RocksDB tuning: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-
Guide
● “Compact, delete” topic:https://issues.apache.org/jira/browse/KAFKA-4015
● Monitoring your Kafka Streams Application:
https://docs.confluent.io/current/streams/monitoring.html
● Join Support in Kafka streams:
https://docs.confluent.io/current/streams/developer-guide/dsl-
api.html#streams-developer-guide-dsl-joins