Creating millions of user sessions using Complex Event Processing Every day, Yelp connects millions of consumers with great local businesses through the website and mobile apps. We strive to provide our users with an ever-evolving, excellent experience by constantly running a plethora of experiments based on user activity. A user session encapsulates all of a single user’s activity until the user has been dormant for 30 minutes. Creating user sessions requires us to process hundreds of millions of log events occurring daily and applying filters on them. Due to the large volume of log events, creation of these sessions presents us with several application level challenges, including: handling of late events, filtering bot traffic, etc. Features like event time and exactly once processing that are provided by Flink made building such a large scale streaming application like ours possible. Our main motivation to move towards streaming from batch processing stemmed from the fact that our feedback on analysis based on user sessions was always a day late and as an added bonus it also meant integrating with our state-of-the-art data-pipeline ecosystem. In this talk we will not only discuss why Yelp moved from creating user sessions using batch jobs to generating them in near-real-time using Apache Flink but also highlight issues we encountered with continuous bot traffic that never closed the session window, adding custom triggers for long running sessions, duplicate events while allowing late events to be processed, auditing of the created sessions etc.

1. Prem Santosh
pxu@yelp.com
@premsantosh
Creating millions of user sessions using complex
event processing

2. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

3. Yelp’s Mission
Connecting people with great
local businesses.

4. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

5. ● Fundamental element of engagement with Yelp
● Understand intent of users
● Generate other product metrics using sessions
Why Sessions?

6. Motivation
Batch
Processing
Raw Data Sessions

7. Motivation
● Slow
● Expensive
● One day late
● Sessions crossing midnight

8. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

9. Sessionizer Specs
● 1 Master
○ M4.xlarge (4 CPU, 16GB)
● 6 Core
○ M4.16xlarge (6 * 64 CPUs, 6 * 256GB)
● 6 Input kafka topics
● Total input throughput ~25K msg/sec

10. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

11. User Session

12. User Session

13. User Session
Activity A Activity B Activity C Activity D
UserID = XYZ

14. Sessionizer
Sessionizer
Event Logs
Sessionized Logs

15. Sessionizer Operators
KeyBy: UserID
.window(EventTimeSessionWindows)

16. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

17. Never ending bot sessions
t1 t2 t2 + 30
No Activity

18. Never ending bot sessions
t1 t2 t3

19. Never ending bot sessions

20. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

21. Duplicate Sessions
Session ID Number of Events
31cca9e6-4207-a323-36c121deaf73 45
95bff7c34-4de5-bc10-ef969a876cf7 27
31cca9e6-4207-a323-36c121deaf73 46
….. ….

22. Duplicate Sessions

23. Duplicate Sessions
t1 t2 t2 + 30
No Activity
t2 + 30 + allowedLateness
Late Event

24. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

25. Stale Topics
● Event time processing
● Event time watermark: to signal progress in event time.
● Watermarks are crucial when events can be out-of-order

26. Stale Topics
Sessionizer
Window
Output
Kafka
Watermark = min(T1, T1, T1)

27. Stale Topics
Sessionizer
Window
Kafka
Watermark = min(T5, T3, T5)

28. Stale Topics (Solution)
● Internal state stores time T since last watermark was
seen.
● If T > 10mins, override and set watermark

29. Stale Topics (Solution)
Sessionizer
Window
Output
Kafka
Watermark =
T5 - maxOutOfOrderness

30. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

31. S3 Throttling/ Checkpointing
● Due to large ingestion rate, we were checkpointing
frequently.
● Caused us to be throttled by S3 , causing checkpoint
failures

32. Fixed by FLINK-9061

33. ● Motivation
● Sessionizer Specs
● Sessionizer
● Issues
○ Bot traffic
○ Duplicate sessions
○ Stale topics
○ S3 Checkpointing / Throttling
○ Flink Internal Timer State
Agenda

34. ● Slow data structure to store timer state
● Stores timers in-memory
Flink Internal Timer State

35. ● Slow data structure to store timer state
○ HeapInternalTimerService O(n) deletion
operation
○ High CPU usage
○ 50 million session mark → super slow processing
Flink Internal Timer State

36. ● Stores timers in-memory
○ HeapInternalTimerService only
implementation
○ Session count ↑ results OOM
○ FLINK-9485 rocksDB implementation
Flink Internal Timer State

37. ● Slow data structure to store timer state
○ FLINK-9423
● Stores timers in-memory
○ FLINK-9485
Flink Internal Timer State

38. Thank You!
pxu@yelp.com
@premsantosh

39. www.yelp.com/careers/
We're Hiring!

40. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp