My use case is to provide monitoring, and improving the overall search data quality, also to find the unusual patterns of user’s search behavior, and notifying the intent on-site back to the respective business stakeholders. To achieve the same, I explored various big data processing engines, which can process huge data with complex business logic in real time. Eventually, I used Flink Stream processing. This talk will showcase how I used Flink to accomplish my goal.

1. Real Time DQMM on Flink
Jaydeep
Staff Engineer in Search Team @ WalmartLabs
Apache Oozie Committer
June, 2019

2. Table of Contents
2
• What is Real Time Aggregation​?
• Use Cases
• Processing engine
• Flink vs Spark
• Flink on Yarn
• Real Time DQMM Architecture
• 100% data completeness
• Demo
• What Next
• Open Items

3. What is Real Time Aggregation​?
3
• What is real time ?​
• What is the processing delay today?​
• What real time offering?​
• Why do we need it?

4. Use Cases
4
• Release with Beacon Bugs​
• Item Catalogue health​
• Item out of stock (specially on event days)​
• Best seller item tracking​
• Top query monitoring​
• Top item tracking​
• Category performance

5. Processing Engine
5
• Support for Real-time processing.
• Support to track the events.
• Easy to recover from failure.
• Exactly once processing
• Backpressure handling
• Support for Event based, Time based and Dynamic Window
• Highly Available

6. Flink on Yarn
6

7. Real Time DQMM Architecture
7

8. 100% Data Completeness
8
Event Arrival Time Actual Event Time Clicks
2019-06-01 10:01:00 2019-06-01 10:01:00 3
2019-06-01 10:02:00 2019-06-01 10:02:00 1
2019-06-01 10:04:00 2019-06-01 10:03:00 4
2019-06-01 10:06:00 2019-06-01 10:04:00 5
2019-06-01 10:08:00 2019-06-01 10:04:00 1
Processed Time Event time Window Clicks
2019-06-01 10:05:00 2019-06-01 10:05:00 8
2019-06-01 10:10:00 2019-06-01 10:10:00 6

9. 100% Data Completeness
9
• Event Time data processing
• Handling the delayed event
• Prevent false anomaly detection
• Probability based Model for data completeness

10. What we deal with?
10
~4 Billion logs
~8 million records per minutes
~600 GB Records
Yarn HA cluster
14 Nodes clusters
1.12 GB memory
112 cores

11. Demo
Walmart Labs PPT Timesaver – Privileged and Confidential11

12. What next
12
• Page Type Onboarding
• Category Onboarding
• Mobile App Onboarding
• Best Seller Onboarding
• Improvement on Notification layer

13. Open Items
13
• Real time Model training
• Handling Seasonality while detecting Anomaly

14. Walmart Labs – Privileged and Confidential14