EnovyCon Presentation

1. Insights and Graph-based ML
Anomaly Detection for eBay Edge
Services
Presenter: Anoop Koloth, Hanzhang Wang
Team: Anirudh Muralidhar, Kalieswaran Rayar, Phuong Nguyen, Saravana Chilla,
Venkatesh Palani
EnovyCon, Nov 18, 2019

2. Agenda
eBay Envoy Ecosystem for Edge
Data, Traffic Insights and Anomaly Detection
Grano - Graph-based Anomaly Detection
© 2018 eBay. All rights reserved.2

3. © 2018 eBay. All rights reserved.3
Traffic Engineering
LATENCY
AVAILABILITY

4. Envoy at eBay

5. © 2018 eBay. All rights reserved.5
L7 - High Level

6. Request
Services
Latency
Zoom Images
Client Error
Client
Interactions
Security
Experimentation
>12 Billion+ records /day
120+ dimensions
60+ metrics
Anomaly Detection
Ramp Up Decision
Traffic
Insights
Bot/Attack
Remediation
© 2018 eBay. All rights reserved.6
Data Set

7. route_name , rlogid , isp ,
actual_pop_dist , actual ,
parent , tags ,
upstream-service-time ,
time_to_first_resp_byte,
response_flag
© 2018 eBay. All rights reserved.7
Raw Record

8. Observability

9. Anycast

10. © 2018 eBay. All rights reserved.10
First AD Solution

11. Xpack( Inverse
Deduction ) , Anadot ,
In-House - PyAnom
Cross Data sets /
metrics / models
Single variant
© 2018 eBay. All rights reserved.11
Our Journey

12. “Leverage ML and graph-based solutions to reduce Mean Time to Detect (MTTD) production
incidents in distributed systems.”
The time between and incident and somebody’s knowing about it.
● “Somebody” is an automated response system or a person
● “Knowing about” Doesn't count if there's an alert that gets lost in a flood of alerts, even if the
monitoring system knew.
● “It” is the incident and the root cause(s).
© 2018 eBay. All rights reserved.12
Grano Vision

13. GRANO is an end-to-end graph-based anomaly detection and root cause analysis system for
distributed cloud-native systems
Anomaly Detection
ML time series anomaly detection on metrics
Adaptive Anomaly Graph
Adaptive analysis of all health signals on a real-time property graph of connect metrics/event (or
project on a system topology) to identify the root cause
Applications
Alerting, interactive Graph-UI, and integration with monitoring system
© 2018 eBay. All rights reserved.13

14. © 2018 eBay. All rights reserved.14
Knowledge Graph (or Heterogeneous Graph)
Knowledge Graph:
1. Represents knowledge domain.
2. Connects things of different types in a systematic way.
3. Encode knowledge arranged in a network of nodes and edges rather
than tables of rows and columns.
Why graph here?
1. Native method to understand the “Unknown”
2. White box and Visualization
3. The platform and support is ready (e.g. Data, Graph DB, Computation
Power)
4. “Get ready for AI” or “translator of AI” analysis (Pattern)
5. Recommendation/Refactoring

15. © 2018 eBay. All rights reserved.15
Motivating Example

16. © 2018 eBay. All rights reserved.16
Data
Preprocess
xPack
ML jobs
Service
Adaptive
GraphGen
Grano Quick
Walkthrough
{
"Compoents":{
"POP": ["LHR"]
"Site": ["EU","US"]
"Flow": ["AddtoCart", "Checkout"]
}
"Links": {
"Agent": ["POP","Flow","Site"]
}
"Alerts":{
"XPack":{
"Latency_90":{
"LHR": [0,0,0,0.4]
"LHR_AddtoCart_EU":
[0.1,0.3,0.5,0.9]
}
}
}
}
General Anomaly Graph Schema
PyAnom
○ Additive decomposition
forecasting models
○ Multivariate clustering
models
○ Statical-based models
GraphDB

17. © 2018 eBay. All rights reserved.17
Pop
Flow
Site EU US
Checkout Buy Sell
LHR DUS SJC ...
...
...
#Type #Example
... ...
Adaptive GraphGen

18. © 2018 eBay. All rights reserved.18
Pop
Flow
Site
Sub-Metrics
Metric_of
Metric_of
Metric_of
SJC
Checkout
US
US_Checkout_SJC
Metric_of
Metric_of
Metric_of
#Type #Example
Adaptive GraphGen

19. © 2018 eBay. All rights reserved.19
SJC
Checkout
US
US_Checkout_SJC
Metric_of
Metric_of
Metric_of
#Type #Example
Pop
Flow
Site
Sub-Metrics
Metric_of
Metric_of
Metric_of
Dependency
US_AddCart_SJC
Dependency
(pearson correlation)
Metric_of
Adaptive GraphGen

20. © 2018 eBay. All rights reserved.20
SJC
Checkout
US
US_Checkout_SJC
#Type #Example
Event/
Alert
Pop
Flow
Site
Sub-Metrics
Metric_of
Metric_of
Metric_of
{Criticality,
Confidence,
Value,
Expected range ...}
Duration_P50
Criticality
{Medium,
40, 104ms,
60-80ms…}
{High,
90, 2000 ms,
105-300ms…
}
Dependency
US_AddCart_SJC
{High,
90, 1700 ms,
80-200ms…}
Metric_of
Metric_of
Metric_of
Metric_of
Dependency
Adaptive GraphGen

21. © 2018 eBay. All rights reserved.21
Data
Preprocess
xPack
ML jobs
Service
GraphDB
ML Models
Grano
Algorithm
Adaptive
GraphGen
Grano Quick
Walkthrough

22. © 2018 eBay. All rights reserved.22
SJC
Checkout
US
US_Checkout_SJC
#Type #Example
Event/
Alert
Pop
Flow
Site
Sub-Metrics
Metric_of
Metric_of
Metric_of
{Criticality,
Confidence,
Value,
Expected range ...}
Duration_P50
Criticality
{Medium,
40, 104ms,
60-80ms…}
{High,
90, 2000 ms,
105-300ms…
}
Dependency
US_AddCart_SJC
{High,
90, 1700 ms,
80-200ms…}
Metric_of
Metric_of
Metric_of
Metric_of
Dependency
Assign edge score based on:
1. Alarm Severity (Metrics Dynamics)
2. Alarm Connectivities (at the Dimension)
Grano Algorithms

23. © 2018 eBay. All rights reserved.23
SJC
Checkout
US
US_Checkout_SJC
#Type #Example
Event/
Alert
Pop
Flow
Site
Sub-Metrics
Metric_of
Metric_of
Metric_of
Duration_P50
Criticality
{Medium,
40, 104ms,
60-80ms…}
{High,
90, 2000 ms,
105-300ms…
}
Dependency
US_AddCart_SJC
{High,
90, 1700 ms,
80-200ms…}
Metric_of
Metric_of
Metric_of
Metric_of
Dependency
Other Alerts
Alarm Criticality (e.g.
Latency vs CPU)
Edge Scores
{Criticality,
Confidence,
Value,
Expected range ...}
Grano Algorithms

24. © 2018 eBay. All rights reserved.24
SJC
Checkout
US
US_Checkout_SJC
#Type #Example
Event/
Alert
Pop
Flow
Site
Sub-Metrics
Metric_of
Metric_of
Metric_of
{Criticality,
Confidence,
Value,
Expected range ...}
Duration_P50
Criticality
{Medium,
40, 104ms,
60-80ms…}
{High,
90, 2000 ms,
105-300ms…
}
Dependency
US_AddCart_SJC
Propagations{High,
90, 1700 ms,
80-200ms…}
Propagation through the topology
from all nodes to their connected
ones.
Grano Algorithms

25. © 2018 eBay. All rights reserved.25
Data
Preprocess
xPack
ML jobs
Service
GraphDB
Knowledge
APIs
UI
Slack/Email
Alerts
Grano Quick
Walkthrough
ML Models
Grano
Algorithm
Adaptive
GraphGen

26. © 2018 eBay. All rights reserved.26
Grano UI

27. Envoy enables us for better Observability , Scale , TLS Termination , TCP Congestion BBR and
Routing.
Today, we are able to serve experiences(closer and faster) to users with higher ATB powered by
our detection.
We should leverage ML graph, and knowledge engineering, rather than a relying on one
method/metric (single point failure).
Graph is powerful to understand, connect, and make sense out of complex heterogeneous data
(e.g., anomalies, metrics, and system event).
© 2018 eBay. All rights reserved.27
Takeaway

28. Thank
You

29. L4 Architecture
● IPVS based dataplane
● custom consistent hashing ko
● k8s driven control plane
● k8s native deployment
● horizontally scalable
● DSR
● BGP

30. Anomaly Detection
© 2018 eBay. All rights reserved.30
Bucket Span + Data History + Exponential Smoothing
m1
--------- d1 ( primary )
--------- d2 ( secondary / influencers )