SEI workshop paper and keynote

1. eBay INAR -
“Intelligent
Architecture”
Hanzhang Wang, Sanjeev Katariya
Nov 15, 2019
© 2019 eBay. All rights reserved.

2. Agenda
eBay INAR Vision and Challenges
INAR Blueprint and Progress
INAR Example: Grano – Cross Level Anomaly Detection
Existing and Future Collaborations

3. © 2018 eBay. All rights reserved.3
Enables the rapid, frequent and reliable delivery of large, complex applications.
Highly maintainable and testable
Loosely coupled
Independently deployable
Organized around business capabilities
Owned by a small team
Micro-Service Architecture

4. © 2018 eBay. All rights reserved.4
Enables the rapid, frequent and reliable delivery of large, complex applications.
Highly maintainable and testable
Loosely coupled
Independently deployable
Organized around business capabilities
Owned by a small team
Everything seems perfect, until...
Micro-Service Architecture
Erhh….
Hi, can I change
this …?

5. eBay INAR Vision
“To build sustainable service architecture by providing
automated visibility, assessment, and governance
Intelligence.”

6. © 2018 eBay. All rights reserved.6
Open Challenges for Large-Scale Service Ecosystem
Blindness
Limited observability on
architectural
knowledge or issues
Primitiveness
Missing diagnostic, engineering
and run-time automation
Ignorance
Lack of measurability
for service architecture,
or technical debts

7. © 2018 eBay. All rights reserved.7
Limited observability on architectural knowledge or issues
Cannot see…
Architectural antipattern problems
Incorrect / Inappropriate dependencies
Low tier services dependent by the critical/secure services
Detection of invalid cross domain endpoint cyclic dependencies
Architectural evolution corruption problems which cause low engineering
productivity and poor sustainability
Popular software and services evolve frequently and prone to become monolithic
Redundant services and duplicated functionalities
No recommendation to support architecture evolution
Easy access to the cross-domain data
Root cause visibility on cross-domain incidents
The “Blindness”

8. © 2018 eBay. All rights reserved.8
Lack of measurability for service architecture quality, or technical debts
Cannot know…
Metrics Suite
Lack of service’s importance and impact metrics
Lack of service quality (e.g., reliability, maintainability) metrics
Automation to generate team / organization product dependencies and symbiosis
System/Ecosystem visualization enhanced/powered by metrics
Estimation of technical debts
The “Ignorance”

9. © 2018 eBay. All rights reserved.9
Missing real time algorithm-driven automation for diagnostic-ability or productivity
Cannot have…
Algorithmic/Artificial Intelligent IT operations or AI for IT Operations
Anomaly detection for operations
Antipattern detection for evolution
Automated mining software code repository solutions
Lack of automated solution to discover code-level knowledge or to support quality analysis.
Governance in the ecosystem
Service ecosystem evolution governance
Optimization and architectural decision support (e.g., legacy code migration)
The “Primitiveness”

10. Blindness
ENHANCE DEVOPS
Enhance code intelligence
for quality assurance and
engineering productivity
PROVIDE VISIBILITY
Improve productivity and
understandability of
service ecosystem through
knowledge graph
Ignorance
METRIC SUITE
Quantitatively measure
service behavior and
metrics. “If you can’t
measure it, you can’t
improve it”
PATTERN/ANOMALY
DETECTION
Automate service laying,
service ranking, service
classification; enforce
rule-based
Primitiveness
AIOPs
Enable observability,
unified analysis and
proactive incident/root
cause detection and
remediation
SYMBIOSIS & DISCIPLINE
Automated governance over service ecosystem for evolution
The eBay INAR Blueprint
{
}

11. © 2018 eBay. All rights reserved.11
Bring DevOPs Into Connected Graph Visibility
Action
Bring in data into graph to connect,
understand, process and visualize
Result: Graph-based Visibility
We can now see by connecting
cloud-native data, hardware,
people, code and business
What kind of data?
● WIRI (What it really is) and WISB
(What it should be)
● Large-scale data within/across
domains
● Analysis (Pattern, Dynamics and
Behavior)
● Recommendation (Improvement)
{
}

12. © 2018 eBay. All rights reserved.12
Knowledge Graph (Heterogeneous Graph)
Knowledge Graph:
1. Represents knowledge domain.
2. Connects things of different types in a systematic way.
3. Encode knowledge arranged in a network of nodes and edges rather
than tables of rows and columns.
Why graph here?
1. Native method to understand the “Unknown”
2. White box and Visualization
3. The platform and support is ready (e.g. Data, Graph DB, Computation
Power)
4. “Get ready for AI” or “translator of AI” analysis (Pattern)
5. Recommendation/Refactoring

13. © 2018 eBay. All rights reserved.13
An Initial Design..

14. © 2018 eBay. All rights reserved.14
A “Baby” Step..

15. © 2018 eBay. All rights reserved.15
A “Baby” Step...

16. © 2018 eBay. All rights reserved.16
Build Metrics Suite and Understand by AI
Action
Build Metrics on previous
connected graph, to develop and
understand patterns and
anti-patterns in raw data and metrics
Result: Intelligence Data
We transform industrial SE data which
is often noisy, low-quality, and large into
clean / intelligence data

17. © 2018 eBay. All rights reserved.17
Metrics (Data-driven Insights)
Structural Metrics (by dependency)
Traffic Metrics (by traffic)
Mid-Activity (13%)
High-Activity Pools
(6%)
Mid-Activity (8%)Low-Activity (72%)
Essential
Consumption
Inbound
Services
Inbound
Endpoints
Traffic Vol.

18. © 2018 eBay. All rights reserved.18
Metrics (Data-driven Insights)
Essential
Consumption
Automated classification of service layering :
1. Back-End Pools
2. Middle-Tier Pools
3. Customer-Facing Pools (Experience, Public APIs)
Service architecture refactoring support:
1. Classifying external-facing pools to support build “large
team” APIs
2. Impact and importance

19. © 2018 eBay. All rights reserved.19
Our current progress
Enable AIOPs With SEI Applications
Dependency system
Pre/post release
quality analysis
Graph-based distributed
system anomaly detection
Capacity forecasting
and planning
Service popularity metrics
and layering
recommendation
Invalid service
dependency detection
Action
Leveraging
graph-based visibility
and intelligence data to
build applications to
support site operation
and engineering
productivity
Result: Symbiosis
Automated discipline
and governance among
software engineering
services

20. © 2018 eBay. All rights reserved.20
Grano Research Motivation Cross Level Anomaly Detection
Univariate Anomaly by Detections Models
(High False-Positive and Busy)
Visibility
Multi-Variate Anomaly Detection
(Black-Box with Bad Interpretability)
ML Models for Anomaly Detection for the components
Ignorance
Topology Example of eBay distributed data
platform

21. © 2018 eBay. All rights reserved.21
Solution: Grano ML +Graph
Project health signals should be in-line with the lines
ML - First level anomaly detection for
components of distributed data platform.
Consists of three different types of detection
models to handle different anomalies
(clustering-based model, forecast-based
mode, statistical model)
Visibility Ignorance
Graph – Constructing the anomaly graph with different (health) signals
Scoring the significance
of every signal based on
its distribution, and
history frequency
Scoring the system
components based
on their signals
Propagating the
scores through the
topology to enhance
the detection
accuracy

22. © 2018 eBay. All rights reserved.22
Solution: Grano System
GRANO is an end-to-end graph-based
anomaly detection and root cause analysis
system for distributed cloud-native data
platform
Anomaly Detection
Time series anomaly detection on metrics
Adaptive Anomaly Graph
Adaptive analysis on system topology and all
signals to identify the root cause
Applications
Alerting, interactive Graph-UI, and integration
with monitoring systems

23. © 2018 eBay. All rights reserved.23
GRANO: Interactive Graph-based Root Cause Analysis for
Cloud-Native Distributed Data Platform link
Hanzhang Wang; Phuong Nguyen; Jun li, Selcuk Kopru, Gene Zhang,
Sanjeev Katariya, Sami Ben-Romdhane
eBay Inc.
E2E Solution: Grano Applications (In Production)
Intelligent anomaly graph view Automated Root cause analysis
Component-based view
Automated RCA graph
High-level comprehensive monitoring
Low-level RCA

24. © 2018 eBay. All rights reserved.24
University Topics Area
Fudan
University
Industry microservice study
Distributed tracing flow analysis
Drexel
University
Service quality metrics suite
University of
Michigan
Intelligent Software Engineering
Existing INAR Collaborations

25. eBay SEI Vision
“To build sustainable service architecture by providing automated visibility,
assessment, and governance Intelligence.”
Challenge Blindness Ignorance Primitiveness
Progress
DevOPs, quality dashboards,
knowledge graph and etc.
Service popularity and behavior
metrics, pre/post release quality
analysis and etc.
Anomaly Detection, RCA and etc.
Future work
Intelligent visibility for developers Prioritize and quantify technical debt
using service metrics
Auto remediation
Collaboration
Service dependency analysis and
governance
Extend service metrics suite Critical flow and anomaly detection
on distributed tracing graph
Please Contact: hanzwang@ebay.com
Request for Future INAR Collaboration