1.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Improving Adobe Experience Cloud Services Dependability with Machine
Learning
Nicolas Brousse | Director, SRE

2.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Cloud — an all-in-one experience solution
Adobe Experience Cloud brings together all of your marketing
tech in a single place, so you can do everything from
managing your content and delivering email campaigns to
automating your ad buying and measuring your success.
One integrated approach for one seamless experience.
2

3.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 3
Dependable Services

4.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 4
Real Time Decisions

5.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5
Increase In Complexity

6.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 6
Growing Amount Of Data

7.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 7
Core Infrastructure
ML & Data Pipeline
Adobe Advertising Cloud
Enable a Machine Learning Pipeline supported by a
Kubernetes Infrastructure to allow fast iterations and
deployment of new models to production.

8.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8
Adobe Analytics
Build models to provide traffic
prediction and improve capacity
planning efforts.
Forecasting
&
Capacity Planning

9.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9
Adobe Experience Platform
Provide anomaly detection on streamed
live timeseries data to enable auto-
remediation rules based of forecasted
dynamic thresholds.
Anomaly Detection
&
Self-Healing

10.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 10
Risk Classification
&
Failure Prediction
Adobe Experience Cloud
Improve change management processes through
dynamic risk classification of changes to reduce
number of incidents due to changes.

11.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 11
Build
&
Deploy
Measure
&
Plan
Verify
&
Test
Operate
&
Monitor

12.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Replacing Human Intuition with Machine Learning
Toward A Better Notion Of Risk During A Release

13.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adoption of Microservices and
Distributed Systems means:
§ Increased complexity for Developers
§ Increased complexity for Operators
§ Increased of noise in DevOps
communications
13
Complexity
Microservices
&
Distributed Systems
Human Cognitive Limit

14.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Analyze Changes and Likelihood of Leading to An Incident
§ Goal: Reduce Incidents due to Changes With Better Risk Classification
of Changes
§ Exported Data of 123 Incidents due to Change and 40,991 Normal
Changes from January 2017 to June 2018
§ Use of Jupyter Notebooks with Python and scikit-learn, etc.
14

15.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Data Findings
§ Most Changes Leading to an Incident
are self reported as Low Risk
§ Most Changes Leading to an Incident
have a short approval deadline <2H
§ Incident Occurrence Increases
Proportionally to Maintenance
Duration
15

16.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Models Comparison
16
Overall
Accuracy
Incident
Accuracy
Notification
Noise
Logistic Regression 85.79% 76.92% 28%
ANN using MLP 83.09% 94.06% 13.8%
Linear SVM 96.04% 99.26% 4.8%

17.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Future Work
§ Link to Existing Monitoring Systems for real-time feedback loop
§ Link to Tickets System and Git repository for better context analysis
§ Link to Service Discovery for Service Dependency awareness
§ Add checkpoints into CI/CD workflow
17