1.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Intelligent DevOps enabling
Enterprise Agility
Harsha Vardhan
Director - Software Engineering
Optum Global Solutions
The contents in this presentation is prepared by Harsha Vardhan in his personal capacity and to the best of his knowledge. No lia bility
whatsoever for the accuracy and completeness of the above information ishowsoever assumed. The opinions/ contents expressed/
provided in this article are the author'sown and do not reflect the views of the Optum Global Solutions(India) Private Limited
(“company”). Neither company nor the author shall not be held liable for any improper or incorrect use of the information described and/or
contained herein and assumes no responsibility for anyone's use of the information.”

2.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Agenda
DevOps relevance in Digital world
Scaling DevOps
DevOps Systems that Do -> Think -> Learn
Q&A

3.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Kuwait: A Desert on Fire © Sebastião Salgado
5 years to 18 months - it’s all about Agility ++

4.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Business Value driven by DevOps

5.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Raise your bar - Be contextual …
*ApplyingAgile andDevOpsprinciple at scale by Gary Gruver

6.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Raise your bar !!

7.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Automation +
Speed +
Collaboration
Adaptive
Automation +
Predictive outcomes
+ CX
Continuous journey (Continuous Integration, Continues Testing,
Continuous Deployment, Continuous Operations etc.) of DevOps to
become Intelligent i.e. Intelligent Integration, Intelligent Testing, Intelligent
Deployment and Intelligent Operations
Intelligent DevOps

8.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Code
Management
Code Merge Packaging
Build (CI)
Server
Unit Testing
Code
Coverage
App Config Publishing Deploy
Auto Test/
ATDD
Dashboard
/ Reporting
Publishing
Dashboard/
Reporting
Smoke
Testing
System/
Integration
SecurityScan
Dashboard/Reporting
Network
Config
Continuous Testing
Continuous Delivery
Continuous Integration
Non-Functional/
Performance
Dev Tech Lead Tech Lead Architect Dev/Architect Dev/Architect
Architect Architect Dev/ArchitectSDETSDETSDETSDET
Architect Architect Architect Architect Architect ArchitectDev/SDET
Medium High One Time ContinuousLow
Efforts /
Complexity
Regression
1 1 1 1 3 1
2113
3
3 3 3
3 3 3 3 3 3
DevOps Systems that Do

9.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
In the spirit of the “Continuous” philosophy in DevOps, we envision a new
solution approach around “Continuous Insight” where analytical insights
are generated and acted upon continuously (and autonomously) as
processes execute
Incident Data
Log Data
Monitoring data
NPS data
Continuous Insight

10.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
DevOps Systems that Think

11.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Traditional CAB ( Change Advisory Board)
Vs Change Predictors

12.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Current Culture
“You are only as good as your last failure”
..Restrict
.. Inconsistent
..Govern
..Focusing on the Negative
Future
“Balancing risk and velocity & focus on outcomes”
..increased throughput & flexibility
..consistency
..focus on the right changes
..focus on success & learn from
failure
Turning the corner on Change Culture

13.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
How might we leverage machine learning to
identify changes with the highest potential risk for
negative outcomes?

14.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com 1
ITSM
Datamart
Data Ingestion
Completed Changes
(Scheduled Batch)
Model Building,
Training, Re-Training,
and Validation
Additional insights to
supplement model
Trained Model
ET
Data Lake
New change
request
Enhanced Risk
assessment
Machine Learning
Model API
Service Now API
Planned Changes
(real-time)
Web Service
Dashboard updated daily
result
input
Source code
Data Processing / Transformation
NLP
Solution Architecture

15.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
• Performance metrics, several models, pick the best
• Gradient-BoostedTrees implementation  XGBoost
• Ensemble  combines many “weak” learners to create strong learner
TPR
FPR
Algorithm AUC Precision Recall F1
Logistic Regression 0.825 0.162 0.135 0.147
Decision Tree 0.831 0.192 0.176 0.184
Random Forest 0.854 0.213 0.191 0.201
Gradient Boosted Trees 0.906 0.292 0.269 0.281
Neural Network - LSTM 0.848 0.089 0.176 0.142
Test-set performance metrics
Higher risk
Model Training & Selection – The Recipe

16.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Via Machine Learning, we can drive down our cost per incident from Low
Risk Changes.
Scenario :
• We average about 10 incidents from low risk changes per month. Currently the machine
learning is upgrading 10% (1) of those changes to high or critical Risk. This higher risk
profile brings about more review and planning. This in turn leads to more positive
outcomes.
• Making those assumptions, we estimate that we can save approximately $17,000 per year
month or about $204,000 annually from reduction of P1/2 incidents.
Additional Insights

17.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Availability
Traditional approach of uptime vs Raw Error Rate
An availability metric that is an accurate indicator of
true customer experience

18.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
HttpResponse RER Dashboards
RER
Dashboard
Inputs
Response Codes
URI
Elapsed Time
Error Rate measuring isn’t new:
•Companies like Google uses Error Rates as a key part of their Error Budgets for
managing risk and their SRE program
•Apps exceeding their ’budget’ aren’t allowed to release features
• In the simplest form, Raw Error Rate is:
[successful transaction count / total transaction count]
• Successful transaction do not include transactions that exceed the response
SLA
Raw Error Rate Example: (Example: outage@ 2 AM vs 2 PM with a 1 hour outage)
ADTM: 98.9% availability for the day where as
Raw Error Rate: An application that serves 10,000 requests in 24 hours
issues 30 errors over an hour at 2am. 99.7% availability for the day.
Continuous Monitoring using RER

19.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Why enhance monitoring?
Customer Experience Monitoring

20.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Sec + Ops
Proactively mitigation of security risks with
automation and holistic visibility

21.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
AI integration in DevOps yielding useful outcomes in SecOps (e.g.
preventing Denial of Service )
Bot to detect abnormal traffic
Captcha (technique to distinguish between humans and computers)
Preventing DoS

22.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
CX-driven DevOps (or CX-Ops) that requires data analytics and cognitive
approaches (such as natural language processing or NLP) to decode
meaningful insight
Transactional NPS feeding into log monitoring
Monitoring triggers drives Bots / Agents availability for chat / call
Peak Season readiness
DevOps System that learns

23.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Intelligent DevOps through Continuous Insight
Investment
decision
analytics
CX-based
requirements
Code quality
analytics
Test Optimization
& Defect
Prediction
A/B Testing
Log & CX
Analytics
Portfolio backlog
prioritization & value
analysis.
Improved Scheduling
& Capacity
Validated backlog
requirements.
Improved estimates
Predict failures
based on code
commit.
Dynamic code
promotion criteria
Predict defects
Optimized tests
Scenario
identification based
on trends
CX scores
Anomaly detection
Better load
balancing
Capture
Phases/Tools
Details
Capture
Environment
Details
Map Phases/Tools
With
Environments
Create
Pipeline Job
DevOps PitStop Process Flow
Tools Assessment
& Selection
Tool Chain Framework (Set-up the tools using DevOps PitStop)
ExecutionFramework(Run the automated phases using DevOps PitStop)
Intelligent DevOps

24.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Engineering Excellence
Optimize for iteration
speed
Push relentlessly toward
automation
Build the right software
abstractions
Develop a focus on high
code quality withcode
reviews
Maintain a respectful
work environment
Build shared
ownershipof code
Invest in automated
testing
Build a culture of
learning and
continuous
improvement
Hire the best talent
Agile Engineering Practices
Its all about Software Engineering Culture…

25.
@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com@ Agile Network India , All Rights Reserved. www.agilenetworkindia.com
Thank You