Classic event, incident, problem and change management are ITSM practices that are getting integrated with DevOps/SRE and ML through a competency known as AIOps. Large streams of data generated through logs, metrics and traces are organized and computed using machine learning algorithms to extract insights on the anomalies of system behavior that could be impacting end-users and business transactions. Businesses cannot afford to see their end-users impacted by those anomalies and therefore would want to proactively predict the likelihood of systems regressing and take corrective action long before any material impact. In this talk, we show the use of simple linear regression and multivariate linear regression techniques to predict the likelihood of system behavior resulting in one or two sigma of standard deviation. We show how to use FOSS tools to predict them using various decision trees that are integrated to high performing streaming platforms like Apache Flink, Apache Beam, Prometheus and Grafana which makes it a lot easier to visualize the various alerts and triage their way back to performing root cause analysis. These high performing systems are also backed by KAFKA for its streaming and distributed computing capabilities by partitioning the data for various staged analysis some of which can be done in parallel and concurrently based on the use cases. We present a fully integrated architecture that helps you realize a commercial AIOps capability without having to license expensive software products. The above open architecture allows you to implement various ML algorithms as needed and its agnostic to programming languages and tools. The talk will combine various techniques with demos and is focused to practicing engineers and developers who are familiar with ML.

1. Effective AIOps using
Open Source Software
...building an AIOps solution in a week.
Murali Kaundinya
Keith Kroculick

2. Suresh Veda Keith Kroculick Sandeep Mehandru
Sarath Parvathaneni Javin Sword Murali Kaundinya

3. Agenda
§ Overview
§ AI Ops Quick Overview
§ Building an Open Source AIOps Platform
§ AIOps Operational Events
§ Legacy Operations Platform Architecture
§ New AIOps Platform “Open” Architecture
§ AIOps Predictive Modeling & Analytics
§ AIOps Data Collection & Visualization

4. Effective AIOps
What is AIOps?
AIOps Quick Overview
• AIOps is the evolution of Operations and uses Artificial Intelligence (AI) and Machine
Learning (ML) technologies to see and predict the health of a company’s mission critical
systems
• AIOps tools harvest data such as logs, alerts, traces and other data from applications, servers
and infrastructure to build datasets for analytics, event correlation and to find relationships
between events
• AIOps technologies can show the graph dependencies across all environments as well as
analyze that data to extract significant events related to slow downs or outages, etc.
• Notifications and alerts can be generated to inform IT staff and predict problems, provide root
causes and recommend solutions

5. Building an Open Source AIOps Platform
Reasons for building a custom solution
• Needed to build a solution to rapidly to solve our operational issues with our mission critical
systems
• A solution was tailored to solve the company’s specific “Ops” related problems
• Almost all COTS AIOps solutions need to be tailored and modified regardless of vendor
• Same amount of work as customizing an COTS AIOps products
• Components can be modified or extended replaced with other solutions
• Maintain (own) the source code and allow people contribute to it
Custom AI Ops Solution vs Vendor Solution

6. AIOps & Open Source Technologies
Technologies used to rapidly build our AIOps Solution
Technologies
• Apache Flink Distributed Stream Processing Framework
• Python Keras Deep Learning Framework
• MySQL Database
• Grafana Observability Platform
• Prometheus Monitoring & Time Series Database (Optional)

7. AIOps & Operational Events and Data
Operational Events & Data from Applications, Web Servers, Network Devices & Hardware
AIOps platforms ingest “Ops” data from applications, web servers, network devices and
hardware
• Logs
• Traces
• Alerts
• Configuration Management
• Reference & Lookup Details
• Summarized Metrics

8. Legacy Operations Platform Architecture
Point to Point Operations (Logging & Monitoring) Architecture
Monitoring Agents
Splunk
Application
Splunk
IBM Tivoli
Netcool
IBM
Netcool
Monitoring
Systems
Other Monitoring
Agents
Remedy
Alert Incident Ticket
Ticket
Logs
Monitoring
• Current Eventing State
Problem, Incident & Ticket
• Complex “Point to Point” monitoring &
eventing are replicated 1000s of times
• Tickets may take several
days to resolve due to
limited correlation between
systems
• Logs are sent to tools
like Splunk with limited
logic on what gets
ingested
• Continuing growth of
applications and monitoring
make this difficult to support
• Multiple monitoring and
logging agents are installed
for an application depending
on application type
Application
Monitoring
Application Landscape

9. New AIOps Platform “Open”
Architecture
Building An AIOps Solution using Apache Flink, Python Keras & Grafana

10. Apache Flink Job
AIOps Event Correlation
Apache Flink and Event Correlation
Example of “Ops” Event(s) Correlation & Logic
Combined “Ops”
Event Dataset
App Logs
Server Logs
Alerts
Configuration
Management
ü AppName
ü AppLogLevel
ü AppLogMessage
ü ServerLogMessage
ü HostType
ü Hostname
ü Process
ü Datacenter
ü IP Address
ü Timestamp
ü AlertErrorMessage
ü Actions
ü ConfigurationItems
AppName, LogLevel, LogMessage, Hostname, IPAddress, Timestamp
AlertErrorMessage, Actions, Hostname, IPAddress, Timestamp
ConfigurationItems, Hostname, IPAddress
ServerLogMessage, LogLevel, HostType, Process, Datacenter, Hostname, IPAddress, Timestamp
Operational
Events
Common Data Sources
**Events can be correlated (joined) on common fields like IP Address or Hostname and Timestamps to create a unified
dataset

11. AIOps Predictive Modeling & Analytics
Correlation & Predictive Analytics
• Events from applications, servers & infrastructure are combined to create a “view” of its
health of the platform at any point in time (i.e. graph of events)
• Our “event” sources (application & server logs, alerts , traces) are combined and transformed
into individual datasets that can be used in ML models (e.g. training, validation, test)
• We use Apache Flink stateful functions with PyFlink and the Python Keras TensorFlow
libraries to apply ML models to process our “Ops” datasets
• We start with a simple Linear Regression approach to modeling the relationships between
applications, middleware, servers based on the events we ingest
• Linear Regression is the a simple starting point for predicting behaviors
Apache Flink with AIOps & Machine Learning

12. AIOps Predictive Modeling and Analytics
Apache Flink & Machine Learning
Using Python Keras & Linear Regression Logic in our Apache Flink Application
Ops Events
Training Dataset
Test Dataset
Ops Events
Validation Dataset
Ops Events Final
Dataset
Using Supervised Learning Models for ML
4. Compare final dataset results
to sample test dataset
2. Compare the training dataset to validation set.
Validation set contains values with expected ranges
3. Tune training dataset models based
on previous results
5. Repeat the process as necessary to harden your models
Combined “Ops”
Event Dataset
1. Our combined dataset
is used to create our
training dataset, validation
dataset and test dataset
Apache Flink Job
Python AI ML Step

13. AIOps Data Collection and Visualization
Grafana & MySQL
• Grafana provides dashboarding, visualization and alerting capabilities for our AIOps platform
• Data produced by our Apache Flink jobs i.e. summarized application & server ”Ops” event
details as well as our Python Keras ML outputs were loaded into a simple MySQL database
• Aggregated Metrics were exposed to Grafana using common SQL VIEWs and SQL idioms i.e.
GROUP BY, COUNT(*), MAX, AVG with Grafana’s MySQL connector
• The database used a very simple schema for our “Ops” event data to allow us to create
queries to show relationships between applications, app components, servers, etc. based on
the the details we ingested from our Flink job
• Grafana allowed us to quickly use various visualizations to display our data using histograms,
pie charts, scatter plots
AIOps Dashboards & Reporting

14. AIOps Data Collection and Visualization
Final AIOps Dashboard with Grafana

15. AIOps Data Collection and Visualization
Grafana with Prometheus Time Series Database
• MySQL was used to house our data for
our AI Ops solution due to its ease of use
and availability
• A time series database is a better fit for
reporting on operational events i.e.
Prometheus
• Aggregated Metrics were exposed to
Grafana using common SQL VIEWs and
SQL idioms i.e. GROUP BY, COUNT(*),
MAX, AVG with Grafana’s MySQL
connector
• Apache Flink can write directly to
Prometheus using via Prometheus
Reporter Library
• Prometheus also supports a native
MySQL exporter which can use our
existing MySQL database and not require
us to change our application logic

16. In closing….
AIOps Overview
• Legacy Operations vs. a new AIOps Solution
• Building our Solution with Open Source Tools, Frameworks and Libaries
• Using Event Correlation & Machine Learning
• Building Dashboards and Reporting
Thanks!
• Thanks to the Databricks team for providing us the opportunity to speak about AIOps!
Discusssion Summary & Thanks
CommentsQuestions:
Murali Kaundinya Murali.Kaundinya@gmail.com
Keith Kroculick kkroculick@gmail.com

17. Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.