Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Dr. Stephen Dodson, Tech Lead Machine Learning,
Elastic
Machine Learning and
the Elastic Stack
2
Overview
•  Background
•  Machine Learning Overview
•  Machine Learning and the Elastic Stack
•  Demo
•  Architecture
Background
•  Me
–  Currently, Tech Lead, Machine Learning @ Elastic
–  Formally, Founder and CTO of Prelert (acquired by ...
4
Machine Learning
•  Algorithms and methods for data driven prediction, decision making, and
modelling1
‒  Learn models f...
5
How is this relevant to the Elastic Stack?
•  Extracting useful, valuable information is hard
Search
Aggregations
Visual...
6
How is this relevant to the Elastic Stack?
•  What if we want to search for:
‒  Has my order rate dropped significantly?...
7
Has my order rate dropped significantly?
8
Has my order rate dropped significantly?
•  Learn models from past
behaviour (training, modelling)
•  Use models to pred...
Demo: Simple Time Series
10
Do my application logs contain unusual messages?
11
Do my application logs contain unusual messages?
Classify unstructured log messages by clustering similar messages
Norm...
Demo: Multiple Data Sources
13
Analytics Outside of Elastic Architecture
Beats
Logstash
Kibana
X-Pack X-Pack
Elasticsearch Prelert analysis node
Data
...
14
Architecture
•  Machine Learning will be part of X-Pack
•  Machine Learning jobs will be automatically distributed acro...
15
Status
•  Demo on Elastic 5.4 available at Elastic{ON} (March 7th 2017)
•  GA shortly after… (ask Sophie!)
•  Focus of ...
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
Elasticsearch 5.0
Next
Download to read offline and view in fullscreen.

3

Share

Download to read offline

Machine Learning and the Elastic Stack

Download to read offline

A talk given by Dr Stephen Dodson, Tech Lead Machine Learning at Elastic at the London Elasticsearch meetup January 2017.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Machine Learning and the Elastic Stack

  1. 1. 1 Dr. Stephen Dodson, Tech Lead Machine Learning, Elastic Machine Learning and the Elastic Stack
  2. 2. 2 Overview •  Background •  Machine Learning Overview •  Machine Learning and the Elastic Stack •  Demo •  Architecture
  3. 3. Background •  Me –  Currently, Tech Lead, Machine Learning @ Elastic –  Formally, Founder and CTO of Prelert (acquired by Elastic September 2016) ‒  Presented overview of Prelert at Elastic London User Group in May 2016 •  Prelert –  VC backed software company, founded 2009 –  Behavioural analytics for machine data based (mainly) on unsupervised machine learning –  100+ customers + OEMs with CA, Bluecoat, NetApp + others ‒  IT Operations, IT Security, Retail analytics, IoT etc..
  4. 4. 4 Machine Learning •  Algorithms and methods for data driven prediction, decision making, and modelling1 ‒  Learn models from past behaviour (training, modelling) ‒  Use models to predict future behaviour (prediction) ‒  Use predictions to make decisions •  Examples ‒  Image Recognition ‒  Language Translation ‒  Anomaly Detection 1Machine Learning Overview, Tommi Jaakkola, MIT
  5. 5. 5 How is this relevant to the Elastic Stack? •  Extracting useful, valuable information is hard Search Aggregations Visualization Machine Learning
  6. 6. 6 How is this relevant to the Elastic Stack? •  What if we want to search for: ‒  Has my order rate dropped significantly? ‒  Do my application logs contain unusual messages? ‒  Are any users behaving unusually? ‒  What transactions are fraudulent? •  Goal of ML at Elastic: Extend the Elastic Stack to allow the user to ask these type of questions and get understandable answers •  Constraints: ‒  Data may be limited: no markup may be available or relevant ‒  Compute resource dedicated to machine learning may be limited ‒  User should not need to be a machine learning expert or data scientist
  7. 7. 7 Has my order rate dropped significantly?
  8. 8. 8 Has my order rate dropped significantly? •  Learn models from past behaviour (training, modelling) •  Use models to predict future behaviour (prediction) •  Use predictions to make decisions Expected value @ 15:05 = 1859 Actual value @ 15:05 = 280 Probability = 0.0000174025
  9. 9. Demo: Simple Time Series
  10. 10. 10 Do my application logs contain unusual messages?
  11. 11. 11 Do my application logs contain unusual messages? Classify unstructured log messages by clustering similar messages NormalLogMessages UnusuallogMessages
  12. 12. Demo: Multiple Data Sources
  13. 13. 13 Analytics Outside of Elastic Architecture Beats Logstash Kibana X-Pack X-Pack Elasticsearch Prelert analysis node Data Kibana Prelert UI •  Issues –  Data Gravity – data from Elasticsearch needs to be sent to Prelert analytics node –  Context – anomalies and data are stored in different data stores and viewed in different Uis –  Scale – Prelert analysis was not easily distributable across nodes –  Resilience – Prelert analysis needed to be restored manually on failover
  14. 14. 14 Architecture •  Machine Learning will be part of X-Pack •  Machine Learning jobs will be automatically distributed across the Elasticsearch cluster •  Machine Learning jobs will be resilient to failover •  Machine Learning results and data can be in the same cluster Beats Logstash Kibana X-Pack X-Pack Elasticsearch Security Alerting Monitoring Reporting Graph Machine LearningICON TBD!! X-Pack
  15. 15. 15 Status •  Demo on Elastic 5.4 available at Elastic{ON} (March 7th 2017) •  GA shortly after… (ask Sophie!) •  Focus of initial ML product is time series analysis in real-time ‒  Metric anomaly detection ‒  Log message classification and anomaly detection ‒  Population analysis (entity profiling) •  Shrink-wrapped configurations on Beats data - full Elastic Stack experience! Beats X-Pack Elasticsearch AlertingMachine LearningICON TBD!! Kibana
  • WalterLee13

    May. 6, 2020
  • smilesmith

    Jul. 30, 2017
  • HaoWang1

    Jun. 20, 2017

A talk given by Dr Stephen Dodson, Tech Lead Machine Learning at Elastic at the London Elasticsearch meetup January 2017.

Views

Total views

4,121

On Slideshare

0

From embeds

0

Number of embeds

2

Actions

Downloads

156

Shares

0

Comments

0

Likes

3

×