Machine Learning for detecting anomalies in your data Steve Dodson

1. Dr Stephen Dodson, Machine Learning, Elastic
Machine Learning in the
Elastic Stack

2. 2
Elastic Stack
Beats
Logstash
Kibana
Elasticsearch
X-PackX-Pack X-Pack
Security
Alerting
Monitoring
Reporting
Graph
Machine Learning

3. 3
Extracting useful, valuable information is hard

4. 4
Search
Aggregations
Visualization
Machine Learning
Extracting useful, valuable information is hard

5. Machine Learning1
Algorithms and methods for data driven prediction, decision making, and modelling
Supervised Learning
Prediction based on examples of
correct behavior
1Machine Learning Overview, Tommi Jaakkola, MIT
Unsupervised Learning
No explicit target, only data, goal to
model/discover
Semi-supervised Learning
Supplement limited annotations with unsupervised learning
Active Learning
Learn to query the examples actually needed for learning
Transfer Learning
How to apply what you have learned from A to B
Reinforcement Learning
Learning to act, not just predict; goal
to optimize the consequences of
actions
Other!
…

6. Machine Learning1
Algorithms and methods for data driven prediction, decision making, and modelling
Supervised Learning
Prediction based on examples of
correct behavior
1Machine Learning Overview, Tommi Jaakkola, MIT
Unsupervised Learning
No explicit target, only data, goal to
model/discover
Semi-supervised Learning
Supplement limited annotations with unsupervised learning
Active Learning
Learn to query the examples actually needed for learning
Transfer Learning
How to apply what you have learned from A to B
Reinforcement Learning
Learning to act, not just predict; goal
to optimize the consequences of
actions
Other!
…
Time Series
Anomaly
Detection

7. Machine Learning - Technical Debt
7
• Machine Learning Systems generally involved a large amount of plumbing
and complexity that incur large ongoing maintenance costs
From Google Paper: Sculley, D., et al. "Machine learning - The high-interest credit card of technical debt." (2014).

8. © Elasticsearch
Has my order rate dropped significantly?

9. Has my order rate dropped significantly?
• Learn models from past
behaviour (training, modelling)
• Use models to predict future
behaviour (prediction)
• Use predictions to make
decisions
Expected value @ 15:05 = 1859
Actual value @ 15:05 = 280
Probability = 0.0000174025

10. Demo: Simple Time Series

11. Has my system changed behaviour?
i-5cfd3dcb
...
…
i-f1e94994
i-ece626ff i-ebc323df

12. Has my system changed behaviour?
i-5cfd3dcb i-f1e94994
i-ece626ff i-ebc323df
...
…

13. Demo: Multiple Time Series

14. Do my application logs contain unusual messages?

15. Do my application logs contain unusual messages?
Classify unstructured log messages by clustering similar messages
NormalLogMessages
UnusuallogMessages

16. Demo: Log Messages

17. 17
Ingest, Enrich, Visualize, Analyse, Alert
Elasticsearch
X-pack
Master Nodes (3)
Ingest Nodes (X)
Data Nodes - Hot (X)
Data Nodes - Warm (X)
Beats
Log Files Metrics
Wire Data your(beat)
Filebeat Module
NGINX
Kibana
X-pack
Instances (X)

18. 18
Coming soon - forecasting…and more…

19. Except where otherwise noted, this work is licensed under
http://creativecommons.org/licenses/by-nd/4.0/
Creative Commons and the double C in a circle are
registered trademarks of Creative Commons in the United States and other countries.
Third party marks and brands are the property of their respective holders.
Please attribute Elastic with a link to elastic.co