Recent years have seen an explosion in the volumes of data that modern production environments generate. Making fast educated decisions about production incidents is more challenging than ever. BigPanda's team is passionate about solutions such as anomaly detection that tackle this very challenge.

1. Guide to
Anomaly
Detection
A Practical
for DevOps

2. 2 categories
Anomaly Detection
log analysis metric analysis

3. log analysis
identify suspicious event
patterns in log files

4. 2 categories
Anomaly Detection
log analysis metric analysis

5. metric analysis
identify misbehaving
time-series metrics

6. Why is anomaly detection worth our time?
1
It reveals dangerous patterns
that previously were undetected
The static nature of rule-based and threshold-based alerts
encourages
a) false positives during peak times
b) false negatives during quieter times 2

7. Why is anomaly detection worth our time?
It reveals dangerous patterns
that previously were undetected
12
The static nature of rule-based and threshold-based alerts
encourages
a) false positives during peak times
b) false negatives during quieter times

8. weapons of
mass detection

9. weapons of
mass detection anomaly

10. Anomaly Detective by Prelert
• Product: Anomaly Detective for Splunk
• Pricing: $0-$225 / month (quote-based pricing > 10GB)
• Setup: On premise (OS X, Windows, Linux & SunOS)
• Installation: Easy (with Splunk Enterprise)
• Main Datatype: Log lines

11. Anomaly Detective by Prelert
Highlights:
• Capable of consuming any stream of machine-data
• Can identify rare or unusual messages.
• A robust REST API, which can process almost any data feed
• Offers an out-of-the-box app for Splunk Enterprise
• Extends the Splunk search language with verbs tailored for anomaly
detection

12. Sumo Logic
• Pricing: Quote-based
• Setup: SaaS (+ on-premise data collectors)
• Ease of Installation: Average (deploy Sumo Logic's full solution)
• Main Datatype: Log lines

13. Sumo Logic
Highlights:
• LogReduce: a useful log crunching capability which consolidates
thousands of log lines into just a few items by detecting recurring patterns.
• Sumo Logic scans your historical data to evaluate a baseline of normal
data rates. Then it focuses on the last few minutes and looks for rates
above or below the baseline.
• Anomaly detection will work even if the log lines are not exactly identical.

14. Grok
• Pricing: $219/month for 200 instances & custom metrics
• Setup: Dedicated AWS instance
• Ease of Installation: Easy
• Main Datatype: System Metrics

15. Grok
Highlights:
• Designed to monitor AWS (works with EC2, EBS, ELB, RDS).
• Grok API for custom metrics (it’s fairly easy to process data from statsd).
• Warns you in real time.
• Customizable alerts for email or mobile notifications.
• Grok uses their Android mobile app as their main UI.
• Installation requires a dedicated Grok instance in your cloud environment.

16. Skyline
• Pricing: Open source
• Setup: On-premise
• Ease of Installation: Average (need python, redis and graphite)
• Main Datatype: System Metrics

17. Skyline
Highlights:
• Etsy’s minimalist web UI lists anomalies & visualizes underlying graphs.
• Horizon accepts time-series data via TCP & UDP inputs.
• Stream Graphite metrics into Horizon. Horizon uploads data to a redis
instance where it is processed by Analyzer - a python daemon helping to find
time-series which are behaving abnormally.
• Oculus, the other half of the Kale stack, is a search engine for graphs. Input
one graph then locate other graphs that behave like it. Detect an anomaly
using Skyline, then use Oculus to search for graphs that are suspiciously
correlated to the offending graph.

18. But detecting anomalies
!
is only half the battle...

19. BigPanda + Anomaly Detection
BigPanda uses an algorithmic,
data science approach to
simplify & automate incident
management
!
!
incident
!
management
!
Anomaly Detection

20. Come take a look at what
BigPanda is building!
http://bigpanda.io
Follow us
online!