1.
Real-time Anomaly Detection with
InfluxDB
Allison Wang - Software Engineer, Robinhood

2.
Overview
● Why anomaly detection
● Algorithm for anomaly detection
● Choosing a time-series database
● Building an end-to-end anomaly detection system

3.
We want intelligent alerts on
critical metrics without human
staring at the dashboards 24/7
Why Building An Anomaly Detection System

4.
● Work well with simple time-series
● Fail to account for seasonality and
trend
Threshold-based Alerting

5.
Normal Distribution
● 1σ - 68.27%
● 2σ - 95.45%
● 3σ - 99.73%

6.
Bucketing Data Points
● Aggregate data points in a
small time interval, e.g every
minute)
● Construct normal distribution
for every minute in the day
for the past 30 days

7.
Anomaly Detection
Check if the incoming data point
(aggregated) is in the range
(μ - 3σ, μ + 3σ)

8.
System Requirements
● A database for fast time-series data ingestion and aggregation
● A system for querying and computing anomaly in real-time
● Visualization and alerting

9.
Time Series Databases
● InfluxDB
● Prometheus
● Elasticsearch
● OpenTSDB
● Postgres
● ...

10.
● Lightweight: OpenTSDB
● Schemaless: Postgres
● Allow indexing via a specific field in the data: Prometheus
● Fast data ingestion (write) and aggregation (read): Elasticsearch
● High availability (enterprise version)
● InfluxData Stack (Kapacitor, Chronograf, Telegraf)
● Incredible community
Why InfluxDB

11.
Real-time Stream Processing
Need a system that can send queries and run anomaly detection algorithm.

12.
Why Faust
● Performant (Asyncio)
● Fault-tolerant (RocksDB)
● Scalable (Kafka)

13.
Combine Faust with InfluxDB

14.
Data Ingestion
Logstash
Kafka

15.
Visualization
● Grafana
● Chronograf
● ...

16.
Boundary Visualization in Grafana

17.
Alerting
Kapacitor
Slack

18.
High Level System Architecture

19.
More Anomaly Detection Algorithms
● Kapacitor anomaly detection library
● InfluxDB’s Holt-Winters Function
● scikit-learn, numpy, pandas
● Deep learning, LSTM
● ...

20.
Conclusion
● Lightweight
● Extensible
● Horizontally scalable

21.
Thank You!