Robinhood is democratizing the financial systems by offering commission-free investing and trading with the use of your phone or desktop. As exciting as that sounds to the outside world, internally, the team at Robinhood must understand the different risk vectors and build engineering solutions to mitigate these risks. In this talk, Allison will talk about how they build a real-time risk monitoring system with InfluxDB and Faust, an open-source Python stream processing library. She will review the architecture behind the system which will involve both the time series anomaly detection part (InfluxDB) and the real-time stream processing part (Faust/Kafka).

1. Real-time Anomaly Detection with
InfluxDB
Allison Wang - Software Engineer, Robinhood

3. Overview
● Why anomaly detection
● Algorithm for anomaly detection
● Choosing a time-series database
● Building an end-to-end anomaly detection system

4. We want intelligent alerts on
critical metrics without human
staring at the dashboards 24/7
Why Building An Anomaly Detection System

5. ● Work well with simple time-series
● Fail to account for seasonality and
trend
Threshold-based Alerting

6. Normal Distribution
● 1σ - 68.27%
● 2σ - 95.45%
● 3σ - 99.73%

7. Bucketing Data Points
● Aggregate data points in a
small time interval, e.g every
minute)
● Construct normal distribution
for every minute in the day
for the past 30 days

8. Anomaly Detection
Check if the incoming data point
(aggregated) is in the range
(μ - 3σ, μ + 3σ)

9. System Requirements
● A database for fast time-series data ingestion and aggregation
● A system for querying and computing anomaly in real-time
● Visualization and alerting

10. Time Series Databases
● InfluxDB
● Prometheus
● Elasticsearch
● OpenTSDB
● Postgres
● ...

11. ● Lightweight: OpenTSDB
● Schemaless: Postgres
● Allow indexing via a specific field in the data: Prometheus
● Fast data ingestion (write) and aggregation (read): Elasticsearch
● High availability (enterprise version)
● InfluxData Stack (Kapacitor, Chronograf, Telegraf)
● Incredible community
Why InfluxDB

12. Real-time Stream Processing
Need a system that can send queries and run anomaly detection algorithm.

13. Why Faust
● Performant (Asyncio)
● Fault-tolerant (RocksDB)
● Scalable (Kafka)

14. Combine Faust with InfluxDB

15. Data Ingestion
Logstash
Kafka

16. Visualization
● Grafana
● Chronograf
● ...

17. Boundary Visualization in Grafana

18. Alerting
Kapacitor
Slack

19. High Level System Architecture

20. More Anomaly Detection Algorithms
● Kapacitor anomaly detection library
● InfluxDB’s Holt-Winters Function
● scikit-learn, numpy, pandas
● Deep learning, LSTM
● ...

21. Conclusion
● Lightweight
● Extensible
● Horizontally scalable

22. Thank You!