Successfully reported this slideshow.
Your SlideShare is downloading. ×

How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitigate Risk | Allison Wang | Robinhood

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 22 Ad

How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitigate Risk | Allison Wang | Robinhood

Download to read offline

Robinhood is democratizing the financial systems by offering commission-free investing and trading with the use of your phone or desktop. As exciting as that sounds to the outside world, internally, the team at Robinhood must understand the different risk vectors and build engineering solutions to mitigate these risks. In this talk, Allison will talk about how they build a real-time risk monitoring system with InfluxDB and Faust, an open-source Python stream processing library. She will review the architecture behind the system which will involve both the time series anomaly detection part (InfluxDB) and the real-time stream processing part (Faust/Kafka).

Robinhood is democratizing the financial systems by offering commission-free investing and trading with the use of your phone or desktop. As exciting as that sounds to the outside world, internally, the team at Robinhood must understand the different risk vectors and build engineering solutions to mitigate these risks. In this talk, Allison will talk about how they build a real-time risk monitoring system with InfluxDB and Faust, an open-source Python stream processing library. She will review the architecture behind the system which will involve both the time series anomaly detection part (InfluxDB) and the real-time stream processing part (Faust/Kafka).

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitigate Risk | Allison Wang | Robinhood (20)

Advertisement

More from InfluxData (20)

Recently uploaded (20)

Advertisement

How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitigate Risk | Allison Wang | Robinhood

  1. 1. Real-time Anomaly Detection with InfluxDB Allison Wang - Software Engineer, Robinhood
  2. 2. Overview ● Why anomaly detection ● Algorithm for anomaly detection ● Choosing a time-series database ● Building an end-to-end anomaly detection system
  3. 3. We want intelligent alerts on critical metrics without human staring at the dashboards 24/7 Why Building An Anomaly Detection System
  4. 4. ● Work well with simple time-series ● Fail to account for seasonality and trend Threshold-based Alerting
  5. 5. Normal Distribution ● 1σ - 68.27% ● 2σ - 95.45% ● 3σ - 99.73%
  6. 6. Bucketing Data Points ● Aggregate data points in a small time interval, e.g every minute) ● Construct normal distribution for every minute in the day for the past 30 days
  7. 7. Anomaly Detection Check if the incoming data point (aggregated) is in the range (μ - 3σ, μ + 3σ)
  8. 8. System Requirements ● A database for fast time-series data ingestion and aggregation ● A system for querying and computing anomaly in real-time ● Visualization and alerting
  9. 9. Time Series Databases ● InfluxDB ● Prometheus ● Elasticsearch ● OpenTSDB ● Postgres ● ...
  10. 10. ● Lightweight: OpenTSDB ● Schemaless: Postgres ● Allow indexing via a specific field in the data: Prometheus ● Fast data ingestion (write) and aggregation (read): Elasticsearch ● High availability (enterprise version) ● InfluxData Stack (Kapacitor, Chronograf, Telegraf) ● Incredible community Why InfluxDB
  11. 11. Real-time Stream Processing Need a system that can send queries and run anomaly detection algorithm.
  12. 12. Why Faust ● Performant (Asyncio) ● Fault-tolerant (RocksDB) ● Scalable (Kafka)
  13. 13. Combine Faust with InfluxDB
  14. 14. Data Ingestion Logstash Kafka
  15. 15. Visualization ● Grafana ● Chronograf ● ...
  16. 16. Boundary Visualization in Grafana
  17. 17. Alerting Kapacitor Slack
  18. 18. High Level System Architecture
  19. 19. More Anomaly Detection Algorithms ● Kapacitor anomaly detection library ● InfluxDB’s Holt-Winters Function ● scikit-learn, numpy, pandas ● Deep learning, LSTM ● ...
  20. 20. Conclusion ● Lightweight ● Extensible ● Horizontally scalable
  21. 21. Thank You!

×