A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix
•
3 likes•1,054 views
Roy Rapoport will discuss the framework Insight Engineering at Netflix uses to think about the real-time operational insight space, the capabilities that any successful organization will eventually need in that space, and what Netflix has done in pursuit of addressing these needs at extremely large scale.
Recommended
Recommended
More Related Content
Similar to A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix
Similar to A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix (20)
More from Outlyer
More from Outlyer (20)
Recently uploaded
Recently uploaded (20)
A Holistic View of Operational Capabilities—Roy Rapoport, Insight Engineering at Netflix
- 1. (one way to think of) Operational Insight Roy Rapoport @royrapoport or rsr@netflix.com March 30, 2017
- 2. Insight Engineering @ Netflix • 17 + 3 people • Two groups • Largest Single Cloud Spend Item Real-Time Operational Insight
- 3. What does that mean?
- 4. What does that mean?
- 5. What does that mean?
- 6. What does that mean?
- 7. What does it all mean?
- 9. John Boyd
- 10. Observe Orient Decide Act OODA “This approach favors agility over raw power in dealing with human opponents in any endeavor” - Wikipedia
- 11. This Is What We Do
- 15. Observe Orient Decide Act OODA Telemetry platforms Data Viz Graphs Dashboards Alerting Decision-Output Systems
- 16. Observe Orient Decide Act OODA Telemetry platforms Data Viz Graphs Dashboards Alerting Decision-Output Systems
- 17. Observe Orient Decide Act OODA Telemetry platforms Data Viz Graphs Dashboards Alerting Decision-Output Systems Remediation
- 18. Observe Orient Decide Act OODA Telemetry platforms Data Viz Graphs Dashboards Alerting Decision-Output Systems Remediation
- 19. OODA* AT
- 20. Observe • Atlas • Time Series Database • Up to ~3B time series • 6H, 3D, 18D tiers • Indefinite Retention • Custom • Expensive
- 21. Observe • Chronos • What the Hell Changed? • Change Reporting System • Entirely Automated • ~18 months* • REST on top of ElasticSearch
- 23. Observe • Salp • Based on Dapper • Distributed Trace • Dependency Discovery
- 24. Orient • Already Showed Some • Lumen: Extensible Dashboards • JSON Construction • Surprising Use Cases • Easily Extensible • Data Sources • Visualizations
- 26. Decide • Alerting: Simple • Alerting: Not Simple sps_all,nf.region,(,us-east-1,),:in,name,COUNTER-playback-action-APIPlaybackStartAction,:eq, :and,isSupplementalVideo,false,:eq,:and,operation,onComplete,:eq,:and,device.operationalName, streaming_stick,,:eq,:and,:sum,:set,sps_1h_offset,sps_all,:get,1h,:offset,:set,entering_trough,sps_1h_ offset,:get,0.95,:mul,sps_all,:get,:gt,:set,smoothed,sps_all,:get,10,0.1,0.02,:des,:set,low_volume, smoothed,:get,-0.005,:mul,0.1,:add,:set,mid_volume,smoothed,:get,-0.00125,:mul,0.1,:add,:set,base, 0.06,:set,min_pct,1,smoothed,:get,20,:lt,low_volume,:get,:mul,smoothed,:get,80,:lt,mid_volume,:get, :mul,:add,entering_trough,:get,0.05,:mul,:add,base,:get,:add,:sub,10,0.1,0.02,:des,:set,sps_all,: get,min_pct,:get,smoothed,:get,:mul,:lt,10,:rolling-count,5,:ge,$device.operationalName Alert Trigger, :legend
- 27. Decide • Alerting: Simple • Alerting: Not Simple • Kepler: Outlier Detection
- 28. Decide • Alerting: Simple • Alerting: Not Simple • Kepler: Outlier Detection
- 29. Decide • Alerting: Simple • Alerting: Not Simple • Kepler: Outlier Detection • Automated Canary Analysis
- 30. Decide • Alerting: Simple • Alerting: Not Simple • Kepler: Outlier Detection • Automated Canary Analysis • Active: • UVAD • MVAD
- 31. Act
- 32. Act
- 33. Act
- 34. Observe Orient Decide Act OODA Telemetry platforms Data Viz Graphs Dashboards Alerting Outlier Detection Anomaly Detection Canary Analysis Remediation
- 35. ?