Downloaded 59 times
Uploaded bySergey Fedorov
Monitorama 2015 Monitoring OpenConnect CDN
At Netflix we are building Content Delivery Network called OpenConnect to power the traffic from Netflix customers (that currently takes up to 36.5% of peak internet traffic in the US). Currently the network consists of thousands of caches spread around the world and we are actively deploying more as Netflix is adding new customers and coming into new markets. Apparently, monitoring is important part of the our work day as we operate and grow the system, make changes to the network and the software powering the caches to make sure Netflix customers are not affected. While we follow 'testing in production' development style, we don't have 24/7 NOC and the whole network is maintained by relatively small operations team. Given the size of the system we have something failing all the time, but the network is resilient to small failures. Therefore, while we want to track all issues, not all of them are equally urgent. Given specifics of the problem domain we decided to build our own monitoring system, optimized for our environment and providing: * Integration with different metric sources to get monitoring signals * Programmable API for automated tools to communicate with the monitoring system * Prioritization of issues * Aggregation of metrics per logical groups representing structure of the monitored system * UI elements providing OPS with control over visualization of data, issues troubleshooting and triage While currently our monitoring system is targeted for our problem domain we believe that our experience in building our monitoring tools will benefit the community and can be adapted to any distributed system. Main topics: * The concept of stateful monitoring and alerting based on state changes * Issues aggregation and prioritization * Building UI that turns your monitoring system into collaborative tool for ops to detect, triage and troubleshoot issues * Lessons learned
![Stream Processing Inside Librato [Monitorama PDX 2015]](https://cdn.slidesharecdn.com/ss_thumbnails/streamprocessingpdf-june2015-150618174253-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Monitorama 2015 Monitoring OpenConnect CDN
- 1. Monitoring OpenConnect CDN SergeyFedorov, Netflix Monitorama 2015 Sergey Fedorov, Netflix, Monitorama 2015
- 2. What is OpenConnect 36.5% USdownstream traffic * * 2015 Sandvine reportSergey Fedorov, Netflix, Monitorama 2015
- 3. OpenConnect Cache Appliance Space/Poweroptimized 10/40Gbs network interface FreeBSD OS NGinx server Bird routing proxy Gizmodo, “This box can hold an entire Netflix” http://gizmodo.com/this-box-can-hold-an-entire-netflix-1592590450 Sergey Fedorov, Netflix, Monitorama 2015
- 4. Network Transit Internet Exchange ISP embedded SergeyFedorov, Netflix, Monitorama 2015
- 5. Sergey Fedorov, Netflix,Monitorama 2015 Intelligent clients
- 6. Control Plane end-user contentrequest router client location network conditions server utilization content distribution Sergey Fedorov, Netflix, Monitorama 2015
- 7. Who we are SergeyFedorov Stefan Praszalowicz Sergey Fedorov, Netflix, Monitorama 2015
- 8.
- 9. Testing in prod* Networkchanges Firmware deployments App pushes Updating content ... Sergey Fedorov, Netflix, Monitorama 2015
- 10. Sergey Fedorov, Netflix,Monitorama 2015 CachesClients Control Plane Micro services Network Capacity Config Content Telemetry (Atlas) Logs (ElasticSearch) Data sources METRICS
- 11. Something breaks allthe time
- 13. Big problems startsmall
- 14. Context matters Sergey Fedorov,Netflix, Monitorama 2015
- 15. Sergey Fedorov, Netflix,Monitorama 2015
- 16.
- 19.
- 20.
- 21. Netflix Clients Caches Network ConfigDatasources ...... ... Sergey Fedorov, Netflix, Monitorama 2015
- 22. Netflix Clients Caches Network ConfigDatasources ...... ... Orchestration Data processing stream processorspollers Sergey Fedorov, Netflix, Monitorama 2015
- 23. FSMState processing Netflix Clients Caches NetworkConfigData sources ...... ... Orchestration Data processing stream processorspollers Sergey Fedorov, Netflix, Monitorama 2015
- 24. MAINTENANCE start fixing endfixing threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 25. start fixing endfixing action: ok from: cpu threshold=75% MAINTENANCE Sergey Fedorov, Netflix, Monitorama 2015
- 26. start fixing endfixing action: ok from: cpu threshold=75% MAINTENANCE Sergey Fedorov, Netflix, Monitorama 2015
- 27. start fixing endfixing action: ok from: cpu threshold=75% MAINTENANCE Sergey Fedorov, Netflix, Monitorama 2015
- 28. start fixing endfixing action: ok from: cpu threshold=75% MAINTENANCE Sergey Fedorov, Netflix, Monitorama 2015
- 29. MAINTENANCE start fixing endfixing action: silence from: config threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 30. MAINTENANCE start fixing endfixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 31. MAINTENANCE start fixing endfixing action: silence from: config threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 32. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 33. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 34. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 35. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 36. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 37. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 38. MAINTENANCE start fixing endfixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 39. MAINTENANCE start fixing endfixing action: unsilence from: config threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 40. MAINTENANCE start fixing endfixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 41. MAINTENANCE start fixing endfixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 42. MAINTENANCE start fixing endfixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 43. MAINTENANCE start fixing endfixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 44. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 45. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 46. MAINTENANCE start fixing endfixing action: start_fix from: user threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 47. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 48. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 49. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 50. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 51. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 52. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 53. MAINTENANCE start fixing endfixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 54. MAINTENANCE start fixing endfixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 55. MAINTENANCE start fixing endfixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 56. MAINTENANCE start fixing endfixing action: end_fix from: user threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 57. MAINTENANCE start fixing endfixing threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 58. FSMState processing Netflix Clients Caches NetworkConfigData sources ...... ... Orchestration Data processing stream processorspollers Sergey Fedorov, Netflix, Monitorama 2015
- 59. FSMState processing Netflix Clients Caches NetworkConfigData sources ...... ... Orchestration Data processing stream processorspollers Events processing Event handlers
- 60. STATE TRANSITION EVENT ● OLDSTATE ● NEW STATE ● Input action ● Metric name ● Action metadata ○ metric value ○ comments ○ tags ○ timestamp ○ ... Event handlers Triggers an event Event handlers RULES Sergey Fedorov, Netflix, Monitorama 2015
- 61. Sergey Fedorov, Netflix,Monitorama 2015 Events priority Escalation Do Never Notice Warning Critical Severity Info Do Next Do Last Do Now 0 1 2 3
- 62. Notice Warning Critical Severity Info 0 1 23Escalation Notice Warning Critical Severity Info 0 1 2 3 Notifications Sergey Fedorov, Netflix, Monitorama 2015
- 63. FSMState processing Netflix Clients Caches NetworkConfigData sources ...... ... Orchestration Data processing stream processorspollers Events processing Event handlers
- 64. Aggregation C Cluster Cache state =aggregation of states of its metrics Cluster state = aggregation of states of its caches OK all OK DEGRADED some BROKEN or DEGRADED BROKEN most BROKEN All caches are OK → cluster state is OK Sergey Fedorov, Netflix, Monitorama 2015
- 65. Aggregation C Cluster OK allOK DEGRADED some BROKEN or DEGRADED BROKEN most BROKEN 2/12 caches are BROKEN → cluster state is DEGRADED Sergey Fedorov, Netflix, Monitorama 2015
- 66. Aggregation C Cluster OK allOK DEGRADED some BROKEN or DEGRADED BROKEN most BROKEN 7/12 caches are BROKEN → cluster state is BROKEN Sergey Fedorov, Netflix, Monitorama 2015
- 67. FSMState processing Netflix Clients Caches NetworkConfigData sources ...... ... Orchestration Data processing stream processorspollers Events processing Event handlers
- 68. Challenges Setup Sergey Fedorov, Netflix,Monitorama 2015
- 69. Challenges Setup Predefined groupings Sergey Fedorov,Netflix, Monitorama 2015
- 70. Challenges Setup Predefined groupings UI Sergey Fedorov,Netflix, Monitorama 2015
- 71. Challenges Setup Predefined groupings UI Issues correlation SergeyFedorov, Netflix, Monitorama 2015
- 72. Challenges Setup Predefined groupings UI Issues correlation Failureforecasting Sergey Fedorov, Netflix, Monitorama 2015
- 73. Challenges Setup Predefined groupings UI Issues correlation Failureforecasting OSS Sergey Fedorov, Netflix, Monitorama 2015
- 74.
- 75.