Network Infrastructure Monitoring @ LinkedIn
•
2 likes•286 views
How does LinkedIn monitor its network infrastructure. Slides from the talk: Infrastructure Engineering @Scale, Meetup (https://www.meetup.com/Infrastructure-Engineering-Scale/events/243011551/) 4th Nov 2017 Bangalore, IN
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Network Infrastructure Monitoring @ LinkedIn
Similar to Network Infrastructure Monitoring @ LinkedIn (20)
Recently uploaded
Recently uploaded (20)
Network Infrastructure Monitoring @ LinkedIn
- 1. Network Infrastructure Monitoring Production Infrastructure Engineering Ashish Gite Infrastructure Development @Linkedin
- 2. Agenda Overview of LinkedIn Scale Tooling Principles Monitoring Solutions - Present Monitoring Solutions - Future Q&A
- 3. Overview of LinkedIn Scale
- 4. Member Engagement Numbers • 530M+ Members • 11M+ Open job listings • 18M+ Companies listed • 29K+ Schools listed • 50K+ Skills listed • 21B+ Sessions in 2017 (20+% QoQ) Global Site Stats
- 5. Production Infrastructure Numbers • 4 Data centers • 18 PoP • 150K+ Hosts • 43 PB+ Local Storage • 14 PB+ Central Storage • 10K+ Network switches • 400K+ Network interfaces Global Infra Stats PIE
- 7. DataCenter Architecture Overview • Leaf & Spine • Parallel Fabrics • ToR → Hosts
- 9. • Build vs Buy - Control our destiny • Integration with rest of infrastructure ecosystem components • Leverage existing internal platforms/tools • Adopt/contribute to open source Principles PIE Tooling
- 10. Ecosystem
- 11. Monitoring Solutions - Present
- 12. Network Monitoring • NetMon SNMP Syslog AutoAlerts Correlation Incidents • inDC BBR view IntraDC view • inFlow Flow sampling Capacity planning
- 13. • SNMP based metrics poller • Metrics dashboards • Syslog listener • Alerts correlation engine • Incident dashboard UI NetMon PIE Network Monitoring
- 14. Metrics Dashboards • inGraphs, AutoAlerts • 200+ OIDs across vendors, device classes • Device classes - Routers (Edge, Core), Switches, FW, LB • Traffic (IPv4, IPv6), Errors, Speed, OperStatus, CPU, Mem, Temp, Load, CoS, MPLS, RPM, LLDP • ~2M+ metrics
- 15. From Noise to Signal • Alert correlation & Incident dashboard • Correlation - topology, maintenance schedule
- 16. Incident Dashboard • Alerts → Incident • InfluxDB for reporting (Open source, time series database)
- 18. Network Monitoring • NetMon SNMP Syslog AutoAlerts Correlation Incidents • inDC BBR view IntraDC view • inFlow Flow sampling Capacity planning
- 19. • Backbone routing views • IntraDC topology views • Partner maintenance schedules inDC PIE DataCenter Network Visualization
- 20. inDC - BBR view
- 21. inDC - IntraDC view
- 22. Network Monitoring • NetMon SNMP Syslog AutoAlerts Correlation Incidents • inDC BBR view IntraDC view • inFlow Flow sampling Capacity planning
- 23. • netflow, sflow based packet sampling • Near real-time traffic analysis • Capacity planning • inFlow Dashboard UI inFlow PIE Network Flow
- 25. Monitoring Solutions - Future
- 26. • Telemetry based metrics collection using Kafka (Falco) • Centralized Syslog • ML based correlation • Nurse auto remediation platform NextGen PIE Tooling Way forward
- 27. Falco • Disaggregated switching platform • Programmable data center fabric • LNOS: LinkedIn Network Operating System • Kafka based Telemetry • Nurse based self-healing
- 28. Nurse • Event driven, Automation platform • Workflow execution, Data gathering and Remediation • Integration with ecosystem • Salt based
- 29. References • https://engineering.linkedin.com/blog/2016/02/falco-decoupling-switching-hardware-and- software-pigeon • https://engineering.linkedin.com/sre/introducing-nurse-auto-remediation-linkedin • https://engineering.linkedin.com/blog/2017/06/open-sourcing-iris-and-oncall • https://engineering.linkedin.com/blog/2016/03/project-altair--the-evolution-of-linkedins- data-center-network
- 30. Q & A