Uploaded byAPNIC
The Age of Data-Driven Network Operations
This document discusses the need for data-driven network operations. It begins by noting how difficult it can be to manage infrastructure using the many disparate operational tools currently available. It then argues that taking a data-driven approach by collecting, fusing, storing and analyzing network data can help network operators better understand issues, plan infrastructure changes, and provide insights for both technical and business stakeholders. Specific use cases mentioned include traffic debugging, anomaly detection, network planning, security analytics and performance analysis. The challenges of obtaining and working with network data at scale are also addressed.
More Related Content
Similar to The Age of Data-Driven Network Operations
![Automating ISP Networks Using Ansible and IPAM as a Source of Truth [SoT]](https://cdn.slidesharecdn.com/ss_thumbnails/automatingispnetworksusingansibleandipamasasourceoftruthsot-v25-1-251124105117-d7d4ca24-thumbnail.jpg?width=640&height=640&fit=bounds)
The Age of Data-Driven Network Operations
- 1.
- 2.
- 3.
- 4.
- 5. But all the views are disparate! How many operational tools+scripts do you run? Active Testing (ping/traceroute) Flow Tools APMLogging BI Metric (App/SNMP) BGP Hijack detection NPM Config Management Policy Analysis Event Correlation Routing Analytics Forensics Traffic Engineering Threat Intelligence And how many instances of each? 5
- 6.
- 7.
- 8. The network is key to delivering revenue Applications generate traffic… But networks deliver it! So why is the network view left out of cross-enterprise visibility stacks? 1) A bit of a chicken-and-the-egg problem. 2) It’s hard to get some network concepts without being hands-on. 3)General lack of vendor innovation from 2003-2013. 4) Immaturity of backend and distributed systems methods. 8
- 9. Why the limitations like… • Not scaling to handle large amounts of data (space/IO/CPU limited). • Not understanding network concepts (classic BI tools). •Limited scope (data source, functionality, target users). • Typically storing only aggregates or pre-filtered data. • Very limited fusing, only 1 or 2 data types per tool. • Limited dimensions and filtering depth, often slow. • Needing lots of tuning and configuration. 9
- 10. Most tools are pre-big data architectures! But with a more modern approach, it’s possible to get: • Large scale, billions of records per hour, no aggregation, complex fusing. • Distributed micro-service architecture. Can scale very “wide”. ++Hardware. •Speed with granularity. Real-time ingest and < 5s queries. • Big challenge is adding fusion, network savvy-ness and speed of back-ends. • Modern data bus (Kafka) and streaming analytics options have gone from 0 to great in the last 5 years. • Whether for DIY, or from a new wave of vendor options. 10
- 11.
- 12. And yes - DIY is hard… 12 Required areasof expertise (because every presentation needs a Vin diagram) Distributed systems engineersNetwork Engineers SREs Low-level Network developers Resilience / Reliability Geo-distributed ingest Flow friendly data-store BGP Daemon Flow inspection & conversion Network protocols hacking Make all of the above work reliably Train all the other teams on the involved network protocols and their usage Unicorn
- 13. But systems likethis are no longer (as) exotic. Ingest & Fusion layer Storage Layer Query Layer Each layer has separate and different scaling characteristics Query engine and UI Query interfaces SQL WWW REST data sources clients SELECT flow FROM router WHERE … >_
- 14.
- 15.
- 16.
- 17. Data-driven operations + business use cases • Network Planning • Peering Analytics and Abuse •Congestion detection • Is it the network? • Where on the network? • Proactive alerting • Distributed DDoS Detection • What Changed Post Deploy? • Security and Breach Detection • Cost Analytics • Revenue Identification (New + @ Risk) • Enabling Internal Groups 17
- 18. Key network operator requirements 18 Key requirements for modern Data-Driven Network Operations: • No data aggregation or pre-filtering. • Correlation (fusing) between data types. •Full resolution searchable and stored for months. • FAST. Less than 10s for results. Cannot wait minutes while spelunking. • Network-savvy UIs and APIs (understanding routing and prefixes). • Detect anomalies. Should not have to watch graphs manually. • Data and alerts available across the company. • “0” to usable in minutes to weeks, not months to years.
- 19.
- 20.
- 21. TCP stats data/ app specific data Where to find this data ? Flow data NetFlow, SFlow, IPFIX SNMP, Streaming telemetry Sys/Event logs TACACS & Syslog App Server, Logs, Metrics BGP, IGP Path info NETWORK + + + = Combinatorially Useful! + Router Router PCAP agent +User tags, Threat Intel, SDN Control, DNS, ping/trace
- 22. A broader view of “NetFlow” You can ALSO get performance data from the infrastructure: • Queue Depth • Retransmits per flow •TCP latency • Application Latency From: • Host software (nProbe) • Sensors / Taps • Webserver logs (Nginx) • Cisco AVC supported routers 22
- 23.
- 24. Fusing data for richer traffic analytics Flow or BGP or SNMP or DNS or logs alone are not enough. This becomes much richer when combined with: • Performance and layer 7 information • BGP attributes •Geography • Tags (rack, department, customer…) • Config changes and software versions • Threat intelligence and known-bad IPs • Fusing should be near real-time, performed at ingest and data specific 24
- 25. DATA FUSION Decoder Modules Mem Table sNetFlow v5 NetFlowv9 IPFIX BGP RIB Custom Tags SNMP Poller BGP Daemons Enrichment DB DATA FUSION Geo ←→ IP ASN ←→ IP SFlow ROUTER TRAFFIC-SAVVY DATASTORE Single flow fused row sent to storage PCAP PCAP agent proxy
- 26.
- 27.
- 28.
- 29. Fusing data for richer traffic analytics Data in a “lake” is not useful! A modern data-driven network operations system should have: • A flexible and spelunk-able UI • Proactive alerting with links to detailed history and trends •Dashboards that instantly link to detailed history and exploration • Complete API availability for bi-directional integration 29
- 30.
- 31.
- 32.
- 33.
- 34.
- 35. Use case: Network planning Flow-based traffic + BGP can be used to help show: • Path, neighbor, transit, origin, and country of traffic. • Strategic peering and transit changes that can improve perfand costs. • Potential new peers and locations to peer. • Evaluate the potential of new peering exchanges or facilities. • Transit relationships that are of high or little value. • Understand ROI before extending backbone links or capacity. 35
- 36.
- 37. Use case: Network security analytics Flow-based traffic +/- threat intelligence can show: • Compromised servers, desktops, and IoTdevices. • From threat intel and anomaly detection. • To help your own, or downstream/internal customers. • And feed DDoS response if there are local sources/sinks. • And find BCP38 violations on-net or on peers, even with simple “how many source /8s” heuristics. 37
- 38. Use case: Network performance analytics Flow-based traffic + BGP + network performance data can show: • Whether issues are in the application or network layer (+ where) • And where? •And in a way expose-able to internal dev + app operations • And to pinpoint performance issues by peer or remote AS path • Or prefix • Or data center • Or provably not in the network J 38
- 39. Perf-enhanced flow: TCPlatency / ASN
- 40. Perf-enhanced flow: TCPlatency / Prefix
- 41.
- 42.
- 43.
- 44. Use case: Revenue enhancement and retention Sometimes not discussed in polite company, but great traffic-based analytics can help with the top line as well as margin: • Offering high-margin customers lower rates to attract more traffic. • Identifying large 2ndand 3rd hop AS sinks or sources behind peers, to convert to customers. • Find large customers that look small (on-boarding/testing) or large customers starting to migrate to competitors. • Create service revenue around security, DDoS, and performance. 44
- 45.
- 46. Summary 46 • It’s not just “flow tools” any more (or just BGP or just SNMP or …). • Networks can produce a lot of (very diverse) data. •You can capture and use it with modern streaming, fusing, and storage components. • Enterprises are looking for and funding turnkey but open tools that integrate and provide cross-group access. • Vendors are starting to innovate. • And DIY options are getting better (but still require a lot of training).
- 47.
- 48. Take-Aways 48 • You deserve, and can have, nice things! • And so can your tech and business peers. •And answer business as well as tech ops questions. • It’s possible (with some work) to see not just traffic flow but performance. • There are a new wave of SaaS and big data-based vendors that integrate. • And if you intend to DIY, start cross-training and hiring now.
- 49.