Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Network visibility and control using industry standard sFlow telemetry

• Find out about the sFlow instrumentation built into commodity data center network and server infrastructure.
• Understand how sFlow fits into the broader ecosystem of NetFlow, IPFIX, SNMP and DevOps monitoring technologies.
• Case studies demonstrate how sFlow telemetry combined with automation can lower costs, increase performance, and improve security of cloud infrastructure and applications.

  • Login to see the comments

Network visibility and control using industry standard sFlow telemetry

  1. 1. Network visibility and control using industry standard sFlow telemetry Peter Phaal InMon Corp.
 March, 2016 Twitter: @sFlow Blog: San Francisco Network Visibility Meetup
  2. 2. Why monitor? “If you can’t measure it, you can’t improve it” Lord Kelvin
  3. 3. Time Capacity Demand Static provisioning $ Unused capacity $$$ Service failure $$ Unused capacity
  4. 4. $$ Savings Time Capacity Demand Dynamic provisioning
  5. 5. Feedback control Measure Control System desired output measured output
  6. 6. Controllability and Observability Basic concept is simple, a stable feedback control system requires: 1. ability to influence all important system states (controllable) 2. ability to monitor all important system states (observable)
  7. 7. It’s hard to stay on the road if you can’t see the road, or keep to the speed limit without a speedometer It’s hard to stay on the road or maintain speed if your brakes, engine or steering fail Controllability and Observability driving example Observability Controllability States location, speed, direction, ... Tule fog in California Central Valley
  8. 8. Effect of delay on stability Measurement delay Planning delay Time Configuration delayDisturbance Response delay EffectLoop delay DDoS launched Identify target, attacker Black hole, mark, re-route? Switch CLI commands Route propagation Traffic dropped Components of loop delay e.g. Slow reaction time causes tired / drunk / distracted driver to weave, very slow reaction time and they leave the road
  9. 9. What is sFlow? “In God we trust. All others bring data.” Dr. Edwards Deming
  10. 10. Industry standard measurement technology integrated in switches
  11. 11. Open source agents for hosts, hypervisors and applications Host sFlow project ( is center of an ecosystem of related open source projects embedding sFlow in popular operating systems and applications Host agent extends network visibility into public / private cloud
  12. 12. Network (maintained by hardware in network devices) - MIB-2 ifTable: ifInOctets, ifInUcastPkts, ifInMulticastPkts, ifInBroadcastPkts, ifInDiscards, ifInErrors, ifUnkownProtos, ifOutOctets, ifOutUcastPkts, ifOutMulticastPkts, ifOutBroadcastPkts, ifOutDiscards, ifOutErrors Host (maintained by operating system kernel) - CPU: load_one, load_five, load_fifteen, proc_run, proc_total, cpu_num, cpu_speed, uptime, cpu_user, cpu_nice, cpu_system, cpu_idle, cpu_wio, cpu_intr, cpu_sintr, interupts, contexts - Memory: mem_total, mem_free, mem_shared, mem_buffers, mem_cached, swap_total, swap_free, page_in, page_out, swap_in, swap_out - Disk IO: disk_total, disk_free, part_max_used, reads, bytes_read, read_time, writes, bytes_written, write_time - Network IO: bytes_in, packets_in, errs_in, drops_in, bytes_out, packet_out, errs_out, drops_out Application (maintained by application) - HTTP: method_option_count, method_get_count, method_head_count, method_post_count, method_put_count, method_delete_count, method_trace_count, method_connect_count, method_other_count, status_1xx_count, status_2xx_count, status_3xx_count, status_4xx_count, status_5xx_count, status_other_count - Memcache: cmd_set, cmd_touch, cmd_flush, get_hits, get_misses, delete_hits, delete_misses, incr_hits, incr_misses, decr_hists, decr_misses, cas_hits, cas_misses, cas_badval, auth_cmds, auth_errors, threads, con_yields, listen_disabled_num, curr_connections, rejected_connections, total_connections, connection_structures, evictions, reclaimed, curr_items, total_items, bytes_read, bytes_written, bytes, limit_maxbytes Standard counters
  13. 13. Simple - standard structures - densely packed blocks of counters - extensible (tag, length, value) - RFC 1832: XDR encoded (big endian, quad-aligned, binary) - simple to encode / decode - unicast UDP transport Minimal configuration - collector address - polling interval Cloud friendly - flat, two tier architecture: many embedded agents → central “smart” collector - sFlow agents automatically start sending metrics on startup, automatically discovered - eliminates complexity of maintaining polling daemons (and associated configurations) Scaleable push protocol
  14. 14. • Counters tell you there is a problem, but not why. • Counters summarize performance by dropping high cardinality attributes: - IP addresses - URLs - Memcache keys • Need to be able to efficiently disaggregate counter by attributes in order to understand root cause of performance problems. • How do you get this data when there are millions of transactions per second? Counters aren’t enough Why the spike in traffic? (100Gbit link carrying 14,000,000 packets/second)
  15. 15. • Random sampling is lightweight • Critical path roughly cost of maintaining one counter:
 if(--skip == 0) sample(); • Sampling is easy to distribute among modules, threads, processes without any synchronization • Minimal resources required to capture attributes of sampled transactions • Easily identify top keys, connections, clients, servers, URLs etc. • Unbiased results with known accuracy Break out traffic by client, server and port (graph based on samples from100Gbit link carrying 14,000,000 packets/second) sFlow also exports random samples
  16. 16. Integrated data model Packet Header Source Destination TCP/UDP Socket TCP/UDP Socket MAC Address MAC Address Sampled Packet Headers + Forwarding State I/F Counters NETWORK HOST CPU Memory I/O Adapter MACs APPLICATION Sampled Transactions Transaction Counters TCP/UDP Socket Independent agents sFlow analyzer joins data for integrated view
  17. 17. Virtual Servers Applications Apache/PHP Tomcat/Java Memcached Virtual Network Servers Network Embedded monitoring of all switches, all servers, all applications, all the time Consistent measurements shared between multiple management tools Comprehensive data center wide visibility
  18. 18. Picking the right tools “This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together.” Doug McIlroy
  19. 19. packets decode hash sendflow cache flushsample Flow Records flow cache embedded on switchswitch NetFlow IPFIX … decode hash sendflow cache flush Flow Records packets send polli/f counters sample multiple switches export sFlow packets send polli/f counters sample ... centralized software flow cache switch switch JSON/REST
 NetFlow IPFIX … • Reduce ASIC cost / complexity • Fast response (data not sitting on switch) • Centralized, network-wide visibility • Increase flexibility → software defined analytics Move flow cache from ASIC to external software Scale-out alternative to SNMP polling Traffic analytics with sFlow
  20. 20. analytics engine • Low latency flow analytics for real-time control applications • Disaggregates flow cache from database. Choose external database(s) for history (InfluxDB, Logstash, etc.) • Programmable analytics pipeline through open APIs
  21. 21. RESTful API for defining flows curl -H "Content-Type:application/json" -X PUT —data '{"keys":"ipsource,ipdestination,tcpsourceport,tcpdestinationport", "value":"bytes", "ipfixCollectors":[""]}' curl -H "Content-Type:application/json" -X PUT --data '{"keys":"ipdestination,icmpunreachableport", "value":"frames"}' • Instantly enables network wide monitoring of flows • All switches, all ports, including hosts and virtual switches • Contrast with task of re-configuring Flexible Netflow/IPFIX caches on every switch in multi-vendor network. How many simultaneous flow definitions are allowed? What key / value combinations are allowed? curl -H "Content-Type:application/json" -X PUT --data '{"value":"frames"}'
  22. 22. InMon sFlow-RT active timeout active timeout NetFlow Open vSwitch SolarWinds Real-Time NetFlow Analyzer • sFlow does not use flow cache, so realtime charts more accurately reflect traffic trend • NetFlow spikes caused by flow cache active-timeout for long running connections Rapid detection of large flows Flow cache active timeout delays large flow detection, limits value of signal for real-time control applications
  23. 23. Counters and packet samples • Packet samples give a fast signal that operates at scale • Counters are maintained in hardware and provide precise traffic totals. • Counters capture rare events, like packet discards, that can severely impact performance. • Counters report important link state information, like link speed, LAG group membership etc.
  24. 24. Metrics and Events Metrics Events Sources SNMP, sFlow, collectd Traps, syslog, IPFIX, NetFlow Crossover Event count Threshold event Collectors InfluxDB, OpenTSDB, Graphite, rrdtool Logstash, flowtools, Splunk Application Performance Security and alerts
  25. 25. Data models and transports sFlow SNMP NetFlow version 5 OpenConfig Telemetry IPFIX syslog Model standard measurements published by, Dataplane focus: based on IEEE, IETF, APIs (MIB-2, LAG-MIB, libvirt, JMX, …) standard MIBs defined by IETF standard tcp / udp / icmp flow record defined by Cisco Telemetry defined as part of YANG configuration models by
 Control plane focus: BGP, MPLS, VLAN, etc. Encoding XDR
 (RFC 4506) ASN1 (IETF) NetFlow (Cisco) protobufs, JSON, NetConf IPFIX (IETF) Syslog (RFC 5424) Transport UDP UDP UDP UDP, HTTP SCTP, TCP, UDP UDP Mode Push Pull Push Push Push Push Easy to combine multiple data sources if you disaggregate tool chain 
 e.g. separate agents from collectors, feed data from all sources into InfluxDB / Logstash
  26. 26. sflowtool sflowtool replicate ascii Perl, Python, awk, grep pcap wireshark, tcpdump Netflow Open source command line tool
  27. 27. Network visibility for DevOps tools • Streaming filtering and summarization reduces data volume and increases scaleability of backend tools • Streaming flow analytics to generate application metrics
  28. 28. Feedback control of cloud infrastructure “You can’t control what you can’t measure” Tom DeMarco
  29. 29. Cloud depends on network • Server costs (both capex and power consumption) far exceed networking costs in the data center. • Network congestion caused server to wait, resulting in poor utilization of cloud infrastructure. • Optimize network to increase data center efficiency “Typically the resource that is most scarce is the network.” Amin Vahdat, Google, ONS2015 Keynote
  30. 30. Elephant flows are the small number of long lived large flows responsible for majority of bytes on network Large “Elephant” flows
  31. 31. ECMP visibility and control
  32. 32. Default sFlow settings Port Speed Large Flow Sampling Rate Polling Interval 1 Gbit/s >= 100 Mbit/s 1-in-1,000 30 seconds 10 Gbit/s >= 1 Gbit/s 1-in-10,000 30 seconds 25 Gbit/s >= 2.5 Gbit/s 1-in-25,000 30 seconds 40 Gbit/s >= 4 Gbit/s 1-in-40,000 30 seconds 50 Gbit/s >= 5 Gbit/s 1-in-50,000 30 seconds 100 Gbit/s >= 10 Gbit/s 1-in-100,000 30 seconds Supports real-time detection of large “Elephant” flows where large flow is defined as flow consuming >10% of link bandwidth for >1 second
  33. 33. Elephant flow collisions
  34. 34. Elephant flow collisions
  35. 35. ECMP monitoring challenge • large number of links, 12 x 10G links • all links need to be monitored continuously, 180G total bandwidth • real-time detection of congested links • real-time detection of Elephant flows
  36. 36. Fabric level performance metrics • Fabric View application runs on sFlow-RT • Downloadable from, includes captured data set from 4 node 10G ECMP fabric • Elephant collisions on spine links occur frequently • Collisions halve throughput • Collisions cause packet discards
  37. 37. Large “Elephant” flows delay small “Mice” flows Separating Elephants and Mice into different queues significantly improves latency and response time Large flow marking
  38. 38. Large flow marking
  39. 39. ONS2015 SDN Showcase
 CORD: Open-source spine-leaf fabric
  40. 40. CORD: Open-source spine-leaf fabric Rack Rack Rack Rack Spine
  41. 41. SDN Central Demo Friday SDN Optimization of Hybrid Packet / Optical Data Center Fabric
  42. 42. Cumulus, Calient and InMon PoC. Large flows detected in real-time and diverted to direct ToR to ToR optical circuit SDN Optimization of Hybrid Packet / Optical Data Center Fabric
  43. 43. Software Defined Internet Router (SIR)
  44. 44. IXP Routes Installed Routes not installed LINX 18672 95711 DECIX 12518 108164 EQIX-ASH 27687 75146
  45. 45. White box Internet router PoC
  46. 46. Internet Router PoC Peer 1 Peer 2 Peer N DDoS Mitigation REST API sFlow-RT controller real-time analytics OpenFlow, REST, BGP, … Large flow marking Routing Offload BGP … BGP Router Switch
  47. 47. Open Networking Summit SDN Idol winning solution Real-time SDN Analytics for DDoS mitigation
  48. 48. DDoS Mitigation Market Opportunity DDoS Attack Megatrends [Reference 1] • High bandwidth, volumetric infrastructure layer (Layer 3 & 4) attacks increased approximately 30 percent • DDoS attack volume also increased month-to-month in 2013, with 10 out of 12 months showing higher attack volume compared to 2012 • Average DDoS attack sizes continued to increase – many over 100 Gbps, the largest peaking at 179 Gbps DDoS Mitigation Market Growth • $870M market by 2017, 18.2% CAGR – Source: IDC:Worldwide DDoS Prevention Products and Services 2013-2017 Forecast • $1049M market by 2017, 25% CAGR – Source: Infonetics: Global DDoS Prevention Appliances 2012-2017 Forecast Reference 1: Top DDoS Attack Trends
  49. 49. DDoS Mitigation Use Case (1) ISP 1 ISP 2 ISP N • ISP/IX is uniquely positioned to protect customers from DDoS flood attacks • New revenue from DDoS mitigation service + differentiates ISP/IX service Attacker User Prevent attack from overwhelming customer access link Filter attack traffic in real-time Customer network DDoS target host Attack on single host can take out entire customer data center. Customer cannot mitigate flood attack without upstream help ISP / IX ISP/IX Market Segment
  50. 50. Customer portal DDoS Mitigation Service Web UI + RESTful programmatic API • real-time TopN analytics • programmable filtering of traffic • set thresholds + automatic blocking Real-time sFlow visibility, Hybrid OpenFlow Control capability of Brocade switches/routers REST API InMonsFlow-RT REST API OpenFlowController DDoS Mitigation Application Customer Network Internet 1. Flood attack overloads customer port 2. Attack maps to large flows [Ref. 2]. sFlow-RT detects attack (maps to large flows) and characterizes attack (srcip, dstip, protocol, ports, etc.) 3. mitigation application takes signature, applies customer policy, selects optimal control and push OpenFlow rule(s) to switch(es) 5. OpenFlow rule(s) applied to switch forwarding path to drop / mark traffic and protect link HTTPS HTTPS 4. Controller pushes OpenFlow rule(s) to switch(es) OpenFlow 1.3 Match Fields line rate filtering using Brocade switches Reference 2: IETF I2RS Working Group Draft -
  51. 51. Demonstration
  52. 52. DDoS Mitigation
  53. 53. Cloudflare DDoS mitigation • sFlow used to identify attack signature • BPF created to match signature • Detailed signatures minimize collateral damage
  54. 54. Big Switch Network Webinar Big Tap sFlow: Enabling Pervasive Flow-level Visibility
  55. 55. Pervasive monitoring
  56. 56. Targeted capture
  57. 57. Comments • sFlow instrumentation is widely available in switches • Host sFlow ( agent extends visibility into servers (works with libpcap, iptables, Open vSwitch to efficiently sample packets in host data plane) • Common data model ensures strong interoperability across sFlow data sources • Streaming counter and packet telemetry across network, compute and application tiers makes data center observable • Observability makes it possible to apply feedback controls
  58. 58. Extra slides
  59. 59. Host sFlow monitoring of Linux datapath Technology Reference Adapter, bridge, macvlan, ipvlan Berkeley Packet Filter (BPF) sampling function 2016/02/linux-bridge- macvlan-ipvlan-adapters.html Open vSwitch Kernel datapath has sFlow support support/config- cookbooks/sflow/ Linux Firewall iptables statistic module random function with ulog 2010/12/ulog.html Top of Rack Switch ASIC provides wirespeed monitoring of attached servers 2010/04/hybrid-server- monitoring.html Efficient monitoring of high traffic production workloads
  60. 60. Open vSwitch Fall Conference New OVS instrumentation features aimed at real-time monitoring of virtual networks
  61. 61. Overlay / Underlay Visibility
  62. 62. Open vSwitch Fall Conference OVN service injection demonstration
  63. 63. Service injection
  64. 64. White Paper Actionable Intelligence in the SDN Ecosystem: Optimizing Network Traffic through FRSA
  65. 65. WAN Optimization
  66. 66. Dell NFV Summit 2015 Demystifying NFV Infrastructure Hotspots End-to-end Monitoring, Analytics & SDN Control
  67. 67. Demystifying NFV Infrastructure Hotspots
  68. 68. Hybrid OpenFlow ECMP testbed
  69. 69. Hybrid OpenFlow ECMP testbed • Simulated ECMP network for developing visibility and control applications • sFlow support in Open vSwitch • OpenFlow for control
  70. 70. The sFlow Standard: Scalable, Unified Monitoring of Networks, Systems and Applications 2012 Velocity Conference
  71. 71. case study