Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flow Monitoring Tools, What do we have, What do we need?

Presentació a càrrec de Maria Isabel Gandia, cap de Comunicacions del CSUC, duta a terme dins la 9a edició del SIG-NOC meeting el dia 8 d'abril de 2019 a Ljubljana, Eslovènia.
La presentació explica l'estudi que s'està duent a terme en el marc de Géant per conèixer les eines utilitzades per monitorar fluxos i les necessitats de les NREN de cara a un possible desenvolupament.

  • Be the first to comment

Flow Monitoring Tools, What do we have, What do we need?

  1. 1. Maria Isabel Gandía Carriedo 9th SIG-NOC Meeting ARNES, Ljubljana, 08-04-2019 Flow Monitoring Tools: What do We Have? What do we need?
  2. 2. Disclaimer ü I’m neither a developer nor an expert on Network Flow Monitoring Tools. I’m a user of some of the following tools and I have put together the information I got as a user with what I have searched about some other popular tools and the input from Jisc, NORDUnet and AMRES. ü The idea is to offer SIG-NOC an abstract with useful information about currently used flow monitoring tools.
  3. 3. Nework Flow Monitoring Tools… Just Monitoring? üSome functionalities go far beyond pure flow monitoring: • DDoS mitigation (blackholing, scrubbing centre, Flowspec…). • SNMP support to sync flows with actual traffic volumes. • BGP-peering with the core to do forward-path analysis. • Open APIs to integrate towards other tools (NSO, Nagios, Stats) • Trend reports to do Capacity Planning • Segregated logins to give filtered views to different users. • Report generation. • …
  4. 4. A Common Path for Many Network Flow Monitoring Tools ü There are good Network Flow monitoring Open Source tools, but they are hard to manage (no GUI, different programs for different actions…). ü A Research Group works on a great monitoring user-friendly Open source or not very expensive tool. ü They create a Company / They develop a different flavour for commercial use / They are bought by another company.
  5. 5. Some Network Flow Monitoring Tools ü Argus ü NFSEN / NFDUMP ü Nimbus (formerly FlowSonar (Team Cymru)) ü SILK / FlowBAT ü PMACCT ü NetVizura - NetFlow Analyzer (formerly ICmynet ü Talaia (Formerly SMARTxAC, now Auvik) ü NTOP/NTOPNG ü Fastnetmon Community / Advanced ü DDPS ü Arbor Networks SP (formerly Peakflow) (+TMS?) (from Netscout)
  6. 6. Argus (Audit Record Generation and Usage System) ü ü GPLv3, with other licensing agreements available for commercial, governmental and educational users. No public Git-like repository. ü Status: Last versions argus- and argus-clients- (from 2016), Mailing list is active, ArgusPro (with commercial hardware and software versions of argus) is under development ü Threat intelligence: No ü Machine learning: No ü Supported protocols/inputs: Netflow (1-8, support for v9 is not clear), Flow-Tools, (Sflow and Jflow, maybe on demand), Port mirroring ü Users: Gloriad
  7. 7. Argus (Audit Record Generation and Usage System) ü Argus is a bidirectional network flow generator and aggregator It’s structured as a server and a suite of supporting clients. ü The server (Argus) retrieves packets, it assembles them into binary data (representing flows) and writes this binary data to disk and/or a network socket (argus data stream). ü The argus-clients package provides a set of more than 30 binaries and scripts that read flow data from files of binary flow data and do actions like printing, processing, sorting, aggregating, tallying, collecting, distributing, archiving, and anonymizing data. ü Argus provides reachability, availability, connectivity, duration, rate, load, good-put, loss, jitter, retransmission and delay metrics for all network flows, and captures most attributes that are available from the packet contents, such as L2 addresses, tunnel identifiers (MPLS, GRE, IPsec, etc...), protocol ids, SAP's, hop-count, options, L4 transport identification (RTP detection), host flow control indications, etc...
  8. 8. NFDUMP/NFSEN ü, ü BSD ü Status: updated ü Threat intelligence: No ü Machine-learning: No ü Supported protocols/inputs: netflow v1, v5/v7,v9,IPFIX and SFLOW ü Users: Uninett, SURFsara, GARR, SWITCH, BelWü, PIONIER, DeIC
  9. 9. NFDUMP/NFSEN ü nfdump is a toolset: • nfcapd collects the data, sent from exporters and stores the flow records into files. Multiple netflow streams can be collected by a single collector. • nfdump reads the netflow data from one or many files stored by nfcapd and displays it and/or creates top N statistics of flows, bytes, packets. All data is stored to disk, before it gets analyzed. • nfanon (for anonymization), • nfexpire (for data expiration), • nfreply (to export the files stores by nfcapd), • sfcapd (for sflow collection), • etc. ü NfSen (NetFlow Sensor) is a graphical web based front end for the nfdump netflow tools. It allows users to display flows, packets and bytes using RRD (Round Robin Database).
  10. 10. Nimbus (formerly FlowSonar, (Team Cymru)) ü ü Type: commercial, cloud-based ü Threat intelligence feeds: yes (IP reputation and Botnet controllers) ü “Price”: access to participant's information (NDA required) ü Users: CSUC (FlowSonar)
  11. 11. Nimbus (formerly FlowSonar, (Team Cymru)) ü Flowsonar is based on nfdump/NfSen and it works on-premises. ü Flowsonar offers graphs for flows, packets and bits/s, per-protocol graphs, alerts and customized filters. ü Nimbus is a cloud-based netflow collection, analysis, and reporting platform. The partner exports their flows to a private IP and port over an encrypted tunnel. ü Nimbus uses a Kibana-based portal and provides with XML versions of the threat intelligence feeds. It is focused on real-time threat monitoring.
  12. 12. SiLK (System for Internet-Level Knowledge) / FlowBAT ü, ü Type: GPLv2 & Government Purpose License Rights (GPLR). However, there is no public Git-like repository for the project, all patches should be sent to the CMU team. ü Status: updated. Last version, silk-3.18.1 (March 2019) ü Threat intelligence feeds: No ü Machine learning: No ü Supported protocols/inputs: IPFIX (from yaf software, not from routers), NetFlow (v5, v9) and sFlow v5, or PDUs from a router ü Users: JISC
  13. 13. SiLK (System for Internet-Level Knowledge) ü SiLK is a tool suite with two categories of applications: • The SiLK Packing System: daemon applications that collect flow data and convert them into a more space efficient format, storing the packed records into service-specific hourly binary flat files. • The SiLK Analysis Suite is a collection of command-line tools that read binary files containing SiLK Flow records and partition, sort, and count these records. The analysis tools interoperate using pipes, allowing a user to develop relatively sophisticated queries. ü It is suited for analyzing traffic on the backbone or border of a large, distributed enterprise or mid-sized ISP. However, it has not been designed for real-time flow analysis. ü Analysis Pipeline is a separate suite which works along with real-time analysing of flow data records. It can take flow records from SiLK files as they are created or IPFIX data from any application. ü FlowBAT is a graphical flow-based analysis tool designed to work with a SiLK-based NetFlow system as a back-end. Other third-party GUI are SiLKWeb and iSiLK.
  14. 14. pmacct ü ü Type: GPLv2 ü Status: Updated, ü Threat intelligence: No ü Machine learning: No ü Supported protocols/inputs: libpcap, Netlink/NFLOG, NetFlow v1/v5/v7/v8/v9, sFlow v2/v4/v5 and IPFIX. -It also Collects Streaming Telemetry data ü Users: PSNC
  15. 15. pmacct ü pmacct is a set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, collect and correlate control-plane data via BGP and BMP; collect infrastructure data via Streaming Telemetry. ü Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). ü pmacct can save data to many types of backends (relational DB, non- SQL DB, flat files, etc). ü It’s able to tag, filter, redirect, aggregate and split captured data. ü It has a BGP daemon for visibility of BGP multi-path routes. ü It does Packet classification via nDPI . ü You can use tools like Project Herber ( gui/index.html) to print graphs with the data.
  16. 16. NetVizura - NetFlow Analyzer (formerly ICmynet) ü References: ü Type: Research/commercial, on-premises ü Status: updated ü Threat inteligence feeds: Yes, through in-depth forensics ü Machine learning: No ü Supported protocols/inputs: NetFlow, IPFIX, NSEL, sFlow and compatible netflow-like protocols. ü Users: AMRES
  17. 17. NetVizura - NetFlow Analyzer (formerly ICmynet) ü Netvizura NetFlow Analyzer helps net admins with bandwidth monitoring, network traffic investigation, analyses and reporting. It supports the following features: • Device Traffic Analysis - traffic distribution analysis per interface, device or subnet, used network planning • Custom Traffic Analysis - analysis per flow type, subnet, organization unit, etc. • End User Traffic Analysis - analysis per end user, apps and protocols used, throughput, etc. • In-Depth Forensics - through raw data analysis and queries • Traffic Reports - PDF traffic report • Threshold Alarms - throughput and volume threshold and alarms, notifications via email • Dashboard Overview - realtime alarm prioritization and presentation • Powerful Settings - Flow sampling and filtering, Top N analysis, managing data and archives • Flexible Data Collection - multi-vendor support
  18. 18. TALAIA (evolution of SMARTxAC, now Auvik) ü, tx-ac-network-polygraph-catnix-publicable, ü Type: commercial, on-premises and Cloud-based ü Status: deprecated as it was (bought by Auvik) ü Threat intelligence feeds: No ü Machine learning: yes ü Supported protocols: Netflow, IPFIX ü Users: CSUC, RedIRIS
  19. 19. TALAIA (evolution of SMARTxAC, now Auvik) ü Talaia was a spin-off of Universitat Politècnica de Catalunya (UPC / BarcelonaTech) for their former project SMARTxAC (Traffic Monitoring System for Anella Científica) and it was bought by Auvik. ü Talaia gets Netflow information, it analyzes it using Deep Packet Inspection information taken from capture linecards in the main connection lines and shows this information in a graphical interface. ü It is a multi-tenant solution (each institution only sees the information associated with its IP addresses). ü It has views for applications, protocols, top N, autozoom, geolocation, anomalies, flows and reports. ü The platform is able to authenticate federated users. ü Auvik offers an integrated solution for Managed Service Providers (MSP), offering several tools in a single platform (Configuration Management, Service Monitoring, IPAM, Inventory Management, Password Management...). Flow monitoring is one of the functionalities, although they don't offer the on-premises solution nor the federated access and it's still work-in-progress to integrate Talia with their platform.
  20. 20. NTOP/NTOPNG ü ü Type: GPLv3 & commercial (but free for Education) ü Status: Updated ü Threat intelligence feeds: No ü Machine learning: No ü Users: TSSG/WIT ü Supported protocols/inputs: sFlow, NetFlow (v5 & v9) and IPFIX support through nProbe
  21. 21. NTOP/NTOPNG ü NTOP-NG is a web-based traffic analysis and flow collection software that provides a web GUI to access monitoring data. It provides detailed views on active hosts, flows, IP addresses, Mac addresses, Autonomous systems. ü It can be used to monitor and report live throughput, network and application latencies, Round Trip Time (RTT), TCP statistics (retransmissions, out of order packets, packet lost), and bytes and packets transmitted. ü It requires to install nProbe as an intermediate flow collector, a probe to install in the middle for detailed L7 application dissection or per- packet realtime analysis. This intermediate step is needed as ntopng does not understand Netflow, so nProbe acts like a translator. ü NTOPNG can listen to a SPAN port directly.
  22. 22. Fastnetmon Community / Advanced ü, ü Type: Community edition Open source (GPLv2), Advanced edition commercial (free one-month trial license available) ü Status: Both editions updated, more development on Advanced edition. ü Threat intelligence: No ü Machine-learning: No ü Supported protocols/inputs: NetFlow (v5, v9), IPFIX, sFlow (v4, v5), Port mirror/SPAN capture with PF_RING, SnabbSwitch, NETMAP and PCAP. Commercial version offers support for more protocols ü Users: DeiC (for DDPS)
  23. 23. Fastnetmon Community / Advanced ü FastNetMon is a volumetric DDoS detector able to perform a configurable action when defined thresholds are exceeded (notifying, blackholing, sending BGP Flowspec rules, switching off a server…) . ü The blocked IPs are announced via BGP with ExaBGP. ü It has support for the most popular attack types (syn_flood, udp_flood, icmp flood, ip_fragmentation_flood, DNS amplification, NTP amplification, SSDP amplification, SNMP amplification). ü It includes an API and a JSON based database for configuration/attacks. It can be integrated with Graphite and InfluxDB. ü The advanced FastNetMon includes bundled support for Grafana, using InfluxDB or ClickHouse. It can expose total bandwidth, per host bandwidth, per network bandwidth and arbitrary traffic reports from traffic persistency database (peering reports, per prefix reports). Some pre-created dashboards are available. ü More differences at advanced/
  24. 24. DDPS (DeiC DDoS Protection Service) ü ü Type: DDPS is copyright 2015-2017 DeiC, Denmark. Licensed under the Apache License, Version 2.0 ü Status: Updated ü Threat intelligence: No ü Machine learning: No ü Users: DeIC
  25. 25. DDPS (DeiC DDoS Protection Service) ü DDPS relies on FastNetMon Community and it is conceived as an automated system for DDoS mitigation: it detects attacks and automatically triggers mitigation. ü Based on BGP Flowspec, it is intended to be used in a system where detection is placed as close as possible to the target (FastNetMon in the customer’s network) and mitigation is placed as close as possible to the source(s) of the attack (DeiC). ü End-users may add, edit, or cancel mitigation rules as well as view archived rules and statistical information. ü The project is split in sub-projects: • DDPS fastnetmon • DDPS database daemon • DDPS NODE • DDPS web-user interface • DDPS Customer Site Simulation • DDPS DDoS simulator
  26. 26. Arbor Networks SP (formerly Peakflow) (+TMS) (Netscout, formerly Arbor) ü ü Type: commercial ü Status: on-premises, evolving to a cloud-based mitigation solution ü Threat intelligence feeds: yes ü Supported protocols/inputs: NetFlow, sFlow, J-Flow, IP FIX, ü Users: BelNET, CSUC, JISC ü Arbor offers a solution for monitoring (Peakflow/SP) and a different product for DDoS mitigation (TMS), although the front-end is in the SP. It is based on Netflow, SNMP and BGP information. It compares Netflow data to SNMP data to set the thresholds. It works with pre- defined managed objects and it has four main functions: • Monitoring (SP) • DDoS detection (SP) • DDoS mitigation (TMS) • Reports (SP)
  27. 27. Arbor Networks SP (formerly Peakflow) (+TMS) (Netscout, formerly Arbor) ü It offers different views, like: • Traffic (per application, AS, customer, protocol, etc) • Alerts (including Summary, Activity reports, etc) • Mitigations • Reports ü The alerts thresholds are defined by the administrators of the platform. ü There are three different types of detection: • Threshold (fixed), in bps and pps. It applies to the whole object • Profile (with different configurable multiplying factors) in bps and pps. It applies to the whole object. • Host, in bps and pps. It applies to each host inside the object. There are many types of protocols and the administrators must define a threshold for each one of them (NTP, ICMP, etc). ü There are different types of users with different permissions, although it is not exactly a multi-tenant platform. ü It has SOAP and REST APIs.
  28. 28. More Network Monitoring Tools ü AlienVault (AT&T Cybersecurity since February 2019) ü Insight2 (based on Argus) ü OSSIM (Open Source Security Information Management) ü Deepfield ü Kentik ü Flowmo ü Scrutinizer (plixer) ü ManageEngine ü SolarWinds NetFlow Traffic Analyzer ü ..
  29. 29. What do we have? What do we need? ü Does your current tool cover your needs? The Incubator subtask under the Network Technologies and Services Development in the Géant Project may propose to contribute to the development of an open source alternative to the commercial tools, from scratch or though contributions to existing open source tools. ü Let’s play with Mentimeter.
  30. 30. Thanks for your attention! Questions?