1. Maria Isabel Gandía Carriedo
9th SIG-NOC Meeting
ARNES, Ljubljana, 08-04-2019
Flow Monitoring Tools:
What do We Have? What do we need?

2. Disclaimer
ü I’m neither a developer nor an expert on Network Flow Monitoring
Tools. I’m a user of some of the following tools and I have put together
the information I got as a user with what I have searched about some
other popular tools and the input from Jisc, NORDUnet and AMRES.
ü The idea is to offer SIG-NOC an abstract with useful information about
currently used flow monitoring tools.

3. Nework Flow Monitoring Tools… Just Monitoring?
üSome functionalities go far beyond pure flow monitoring:
• DDoS mitigation (blackholing, scrubbing centre, Flowspec…).
• SNMP support to sync flows with actual traffic volumes.
• BGP-peering with the core to do forward-path analysis.
• Open APIs to integrate towards other tools (NSO, Nagios, Stats)
• Trend reports to do Capacity Planning
• Segregated logins to give filtered views to different users.
• Report generation.
• …

4. A Common Path for Many Network Flow Monitoring Tools
ü There are good Network Flow monitoring Open Source tools, but they
are hard to manage (no GUI, different programs for different actions…).
ü A Research Group works on a great monitoring user-friendly Open
source or not very expensive tool.
ü They create a Company / They develop a different flavour for commercial
use / They are bought by another company.

5. Some Network Flow Monitoring Tools
ü Argus
ü NFSEN / NFDUMP
ü Nimbus (formerly FlowSonar (Team Cymru))
ü SILK / FlowBAT
ü PMACCT
ü NetVizura - NetFlow Analyzer (formerly ICmynet
ü Talaia (Formerly SMARTxAC, now Auvik)
ü NTOP/NTOPNG
ü Fastnetmon Community / Advanced
ü DDPS
ü Arbor Networks SP (formerly Peakflow) (+TMS?) (from Netscout)

6. Argus (Audit Record Generation and Usage System)
ü https://qosient.com/argus/
ü GPLv3, with other licensing agreements available for commercial,
governmental and educational users. No public Git-like repository.
ü Status: Last versions argus-3.0.8.2 and argus-clients-3.0.8.2 (from
2016), Mailing list is active, ArgusPro (with commercial hardware and
software versions of argus) is under development
ü Threat intelligence: No
ü Machine learning: No
ü Supported protocols/inputs: Netflow (1-8, support for v9 is not clear),
Flow-Tools, (Sflow and Jflow, maybe on demand), Port mirroring
ü Users: Gloriad

7. Argus (Audit Record Generation and Usage System)
ü Argus is a bidirectional network flow generator and aggregator It’s
structured as a server and a suite of supporting clients.
ü The server (Argus) retrieves packets, it assembles them into binary
data (representing flows) and writes this binary data to disk and/or a
network socket (argus data stream).
ü The argus-clients package provides a set of more than 30 binaries and
scripts that read flow data from files of binary flow data and do actions
like printing, processing, sorting, aggregating, tallying, collecting,
distributing, archiving, and anonymizing data.
ü Argus provides reachability, availability, connectivity, duration, rate,
load, good-put, loss, jitter, retransmission and delay metrics for all
network flows, and captures most attributes that are available from the
packet contents, such as L2 addresses, tunnel identifiers (MPLS,
GRE, IPsec, etc...), protocol ids, SAP's, hop-count, options, L4
transport identification (RTP detection), host flow control indications,
etc...

8. NFDUMP/NFSEN
ü https://github.com/phaag/nfdump, http://nfsen.sourceforge.net/
ü BSD
ü Status: updated
ü Threat intelligence: No
ü Machine-learning: No
ü Supported protocols/inputs: netflow v1, v5/v7,v9,IPFIX and SFLOW
ü Users: Uninett, SURFsara, GARR, SWITCH, BelWü, PIONIER, DeIC

9. NFDUMP/NFSEN
ü nfdump is a toolset:
• nfcapd collects the data, sent from exporters and stores the flow records into
files. Multiple netflow streams can be collected by a single collector.
• nfdump reads the netflow data from one or many files stored by nfcapd and
displays it and/or creates top N statistics of flows, bytes, packets. All data is
stored to disk, before it gets analyzed.
• nfanon (for anonymization),
• nfexpire (for data expiration),
• nfreply (to export the files stores by nfcapd),
• sfcapd (for sflow collection),
• etc.
ü NfSen (NetFlow Sensor) is a graphical web based front end for the nfdump
netflow tools. It allows users to display flows, packets and bytes using RRD
(Round Robin Database).

10. Nimbus (formerly FlowSonar, (Team Cymru))
ü http://www.team-cymru.com/nimbus.html
ü Type: commercial, cloud-based
ü Threat intelligence feeds: yes (IP reputation and Botnet controllers)
ü “Price”: access to participant's information (NDA required)
ü Users: CSUC (FlowSonar)

11. Nimbus (formerly FlowSonar, (Team Cymru))
ü Flowsonar is based on nfdump/NfSen and it works on-premises.
ü Flowsonar offers graphs for flows, packets and bits/s, per-protocol
graphs, alerts and customized filters.
ü Nimbus is a cloud-based netflow collection, analysis, and reporting
platform. The partner exports their flows to a private IP and port over
an encrypted tunnel.
ü Nimbus uses a Kibana-based portal and provides with XML versions
of the threat intelligence feeds. It is focused on real-time threat
monitoring.

12. SiLK (System for Internet-Level Knowledge) / FlowBAT
ü https://tools.netsa.cert.org/silk/, http://www.flowbat.com/
ü Type: GPLv2 & Government Purpose License Rights (GPLR).
However, there is no public Git-like repository for the project, all
patches should be sent to the CMU team.
ü Status: updated. Last version, silk-3.18.1 (March 2019)
ü Threat intelligence feeds: No
ü Machine learning: No
ü Supported protocols/inputs: IPFIX (from yaf software, not from
routers), NetFlow (v5, v9) and sFlow v5, or PDUs from a router
ü Users: JISC

13. SiLK (System for Internet-Level Knowledge)
ü SiLK is a tool suite with two categories of applications:
• The SiLK Packing System: daemon applications that collect flow data and
convert them into a more space efficient format, storing the packed records
into service-specific hourly binary flat files.
• The SiLK Analysis Suite is a collection of command-line tools that read
binary files containing SiLK Flow records and partition, sort, and count
these records. The analysis tools interoperate using pipes, allowing a user
to develop relatively sophisticated queries.
ü It is suited for analyzing traffic on the backbone or border of a large,
distributed enterprise or mid-sized ISP. However, it has not been
designed for real-time flow analysis.
ü Analysis Pipeline is a separate suite which works along with real-time
analysing of flow data records. It can take flow records from SiLK files
as they are created or IPFIX data from any application.
ü FlowBAT is a graphical flow-based analysis tool designed to work with
a SiLK-based NetFlow system as a back-end. Other third-party GUI
are SiLKWeb and iSiLK.

14. pmacct
ü http://www.pmacct.net/
ü Type: GPLv2
ü Status: Updated,
ü Threat intelligence: No
ü Machine learning: No
ü Supported protocols/inputs: libpcap, Netlink/NFLOG, NetFlow
v1/v5/v7/v8/v9, sFlow v2/v4/v5 and IPFIX. -It also Collects Streaming
Telemetry data
ü Users: PSNC
http://uowits.github.io/herbert-gui/index.html

15. pmacct
ü pmacct is a set of multi-purpose passive network monitoring tools. It
can account, classify, aggregate, replicate and export forwarding-plane
data, collect and correlate control-plane data via BGP and BMP;
collect infrastructure data via Streaming Telemetry.
ü Each component works both as a standalone daemon and as a thread
of execution for correlation purposes (ie. enrich NetFlow with BGP
data).
ü pmacct can save data to many types of backends (relational DB, non-
SQL DB, flat files, etc).
ü It’s able to tag, filter, redirect, aggregate and split captured data.
ü It has a BGP daemon for visibility of BGP multi-path routes.
ü It does Packet classification via nDPI .
ü You can use tools like Project Herber (http://uowits.github.io/herbert-
gui/index.html) to print graphs with the data.

16. NetVizura - NetFlow Analyzer (formerly ICmynet)
ü References: https://www.netvizura.com/netflow-analyzer
ü Type: Research/commercial, on-premises
ü Status: updated
ü Threat inteligence feeds: Yes, through in-depth forensics
ü Machine learning: No
ü Supported protocols/inputs: NetFlow, IPFIX, NSEL, sFlow and
compatible netflow-like protocols.
ü Users: AMRES

17. NetVizura - NetFlow Analyzer (formerly ICmynet)
ü Netvizura NetFlow Analyzer helps net admins with bandwidth
monitoring, network traffic investigation, analyses and reporting. It
supports the following features:
• Device Traffic Analysis - traffic distribution analysis per interface, device or
subnet, used network planning
• Custom Traffic Analysis - analysis per flow type, subnet, organization unit,
etc.
• End User Traffic Analysis - analysis per end user, apps and protocols used,
throughput, etc.
• In-Depth Forensics - through raw data analysis and queries
• Traffic Reports - PDF traffic report
• Threshold Alarms - throughput and volume threshold and alarms,
notifications via email
• Dashboard Overview - realtime alarm prioritization and presentation
• Powerful Settings - Flow sampling and filtering, Top N analysis, managing
data and archives
• Flexible Data Collection - multi-vendor support

18. TALAIA (evolution of SMARTxAC, now Auvik)
ü https://www.talaia.io/, https://es.slideshare.net/CSUC_info/1127-smar-
tx-ac-network-polygraph-catnix-publicable, https://www.auvik.com/
ü Type: commercial, on-premises and Cloud-based
ü Status: deprecated as it was (bought by Auvik)
ü Threat intelligence feeds: No
ü Machine learning: yes
ü Supported protocols: Netflow, IPFIX
ü Users: CSUC, RedIRIS

19. TALAIA (evolution of SMARTxAC, now Auvik)
ü Talaia was a spin-off of Universitat Politècnica de Catalunya (UPC /
BarcelonaTech) for their former project SMARTxAC (Traffic Monitoring
System for Anella Científica) and it was bought by Auvik.
ü Talaia gets Netflow information, it analyzes it using Deep Packet
Inspection information taken from capture linecards in the main
connection lines and shows this information in a graphical interface.
ü It is a multi-tenant solution (each institution only sees the information
associated with its IP addresses).
ü It has views for applications, protocols, top N, autozoom, geolocation,
anomalies, flows and reports.
ü The platform is able to authenticate federated users.
ü Auvik offers an integrated solution for Managed Service Providers
(MSP), offering several tools in a single platform (Configuration
Management, Service Monitoring, IPAM, Inventory Management,
Password Management...). Flow monitoring is one of the functionalities,
although they don't offer the on-premises solution nor the federated
access and it's still work-in-progress to integrate Talia with their platform.

20. NTOP/NTOPNG
ü https://www.ntop.org/
ü Type: GPLv3 & commercial (but free for Education)
ü Status: Updated
ü Threat intelligence feeds: No
ü Machine learning: No
ü Users: TSSG/WIT
ü Supported protocols/inputs: sFlow, NetFlow (v5 & v9) and IPFIX
support through nProbe

21. NTOP/NTOPNG
ü NTOP-NG is a web-based traffic analysis and flow collection software
that provides a web GUI to access monitoring data. It provides detailed
views on active hosts, flows, IP addresses, Mac addresses,
Autonomous systems.
ü It can be used to monitor and report live throughput, network and
application latencies, Round Trip Time (RTT), TCP statistics
(retransmissions, out of order packets, packet lost), and bytes and
packets transmitted.
ü It requires to install nProbe as an intermediate flow collector, a probe
to install in the middle for detailed L7 application dissection or per-
packet realtime analysis. This intermediate step is needed as ntopng
does not understand Netflow, so nProbe acts like a translator.
ü NTOPNG can listen to a SPAN port directly.

22. Fastnetmon Community / Advanced
ü https://github.com/pavel-odintsov/fastnetmon, https://fastnetmon.com/
ü Type: Community edition Open source (GPLv2), Advanced edition
commercial (free one-month trial license available)
ü Status: Both editions updated, more development on Advanced edition.
ü Threat intelligence: No
ü Machine-learning: No
ü Supported protocols/inputs: NetFlow (v5, v9), IPFIX, sFlow (v4, v5), Port
mirror/SPAN capture with PF_RING, SnabbSwitch, NETMAP and
PCAP. Commercial version offers support for more protocols
ü Users: DeiC (for DDPS)
https://fastnetmon.com/screenshoots-fastnetmon-advanced/

23. Fastnetmon Community / Advanced
ü FastNetMon is a volumetric DDoS detector able to perform a
configurable action when defined thresholds are exceeded (notifying,
blackholing, sending BGP Flowspec rules, switching off a server…) .
ü The blocked IPs are announced via BGP with ExaBGP.
ü It has support for the most popular attack types (syn_flood, udp_flood,
icmp flood, ip_fragmentation_flood, DNS amplification, NTP
amplification, SSDP amplification, SNMP amplification).
ü It includes an API and a JSON based database for
configuration/attacks. It can be integrated with Graphite and InfluxDB.
ü The advanced FastNetMon includes bundled support for Grafana,
using InfluxDB or ClickHouse. It can expose total bandwidth, per host
bandwidth, per network bandwidth and arbitrary traffic reports from
traffic persistency database (peering reports, per prefix reports). Some
pre-created dashboards are available.
ü More differences at https://fastnetmon.com/compare-community-and-
advanced/

24. DDPS (DeiC DDoS Protection Service)
ü https://github.com/deic-dk/DDPS-documentation
ü Type: DDPS is copyright 2015-2017 DeiC, Denmark. Licensed under
the Apache License, Version 2.0
ü Status: Updated
ü Threat intelligence: No
ü Machine learning: No
ü Users: DeIC
https://github.com/deic-dk/DDPS-documentation

25. DDPS (DeiC DDoS Protection Service)
ü DDPS relies on FastNetMon Community and it is conceived as an
automated system for DDoS mitigation: it detects attacks and
automatically triggers mitigation.
ü Based on BGP Flowspec, it is intended to be used in a system where
detection is placed as close as possible to the target (FastNetMon in
the customer’s network) and mitigation is placed as close as possible
to the source(s) of the attack (DeiC).
ü End-users may add, edit, or cancel mitigation rules as well as view
archived rules and statistical information.
ü The project is split in sub-projects:
• DDPS fastnetmon
• DDPS database daemon
• DDPS NODE
• DDPS web-user interface
• DDPS Customer Site Simulation
• DDPS DDoS simulator

26. Arbor Networks SP (formerly Peakflow) (+TMS) (Netscout, formerly Arbor)
ü https://www.netscout.com/arbor-ddos
ü Type: commercial
ü Status: on-premises, evolving to a cloud-based mitigation solution
ü Threat intelligence feeds: yes
ü Supported protocols/inputs: NetFlow, sFlow, J-Flow, IP FIX,
ü Users: BelNET, CSUC, JISC
ü Arbor offers a solution for monitoring (Peakflow/SP) and a different
product for DDoS mitigation (TMS), although the front-end is in the SP.
It is based on Netflow, SNMP and BGP information. It compares
Netflow data to SNMP data to set the thresholds. It works with pre-
defined managed objects and it has four main functions:
• Monitoring (SP)
• DDoS detection (SP)
• DDoS mitigation (TMS)
• Reports (SP)

27. Arbor Networks SP (formerly Peakflow) (+TMS) (Netscout, formerly Arbor)
ü It offers different views, like:
• Traffic (per application, AS, customer, protocol, etc)
• Alerts (including Summary, Activity reports, etc)
• Mitigations
• Reports
ü The alerts thresholds are defined by the administrators of the platform.
ü There are three different types of detection:
• Threshold (fixed), in bps and pps. It applies to the whole object
• Profile (with different configurable multiplying factors) in bps and pps. It
applies to the whole object.
• Host, in bps and pps. It applies to each host inside the object. There are
many types of protocols and the administrators must define a threshold for
each one of them (NTP, ICMP, etc).
ü There are different types of users with different permissions, although
it is not exactly a multi-tenant platform.
ü It has SOAP and REST APIs.

28. More Network Monitoring Tools
ü AlienVault (AT&T Cybersecurity since February 2019)
ü Insight2 (based on Argus)
ü OSSIM (Open Source Security Information Management)
ü Deepfield
ü Kentik
ü Flowmo
ü Scrutinizer (plixer)
ü ManageEngine
ü SolarWinds NetFlow Traffic Analyzer
ü ..

29. What do we have? What do we need?
ü Does your current tool cover your needs? The Incubator subtask under
the Network Technologies and Services Development in the Géant
Project may propose to contribute to the development of an open
source alternative to the commercial tools, from scratch or though
contributions to existing open source tools.
ü Let’s play with Mentimeter.

30. Thanks for your attention!
Questions?
mariaisabel.gandia@csuc.cat