This document discusses how Freie Netze München used IPFIX and ElastiFlow to monitor their network traffic and make better peering decisions. They exported encrypted IPFIX data from DE-CIX routers via UDP to ElastiFlow, which provided analytics on traffic flows. This identified their top sources and destinations by ASN, mainly eyeball networks, cloud providers, and content providers. Freie Netze München was then able to set up direct peering sessions with most of these ASNs to improve routing resilience and learn more about their network traffic patterns.

1. Flow monitoring explained
From packet capture to data analysis - the use of IPFIX Exporter
DE-CIX 2021

2. Annika Wickert
● Second Chair of Board Freie Netze München e.V.
● Senior Network Engineer / OpenSource since 2010
● Twitter @awlnx / Github @awlx
2
Who am I?

3. 3
FFMUC?
• Freie Netze München e.V. since 2014
• Community Freifunk München since 2004
• Wifi
• #FFMEET
• DoH/DoT/DNSCrypt/DNS
• Streaming

4. 4
Preface
• FFMUC moved from donated uplinks to its own ASN (AS212567)
• 2 datacenters, each announcing a /48 IPv6 and a /24 IPv4
• Due to RIB/FIB limitations only with default routes from transit
• DE-CIX offered (remote-)peering in FRA and MUC

5. 5
Motivation
• Peering with the DE-CIX Route Server(RS) is great, and yields many routes
• Some prefixes however are only announced on direct sessions
• Route Servers remove some resilience from the internet, since they become a
SPOF (though DE-CIX RS has been stable)
• We’re nerds and want to play & learn
So, where is our traffic going? Who should we peer with?

6. 6
Setup
Munich
Icons: icons8.com
DE-CIX FRA
Remote
Peering

7. 7
IPFIX Ingest
Munich
Icons: icons8.com
DE-CIX FRA
Remote
Peering
IPFIX Export
UDP DTLS
wrapper
DTLS encrypted
IPFIX Data
unified flow
collector

8. 8
What is IPFIX?
• IP Flow Information Export (RFC7011)
• (sampled) information about traffic flows
• Transported via UDP
• contains information like
■ Source / Destination IP
■ Source / Destination Port
■ Packet Size

9. 9
IPFIX Export at DE-CIX
• IPFIX data is generated by the DE-CIX platform and sent to a destination IP
• Sampling Rate 10.000:1
• Since IPFIX data contains plain flows it’s encrypted using DTLS
• The receiver decrypts the DTLS encrypted UDP datagrams and forwards the
plain IPFIX data to a flow pipeline(such as elastiflow)
• Can be configured in the DE-CIX Portal

10. 10
ElastiFlow
● ElastiFlow is a open source network performance analytics platform
● ingests IPFIX/NETFLOW/sFLOW into Elasticsearch
● Enriches flow-data with GeoIP and threat Information
● Provides beautiful pre-built dashboards & sankey diagrams
● Soon offers obfuscation of flow data (only show /24 or /48, not exact prefix)
○ Important for us since we value our users privacy
● Offers commercial support
● Very good community support too :)

11. 11
ElastiFlow @ FFMUC

12. 12
ElastiFlow @ FFMUC

13. 13
ElastiFlow @ FFMUC

14. 14
Better peering decisions
● Based on flow information we were able to confirm our Top ASNs
● To no surprise they were mainly
○ Eyeball Networks
○ Cloud Providers
○ Content Providers / CDNs
● We’ve set up direct sessions with most of them

15. 15
Resources
● DTLS IPFIX Wrapper
https://github.com/DE-CIX/udp-dtls-wrapper
● ElastiFlow
https://elastiflow.com/

16. 16
Questions? Feedback? Support?
● @freifunkMUC
● ffmuc.net/kontakt/
● ffmuc.net/spenden/
● peering@fnmuc.net