Introducing Peering LAN 2.0 at DE-CIX Daniel Spierling

1. Introducing Peering LAN 2.0 at DE-CIX
14th June 2023 MyNOG 10
Daniel Spierling, Senior Interconnection Consultant
daniel.spierling@de-cix.net

2. who am I ?
Ú 2011 – 2017: DE-CIX network & datacenter engineering, helping to
plan, build & operate the IX infrastructure in Frankfurt and role out of
new IX infrastructure in EMEA + US
Ú 2018 – today: DE-CIX Peering Manager & Consultant: sharing best
practices of IP peering & IP transit architectures, BGP traffic
engineering principles and routing security
Ú 2021 – today: DE-CIX Solution Architecture for “non-standard projects”
& cloud connectivity portfolio
based in Frankfurt,
Germany 🇩🇪

3. Peering LAN 2.0 – why?
Reduce the amount of
broadcast traffic (ARP/ND)
inside Peering LANs
Prevent IP spoofing
inside Peering LAN
Simplify our software
stack & increase
automation

4. Ú DE-CIX operates 30+ Peering LANs, usually
one for each metro network
Ú With an increased amount of peers,
broadcast traffic in each LAN increases
Overview of DE-CIX Peering LANs

5. Ú Within e.g. the Frankfurt Peering LAN, Broadcast, Unknown Unicast &
Multicast traffic exceeded 1,5 Mbps
Broadcast Noise inside a Peering LAN

6. Ú A router answering to address resolution requests for
addresses that do not belong to his own interface
Ú Effectively spoofing other IX participants traffic and
interrupting all layer 3 based communications of other
peers within the Peering LAN
Dealing with Proxy ARP / ND

7. Port Security at DE-CIX
Egress
Ingress
IXP
Unicast (BE)
Broadcast (L2, 1Mbit/s)
Multicast (L2, 1Mbit/s)
Unicast (BE)
BUM (L2, 5Mbit/s)
U-Unicast (L2, 1Mbit/s)
Static MAC + MAC ACLs

8. Long Lasting Flows
VPLS VPLS
VPLS

9. Long Lasting Flows
VPLS VPLS
VPLS

10. Long Lasting MACs
VPLS VPLS
VPLS VPLS

11. Long Lasting MACs
VPLS VPLS
VPLS VPLS

12. Long Lasting MACs
VPLS
VPLS VPLS
VPLS

13. Long Lasting MACs - EVPN
VPLS
VPLS
VPLS VPLS

14. VLAN Loops at IXPs
IXP

15. VLAN Loops at IXPs
IXP

16. VLAN Loops at IXPs
IXP
Step 1
Routers continously
injecting BUM traffic

17. VLAN Loops at IXPs
IXP
Step 1
Routers continously
injecting BUM traffic
Step 2
BUM traffic in fabric
circulating and increasing

18. VLAN Loops at IXPs
IXP
Step 1
Router continously injecting
BUM traffic
Step 2
BUM traffic in fabric
circulating and increasing
Step 3
BUM traffic level exceeding
egress rate limit and
blocking valid traffic

19. Introducing an ARP/ND Agent
IXP
Peering LAN
Proxy
Agent
Bindings
IP – MAC
IP – MAC
IP – MAC
IP – MAC
ARP Request
ARP Reply
Ú RFC 9161: Enhancing EVPN for the Peering LAN use-case:

20. Migration Process
Which path did we
choose?

21. How does a DE-CIX Metro fabric look like?

22. different migration scenarios
Scenario Small Steps Reduce
Complexity
Reduce
Downtimes
Router by Router
Location by Location
Service by Service

23. Simulating & Testing the network

24. Testing – Configuration Systems

25. Looking into the
results

26. BUM traffic at DE-CIX Frankfurt

27. DE-CIX Frankfurt
BUM traffic packets per second by protocol
798
1378
0
200
400
600
800
1000
1200
1400
1600
IPv4 IPv6
Flood & Learn
8
53
0
200
400
600
800
1000
1200
1400
1600
IPv4 IPv6
EVPN Agent

28. DE-CIX Frankfurt
IPv4 BUM traffic distribution in %
68%
31%
0,01%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Broadcast Unknown
Unicast
Multicast
Flood & Learn
85%
13%
0,6%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Broadcast Unknown
Unicast
Multicast
EVPN Agent

29. DE-CIX Frankfurt
IPv6 UM traffic distribution in %
1%
99%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Unknown Unicast Multicast
Flood & Learn
2%
98%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Unknown Unicast Multicast
EVPN Agent

30. DE-CIX Frankfurt
BUM traffic overall distribution by protocol
37%
63%
Flood & Learn
IPv4
IPv6
13%
87%
EVPN Agent
IPv4
IPv6

31. Bugs & Incidents
Ú Traffic distribution with LDP entropy labels not working
as expected
Ú IPv6 ND Agent replied with wrong source IP Address,
Software upgrade required

32. What is next for our Peering LAN?
Customer Statistics &
Service Insights

33. Creating yet another new software stack

36. Thank you
for your attention!