1) The document discusses using VXLAN, BGP and EVPN to implement a layer 3 network for a cloud deployment using Ceph and CloudStack. This allows scaling beyond the limits of layer 2 networks and VLANs. 2) Key infrastructure components discussed include Dell S5232F-ON switches running Cumulus Linux, SuperMicro hypervisors and Ceph storage servers using NVMe SSDs. 3) The deployment provides high performance private and public cloud infrastructure with scalable networking and over 650TB of reliable Ceph storage per rack.

1. Ceph and CloudStack
24-10-2019, London

2. WiFi
UoL Conferences

3. Social Media
#cephcloudstack

4. Thanks to today’s sponsors

5. Schedule
- Online at https://ceph.io/cephdays
- Highlights:
- Short break at 10:45
- Lunch at 12:00
- Short break at 15:15
- Pub (the Gallery, 1st floor) at 17:00

6. VXLAN+BGP+EVPN
CloudStack & Ceph
Wido den Hollander

7. Who am I?
- Wido den Hollander (1986)
- Dutch, live in the Netherlands
- Ceph, CloudStack and IPv6 guru
- Open Source lover!
- Ceph Trainer and Consultant
- CTO at 42on

8. 42on
- Founded the company in 2012
- Company specialized in Ceph
- Consultancy
- Training
- Emergency Assistance
- https://42on.com/

9. PCextreme
- Co-founded this Dutch hosting company in 2004
- Traditionally a domain + webhoster
- Transitioned into providing infrastructure for many other hosting companies
- Running cloud deployment with Ceph and CloudStack
- Using the KVM hypervisor
- Numbers:
- 80k customers
- >100k domain names
- ~20k Virtual Machines
- ~3PB of Ceph storage

10. Layer 2 networks
- When trying to scale a network Layer 2 becomes difficult
- Redundancy is a problem
- Reliability is a challenge
- L2 works great in smaller scale networks
- We are trying to eliminate them on our cloud deployment
- Usually a 4k limit on the total amount of VLANs

11. Layer 3 networks are better!
- More flexible
- More reliable
- More scalable
- They are just better! (imho)

12. Compute 2.0
● Initiated end of 2018
● Existing Cloud(Stack) and Ceph clusters were based on Layer 2 networking
● Goals:
○ Increase flexibility
○ Better scalability (more virtual networks)
■ Less (or no) Layer 2 networking
○ As Open Source as it can be
○ Underlying network should be IPv6-only!

13. IPv6
Why use technologies of the future with the IP protocol from the past?

15. VXLAN
"Virtual Extensible LAN (VXLAN) is a network virtualization technology that
attempts to address the scalability problems associated with large cloud
computing deployments. It uses a VLAN-like encapsulation technique to
encapsulate OSI layer 2 Ethernet frames within layer 4 UDP datagrams, using
4789 as the default IANA-assigned destination UDP port number."

16. VXLAN
Ethernet packets encapsulated inside UDP
● This means you can route them through an existing Layer 3 network
● MTU of underlying network needs to be >1580
○ We use 9216
● VXLAN segments are called VNI
○ You can create 16M VNIs!

17. VXLAN & Multicast
● Default VXLAN uses Multicast to exchange VNI, IP and MAC information with
other hosts
● Multicast has scalability problems
● Difficult to route Multicast to different Layer 2 segments

18. The solution: VXLAN+BGP+EVPN
● Use the Border Gateway Protocol (BGP) for exchanging VNI, IP and MAC
information between hosts
● Allows for scaling VXLAN over multiple Layer 3 segments and further
○ Multi DC if required
● Good blog: https://vincent.bernat.ch/en/blog/2017-vxlan-bgp-evpn

19. VXLAN & EVPN
Ethernet VPNs (EVPNs) enable you to connect groups of dispersed customer
sites using Layer 2 virtual bridges, and Virtual Extensible LANs (VXLANs) allow
you to stretch Layer 2 connectivity over an intervening Layer 3 network, while
providing network segmentation like a VLAN, but without the scaling limitation of
traditional VLANs.
Source: Juniper.net

20. VXLAN+BGP+EVPN

21. VXLAN & CloudStack
● Properly supported in 4.12 release
○ We (PCextreme) pushed multiple commits to polish and improve support
● BGP+EVPN only supported with customized script
○ https://github.com/PCextreme/cloudstack/blob/vxlan-bgp-evpn/scripts/vm/network/vnet/modifyvxlan.sh
● Configuring Frrouting (BGP) is not done by CloudStack
○ Frr learns new VNIs automatically when CloudStacks spawns a new VM
● Security Grouping (Firewalling) works
○ With IPv6 support!

22. Cumulus Linux
● Debian Linux based Operating System for ONIE switches
○ "Open networking software for the modern data center"
● Support for
○ VXLAN
○ BGP
○ EVPN
● You can run your own (Linux) scripting/tooling on the switch!
○ We run the Salt Minion for provisioning on these switches

23. Dell S5232F-ON
● 32x100Gbit switch with ONIE support
○ Atom CPU
○ 4GB Memory
● Modern Broadcom Trident3 chipset
○ VXLAN offloading in the chipset
○ This does all the actual heavy lifting. Cumulus just programs the chipset
● Affordable
○ ~EUR 3.500,00 (ex VAT) for a single switch
● Supported by Cumulus Linux
● We use them for two purposes
○ Core routers for this cloud deployment
○ Top-of-Rack for Hypervisors

24. Dell S5232F-ON

25. Hypervisors
● 1U SuperMicro with Dual AMD Epyc
○ AS-1123US-TR4
● 1TB DDR4 Memory
● 10x Samsung 2TB NVMe for local storage
● 2x 100Gbit Mellanox ConnectX-5
○ VXLAN offloading support
● Affordable!
○ ~EUR 17.500 per server

26. Hypervisors

27. Ceph OSD
● 1U SuperMicro with Single AMD Epyc 7351P
○ 1113S-WN10RT
● 128GB DDR4 Memory
● 10x Samsung 4TB NVMe
○ PM983 SSDs
● 2x 10Gbit to top of rack
● Affordable
○ ~EUR 8.500 for a single system

28. Ceph OSD

29. Networking on each host
● Hypervisors and Ceph servers run BGP on the host
○ We are using Frrouting
● No LACP, Bonding, Teaming to servers
○ This means no Layer 2 to hosts!
● All hosts announce their IPv6 loopback address through BGP
○ This is a /128 route
● Jumbo Frames enabled on all hosts
○ MTU is set to 9000
○ IPv6 MTU Path Detection makes sure this also works towards the internet

30. BGP configuration
hostname hv-138-a05-23.xxx.yyy.eu
!
interface lo
ipv6 address 2a05:1500:601:2::9/128
!
router bgp 4200100123
neighbor enp81s0f0 interface peer-group uplinks
neighbor enp81s0f1 interface peer-group uplinks
!
address-family ipv6 unicast
network 2001:db8:601:2::9/128
neighbor uplinks activate
exit-address-family

31. BGP Unnumbered
● Each host creates a BGP session with both Top-of-Rack switches
● Using BGP Unnumbered the BGP sessions are created over IPv6 Link-Local
○ No need to manually assign IP(v6) addresses to hosts and switches
○ VXLAN and IPv4 traffic is routed over IPv6
○ Super easy to scale!
BGP neighbor on enp81s0f0: fe80::225:90ff:feb2:bcdf, remote AS 4200100002, local AS
4200100123, external link
Hostname: tor-138-a05-46
..
..
Local host: fe80::ba59:9fff:fe20:6e22, Local port: 39020
Foreign host: fe80::225:90ff:feb2:bcdf, Foreign port: 179

33. Scaling
● We scale by expanding and building multiple Ceph clusters
○ We choose to NOT grow a Ceph cluster larger than a Single 19" rack
○ This way we prevent that our whole cloud environment goes down due an issue with a single
Ceph cluster
○ CloudStack supports multiple Ceph clusters
● Layer 2 (VLAN) scaling is no longer a problem!
○ We can add as many hypervisors as we think CloudStack (safely) supports
○ No need to stretch VLANs over multiple racks
● Hypervisors can be spread out over multiple racks
○ They consume about 0,5kW each. Maximum per rack is ~5KW

34. Scaling
● With VLANs we had to be conservative
● Now we can provide isolated networks for customers on request
○ We can route their own IP-space
○ Additional traffic policies can be set

35. Ceph clusters
● One cluster per rack
● 3 MONs
● As much OSD machines as we can safely fit
○ Power consumption restrictions (5~6kW) apply
● ~650TB of RAW Ceph storage per rack
○ 100% NVMe storage
○ We are considering using 8TB NVMe drives instead of 4TB
● If needed we could deploy a HDD-only Ceph cluster for slow bulk storage

36. Performance
● Excellent!
● We can saturate the 100Gbit links
● We achieve a ~0.8ms write latency
○ 4k write size
○ 3x replication

37. Ceph and BGP
● Just works
○ Ceph only wants a working Layer 3 route to other hosts
● Need to define public_addr in ceph.conf
○ public_addr = 2001:db8:601:2::7
● No further configuration needed for Ceph

38. Conclusion
● Scalable networking solution without the use of VLANs
● We can easily create Virtual Networks for customers on our CloudStack
deployment(s)
○ Announce their (public) IP-space (v4/v6) and route it to their VMs
● High performance Ceph environment providing reliable and fast storage
○ Risks mitigated by creating multiple Ceph clusters

39. Thanks!
Wido den Hollander
wido@denhollander.io
@widodh
https://blog.widodh.nl/