Audience Level Beginner Synopsis Layer 2 versus Layer 3, MLAG, Spanning-Tree, switch mechanism drivers, overlays and routing-on-the-host — What scales and what does not? The underlying plumbing of an OpenStack network is something you’d rather not have to think about. This presentation examines the network architectures of web-scale and large enterprise OpenStack users and how those same efficiencies can be used in deployments of all sizes. Speaker Bio: Scott is a Member of Technical Staff at Cumulus Networks where he designs, supports and deploys web-scale technologies and architectures in enterprise networks globally. Prior to becoming a founding member of the Cumulus office in Australia, Scott started his career as a network administrator before joining Cisco Systems to support their data centre products. OpenStack Australia Day Melbourne 2017 https://events.aptira.com/openstack-australia-day-melbourne-2017/

1. 1
June 1st, 2017
Scott Laffer | Cumulus Networks
OpenStack Networks
The Web-Scale Way

2. 2
Agenda
Who are we?
Why should you care about the network?
Tenant networks – What choices are there?
What do you need from your network?
Demo

3. 3
Cumulus Networks Confidential 3
Cumulus Networks brings Web-Scale Networking to
Enterprise Cloud
Economical scalability
With commodity hardware and a
standardized Linux stack, achieving
a lower TCO by up to 60%
Network OS
Open Hardware
Apps Apps Apps
1 2 3 4Built for the automation age Standardized toolsets Choice and flexibility
Making networking repeatable
and consistent
Easily enable Linux tools:
automation, monitoring, analytics…
50+ hardware platforms, from
11 vendors, and 2 silicon
BLACK BOX
Arista
Juniper
Cisco
CUSTOMER CHOICE
Unlocking the vertical network stack to build the modern data center
Cumulus Linux

4. 4
Why should you care about the network?

5. 5
ML2 Type Drivers
What type of networking is used to separate tenants?
Other
Linux
Bridge
OvS VendorOther GRE VLAN VxLAN
Core Plugin (ML2)
Type Manager
Type Driver
Mechanism Manager
Mechanism Driver
Flat Type Driver
All subnets in the same broadcast
domain. Commonly used for provider
networks with floating IP pools.
VLAN Type Driver
Every tenant network is assigned a
unique VLAN.
VXLAN Type Driver
Every tenant network is assigned a
unique VXLAN.

6. 6
ML2 Mechanism Drivers
What devices are we configuring the networks on?
Other
Linux
Bridge
OvS VendorOther GRE VLAN VxLAN
Core Plugin (ML2)
Type Manager
Type Driver
Mechanism Manager
Mechanism Driver
Open vSwitch
Dedicated network stack on compute
node using both kernel and user
space to provide VM connectivity.
Linux Bridge
Uses native Linux kernel networking
on compute node to provide Layer 2
and Layer 3 connectivity.
Switch Drivers
Installed on network node to work
with network switches.

7. 7
Compute: Linux Bridge with VLAN
VMVM
bridge - <>bridge - <>
subinterface
taptap
subinterface
802.1q trunk 802.1q trunk
802.1q bond
VMVM
bridge - <>bridge - <>
subinterface
taptap
tap tap
taptap
subinterface
vRouter
L3 Agent
DHCP AgentDHCP Agent

8. 8
Network: Linux Bridge with VLAN
Technologies Used:
§ MLAG between all layers
Considerations:
§ VLAN Scale
§ New tenant networks
PRE-PROVISION
MANUAL
SWITCH MECHANISM DRIVER
§ MLAG and STP scale
SWITCH OVERHEADS
§ Complex switch upgrade and failure scenarios
A time-tested and proven design common with many existing OpenStack deployments
ML2 Pair

9. 9
Network: Linux Bridge with VLAN
Technologies Used:
§ MLAG between compute and switch
§ IP routed Fabric
VXLAN between network layers
§ Switch Mechanism Driver
Hierarchical Port Binding
Considerations:
§ Higher networking complexity
MORE MOVING PARTS
§ MLAG Scale still of concern
§ VLAN and STP Scale concerns reduced
Introduces scalable L3 features and removes some L2 concerns at the expense of
complexity.
ML2 Pair
ECMP
L2
L3

10. 10
Compute: Linux Bridge with VXLAN
3
5
6
2
1
4
VXLAN –> Tunnel IP
Server1 Network Node
172.16.1.1
172.16.1.2
192.168.40.2192.168.40.3/24
VM
br-<random> br-<random> br-external
TAP
VXLAN-2061
eth0
eth0 eth0 eth0
swp1 swp8
swp47
vRouter
VXLAN-2061
Mgmt Network
203.0.113.1/24
203.0.113.2/24
1 1 2
1 1

11. 11
Network: Linux Bridge with VXLAN
Technologies Used:
§ IP routed Fabric
VXLAN between the hosts
§ Quagga routing daemon
SAME PACKAGE FROM CUMULUS LINUX
ENABLES UN NUMBERED ROUTING
Considerations:
§ VXLAN Offload NICs
NEED VXLAN AND TCP SEGMENTATION OFFLOADS
Simple, flexible and incredibly scalable. The best overall solution for modern OpenStack and
Docker deployments.
L3
ECMP
ECMP

12. 12
Unnumbered Routing – Simple DC Underlay
OSPF/BGP Unnumbered
§ No more /30 or /31s
§ Simple configuration
§ Just define the interface
§ Traceroute still works
router bgp 100
neighbor swp1 remote-as external
neighbor swp2 remote-as external
neighbor swp3 remote-as internal
Plug-and-play Layer 3 networking right down to the host

13. 13
Compare the pair
Challenge Layer 2 Layer 3 with Overlay
BUM traffic handling Flood and learn over the network L2 Population with ARP
suppression
Network redundancy MLAG and STP Equal Cost Multi-Path
Number of broadcast domains 4096 16.7 Million
Multi-site Network L2 extension required L3 reachability between hosts
Network switch involvement for new
tenant network creation
Switch mechanism driver.
TOR port to host mappings.
None
Bandwidth loss on network switch
failure
50% 1/N
Gracefully remove network switch
from fabric
No Yes

14. 14
Demo Time

15. 15
Thank you!
Visit us at cumulusnetworks.com or follow us @cumulusnetworks
© 2017 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus
Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark
Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.