Configuration & Routing of Clos Networks

v
Running the House That CLOS Built
Configuration of CLOS Fabrics
Dinesh Dutt, Chief Scientist
17th September, 2014

Cumulus Networks: Making Networks Accessible
Vision
Bringing the Linux Revolution to Networking
Mission
To enable high capacity networks that are easy to deploy and affordable helping
customers realize the full promise of the software-defined data center.
September 17, 2014 cumulusnetworks.com 2

A Key Transformation
SPINE
LEAF
AGGREGATION L3
cumulusnetworks.com 3
CORE
L2
ACCESS
September 17, 2014

Simplifications Wrought By L3
§ Fewer protocols to configure
§ Single routing protocol vs many L2 protocols
§ No FHRP required
§ Standard, inter-operable protocols
§ Fewer protocols to troubleshoot
§ You can traceroute across the network

Agenda
§ Intro
§ Configuration & Automation
§ PTM
§ Outro

• Intro
Picture courtesy Nanoer.com @flickr
September 17, 2014

Routing vs Bridging
§ Forwarding based on IP address, not MAC
§ Loop-free protection in data plane and control plane
§ Not just determined by a control protocol (STP)
§ Routing protocol assumes peer is down and not
reachable if it doesn’t hear from it
§ STP takes ports into forwarding if it doesn’t hear from the
other end
§ Routing protocols can work with simple and
sophisticated use cases

Basic Parts To A Routing Routing Protocol
§ Three basic parts
§ Who do I communicate with (neighbor, peer etc.)
§ What do I tell them (IP prefixes usually)
§ Tuning the conversation (timers, various protocol
specific knobs)
§ But first, who am I ?
§ Router ID

Comparing STP vs Routing Configuration
§ auto br0
§ iface br0 inet static
§ bridge-ports swp1 swp2 swp3
§ bridge-stp on
§ mstpctl-treeprio 32768
§ mstpctl-portautoedge swp1=yes
§
§ router ospf
§ router-id 0.0.0.1
§ interface swp1
§ ip ospf area 0.0.0.0
§ ip ospf network point-to-point
§ router bgp 65535
§ bgp router-id 0.0.0.7
§ neighbor 1.2.3.4 remote-as 65534
§ neighbor 1.2.3.4 activate
§ redistribute connected
Who Am I
Who Do I Talk To
What Do I Tell Them
Fine Points Of Conversation
September 17, 2014

The Nice Thing About Routing Protocols…
§ Link state vs Distance Vector
§ In distance vector, you tell your neighbors what you know
about the world
• More precisely, a node provides a summarized view of the world to its
neighbors and each neighbor then examines the summaries from all
its available neighbors to pick the best paths to use for
communication
§ In link state, you tell the world about your neighbors
• More precisely, every node has an exact copy of what every other
node said and uses this to compute reachability to each of the other
nodes
• Comparatively chatty protocol
• Network design to configure protocol to reduce chattiness

• Configuring CLOS Networks
September 17, 2014

Sample Topology
CLUSTER or POD CLUSTER or POD
S1 S2
M1 M2 M3 M4
INTER-CLUSTER SPINE
T1 T2 T3 T4
September 17, 2014

Using OSPF
§ Popular link-state protocol used in
enterprises
§ Separate versions for IPv4 and IPv6
§ Chattiness is reduced by defining
neighborhoods (called areas)
§ All nodes in a neighborhood have a
precise copy of the info from the other
neighbors in the neighborhood
§ Information outside the neighborhood
is summarized by the border nodes
§ M1& M2 are the border nodes for T1
and T2. Similarly M3 and M4 for T3
and T4.
§ Two level neighborhood hierarchy
support only
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
September 17, 2014

Using OSPF (example configuration)
§ router ospf
§ ! On M1
§ log-adjacency-changes detail
§ router-id 0.0.1.1
§ passive-interface lo
§ interface lo
§ ip ospf network loopback
S1 S2
area 0.0.0.0
M1 M2 M3 M4
Area 0.0.0.1. Area 0.0.0.1.
T1 T2 T3 T4
September 17, 2014

Using BGP
§ Swiss-army knife of routing
§ Powers the Internet
§ Quite a simple path-vector (variant of
distance vector) protocol
§ Configuration can be made hairy, but
can be kept simple
§ Multi-protocol support
§ Used without any other routing
protocol in play in the DC
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
September 17, 2014

Using BGP
§ ! M1 config
§ ! As defined by draft-lapukhov-bgp-..txt
§ bgp log-neighbor-changes
§ maximum-paths 64
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
September 17, 2014

• Simplifying The Configuration
September 17, 2014

Key Characteristics of Any Automatable Configuration
§ Cookie cutter configuration
§ As little node-specific variation as possible
• Nothing more than a single IP address, node name, for example
§ As little duplication of information as possible
• Specifying IP addresses of interfaces AND in OSPF/BGP network
statements
§ As much configuration as necessary, not more

OSPF Unnumbered Interfaces
§ IP links are configured to have an IP address at either end in
traditional network design
§ Adds to node-specific config
§ Adds to IP addresses to be allocated
§ IPv6 is very cool with link local address
§ If these addresses are advertised, they can become attack vectors
§ OSPFv2 has a concept called unnumbered interfaces
§ Single IP address per node, usually assigned to loopback interface,
inherited by all the others
§ Simplifies interface configuration, if not routing configuration

Simplifying BGP Configuration
§ BGP uses TCP as its transport which implies the
need for an IP address in the specification
§ Violates multiple cookie cutter configuration
metrics:
§ IP address specified in multiple places (neighbor
statement, IP interface definition etc.)
§ Configuration all become node specific

Unnumbered BGP Configuration
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
§ ! M1 config
§ neighbor swp1 remote-as 64512
September 17, 2014

Unnumbered BGP: More Details
§ Uses IPv6 RA to determine link local address of
remote peer
§ Sets up session over link local address
§ Works well for IPv6 today, IPv4 in the works
§ Interoperates with other implementations that
can’t support this mechanism, but can peer over
link local address

Using iBGP For Further Simplification
§ ASN forces routing configuration to be node-specific
§ iBGP can be used to force configuration to be a
simple three template model:
§ One set of configuration each for leaf, pod-level spine
and inter-pod level spine
§ All leaves get the same configuration
§ All pod-level spines get the same configuration etc.

Using iBGP to Get Rid of ASN
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
§ ! M1 config
§ neighbor TOR peer-group
§ neighbor TOR remote-as 64500
§ neighbor TOR route-reflector-client
§ neighbor TOR next-hop-self-all
§ neighbor TOR activate
§ neighbor swp1 TOR
§ neighbor swp2 TOR
§ neighbor IPS peer-group
§ neighbor IPS remote-as 64500
§ neighbor IPS next-hop-self
§ neighbor IPS activate
§ neighbor swp3 IPS
§ neighbor swp4 IPS
§ maximum-paths ibgp 64
September 17, 2014

• PTM (Prescriptive Topology Manager)
September 17, 2014

Sorting through the Cables
§ Networking cabling complexity
§ As networks scale, the number of cables increases
§ How do we ensure cabling correctness in an easy way?
Expected
Topology
Design
Network
Blueprint
Cable
Install
Observed
Steady
State Failures Re-
Cabling
September 17, 2014

Network Cabling Errors
§ Improper cabling causes
§ Reachability issues
§ Unpredictable and low
performance
“To err is human”
--Alexander Pope
September 17, 2014

Prescriptive Topology Manager (PTM)
§ Define expected topology using DOT language
§ Verify connectivity per topology plan using LLDP
§ Take dynamically defined actions based on match of expected & actual
Graph
G
{
S1:p1
–
M1:p3;
S1:p2
–
M2:p3;
S1:p3
–
M3:p3;
S1:p4
–
M4:p3;
S2:p3
–
M3:p4;
S2:p4
–
M4:p4;
M1:p1
–
T1:p1;
...
M4:p2
–
T4:p2;
}
Topology Graph
S1 S2
M1 M2 M3 M4
T1 T2 T3 T4
September 17, 2014

Prescriptive Topology Manager (PTM) – Work Flow
Graphviz
Topology.dot Configuration
Management
Tool
Matching
design workflow
NOT matching
design workflow
September 17, 2014

Prescriptive Topology Manager (PTM) – Work Flow (contd)
Matching design workflow
• Notify engineers
• Work around problem
LLDP Topology.dot
PTMd
neighbors
• Configure interfaces
• Add self to monitoring…
NOT matching design workflow
September 17, 2014

PTM Pre-defined Hookups
§ With Quagga
§ Quagga acts as PTM client
§ Listens to link-state and BFD notifications
cumulus@S1:~$
sudo
vtysh
-‐c
'conf
t'
-‐c
'ptm-‐enable'
cumulus@S1:~$
sudo
vtysh
-‐c
'show
interface
swp1'
Interface
swp1
is
up,
line
protocol
is
up
PTM
status:
pass
index
3
metrix
1
mtu
1500
flags:
<UP,BROADCAST,RUNNING,MULTICAST>
HWaddr:
00:02:00:00:00:11
inet
21.0.0.2/24
broadcast
21.0.0.255
inet6
fe80::202:ff:fe00:11/64
cumulus@S1:~$

PTM: Running State
§ ptmctl (contd)
§ Shows expected and observed topology
§ Comparison test results – pass or fail
root@cumulus:~# ptmctl
------------------------------------------
port cbl BFD BFD
status status peer
------------------------------------------
swp13 pass pass 51.0.0.41
swp20 fail N/A N/A

ptmd: Running At A Git Near You
§ Open
source,
published
under
Eclipse
Public
License
(EPL)
§
h6ps://github.com/CumulusNetworks/ptm

• PTM 2.0
September 17, 2014

Mapping Gods Eye View To Box Specific Config
§ A fundamental problem with networking
configuration is the inability to map network-wide
view to box-specific configuration
§ PTM provides a network-wide configuration
mechanism

Generate Routing Configuration From PTM
Graph
G
{
S1:p1
–
M1:p3;
S1:p2
–
M2:p3;
S1:p3
–
M3:p3;
S1:p4
–
M4:p3;
S2:p3
–
M3:p4;
S2:p4
–
M4:p4;
M1:p1
–
T1:p1;
...
M4:p2
–
T4:p2;
}
§ ! M1 config

A Summing Up
§ CLOS topology allows for a simplified configuration
§ Routing configuration can be fairly simple
§ Coupled with PTM, BGP unnumbered interface and
other like innovations can make configuring L3
networks a snap

Resources available
§ Cumulus WB:
http://cumulusnetworks.com/cumulus-workbench/
§ cldemo: github.com/CumulusNetworks/cldemo
§ Website: cumulusnetworks.com
§ Blog: cumulusnetworks.com/blog
§ Twitter: @CumulusNetworks @ddcumulus

Bringing the Linux Revolution to Networking
§ Thank You!
CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other
countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from
LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their respective owners.
September 17, 2014

Configuration & Routing of Clos Networks

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (18)

Similar to Configuration & Routing of Clos Networks

Similar to Configuration & Routing of Clos Networks (20)

More from Cumulus Networks

More from Cumulus Networks (20)

Recently uploaded

Recently uploaded (20)

Configuration & Routing of Clos Networks