1. Network Fabrics, L2 Multipath and L3
The traditional 3-tier (access/aggregation/core) architecture is undergoing a transformation due
to the adoption of server virtualization and higher speed Ethernet (10GbE and higher). The
general term for the new architectures are fabrics – a flatter architecture that looks to overcome
the limitations of Spanning Tree Protocol (STP). Each networking vendor has different
terminology and often multiple solutions to transform networking design. Standards implications
include: Layer 2 Multipath (L2MP) alternatives are being worked on by IETF (TRILL) and IEEE
(SPB) groups. An alternative to changing L2 to accommodate the new requirements is L3
routing. Below are some of the considerations that distinguish the various offerings and some of
the criteria on which to decide among the different architectural decisions. Note that this
discussion is orthogonal to FCoE; L2MP is not a prerequisite of multi-hop FCoE, especially if
routing from FCF to FCF.
Edits and comments to clarify the technology are welcome; please keep vendor-specific
comparisons to a minimum.
Contents
1 Layer 2 Multipath
o 1.1 Scalability
o 1.2 Virtualization
o 1.3 Status of the standards
o 1.4 Vendor solutions
2 Layer 3
o 2.1 Vendor solutions
3 What works best in your datacenter?
Layer 2 Multipath
Scalability
STP limits connectivity to a single connection for each device. As 10Gb Ethernet is adopted and
devices demand more data, inactive links become much more expensive and inefficient. Data
center architects typically will limit the scalability of a L2 environment and use vLANs to limit
performance issues from broadcasts or reliability issues from failures.
Virtualization
Virtualization increases the bandwidth between servers (edge-to-edge rather than edge-to-core –
see a from Greg Ferro, aka Etherealmind, provides a good discussion with diagrams.) vMotion is
limited to L2 (see Ivan Pepelnjak aka IOSHints discussion of why not L3). In general, vMotion
2. is typically done in very local areas (first uses were typically between blades of a blade server or
servers in a rack) inside of a cluster which can be contained to a single L2 domain. vMotion
between clusters or even between data centers while possible, requires careful implementation
and should not be the primary driver for a network architecture.
Status of the standards
IETF (TRILL) was ratified in July 2011 (RFC 5556), while IEEE (SPB) is not yet a ratified
standard yet, but is generally complete from a technology standpoint as they work through the
machinations of the standards process.
Vendor solutions
Cisco has stated that TRILL will be supported when the standard is done, in addition they
are shipping FabricPath, which they consider a superset of TRILL.
Brocade VCS includes both DCB and data plane TRILL functionality. They use FSPF
instead of IS-IS in the control plane, making their current implementation of VCS fabric
proprietary. Brocade does plan to support standards-based TRILL in the future.
HP will support both SPB and TRILL (see this HP blog post).
Juniper QFabric is architected as a “single tier”, so does not require L2MP between what
is traditionally the edge and core.
Dell/Force10 Open Cloud Networking includes FTOS advanced software features
including TRILL (Transparent Interconnection of Lots of Links), DCB (Data Center
Bridging), EVB (Ethernet Virtual Bridging), and VEPA (Virtual Ethernet Port
Aggregation).
Layer 3
Layer 3 functionality is supported in most core switches and many edge switches. Using L3 for
scalability allows administrators to limit L2 domains to small islands that will not require L2MP.
As noted above, vMotion is limited to L2.
Vendor solutions
Cisco Nexus 7000, Nexus 5548/5596 (earlier Nexus 5000 series switches are L2 only)
and Nexus 3000 switches support L2/L3.
Juniper QFX3500 supports L2/L3.
Dell/Force10 switches support L2/L3.
HP IRF is a combination of L2/L3.
Arista Networks switches support L2/L3
What works best in your datacenter?
Just like the protocol arguments in the storage world, customers will need to choose the solution
that works best for the scale, skill-set, equipment and applications for their environments. There
3. are also implications of the choice to L4-L7 services, see this site from Intel/Cisco as a reference.
Data centers are going to be a mix of L2, L3, next-gen-L2 for a long time.
Action Item: While every vendor is discussing the latest "fabric" technology, IT
organizations are grappling with the ripple effects of virtualization on networks and
determining how to roll out new architectures with 10Gb Ethernet (and watching
40Gb/100Gb Ethernet coming soon). CIOs and Network Architects should look for proof
points of proven architectures that have been implemented by peers. Vendors need to focus
on delivering best practices and services to help customers make an informed decision on
designing a flexible network architecture.
Footnotes: More on Data Center Ethernet Fabrics
categories
Arista Networks, Cisco, Ethernet, HP, Juniper, Networking, Professional alerts
