Form follows function is a modern architectural principle that has been used to design and understand the workings of organisms and buildings. Computer networks are also an example of this principle. The classical networking topology (or form) inside an enterprise (and data center) has been the access-aggregation-core model that was designed to serve the needs of the applications then vogue in the enterprise. Enter the 21st century: companies like Google and Amazon, and applications based on cloud, big data and web 2.0 are redefining the fundamental morphology of data center networks. This talk introduces the fundamental form of modern data center networking and discusses how form follows function in this brave new world. The talk will range from the application needs of the modern data center and how they redefine the network requirements to the most common topology in modern data centers to the protocols used and new technologies.

1. Data Center Topologies
Morphology of Modern Data Center Networks
Dinesh G Dutt | Chief Scientist

2. Dawn of the Modern Data Center
Network 2.0
Routing
Taming the Configuration Beast
Agenda
10/1/13 2YAC 2013
(Pictures courtesy of Wikimedia, where not stated)

3. Dawn of the Modern Data Center
Network 2.0
Routing
Taming the Configuration Beast
Agenda
10/1/13 3YAC 2013

4. Evolution of the Data Center Application
10/1/13 4YAC 2013
Traditional Enterprise
Applications
§ L2-centric
§ Sensitive to network failures
§ Mostly static
§ VLANs
§ No Server Virtualization
§ Mostly North-South
Lower Capacity
100s-few thousand endpoints
Modern Data Center
Applications
§ IP-centric
§ Workaround network failures
§ Dynamic
§ Clouds
§ Server Virtualization
§ Mostly East-West
High Capacity
Thousands to millions of
endpoints

5. Challenges:
§ Large failure domain
§ Agg box failure
§ Unscalability of agg boxes
§ MAC/ARP
§ VLANs
§ Choke point for E-W
§ Complex
§ HA
§ Too many protocols
§ Many proprietary enhancements
§ Each vendor has their version of the same feature
10/1/13 5YAC 2013
Traditional Enterprise DC Network Design
L3
L2
Access
Aggregation
Core
VRRP VRRP
STP/VTP/GVRP/UDLD
ECMP
STP/VTP/GVRP/UDLD

6. 10/1/13 6YAC 2013
Network's Function is to Serve the Application Needs

7. Dawn of the Modern Data Center
Network 2.0
Routing
Taming the Configuration Beast
Agenda
10/1/13 7YAC 2013

8. Folded CLOS Network
10/1/13 8YAC 2013

9. § ECMP
§ IP fabric ubiquitous
§ Better Failure Handling
§ Predictable Latency
§ Simple Feature Set
§ Scalable
§ L2/L3 Boundary
§ ToR vs. EoR design
Characteristics Of CLOS Network
10/1/13 9YAC 2013
LEAF
SPINE

10. Calculating Network Size
10/1/13 10YAC 2013
TIER-1
TIER-2
TIER-3
2 Tier Fabric
For smaller environments
3 Tier Fabric
For large-sale environments
Pods can be of dissimilar size
LEAF
SPINE

11. Calculating Network Size
10/1/13 11YAC 2013
2 Tier Fabric
• #ports @ToR = (m*n)/2
• Max #ports @ToR= 2K with
64px10GE at Tor/Spine
• Max #ports @ToR= 4608 with
96px10GE at Tor/Spine
3 Tier Fabric
• #ports @ToR = (m*n*o)/4
• Max #ports @ToR= 65K with
64px10GE at Tor/Spine/Spine
• Max #ports @ToR= 884K with
96x10GE at Tor/Spine/Spine
m
m
n
on

12. Oversubscription & Such
10/1/13 12YAC 2013
Number of servers: Number of uplinks
Non-blocking after this first layer
Using Trident and 40 servers per rack:
Oversubscription is 2.5
Using Trident2 in same config:
Oversubscription can be 1

13. Paganini Variations
10/1/13 13YAC 2013

14. Size Does Matter
Fine grained failure domain
Large boxes vs small boxes
Interconnect link
Scheduling Downtime
Trying on new clothes
Multi-vendor
10/1/13 14YAC 2013
Failure Analysis

15. Dawn of the Modern Data Center
Network 2.0
Routing
Taming the Configuration Beast
Agenda
10/1/13 15YAC 2013
Picture courtesy Nanoer.com @flickr

16. What Protocol
Link state (OSPF/ISIS) or BGP
Managing IPv4/v6
Separate session/protocol or unified
Multi-Vendor Support
Deployment Experience
10/1/13 16YAC 2013
Questions That Affect Routing Protocol

17. Commonly deployed protocol within
enterprises
Simplify config:
Only 2 area IDs, backbone and non-BB
Unnumbered interfaces
Run OSPFv3 also if you have IPv6
Route summarization possible, not desired
due to non-optimal routing
10/1/13 17YAC 2013
OSPF
Backbone area
Area 0.0.0.1 Area 0.0.0.1

18. Simple up-down routing
Use Private AS numbers
Route summarization not possible
Interface addresses only
Single BGP session for v4/v6 or
separate sessions
10/1/13 18YAC 2013
eBGP
ASx ASx1 ASx2 ASxn
ASy1 ASy1 ASy1 ASy1ASy ASy ASy ASy
ASz ASz ASz ASz
ASx3 ASx ASx1 ASx2 ASxnASx3

19. Simple up-down routing
No IGP
Eliminates AS number distraction
Use of NH Self with RR
Single-hop BGP peer, use interface
address
Single BGP session for v4/v6 or
separate sessions
10/1/13 19YAC 2013
iBGP
RR RR RR RR RR RR RR RR
RR RR RR RR

20. VM
VM VM
Logical
switch
Great fit for modern data center apps
Layer complex applications such as
clouds as an overlay
L2 as a service
10/1/13 20
Network Virtualization
YAC 2013

21. Dawn of the Modern Data Center
Network 2.0
Routing
Taming the Configuration Beast
Agenda
10/1/13 21YAC 2013

22. To err is human, to automate divine
But traditional networking gear is a black box
OS functions more like an embedded OS
No programmable way to configure the box
Primitive network management tool chain
Vendor-specific
10/1/13 22YAC 2013
Automate Configuration

23. Turn Black box into White & use Linux as the network OS
Why Linux ?
Well established and open API
Vibrant community fueling innovation
Sophisticated management tool chain
Excellent networking support
Linux As The Network OS
10/1/13 23YAC 2013

24. Server management tools to manage networks
Puppet, Chef, Ansible or in house
Common Toolset
10/1/13 24YAC 2013

25. Verify connectivity is as per operator
specified cabling plan
User defined actions on topology check
result
§ For example, routing adjacency is brought up
only if physical connectivity check passes
Example:
§ T1, port1 is connected to M1, port1
§ T1, port2 is connected to M2, port1
§ …
§ M1, port 3 is connected to S1, port1
§ M1, port 4 is connected to S2, port1
…
10/1/13 25YAC 2013
Validating Physical Topology
S2
M2
M1
T2
T1
M4
M3
T4
T3
S1

26. Graphviz: Network topology specified via
DOT language
Well understood graph modeling language
Wide range of supported tools
Open source
Central management tool: Network
topology is pushed out to all nodes
Each node determines its relevant information
LLDP: Use the discovery protocol to
verify connectivity
Graph G {
S1:p1 – M1:p3;
S1:p2 – M2:p3;
S1:p3 – M3:p3;
S1:p4 – M4:p3;
S2:p1 – M1:p4;
S2:p2 – M2:p4;
S2:p3 – M3:p4;
S2:p4 – M4:p4;
M1:p1 – T1:p1;
M1:p2 – T2:p2;
…
M4:p2 – T4:p2;
}
10/1/13 26YAC 2013
ptmd: Prescriptive Topology Manager
h*ps://github.com/CumulusNetworks/ptm

27. CLOS Fabric as the foundation for modern data center networks
Layer Complex applications such as Clouds on top with overlays
Automate Configuration & Simplify Networking
Linux as the network OS to use sophisticated management tools
Simplify networking further with tools such as ptmd
10/1/13 27YAC 2013
Conclusion

28. 10/1/13 28YAC 2013
www.cumulusnetworks.com
ddutt@cumulusnetworks.com
@cumulusnetworks
Spasibo!
Web:
Email:
Twitter: