The document discusses the rise of the modern data center and CLOS networks as the new architecture that is well-suited for modern data center needs. A CLOS network topology is scalable, provides fine-grained failure domains, and simplifies network design using only IP without other complex protocols. This architecture coupled with network virtualization enables agility, flexibility, and simplified management of large scale data center networks.

1. v
The House That CLOS Built
Network Architecture For the Modern Data Center
Dinesh Dutt, Chief Scientist
20th August, 2014

2. Cumulus Networks: Making Networks Accessible
Mission
To enable high capacity networks that are easy to deploy and affordable helping
customers realize the full promise of the software-defined data center.
Vision
Bringing the Linux Revolution to Networking

3. cumulusnetworks.com 3
• The Rise of the Modern Data Center
• Networks 2.0: The New World
Agenda

4. cumulusnetworks.com 4
• The Rise of the Modern Data Center
• Networks 2.0: The New World
Agenda

5. The Winds Of Change Are Blowing Through Networking
cumulusnetworks.com 5
Image credit: http://anciss.deviantart.com/art/you-can-change-the-world-81681894
SDN
Bare Metal Switching
DevOps
Commoditization of Networking
Network Virtualization

6. A Key Transformation
cumulusnetworks.com 6
SPINE
LEAF
CORE
L2
ACCESS
L3
AGGREGATION

7. What Changed ?
§ A new generation of applications:
§ Search
§ Big Data
§ Clouds
§ Other Web 2.0 Applications
cumulusnetworks.com 7

8. Characteristics of the New Applications
§ Traffic Pattern
§ Between servers (East-West) instead of client-server (North-South)
§ Scale
§ 10s of thousands to 100s of thousands of endpoints
§ Agility
§ New endpoints and racks powered up in hours instead of weeks
§ New networks spun up in seconds instead of weeks
§ Flexibility
§ Ability to reuse same infrastructure for different applications
§ Resilience
§ Fine grained failure domain
cumulusnetworks.com 8

9. Where The Existing Topology Falls Short
cumulusnetworks.com 9
§ Not suited for E-W traffic
§ Heavy-core, lean edge design
is not scalable
§ Not Agile
§ Inflexible Design
§ Coarse-grained failure domain
§ Unpredictable Latency
CORE
L2
ACCESS
L3
AGGREGATION

10. The Case Against Complexity
cumulusnetworks.com 10
§ Too many protocols
§ Many proprietary (MLAG, vPC, for
example)
§ STP and its variants, its myriad
knobs, UDLD, Bridge Assurance,
LACP, FHRP (VRRP, HSRP, GLBP),
VTP, MVRP, etc. etc.
§ Dual redundancy only adds to
the complexity mess
§ Dual control planes
§ HA
§ ISSU etc.
§ Complex Failure Modes
CORE
L2
ACCESS
L3
AGGREGATION

11. Twilight In The Land of Godboxes
§ Network's Function is to serve the application needs
§ Existing Network design is a bad fit for the modern DC
application
11
Image credit: http://bestandworstever.blogspot.com/2012/07/best-lane-ending-sign-ever.html

12. cumulusnetworks.com 12
• The Rise of the Modern Data Center
• Networks 2.0: The New World
Agenda

13. CLOS Network
cumulusnetworks.com 13
§ Invented by Charles Clos
in 1953
§ How to build ever larger
telephony networks
without building ever
larger telephony switches
§ http://en.wikipedia.org/
wiki/Clos_network
SPINE
LEAF

14. Characteristics Of CLOS Network
cumulusnetworks.com
§ Well matched for E-W traffic pattern
§ Scalable network topology
§ Reliance on ECMP leads to simple IP-
based fabrics
§ Fine grained failure domain
§ Predictable latency
§ Coupled with network virtualization,
serves as a basis for agility and
flexibility
SPINE
LEAF
14

15. Scalable Network Topology
15cumulusnetworks.com

16. Some Common Terminology
16cumulusnetworks.com
CLUSTER or POD CLUSTER or POD
INTER-CLUSTER SPINE

17. Calculating Number of Attachable Servers: Non-Blocking Math
§ If m, n, o are number of ports on a switch at a given tier
§ Total number of servers attached to a 2-tier fabric:m*n/2
§ Total number of servers attached to a 3-tier fabric: m*n*o/4
cumulusnetworks.com 17
m
n
o

18. Port Math From A Different Perspective
§ Number of spines and ISL link speed is a function of
failure characteristics, cost and cabling simplification
cumulusnetworks.com 18
Number of pods/DC
Number of servers/rack
Number of racks/pod

19. Calculating Number of Servers: Some Concrete Numbers
2-tier 3-tier
Theoretical Non-Blocking
Trident+ 64 port 10GE 2K 64K
Trident2 96 port 10GE 4608 216K
More Pragmatic Numbers: 40 servers/rack
Trident+ (48x10GE, 4x40GE)
Oversubscription: 2.5 (with 4 spines)
1920 (40*48) 60K(40*24*6
4)
Trident2 (48x10GE, 6x40GE)
Oversubscription: 1.6
1920 184K
cumulusnetworks.com 19

20. Paganini Variations
20cumulusnetworks.com

21. Fine Grained Failure Domain
cumulusnetworks.com 21
§ Any link or spine failure
leads to only a 25%
loss instead of 50%
with traditional design
§ Webscale companies
do 8-way and 16-way
ECMP to mitigate single
failure effects

22. The Case for Simplicity
§ Reduced number of protocols
§ Single IP protocol is sufficient
§ No FHRP, STP, the myriad L2 variants
cumulusnetworks.com 22

23. Coda For The Godboxes
§ All boxes are the same
§ Simplified inventory control
§ Simplified management
§ Reduced latency compared to chassis-based boxes
§ Simple 1RU boxes means simple failures
§ Replace failed box and continue instead of debugging in
place
§ No ISSU, HA, etc.
§ Developing new apps to run on platform is simpler
cumulusnetworks.com 23

24. What About Server Attach
§ Most, if not all, webscale companies do single
attach
§ They have so many racks, failure of one doesn’t bother
them
§ Smaller enterprises that cannot sustain the loss of
a rack dual-attach servers
§ Switches are in the same rack or cross connected pair
of racks
cumulusnetworks.com 24

25. Does This Apply Only To Big Guys ?
§ Depends
§ Many small guys are moving to cloud
§ If the application requirements are not L2-specific, this
applies to small guys too
§ Simply think of the cost of bare metal switches to boxes from
traditional vendors
§ Many small IT shops are run by Linux/server admins
§ Script savvy and so can setup networks faster
§ More in next webinar
cumulusnetworks.com 25

26. Fitting Existing Applications: Network Virtualization
§ With network virtualization technologies such as
VxLAN, you can create L2 overlays over the L3
fabric
§ Separating virtual network from physical network
provides for agile network management
§ Can run both new applications such as Hadoop and
memcached along with more traditional apps on the
same network: flexibility
cumulusnetworks.com 26

27. And In Closing..
§ Existing Access-Agg-Core design is slowly making
way for CLOS fabrics
§ CLOS, L3-based fabrics are simple, scalable,
flexible and agile
§ Managing a CLOS fabric including routing and
such is the topic of the next webinar
cumulusnetworks.com 27

28. Resources available
§ Website: cumulusnetworks.com
§ Blog: cumulusnetworks.com/blog
§ Twitter: @CumulusNetworks @ddcumulus
cumulusnetworks.com 28

29. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other
countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from
LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their respective owners.
§ Thank You!
cumulusnetworks.com 29
Bringing the Linux Revolution to Networking