Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modern Data Center Network Architecture - The house that Clos built


Published on

Presentation can be found at:

Listen in on an engaging discussion about network architecture that underpins virtually all of the modern data center. We'll delve into why people chose this architecture and several interesting emergent behaviors that this architecture gives rise to. We will also discuss how the modern data center architecture eases DevOps, failure characteristics and characteristics of network boxes. Featuring Cumulus Networks Chief Scientist, Dinesh Dutt

Published in: Software

Modern Data Center Network Architecture - The house that Clos built

  1. 1. v The House That CLOS Built Network Architecture For the Modern Data Center Dinesh Dutt, Chief Scientist 20th August, 2014
  2. 2. Cumulus Networks: Making Networks Accessible Mission To enable high capacity networks that are easy to deploy and affordable helping customers realize the full promise of the software-defined data center. Vision Bringing the Linux Revolution to Networking
  3. 3. 3 • The Rise of the Modern Data Center • Networks 2.0: The New World Agenda
  4. 4. 4 • The Rise of the Modern Data Center • Networks 2.0: The New World Agenda
  5. 5. The Winds Of Change Are Blowing Through Networking 5 Image credit: SDN Bare Metal Switching DevOps Commoditization of Networking Network Virtualization
  6. 6. A Key Transformation 6 SPINE LEAF CORE L2 ACCESS L3 AGGREGATION
  7. 7. What Changed ? § A new generation of applications: §  Search §  Big Data §  Clouds §  Other Web 2.0 Applications 7
  8. 8. Characteristics of the New Applications §  Traffic Pattern §  Between servers (East-West) instead of client-server (North-South) §  Scale §  10s of thousands to 100s of thousands of endpoints §  Agility §  New endpoints and racks powered up in hours instead of weeks §  New networks spun up in seconds instead of weeks §  Flexibility §  Ability to reuse same infrastructure for different applications §  Resilience §  Fine grained failure domain 8
  9. 9. Where The Existing Topology Falls Short 9 § Not suited for E-W traffic § Heavy-core, lean edge design is not scalable § Not Agile § Inflexible Design § Coarse-grained failure domain § Unpredictable Latency CORE L2 ACCESS L3 AGGREGATION
  10. 10. The Case Against Complexity 10 § Too many protocols §  Many proprietary (MLAG, vPC, for example) §  STP and its variants, its myriad knobs, UDLD, Bridge Assurance, LACP, FHRP (VRRP, HSRP, GLBP), VTP, MVRP, etc. etc. § Dual redundancy only adds to the complexity mess §  Dual control planes §  HA §  ISSU etc. § Complex Failure Modes CORE L2 ACCESS L3 AGGREGATION
  11. 11. Twilight In The Land of Godboxes § Network's Function is to serve the application needs § Existing Network design is a bad fit for the modern DC application 11 Image credit:
  12. 12. 12 • The Rise of the Modern Data Center • Networks 2.0: The New World Agenda
  13. 13. CLOS Network 13 § Invented by Charles Clos in 1953 § How to build ever larger telephony networks without building ever larger telephony switches § wiki/Clos_network SPINE LEAF
  14. 14. Characteristics Of CLOS Network §  Well matched for E-W traffic pattern §  Scalable network topology §  Reliance on ECMP leads to simple IP- based fabrics §  Fine grained failure domain §  Predictable latency §  Coupled with network virtualization, serves as a basis for agility and flexibility SPINE LEAF 14
  15. 15. Scalable Network Topology
  16. 16. Some Common Terminology CLUSTER or POD CLUSTER or POD INTER-CLUSTER SPINE
  17. 17. Calculating Number of Attachable Servers: Non-Blocking Math § If m, n, o are number of ports on a switch at a given tier § Total number of servers attached to a 2-tier fabric:m*n/2 § Total number of servers attached to a 3-tier fabric: m*n*o/4 17 m n o
  18. 18. Port Math From A Different Perspective § Number of spines and ISL link speed is a function of failure characteristics, cost and cabling simplification 18 Number of pods/DC Number of servers/rack Number of racks/pod
  19. 19. Calculating Number of Servers: Some Concrete Numbers 2-tier 3-tier Theoretical Non-Blocking Trident+ 64 port 10GE 2K 64K Trident2 96 port 10GE 4608 216K More Pragmatic Numbers: 40 servers/rack Trident+ (48x10GE, 4x40GE) Oversubscription: 2.5 (with 4 spines) 1920 (40*48) 60K(40*24*6 4) Trident2 (48x10GE, 6x40GE) Oversubscription: 1.6 1920 184K 19
  20. 20. Paganini Variations
  21. 21. Fine Grained Failure Domain 21 § Any link or spine failure leads to only a 25% loss instead of 50% with traditional design §  Webscale companies do 8-way and 16-way ECMP to mitigate single failure effects
  22. 22. The Case for Simplicity § Reduced number of protocols §  Single IP protocol is sufficient §  No FHRP, STP, the myriad L2 variants 22
  23. 23. Coda For The Godboxes § All boxes are the same §  Simplified inventory control §  Simplified management §  Reduced latency compared to chassis-based boxes § Simple 1RU boxes means simple failures §  Replace failed box and continue instead of debugging in place §  No ISSU, HA, etc. § Developing new apps to run on platform is simpler 23
  24. 24. What About Server Attach § Most, if not all, webscale companies do single attach §  They have so many racks, failure of one doesn’t bother them § Smaller enterprises that cannot sustain the loss of a rack dual-attach servers §  Switches are in the same rack or cross connected pair of racks 24
  25. 25. Does This Apply Only To Big Guys ? § Depends § Many small guys are moving to cloud § If the application requirements are not L2-specific, this applies to small guys too §  Simply think of the cost of bare metal switches to boxes from traditional vendors § Many small IT shops are run by Linux/server admins §  Script savvy and so can setup networks faster §  More in next webinar 25
  26. 26. Fitting Existing Applications: Network Virtualization § With network virtualization technologies such as VxLAN, you can create L2 overlays over the L3 fabric §  Separating virtual network from physical network provides for agile network management §  Can run both new applications such as Hadoop and memcached along with more traditional apps on the same network: flexibility 26
  27. 27. And In Closing.. § Existing Access-Agg-Core design is slowly making way for CLOS fabrics § CLOS, L3-based fabrics are simple, scalable, flexible and agile § Managing a CLOS fabric including routing and such is the topic of the next webinar 27
  28. 28. Resources available § Website: § Blog: § Twitter: @CumulusNetworks @ddcumulus 28
  29. 29. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their respective owners. § Thank You! 29 Bringing the Linux Revolution to Networking