Architecting Data Center Networks in the Era of Big Data and

Architecting Data CenterNetworks in the era of Big Dataand CloudBrad HedlundSpring Interop—May 2012 Global Marketing

Two approaches to DC Networking THE SAME OLD• Centralized, Scale-up Layer 2 networks• Monstrous chassis switches TRILL OpenFlow VEPA SPB Or a Different Approach Distributed, Scale-out Layer 3 fabrics Efficient fixed switches Open, industry standard protocols2 Brad Hedlund Global Marketing

Networks that suck for Cloud & Big Data Core Network Dist Topology Access VM Capacity Topology PARTITIONED CAPACITY “Data center networks are in my way” -James Hamilton, AWS3 Brad Hedlund Global Marketing

Networks that Don’t suck for Cloud & Big Data Spine Network Topology Leaf VM Capacity Topology UNIFORM CAPACITY  All points equidistant4 Brad Hedlund Global Marketing

Big Data TCP switch switch TCP switch switch switch switch switch Name Node Job Tracker Secondary NN Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Client Client TCP TCP Rack 1 Rack 2 Rack 3 Rack 4 Rack N World• Inverse Virtualization• Workloads orchestrated like cattle• L2 or L3 network. Does it matter?5 Brad Hedlund Global Marketing

Basic requirements of Cloud (IaaS) switch switch Physical Network switch switch switch switch FW LBWorld Virtual Network VM VM VM VM • Secure, Scalable Multi Tenancy • Location independence • On Demand virtual networks6 Brad Hedlund Global Marketing

Blend the Virtual and Physical Networks VLAN 20 switch VLAN 10 switch switch VM VM vSwitch vSwitch VM VM VM VM VM VM Host Host•Tenant subnet = Network VLAN7 Brad Hedlund Global Marketing

Abstract the Virtual Network from Physical Segment ID 20 switch Segment ID 10 switch switch VM VM vSwitch vSwitch VM VM VM VM VM VM Host Host•Network Virtualization Overlay• Tenant subnet = Software VLAN8 Brad Hedlund Global Marketing

Scale-up centralized Layer 2 L3• 2-post Rooted Architecture L2• Centralized L2/L3• L2/L3/ARP table scale?• Scale w/ Bigger Boxes• Precious Pets• VLAN Provisioning? vSwitch vSwitch• Broadcasts VM VM VM VM9 Brad Hedlund Global Marketing

Scale-out Layer 3 Leaf/Spine Fabric L3 (16) (8) (2) L2(128) (64) (16) 1980 Server ports 3072 Server ports 768 6144 Server ports• Mesh from Leaf to Spine • Non-blocking Spine• OSPF, ISIS, BGP, TRILL • 3:1 @ ToR• ToR w/ 16 uplinks (ECMP) • 128 port 2RU Spine 10 Brad Hedlund Global Marketing

Uniform fabric for Cloud & Big Data (16) (8) (2) Storage Access Hadoop(128) L3 (64) (16) L2 Name Node Job Tracker Secondary NN Node Database ports 6144 Server Node Node Node Node Node Node Node Node vSwitch vSwitch Node Node Block I/O Node Node Node Node Node Node Node Node NAS Node Node VM VM VM VM Client Client Object Node Node Rack 1 Rack 2 Rack 3 Rack N 11 Brad Hedlund Global Marketing

Attaching Services & North/South (16) (8) (2)(128) L3 (64) (16) L2 LB Firewall x86 Gateways Name Node Node Job Tracker Secondary NN LB Firewall Node Node Node Node Node Node Node Node vswitch vswitch vswitch World VM VM VM VM VM VM VM VM VM Client Client vswitch vswitch vswitch Rack 1 Rack N VM VM VM VM VM VM VM VM VM 12 Brad Hedlund Global Marketing

Generic Logical Architecture 1 World • Overlay based L2 Fabric DC router • Physical/Static FW L3 L3 FW NAT FW NAT L2 L2 LB LB L3 VM VM L2 L2 VM VM VM Big Data Green Co. Orange Co.13 Brad Hedlund Global Marketing

Generic Logical Architecture 2 World • Overlay based L2 • Virtual/Mobile FW Fabric DC router • Overlay Gateway Pub DMZ L3 L3 FW NAT FW NAT L2 L2 LB LB L3 VM VM L2 L2 VM VM VM Big Data Green Co. Orange Co.14 Brad Hedlund Global Marketing

Generic Logical Architecture 3 World • No Overlays • TRILL based L2 Fabric DC router • Virtual/Mobile FW Pub DMZ TRILL L3 L3 FW NAT FW NAT L2 L2 LB LB L3 VM VM L2 L2 VM VM VM Big Data Green Co. Orange Co.15 Brad Hedlund Global Marketing

Density: Fixed vs. Chassis 10G per RU @ Line Rate (L3) 140 120 100 80 Chassis 60 Fixed 40 20 0 2008 2010 2012 201416 Brad Hedlund Global Marketing

Power: Fixed vs. Chassis 18 Max Watts / Line Rate 10G (L3) 16 14 12 10 Chassis 8 Fixed 6 4 2 0 2010 2012 201417 Brad Hedlund Global Marketing

What are the Challenges? (16) (8) (2) Dell Fabric Manager  Design Templates(128)  Validate deployment  Automate fabric configuration L3 (16)  Monitoring & Operations L2• Deployment & Cabling • Layer 2 (TRILL?)• Configuration & Policy • Design Best Practices• Monitor & Troubleshoot Dell Fabric Manager 18 Brad Hedlund Global Marketing

Webinar: CLOS Fabrics Explained http://closfabric.eventbrite.com/ Wednesday, June 20, 2012 from 10:00 AM to 1:00 PM (ET) HOST CO-HOST Brad Hedlund Ivan Pepelnjak19 Brad Hedlund Global Marketing

The power to do more20

Three Stage Layer 3 Leaf/Spine Fabric (64) (8) (2) /26(128) 0/0 /26 /26(512) 0/0 L3 L2 24,576 Server ports• Non-blocking @ top tiers • Leaf+ToR mesh groups• Default route @ ToR & Leaf • ~8usec worst case 21 Brad Hedlund Global Marketing

The case for 40G QSFP switch ports 10G 10G 10G 10G 10G 10G 10G 10G VS SFP+ SFP+ SFP+ SFP+ QSFP $1K $1K $1K $1K $1,800 32 ToR $230K $512K22 Brad Hedlund Global Marketing

Comparing Fabric efficienciesof Fixed vs Chassis designsBrad HedlundMay 2012 Global Marketing

Total Fabric Power: Chassis vs. Fixed KW non-blocking 250 200 150 Chassis 100 Fixed 50 0 384 2048 4096 8192 Fabric size24 Brad Hedlund Global Marketing

Total Fabric RU: Chassis vs. Fixed RU non-blocking 700 600 500 400 Chassis 300 Fixed 200 100 0 384 2048 4096 8192 Fabric size25 Brad Hedlund Global Marketing

8192 non-blocking Fabric (64) (8) (2)(128) 8192 non-blocking • 384RU • 153.6KW • 8192 ISL • 192 switches 26 Brad Hedlund Global Marketing

8192 non-blocking Fabric(32)(256) 8192 non-blocking • 608RU • 216.3KW • 8192 ISL • 288 switches 27 Brad Hedlund Global Marketing

4096 non-blocking Fabric (32) (2) (64) 4096 non-blocking • 192RU • 76.8KW • 4096 ISL • 96 switches28 Brad Hedlund Global Marketing

4096 non-blocking Fabric(16)(128) 4096 non-blocking • 304RU • 108.1KW • 4096 ISL • 144 switches 29 Brad Hedlund Global Marketing

2048 non-blocking Fabric (16) (2) (32) 2048 non-blocking • 96RU • 38.4KW • 512 ISL • 48 switches30 Brad Hedlund Global Marketing

2048 non-blocking Fabric(8)(64) 2048 non-blocking • 152RU • 54KW • 2048 ISL • 72 switches 31 Brad Hedlund Global Marketing

384 non-blocking Fabric (4) (2) (6) 384 non-blocking • 20RU • 8KW • 96 ISL • 10 switches32 Brad Hedlund Global Marketing

384 non-blocking Fabric (2) (12) 384 non-blocking • 26RU • 18KW • 384 ISL • 14 switches33 Brad Hedlund Global Marketing

256 non-blocking Fabric (2) (4) 256 non-blocking • 12RU • 4.8KW • 64 ISL • 6 switches34 Brad Hedlund Global Marketing

256 non-blocking Fabric (2) (8) 256 non-blocking • 22RU • 6.7KW • 256 ISL • 10 switches35 Brad Hedlund Global Marketing

Total Fabric Power: Chassis vs. Fixed KW 3:1 oversubscribed 80 70 60 50 40 Chassis Fixed 30 20 10 0 768 1536 3072 6144 Fabric size36 Brad Hedlund Global Marketing

Total Fabric RU: Chassis vs. Fixed RU 3:1 oversubscribed 250 200 150 Chassis 100 Fixed 50 0 768 1536 3072 6144 Fabric size37 Brad Hedlund Global Marketing

768 @ 3:1 oversubscribed Fabric (2) (16) 768 @ 3:1 • 20RU • 7.2KW • 64 ISL • 18 switches38 Brad Hedlund Global Marketing

768 @ 3:1 non-blocking Fabric (2) (16) 256 non-blocking • 30RU • 7.5KW • 256 ISL • 18 switches39 Brad Hedlund Global Marketing

1536 @ 3:1 oversubscribed Fabric (4) (2) (32) 1536 @ 3:1 • 40RU • 14.4KW • 128 ISL • 36 switches40 Brad Hedlund Global Marketing

1536 @ 3:1 oversubscribed Fabric (2) (32) 1536 @ 3:1 • 54RU • 17KW • 512 ISL • 34 switches41 Brad Hedlund Global Marketing

3072 @ 3:1 oversubscribed Fabric (8) (2) (64) 3072 @ 3:1 • 80RU • 28.8KW • 1024 ISL • 72 switches42 Brad Hedlund Global Marketing

3072 @ 3:1 oversubscribed Fabric (4) (64) 3072 @ 3:1 • 108RU • 34KW • 1024 ISL • 68 switches43 Brad Hedlund Global Marketing

6144 @ 3:1 oversubscribed Fabric (16) (8) (2)(128) 6144 @ 3:1 • 160RU • 57.6KW • 2048 ISL • 144 switches 44 Brad Hedlund Global Marketing

6144 @ 3:1 oversubscribed Fabric(8)(128) 6144 @ 3:1 • 216RU • 68KW • 2048 ISL • 136 switches 45 Brad Hedlund Global Marketing

Total Fabric Power: Chassis vs. Fixed KW 3:1 oversubscribed - Massive 1400 1200 1000 800 Chassis 600 Fixed 400 200 0 12288 24576 49152 98304 Fabric size46 Brad Hedlund Global Marketing

Total Fabric RU: Chassis vs. Fixed RU 3:1 oversubscribed - Massive 5000 4500 4000 3500 3000 2500 Chassis 2000 Fixed 1500 1000 500 0 12288 24576 49152 98304 Fabric size47 Brad Hedlund Global Marketing

12,288 @ 3:1 oversubscribed Fabric (32) (64) (8) (2)(256) 12,288 @ 3:1 • 448RU • 166.4KW • 5120 ISL • 352 switches 48 Brad Hedlund Global Marketing

12,288 @ 3:1 oversubscribed Fabric(16)(256) 12,288 @ 3:1 • 432RU • 136.3KW • 4096 ISL • 272 switches 49 Brad Hedlund Global Marketing

24,576 @ 3:1 oversubscribed Fabric (64) (128) (8) (2)(512) 24,576 @ 3:1 • 896RU • 332.8KW • 10,240 ISL • 704 switches 50 Brad Hedlund Global Marketing

24,576 @ 3:1 oversubscribed Fabric (32) (256)(512) 24,576 @ 3:1 • 1120RU • 328.9KW • 16,384 ISL • 800 switches 51 Brad Hedlund Global Marketing

49,152 @ 3:1 oversubscribed Fabric (128)(256) (1024) 49,152 @ 3:1 • 1792RU • 665.6KW • 20480 ISL • 1408 switches 52 Brad Hedlund Global Marketing

49,152 @ 3:1 oversubscribed Fabric (64)(512) (1024) 49,152 @ 3:1 • 2240RU • 657.9KW • 32,768 ISL • 1600 switches 53 Brad Hedlund Global Marketing

98,304 @ 3:1 oversubscribed Fabric (256)(512) (2048) 98,304 @ 3:1 • 3584RU • 1331.2KW • 40960 ISL • 2816 switches 54 Brad Hedlund Global Marketing

98,304 @ 3:1 oversubscribed Fabric (128)(1024) (2048) 98,304 @ 3:1 • 4480RU • 1315.8KW • 65,536 ISL • 3200 switches 55 Brad Hedlund Global Marketing

The power to do more56

Architecting Data Center Networks in the Era of Big Data and

by Dell Enterprise on Jun 13, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Architecting Data Center Networks in the Era of Big Data and Presentation Transcript