PLNOG 13: Alexis Dacquay: Handling high-bandwidth-consumption applications in a modern DC design
•
2 likes•520 views
Alexis Dacquay – is CCIE with over 10 years experience in the networking industry. He has in the past been designing, deploying, and supporting some large corporate LAN/WAN networks. He has in the last 4 years specialised in high performance datacenter networking to satisfy the needs of cloud providers, web2.0, big data, HPC, HFT, and any other enterprise for which high performing network is critical to their business. Originally from Bretagne, privately a huge fan of polish cuisine. Topic of Presentation: Handling high-bandwidth-consumption applications in a modern DC design Language: English Abstract: Modern Data Centre requires proper handling of high-bandwidth consuming applications, like BigData or IP Storage. To achieve this, next generation Ethernet speeds of 25, 50 and 100Gbps are being pursued. We are to show _why_ these new Ethernet speeds are vital from technology standpoint and _how_ to cope with the those sparkling new requirements by networking hardware enablements. We are to share ethernet switches’ design considerations, with the biggest emphasis put on the importance of big buffers and how they accommodate this bursty traffic. Throughout the presentation we will additionally elaborate on the evolution of variety of modern applications, and how we can handle those with the properly designed hardware, software, and Data Centre itself.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to PLNOG 13: Alexis Dacquay: Handling high-bandwidth-consumption applications in a modern DC design
Similar to PLNOG 13: Alexis Dacquay: Handling high-bandwidth-consumption applications in a modern DC design (20)
Recently uploaded
Recently uploaded (20)
PLNOG 13: Alexis Dacquay: Handling high-bandwidth-consumption applications in a modern DC design
- 1. Handling High-Bandwidth Applications in a Modern DC design Alexis Dacquay (ad@arista.com) Arista
- 3. Drivers for bandwidth increase ü Application Clustering ü High Density non-blocking scale ü ECMP to provide scale and fault tolerance ü IP Storage / Big Data and Hadood ü 2 tier active / active with low oversubscription ratio ü Dual home or single home server ü Distributed traffic, Mesh, anything-anywhere ü Fan-in, Fan-out ü Virtualized Cloud – Scale ü VXLAN with Equal Cost Multipathing
- 4. Storage – Arista european customer case 40G Ethernet storage 8 x 40G Ethernet 6PB Storage (GPFS) High density 10/40/100G Ethernet Compute Workstations 1000+ compute Replication Users • 39.5 Gb/s utilization per 40G Ethernet link, on all 8 simultaneously (=316Gbps) • 30 GB/s GPFS aggregate throughput, with some disk drawers still unpopulated • Low latency, large buffers. Highest performance without tuning on the network
- 5. 2000 0 0 1 2 3 4 5 6 7 8 9 10 Time Eth1 Eth2 Eth3 Non stressful traffic Buffer limit Packet segments Eth 4 100 80 60 40 20 0 120 10 20 30 40 50 60 70 80 90 100 Utilisation (%) Time (ms) Av Throughput Buffer Usage Current Throughput
- 6. Buffering Visibility with LANZ (trigger-based) • Offers visibility of μburst • Impact of congestion on latency, drops • Trigger-based • Guaranteed visibility (vs Polls) • Configurable high/low threshold Arista 7150S#show queue-monitor length drops Report generated at 2013-01-16 20:48:09 Time Interface TX Drops ----------------------------------------------------------------- 0:02:32.18999 ago Et46 32755054 0:02:35.29710 ago Et46 53552534 0:02:40.29720 ago Et46 53552633 High Threshold Eth8 Eth9 Eth10 Eth1 Eth2 Eth3 Congestion Low Threshold Congestion Event triggered by an Over-threshold event Packet buffering on Eth8 queue due to temp μburst from eth1 and eth2 EOS LANZ Agent
- 7. What Causes Congestion? Buffer starvation à TCP collapse • Oversubscribed networks with bursts > available bandwidth • Multiple nodes trying to read/write to one node (e.g.: Storage) • Lack of buffers means drops, which result in lower goodput 1 2 3 4 Data Block Storage Servers Client Switch SRU (Server Request Unit)
- 8. 2000 0 0 1 2 3 4 5 6 7 8 9 10 Time Eth1 Eth2 Eth3 Bursty traffic on shallow buffer Buffer limit Packet segments Eth 4 140 120 100 80 60 40 20 0 160 10 20 30 40 50 60 70 80 90 100 Utilisation (%) Time (ms) Av Throughput Buffer Usage Current Throughput
- 9. 3 second response time 5 second response time Bandwidth 100% Utilization Time Why are deep buffers required? Packet Loss Backoff and Slowstart Window Increasing Greater than 3 second screen paint time will cause you to lose 43% of your customers! Akamai report on page response time
- 10. Deep Buffers Matter! … Fairness • 20 node test with 10 flows per node (200 flows) • Two tests: • 4MB buffer • 256MB buffer Results: • Complete fairness with large buffers • Small buffers caused confusing flow transmission rates
- 11. 2000 0 0 1 2 3 4 5 6 7 8 9 10 Time Eth1 Eth2 Eth3 Bursty traffic on deep buffer Buffer limit Packet segments Eth 4 160 140 120 (%) 100 Utilisation 80 60 40 20 0 10 20 30 40 50 60 Time (ms) 70 80 90 100 Av Throughput Buffer Usage Current Throughput
- 12. BUFFERING: HOW MUCH IS NEEDED ?
- 13. Deep Buffers Matter! …Hadoop Test 1000000 750000 500000 250000 0 1MB 4:1 1MB 5.33:1 48MB 5.33:1 Zero! Packets dropped per TeraGen 4x10G ⇒ 4:1 3x10G ⇒ 5.33:1 ... 16 hosts x 10G ... 16 hosts x 10G 1k TCP slow start/sec
- 14. Buffer Impact to High Performance Deep Buffer Shallow Buffer Oversubscription Goodhput • Use Cases: ü Optimizing multi-speed: 40à10G , 100Gà10,40G, 10Gà1G ü Improving uplink contention in mixed speed networks ü High Density in core/spine (many-to-one, in-cast, fan-in)
- 15. 130 Buffer Utilization per Port – High Perf Networking Buffer consumed (MB) per port th percentile 120 110 100 90 80 70 60 50 40 30 20 10 0 2 10 18 26 34 42 50 58 66 74 82 90 98 Arista 7500E : 125MB Per 10G Trident+ ASIC 9MB per 64 ports
- 16. How much Buffer Memory do you need ? NS3 Network simulations match real-world data showing TCP incast issue - large # of TCP flows create microburst congestion Customer Real Buffer Utilization Observations Max Buffer Used per Port HPC Storage Cluster – Medium 33 MB Animation Storage Filer (NFS) 6.2 MB Software vendor Engineering Build Servers (Perforce) 14.9 MB Online shopping Hadoop 2K servers – Big Data 52.3 MB Educational Enterprise Data Center (Virtualization) 52.4 MB Real World Data
- 17. LANZ Revolutionizes Network Visibility Switch detects potential congestion 7150S! 7150S! 7150S! 7150S! 7150S! 7150S! Application reacts to conditions NMS Identifies hotspots 7150S! 7150S! 7150S! 7150S! Applications NMS Precision analysis of queues, ports and buffers + congestion capture!
- 18. How to catch microbursts? LANZ – Trigger-based vs Polling • Microbursts occur in very short periods, micro or even nanoseconds, they are undetectable using standard polling methods. • LANZ on 7150, is event-driven, offer real-time visibility of microbursts poll SNMP Polling Rate (1/sec) poll Average u6liza6on based on 1 second polling: 0% At 10Gbps 1 Second = ~30 Million Packets !
- 19. 100G
- 20. Cloud Data Center 100G Requirements Increasing port density choice and transceiver distance will accelerate 100G adoption Customers will only deploy 100G Ethernet in volume once it is cost-competitive, i.e. 100 GigE less or equal to 10 x 10 GigE
- 21. 100G Deployment in Data Center 100G Any-scale Pods 10 G 10 G 100G 10 G 100G 10 G 100G Rack to DC Spine High Performance Storage Mix 1km and 10km Broadest choice of 100G ports Highest Density for DC Spines Mix and match 40G and 100G 100G at the PoP Data Center Interconnecting Data Centers & POPs 10km Long Distances Data Center Small DC or PoP Smaller Footprint Option IEEE LR4 and SR10 Optical interconnect Metro, Core, Edge Routers Mix and Match SM and MM 100G to the ToR Up to 400m 1/10G Scale Built Spine 10 G 40 G Server/Storage Expansion Leaf and Spine Mix 10/40/100G 400m reaches Investment Protection 10G to 40G Server Transition
- 22. The Foundation for Virtualized Clouds Arista 7500E Series • Architected to operate at massive network scale • Designed and Optimized for Virtualization and Cloud • Energy Efficient • 1,152 10GbE / 288 40GbE / 96 100GbE • 30 Tbps Highest Density 10/40/100GbE Switch
- 23. Pay As You Grow 100G Deployment Flexibility Cost Effective MXP Integrated Triple Speed SR10 7500E-12CM-LC Optics 10/40/100G 7500E-6C2-LC Flexible short and long reach CFP2 LR4 - 100GbE over 10km / SR10 - 300m 7500E-12CQ-LC High Density QSFP-100G Broad 10/40/100G QSFP Optics Dense 100/40/10G Ÿ Deep Buffers Ÿ Feature Parity Ÿ Investment Protection
- 24. Wire Speed 10/40/100G with Deep Buffers 7280SE Fixed Configuration Switches 900 Million Packets per Second 1.44 Terabits per second Less than 4usec Latency Ultra deep 9GB packet buffers VOQ architecture for lossless forwarding Wire speed L2 and L3 forwarding 40G and 100G uplinks for HPC and CDN Leaf and Spine 40/100G ECMP and MLAG Integrated SSD for local traffic analysis Reversible airflow and AC / DC Power Options
- 25. Flexible Optics: 100G CFP2 & QSFP CFP2 • Hot pluggable transceiver for 100GbE • Full support for IEEE 100G standards – SR, CR, LR, ER • Interoperable with IEEE compliant 100G optics • Half the size of CFP – allows higher density • Lower power consumption than CFP reducing concerns on optic cooling QSFP100 • Smallest form factor transceiver for 40/100GbE • Support for IEEE 100G standards – SR, CR, LR • Interoperable with IEEE compliant 40G and 100G optics • Power efficient with only 3.5W/port • Low power and size allows for high 100G density Highest density, lowest cost Broad MM and SM choice
- 26. 7500E-12CQ Use Case: Long Distance Single Mode Direct Data Center Interconnect Data Center Interconnecting Data Centers & POPs 10km Data Center Small DC or PoP • Up to 10km reach over Single Mode Fiber • Connect to optical transport and core routers • IEEE Standards for multi-vendor interoperability • Broad range of pluggable CFP2 optics • Lowest cost solution for cross-site 100GbE 7280SE-68 Small Data Center/PoP Interconnect • QSFP100 drives up to 10km distance • Provide up to 2x100G bandwidth • 1RU form factor ensures minimal space and very low power requirements
- 27. SPEED UP : 25G AND 50G ETHERNET
- 28. 25G and 50G Ethernet Consortium • Founded by Arista, Broadcom, Google, Mellanox & Microsoft • 25gEthernet.org consortium website • An open specification for the new speeds • Consortium open to everyone in the industry
- 29. Cloud Applications that drive bandwidth 25G • Compute/BigData that needs lowest cost per Gbps • Servers can push more than 10Gbps but not willing to pay a premium • Need same port density as 10G 50G • IP Storage • 2x25G is most cost effective • Higher port density than 40G, so single Leaf switch sufficient • Easier to scale on NICs too Arista is leading the industry here 25G and 50G support needed in silicon Products expected in the next 18 to 36 months on both switches and NICs
- 30. Why need another speed? 1G 10G 40G: 4x10G 100G: 4x25G • 1G and 10G use single lanes (1 pair) • 40G and 100G use parallel lanes (4 pairs) • 40G and 100G ports need more SerDes, consume more power and reduce port density • The Cloud needs to hit the sweet-spot of lowest price per Gigabit vs optimal performance
- 31. 25G and 50G Ethernet 25G 50G: 2x25G • 25G is a single lane specification, just like 10G • Leverages IEEE 802.3 ethernet framing • Offers 2.5X the speed at a cost structure closer to 10G • Same port density & connectors as 10G SFP+ • 50G is dual-lane • Offers 1.25X the speed of 40G • Cost structure is closer to 2X of 10G • 2X the port density as 40G using splitter cables from QSFP
- 32. The Sweet Spots 120" 100" 80" 60" 40" 20" Price&per&Gbps& • 25G is a single lane specification, just like 10G • Leverages IEEE 802.3 ethernet framing • Offers 2.5X the speed at a cost structure closer to 10G • 50G is dual-lane • Offers 1.25X the speed of 40G • Cost structure is closer to 2X of 10G 0" 1G" 10G" 25G" 40G" 50G" 100G"
- 33. Thank You