Download to read offline
Uploaded byMarian Marinov
Challenges with high density networks
Marian Marinov is the chief system architect and head of the DevOps department at SiteGround.com. He discussed the challenges of high-density networks including large broadcast domains, limited MAC/ARP tables, and bandwidth constraints. Some solutions proposed were using VLANs, layered network designs, and overlay technologies like VXLAN and NVGRE to divide the network into smaller segments and increase scalability.
More Related Content
![Webinar: Network Automation [Tips & Tricks]](https://cdn.slidesharecdn.com/ss_thumbnails/webinar-nw-automation-august-2016-160901213303-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Challenges with high density networks
Challenges with high density networks
- 1. Marian MarinovMarian Marinov mm@siteground.commm@siteground.com ChiefSystem ArchitectChief System Architect Head of the DevOps departmentHead of the DevOps department Challenges with Challenges with Highdensity Highdensity networksnetworks
- 2. ❖❖ Who amI?Who am I? - Chief System Architect of SiteGround.com- Chief System Architect of SiteGround.com - Sysadmin since 1996- Sysadmin since 1996 - Organizer of OpenFest, BG Perl- Organizer of OpenFest, BG Perl Workshops, LUG-BG and othersWorkshops, LUG-BG and others - Teaching Network Security and- Teaching Network Security and Linux System AdministrationLinux System Administration courses in Sofia Universitycourses in Sofia University and SoftUniand SoftUni
- 3. - Physical Machines -with a few hundreds IPs per machine A bit of historyA bit of history
- 4. - Physical Machines -with a few hundreds IPs per machine - Virtual Machines - with a tens of IPs per VM - with hundreds of VMs per machine A bit of historyA bit of history
- 5. - Physical Machines -with a few hundreds IPs per machine - Virtual Machines - with a tens of IPs per VM - with hundreds of VMs per machine - Containers A bit of historyA bit of history
- 6. - Containers - withseveral IPs per container - with thousand containers per machine A bit of historyA bit of history
- 7. - ONE 42URack - with physical machines 42 * 100 = 4200 IPs - with VMs 42 * (100 * 10) = 42 000 IPs - with containers 42 * (1000 * 2) = 84 000 IPs A bit of historyA bit of history
- 8. ❖❖ ProblemsProblems - broadcastdomains- broadcast domains - mac/arp address tables- mac/arp address tables - bandwidth- bandwidth
- 9. - ARP/ICMPv6 - DHCP/mdns -HA and Gossip like protocols - 42 machines... “Not Great, Not Terrible” - 420 or 4200... that is a problem Broadcast domainsBroadcast domains
- 10. MAC/ARP address tablesMAC/ARP address tables - A typicalDataCenter switch has around: ~ 1Tbps throughput
- 11. MAC/ARP address tablesMAC/ARP address tables - A typicalDataCenter switch has around: ~ 1Tbps throughput ~ 600-700 Mpps switching
- 12. MAC/ARP address tablesMAC/ARP address tables - A typicalDataCenter switch has around: ~ 1Tbps throughput ~ 600-700 Mpps switching ~ 48-64k MAC table
- 13. MAC/ARP address tablesMAC/ARP address tables - A typicalDataCenter switch has around: ~ 1Tbps ~ 600-700 Mpps ~ 48-64k MAC table Remember: 84k IPs with 1000 containers per machine
- 14. MAC/ARP address tablesMAC/ARP address tables ❖❖ What if...Whatif... we connect 10 racks?we connect 10 racks?
- 15. MAC/ARP address tablesMAC/ARP address tables ❖❖ What if...Whatif... we connect 10 racks?we connect 10 racks? 84k IPs per rack 840k IPs for this network
- 16. MAC/ARP address tablesMAC/ARP address tables ❖❖ What aboutthe servers?What about the servers? Linux ARP table defaults to 1024 entries
- 17. MAC/ARP address tablesMAC/ARP address tables - If youwant to change the defaults on Linux: net.ipv4.neigh.default.gc_thresh1 net.ipv4.neigh.default.gc_thresh2 net.ipv4.neigh.default.gc_thresh3 IP sysctl options
- 18. BandwidthBandwidth Assumptions: - a singlephysical machine avg. 200Mbps - a single VM avg. 100Mbps (req. 10Gbps physical uplink) - a single container avg. 100Mbps (req. 100Gbps physical uplink)
- 19. BandwidthBandwidth 42 * 200Mbps= 8.4 Gbps 42 * 100VM * 100Mbps = 420 Gbps 42 * 1000LXC * 100Mbps = 4.2 Tbps - a single physical machine avg. 200Mbps - a single VM avg. 100Mbps - a single container avg. 100Mbps
- 20. BandwidthBandwidth - linking theseswitches - typical DataCenter switches link with multiple 10 or 40Gbps uplinks - its rear to see 100Gbps uplinks
- 21. BandwidthBandwidth - getting 420Gbps out of a single switch, would require at least 4 100Gbps uplinks
- 22. ❖❖ SolutionsSolutions - VLANs/QinQ/MPLS-VLANs/QinQ/MPLS - Layered network designs- Layered network designs - VXLAN/NVGRE- VXLAN/NVGRE
- 23. VLANs/QinQ/MPLSVLANs/QinQ/MPLS - reduces thebroadcast domains - if switches support mac address tables per-vlan, it also reduces the mac address table issues - it does not solve the capacity/BW issues - it introduces complexity in the setup
- 24. Layered DesignsLayered Designs Cisco's Spine andLeaf topology
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. VXLAN/NVGREVXLAN/NVGRE 192.168.0.15 192.168.0.28 10.0.1.12 10.0.1.17 10.1.5.10010.1.5.204 5a:7a:de:23:0b:27 192.168.0.15 00:00:5e:00:01:2a
- 31. VXLANVXLAN - Point-to-Point orMulticast - 50 bytes overhead - Jumbo frames are preferred - statically with iproute2 - Linux Documentation vxlan.txt - dynamically with OpenVswitch - Supported in switches from Arista and Brocade
- 32. NVGRENVGRE - Developed byMicrosoft supported only by Microsoft - working over GRE tunnels - Point-to-Point or Multicast - 42 bytes
- 33. GeneveGeneve - Point-to-Point orMulticast - Unify NVGRE, VXLAN and STT(Stateless Transport Tunneling) - supported in OpenVswitch
- 34. Overlay technologiesOverlay technologies Cisco made agood comparison of the overlay technologies. You can read the paper here.
- 35. ConclusionConclusion - Layer 2should end in ToR switches - Layer 3 should be used for anything more complex - Overlays can be used to accommodate specific client requirements
- 36.