Successfully reported this slideshow.

Hardware accelerated switching with Linux @ SWLUG Talks May 2014

3,054 views

Published on

Nat Morris will take us through the use of Linux on a new generation of hardware accelerated network switches

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Hardware accelerated switching with Linux @ SWLUG Talks May 2014

  1. 1. v Hardware accelerated switching with Linux Nat Morris 26th April 2014 @ South Wales Linux User Group
  2. 2. About me Nat Morris • Based in Haverfordwest (beyond the M4) • Team lead, Cumulus Networks • Director & Board Member, UK Network Operators Forum (UKNOF) • Feeder of dogs • Attended first SWLUG meeting in 2001 Twitter • @natmorris cumulusnetworks.com 2
  3. 3. About Cumulus Networks Team  JR Rivers, co-founder and CEO  Nolan Leake, co-founder and CTO  Shrijeet Mukherjee, VP Engineering  Reza Malekzadeh, VP Business  Jason Martin, VP Customer Experience Investors  Andreessen Horowitz  Battery Ventures  Sequoia Capital  Wing. VC (Peter Wagner)  Ed Bugnion, Diane Greene and Mendel Rosenblum (VMware founders) cumulusnetworks.com 3
  4. 4. cumulusnetworks.com 4
  5. 5. IP Fabric Networking Landscape cumulusnetworks.com 5 Network Hardware NetworkOS Open Closed
  6. 6. The Expanding Landscape hardware operating system appapp hardware operating system app app Single Vendor Blob Multi-Vendor Ecosystem app app cumulusnetworks.com 6
  7. 7. Expanding Ecosystem The missing piece: Cumulus® Linux® , bringing the Linux revolution to networking cumulusnetworks.com 7
  8. 8. Understanding Characteristics of a Leaf Switch 8cumulusnetworks.com 10/40 Gigabit spine uplink ports Serial console port Ethernet Out-of- Band Management Port * SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options * QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options
  9. 9. Understanding Characteristics of a Spine Switch 9cumulusnetworks.com Serial console port Ethernet Out-of- Band Management Port * QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options
  10. 10. Add leaf switches incrementally Connecting 40G Uplinks to Spine Layer 10cumulusnetworks.com Spine Switch 1 Leaf Switch 1 uplink 1 uplink 2 uplink 3 uplink 4 Spine Switch 2 Spine Switch 3 Spine Switch 4
  11. 11. Anatomy of a Network Switch cumulusnetworks.com 11 ( Management Interfaces ) ( Data Plane ) CPU SoC DRAM Boot Flash Mass Storage Switchin g ASIC Serial Console Ethernet Mgmt Port 10Gb Port 40Gb Port… 10Gb Port 40Gb Port … PCIe
  12. 12. Bare Metal Switch Provisioning Similar approach to installing OS on server  BIOS + PXE = U-Boot + ONIE (Open Network Install Environment)  Supported hardware (HCL) preloaded with ONIE  ONIE available on GitHub • http://onie.github.io/onie/ cumulusnetworks.com 12 bare metal server operating system app app app BIOS and PXE bare metal switch operating system app app app U-Boot and ONIE
  13. 13. Hardware Vendors cumulusnetworks.com 13
  14. 14. Operating System Vendors cumulusnetworks.com 14
  15. 15. Hardware Compatibility List (HCL) cumulusnetworks.com 15 Switch Model Number Description Merchant Silicon Cumulus Linux Release Dell S6000-ON 32 x 40G-QSFP+ Trident II 2.1 or later Edge-Core AS6700-32X with ONIE 32 x 40G-QSFP+ Trident II 2.0.1 or later Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later Quanta QCT QuantaMesh T5032-LY6 32 x 40G-QSFP+ Trident II 2.0.1 or later Agema AG-7448CU 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 1.5.0 or later Dell S4810-ON 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 2.0.2 or later Edge-Core AS5600-52X with ONIE 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later Edge-Core AS5610-52X with ONIE 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 2.0.1 or later Edge-Core AS5710-54X with ONIE 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1.x or later Penguin Computing Arctica 4804X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.1 or later Quanta QCT QuantaMesh T-3048-LY2 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later Quanta QCT QuantaMesh T-3048- LY2R 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later Quanta QCT QuantaMesh T5048-LY8 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1.x or later* Edge-Core AS4600-54T with ONIE 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later Penguin Computing Arctica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2 1.5.1 or later Quanta QCT QuantaMesh T1048-LB9 48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or later 40G10G1G
  16. 16. Choice cumulusnetworks.com 16
  17. 17. Choice cumulusnetworks.com 17
  18. 18. ONIE: Bare Metal Install – First Time Boot Up cumulusnetworks.com 18 Boot Loader (HW Vendor Supplied) ONIE (HW Vendor Supplied) Installer (OS Vendor) Boot Loader • Low Level boot loader, configures CPU complex • Loads and boots ONIE ONIE • Linux Kernel with Busybox • Configures management Ethernet interface • Locates and executes an OS installer • Provides tools and environment for installer OS Installer • Available from network or USB • Linux executable • Installs vendor OS into mass storage Network OS (OS Vendor Supplied) Fetches Installs
  19. 19. ONIE: Network OS Installer Discovery and Install Behavior cumulusnetworks.com 19 Configure Network Interface Locate Installer Run Installer • Uses DHCPv4, DHCPv6 • Configures Ethernet interface for IPv4 / IPv6 • Configures DNS and hostname • Determines the location of an installer executable • Examines local file systems, e.g. USB flash drives • Uses DHCP options, DNS Service Discovery, Multicast DNS and IPv6 Neighbors • Downloads installer via URL • Passes various environment variables to installer • Launches installer
  20. 20. Networking Interfaces in Linux cumulusnetworks.com 20 Interface Description eth0 Physical interface for out-of-band management lo Loopback (logical interface redirecting to switch) 127.0.0.1 in /etc/hosts Debian lists secondary 127.0.1.1 swpN Physical interface for data plane traffic N corresponds to port number bridge Logical interface creating a single Layer 2 broadcast domain Traffic on sub-interfaces can be untagged or tagged Commonly called “VLAN” bond Logical interface aggregating two or more interfaces Commonly called “LAG” or “port channel”
  21. 21. Pushing Changes Down cumulusnetworks.com 21 CPU, RAM, Flash, etc. Switch Silicon Front Panel Ports lldpd Routing Tables ARP Table Devices Bridge FDB Filter Tables Bonds VLANs LinuxKernel Virtual Kernel Ports Bridging mstpd ACLRouting Suite Quagga snmpd vconfig iptable ebtable ip6tableiproute2 VXLAN Bridges Switch HAL brctl Switch Driver UserSpace Quagga daemon, Quagga.conf, and vtysh CLI and /etc/network/interfaces switchd
  22. 22. Show Interface Statistics cumulusnetworks.com 22 High level statistics for an interface cumulus@switch:~$ ip -s link show dev swp1 3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 500 link/ether 44:38:39:00:03:c1 brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped overrun mcast 21780 242 0 0 0 242 TX: bytes packets errors dropped carrier collsns 1145554 11325 0 0 0 0 Low level statistics for an interface cumulus@switch:~$ sudo ethtool -S swp1
  23. 23. Deconstructing /etc/network/interfaces  auto swp1  iface swp1 inet static  address 192.168.0.11/30  gateway 192.168.0.1  up ip link set $IFACE up  down ip link set $IFACE down cumulusnetworks.com 23 Bring up interface during boot up or service network reload Interface name Method: manual, static, dhcp ifup verse to bring up interface ifdown verse to bring down interface IP address settings for interface, only if using static Metho d Action manual No IP address configured by default static IP address configured using address and gateway options dhcp Obtain IP address using DHCP server
  24. 24. Bridging Bridge = single isolated Layer 2 broadcast domain  Allows hosts connected to bridge ports (members) to discover each other without having to define routes  Traffic on ports is tagged (802.1q VLAN ID) or untagged (native) • Tagging involves using sub-interfaces, e.g. swpN.ID  Commonly called “VLAN” in traditional networking cumulusnetworks.com 24
  25. 25. Defining a Bridge  auto br-vlan100  iface br-vlan100 inet manual  bridge_ports swp4.100 swp5.100  up ip link set $IFACE up  down ip link set $IFACE down cumulusnetworks.com 25 Bring up interface during boot up or service network reload Interface name Method: manual, static, dhcp ifup verse to bring up interface ifdown verse to bring down interface Bridge members. swp4, swp4.100, swp5, and swp5.100 must be defined first .100 creates sub-interface (turning swp into trunk port)
  26. 26. Show Bridge cumulusnetworks.com 26 Show bridges Show bridge MAC addresses cumulus@switch:~$ brctl showmacs br-red port name mac addr is local? ageing timer swp4 06:90:70:22:a6:2e no 19.47 swp1 12:12:36:43:6f:9d no 40.50 swp1 44:38:39:00:12:9b yes 0.00 swp2 44:38:39:00:12:9c yes 0.00 cumulus@switch:~$ brctl show bridge name bridge id STP enabled interfaces br-vlan100 8000.089e01f89511 no swp5 swp6
  27. 27. Cumulus Linux Packaging and Support cumulusnetworks.com 27 main updates security-updates addons testing  250 packages  ~ 20 Cumulus Linux packages  Examples:  Ruby, Perl, Python, Bash, IPtables, LLDP  Updates: packages revised  Security: known concerns, CVEs  User-identified utilities + libraries  Puppet, Factor, Chef, collectd  Early access utilities and libraries  Bird (CL 1.5)  40K+ packagesDebian.org Fully Supported Fully Supported* Best Effort Best Effort* *packages not controlled by Cumulus
  28. 28. Traditional Hierarchical Network Topology L3 L2 Access Aggregation Core Legacy and limitations  Not designed for today’s data center running modern workloads • Server density • Increased server-to-server traffic  Numerous proprietary protocols • STP/RSTP/PVSTP, VTP, HSRP, MLAG, LACP  “This is what we’ve been taught” 28
  29. 29. L3 Is the Future L3 L2 ECMP Clos network (“spine/leaf”) 1. Simpler network 1. Fewer protocols 2. Standards-based 1. Fewer proprietary features 3. Predictable latency 1. Every leaf is 1 hop away 4. Horizontally scalable Leaf Spine Core 29
  30. 30. Basic Clos Architecture (2-Tier Spine/Leaf) 30cumulusnetworks.com Optimized for high bandwidth East to West traffic patterns compute and storage network services Core or WAN Spine Layer Leaf Layer
  31. 31. Basic Clos Architecture (3-Tier or 5-Stage) 31cumulusnetworks.com Leaf Spin e InterPod Spine Network Services Leaf
  32. 32. Ansible demo 32 spine 1 swp1 - 4 swp1 - 4 swp1 - 4 swp1 - 4 leaf 2 swp17 - 20 swp17 - 20 swp17 - 20 swp17 - 20 wbench leaf 1 spine 2eth0 eth0 eth0 eth0 eth1 eth0 192.168.0.0/24 10.2.1.3/32 10.2.1.4/32 10.2.1.1/32 10.2.1.2/32 swp30-33 swp34-37 10.4.1.1/25 10.4.1.129/25 swp30-33 swp34-37 10.4.2.1/25 10.4.2.129/25 192.168.0.1
  33. 33. Questions 33
  34. 34. © 2014 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. Thank You! Bringing the Linux Revolution to Networking 34

×