5. Understanding Characteristics of a Leaf Switch
cumulusnetworks.com
10/40 Gigabit
spine uplink ports
Serial
console port
Ethernet Out-of-
Band
Management Port
* SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options
* QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options
6. Understanding Characteristics of a Spine Switch
cumulusnetworks.com
Serial
console port
Ethernet Out-of-
Band
Management Port
* QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options
7. Bare Metal Switch Provisioning
Similar approach to installing OS on server
BIOS + PXE = U-Boot + ONIE (Open Network Install
Environment)
Supported hardware (HCL) preloaded with ONIE
ONIE available on GitHub
• http://onie.github.io/onie/
bare metal server
operating
system
app app app
BIOS and PXE
bare metal switch
operating
system
app app app
U-Boot and ONIE
cumulusnetworks.com
10. What Is ONIE?
Network OS Install Environment
Provides an environment for network OS installer
discovery and execution
Like a pre-installed kickstarter
Defined by its behaviors
Implemented using a modern Linux kernel and BusyBox
An Open Source Project within OCP
http://www.onie.org/
Evolving …
cumulusnetworks.com
13. ONIE: Anatomy of a Network Switch
( Management Interfaces ) ( Data Plane )
CPU
SoC
DRAM
Boot
Flash
Mass
Storage
Switchin
g
ASIC
Serial
Console
Ethernet
Mgmt Port
10Gb
Port
40Gb
Port…
10Gb
Port
40Gb
Port
…
PCIe
cumulusnetworks.com
14. ONIE: Uses Management Interfaces
( Management Interfaces ) ( Data Plane )
CPU
SoC
DRAM
Boot
Flash
Mass
Storage
Switchin
g
ASIC
Serial
Console
Ethernet
Mgmt Port
10Gb
Port
40Gb
Port…
10Gb
Port
40Gb
Port
…
PCIe
cumulusnetworks.com
15. ONIE: Bare Metal Install – First Time Boot Up
Boot Loader
(HW Vendor Supplied)
ONIE
(HW Vendor Supplied)
Installer
(OS Vendor)
Boot Loader
• Low Level boot loader, configures CPU complex
• Loads and boots ONIE
ONIE
• Linux Kernel with Busybox
• Configures management Ethernet interface
• Locates and executes an OS installer
• Provides tools and environment for installer
OS Installer
• Available from network or USB
• Linux executable
• Installs vendor OS into mass storage
Network OS
(OS Vendor Supplied)
Fetches
Installs
cumulusnetworks.com
16. ONIE: Subsequent Reboots – Vendor’s OS is Already
Installed
Boot Loader
(HW Vendor Supplied)
ONIE
(HW Vendor Supplied)
Boot Loader
• Low Level boot loader, configures CPU complex
• Loads and boots OS vendor’s installed OS
Network OS
• Configures Switching ASIC
• Runs Network Protocols
• Provides CLI
Network OS
(OS Vendor Supplied)
ONIE
• Still exists, but is not used
• Available for uninstall / re-install operations
cumulusnetworks.com
17. ONIE: Network OS Installer Discovery and Install
Behavior
Configure Network
Interface
Locate Installer
Run Installer
• Uses DHCPv4, DHCPv6
• Configures Ethernet interface for IPv4 / IPv6
• Configures DNS and hostname
• Determines the location of an installer executable
• Examines local file systems, e.g. USB flash drives
• Uses DHCP options, DNS Service Discovery, Multicast
DNS and IPv6 Neighbors
• Downloads installer via URL
• Passes various environment variables to installer
• Launches installer
cumulusnetworks.com
18. Why Not Use PXE?
• IPv6 and HTTP out of the box
• Use existing Linux device drivers. No need to write new ones
• Integrated automation
cumulusnetworks.com
19. ONIE: Other Behaviors
Provides a mechanism for a network OS to
invoke the above behaviors.
Reinstall Remove currently installed OS and return to
the “out of box” provisioning state
Uninstall Completely wipe out everything, except ONIE
Rescue Reboot box into ONIE for repair, debug and
forensics
Update Install a new ONIE version
Diag Run HW Vendor’s diag (optional)
cumulusnetworks.com
20. Lessons Learned
Hardware Platforms
Diverse mix of CPU complex designs, even within a
single CPU family.
HW Vendors need freedom to customize ONIE
Need mechanism to run HW vendor diag
Recurring Themes
TLV based EEPROM format widely adopted
Common HW designs reduce development time
cumulusnetworks.com
21. ONIE Ongoing Development
Support Multiple CPU Architectures
PowerPC – Today
x86 – VM Prototype available today
Thinking about ARM, MIPS
Maintain ONIE behaviors across architectures
Testing and Compliance
Enhance and extend regression test suites
Develop compliance test suites
cumulusnetworks.com
22. ONIE Ongoing Development
New Features
DHCPv6
DNS Service Discovery / Multicast DNS discovery
HW Vendor diagnostic
Releases
Quarterly release cadence
Ongoing maintenance
Enhancements
cumulusnetworks.com
23. ONIE x86 Strategy
Use existing BIOS from hardware vendors
During manufacturing install GRUB2 and ONIE-x86
on the mass storage block device
NOS Installer adds partitions, installs software and
updates GRUB2 configuration.
cumulusnetworks.com
24. Block Device Partitioning
Partition using GUID Partition Table (GPT) format
GPT supported by Linux, GRUB, UEFI
Supports dual-booting multiple operating systems
Partition # Name R/W Notes
1 GRUB Boot r/o Used by GRUB and
GPT
2 ONIE BOOT r/o ONIE Kernel
3 ONIE CONFIG r/w Configuration
4+ For NOS use
cumulusnetworks.com
30. Pushing Changes Down
CPU, RAM, Flash, etc. Switch Silicon
Front Panel Ports
lldpd
Routing Tables
ARP
Table
Devices
Bridge FDB Filter Tables
Bonds VLANs
LinuxKernel
Virtual Kernel Ports
Bridging
mstpd
ACLRouting Suite
Quagga
snmpd
vconfig
iptable
ebtable
ip6tableiproute2
VXLAN
Bridges
Switch HAL
brctl
Switch
Driver
UserSpace
Quagga daemon,
Quagga.conf, and vtysh
CLI and
/etc/network/interfaces
switchd
cumulusnetworks.com
31. Cumulus Linux Hardware Compatibility List
40G Portfolio Part number Description
Merchant
Silicon
Cumulus Linux Release
Dell S6000 32 x 40G-QSFP+ Trident II 2.1 or later
Edge-Core AS6700-32X 32 x 40G-QSFP+ Trident II 2.0 or later
Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later
Quanta QCT T5032-LY6* 32 x 40G-QSFP+ Trident II 2.0 or later
10G Portfolio Part number Description
Merchant
Silicon
Cumulus Linux Release
Agema AG-7448CU 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 1.5.0 or later
Dell S4810 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 2.0.x or later**
Edge-Core AS5600-52X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later
Edge-Core AS5710-54X 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later**
Edge-Core AS5710-96X 96 x 10G-SFP+ and 8 x 40G-QSFP+ Trident II 2.1 or later**
Penguin Computing Arctica 4804X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.1 or later
Quanta QCT T-3048-LY2* 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later
Quanta QCT T5048-LY8* 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later**
1G Portfolio Part number Description Merchant Silicon Cumulus Linux Release
Edge-Core AS4600-54T 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later
Penguin Computing Artica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2 1.5.1 or later
Quanta QCT T1048-LB9* 48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or later
cumulusnetworks.com
Editor's Notes
Cumulus Network’s HCL focused on fixed boxes (Leaf/Spine)Same Broadcom silicon as Arista switches, same hardware performance at lower price point.Arista has additional hardware platforms for special purposes Choice – Cumulus focuses on breadth of platforms/vendors for best of breed.Arista supports black boxesArista and supports many different configurations – Cumulus doesn’t need differentiated price points for low end configurations, they are already cheaperCumulus Linux is a Linux OS, and network services apps run on top of it are very rich.Arista in contrast is a Linux-based OS, EoS integrates all apps in one image and control is limited to some Linux containerCloud Networking designs – includes L2/Host Multi-homing*, L3/ECMP, L2 over L3 VXLAN.Customers are moving to L3 CLOS fabrics so L2/Host multi-homing is all that’s needed, not MLAGOrchestration – Comprehensive set of tools today on par with Arista and rapid innovationOur model offers the same Orchestration tool and more due to rapid pace of innovation (ex. Midokura)OpenFlow is supported with other OS such as Big SwitchAutomation.Cumulus Linux has Zero Touch Provisioning, automated install, better DevOps integration (due to unmodified Linux/scripting languages)Application visibility – Leverage server style tools & hardware counters/functionalityArista may have stronger networking tracers, advanced mirroring (DANZ), advanced congestion management (LANZ) tools today. Congestion management/counters will be enabled with switchd file system, more can be done for simplification, but similar capability can be enabled through scriptingProgrammable foundation – drivers abstractions, eAPI, Unmodified Linux Cumulus Linux drivers abstractions are unchanged (in contrast Arista uses sysDB to provide visibility to their own driver), Cumulus Linux networking data structures are unchanged (Arista uses its own so user is limited to management plane/control plane box changes)
Just like BIOS and PXE allows you to install an OS on a server using a remote image, the combination of U-Boot and ONIE allows that for bare metal switches.We require ONIE preloaded on HCL because U-Boot is different across vendor devices, and U-Boot itself is not very user friendly.We created ONIE and gave it to the Open Compute Project (OCP); it facilitates easy network OS installation of not just Cumulus Linux (Pica8 is a competitive example). Now you have your choice of installing whatever OS you want, not just what comes with the switch (e.g. Cisco IOS– OEM example, or FASTPATH– Broadcom’s OS).Think of ONIE as PXE on steroids. ONIE is a small BusyBox Linux distribution, with a bunch of fetch and execution Bash scripts. It leverages modern ways of discovering networks using what was built into Linux—e.g., IPv6 neighbor discovery, DHCPv6, DHCPv4.U-Boot is very good at probing the bus. U-Boot takes about 1MB. It has boot flash that’s dedicated to booting the hardware, separate from the Operating System flash. ONIE is a way to build on top of this. Takes about 3.5 MB.ONIE is extremely well documented and flexible, and embraced by the open source community. (Source is on GitHubsince summer 2013).
Within Linux is a construct called netlink,the communication channel between user space and Linux kernel. Everything we see in the User Space box talks to the Kernel through netlink (not shown on diagram). switchd snoops the netlink traffic and can react (e.g. whenever you add or remove a route)Color decode:Green with orange border pushes things down to the kernel