OSDC 2014 ONIE by Nat Morris

1,754
-1

Published on

Nat Morris' deck during his ONIE presentation at OSCD 2014 in Berlin, Germany.

Published in: Software, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,754
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
89
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Cumulus Network’s HCL focused on fixed boxes (Leaf/Spine)Same Broadcom silicon as Arista switches, same hardware performance at lower price point.Arista has additional hardware platforms for special purposes Choice – Cumulus focuses on breadth of platforms/vendors for best of breed.Arista supports black boxesArista and supports many different configurations – Cumulus doesn’t need differentiated price points for low end configurations, they are already cheaperCumulus Linux is a Linux OS, and network services apps run on top of it are very rich.Arista in contrast is a Linux-based OS, EoS integrates all apps in one image and control is limited to some Linux containerCloud Networking designs – includes L2/Host Multi-homing*, L3/ECMP, L2 over L3 VXLAN.Customers are moving to L3 CLOS fabrics so L2/Host multi-homing is all that’s needed, not MLAGOrchestration – Comprehensive set of tools today on par with Arista and rapid innovationOur model offers the same Orchestration tool and more due to rapid pace of innovation (ex. Midokura)OpenFlow is supported with other OS such as Big SwitchAutomation.Cumulus Linux has Zero Touch Provisioning, automated install, better DevOps integration (due to unmodified Linux/scripting languages)Application visibility – Leverage server style tools & hardware counters/functionalityArista may have stronger networking tracers, advanced mirroring (DANZ), advanced congestion management (LANZ) tools today. Congestion management/counters will be enabled with switchd file system, more can be done for simplification, but similar capability can be enabled through scriptingProgrammable foundation – drivers abstractions, eAPI, Unmodified Linux Cumulus Linux drivers abstractions are unchanged (in contrast Arista uses sysDB to provide visibility to their own driver), Cumulus Linux networking data structures are unchanged (Arista uses its own so user is limited to management plane/control plane box changes)
  • Just like BIOS and PXE allows you to install an OS on a server using a remote image, the combination of U-Boot and ONIE allows that for bare metal switches.We require ONIE preloaded on HCL because U-Boot is different across vendor devices, and U-Boot itself is not very user friendly.We created ONIE and gave it to the Open Compute Project (OCP); it facilitates easy network OS installation of not just Cumulus Linux (Pica8 is a competitive example). Now you have your choice of installing whatever OS you want, not just what comes with the switch (e.g. Cisco IOS– OEM example, or FASTPATH– Broadcom’s OS).Think of ONIE as PXE on steroids. ONIE is a small BusyBox Linux distribution, with a bunch of fetch and execution Bash scripts. It leverages modern ways of discovering networks using what was built into Linux—e.g., IPv6 neighbor discovery, DHCPv6, DHCPv4.U-Boot is very good at probing the bus. U-Boot takes about 1MB. It has boot flash that’s dedicated to booting the hardware, separate from the Operating System flash. ONIE is a way to build on top of this. Takes about 3.5 MB.ONIE is extremely well documented and flexible, and embraced by the open source community. (Source is on GitHubsince summer 2013).
  • Within Linux is a construct called netlink,the communication channel between user space and Linux kernel. Everything we see in the User Space box talks to the Kernel through netlink (not shown on diagram). switchd snoops the netlink traffic and can react (e.g. whenever you add or remove a route)Color decode:Green with orange border pushes things down to the kernel
  • OSDC 2014 ONIE by Nat Morris

    1. 1. Open Network Install Environment NETWAYS OSDC 2014, Berlin Nat Morris 9th April 2014
    2. 2. •Overview •What Is ONIE? •Lessons Learned •ONIE Development •Demo Agenda
    3. 3. IP Fabric Networking Landscape Network Hardware NetworkOS Open Closed cumulusnetworks.com
    4. 4. The Expanding Landscape hardware operating system appapp hardware operating system app app Single Vendor Blob Multi-Vendor Ecosystem app app cumulusnetworks.com
    5. 5. Understanding Characteristics of a Leaf Switch cumulusnetworks.com 10/40 Gigabit spine uplink ports Serial console port Ethernet Out-of- Band Management Port * SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options * QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options
    6. 6. Understanding Characteristics of a Spine Switch cumulusnetworks.com Serial console port Ethernet Out-of- Band Management Port * QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options
    7. 7. Bare Metal Switch Provisioning Similar approach to installing OS on server  BIOS + PXE = U-Boot + ONIE (Open Network Install Environment)  Supported hardware (HCL) preloaded with ONIE  ONIE available on GitHub • http://onie.github.io/onie/ bare metal server operating system app app app BIOS and PXE bare metal switch operating system app app app U-Boot and ONIE cumulusnetworks.com
    8. 8. Choice cumulusnetworks.com
    9. 9. Choice cumulusnetworks.com
    10. 10. What Is ONIE? Network OS Install Environment  Provides an environment for network OS installer discovery and execution  Like a pre-installed kickstarter  Defined by its behaviors  Implemented using a modern Linux kernel and BusyBox An Open Source Project within OCP  http://www.onie.org/ Evolving … cumulusnetworks.com
    11. 11. Hardware Vendors cumulusnetworks.com
    12. 12. Operating System Vendors cumulusnetworks.com
    13. 13. ONIE: Anatomy of a Network Switch ( Management Interfaces ) ( Data Plane ) CPU SoC DRAM Boot Flash Mass Storage Switchin g ASIC Serial Console Ethernet Mgmt Port 10Gb Port 40Gb Port… 10Gb Port 40Gb Port … PCIe cumulusnetworks.com
    14. 14. ONIE: Uses Management Interfaces ( Management Interfaces ) ( Data Plane ) CPU SoC DRAM Boot Flash Mass Storage Switchin g ASIC Serial Console Ethernet Mgmt Port 10Gb Port 40Gb Port… 10Gb Port 40Gb Port … PCIe cumulusnetworks.com
    15. 15. ONIE: Bare Metal Install – First Time Boot Up Boot Loader (HW Vendor Supplied) ONIE (HW Vendor Supplied) Installer (OS Vendor) Boot Loader • Low Level boot loader, configures CPU complex • Loads and boots ONIE ONIE • Linux Kernel with Busybox • Configures management Ethernet interface • Locates and executes an OS installer • Provides tools and environment for installer OS Installer • Available from network or USB • Linux executable • Installs vendor OS into mass storage Network OS (OS Vendor Supplied) Fetches Installs cumulusnetworks.com
    16. 16. ONIE: Subsequent Reboots – Vendor’s OS is Already Installed Boot Loader (HW Vendor Supplied) ONIE (HW Vendor Supplied) Boot Loader • Low Level boot loader, configures CPU complex • Loads and boots OS vendor’s installed OS Network OS • Configures Switching ASIC • Runs Network Protocols • Provides CLI Network OS (OS Vendor Supplied) ONIE • Still exists, but is not used • Available for uninstall / re-install operations cumulusnetworks.com
    17. 17. ONIE: Network OS Installer Discovery and Install Behavior Configure Network Interface Locate Installer Run Installer • Uses DHCPv4, DHCPv6 • Configures Ethernet interface for IPv4 / IPv6 • Configures DNS and hostname • Determines the location of an installer executable • Examines local file systems, e.g. USB flash drives • Uses DHCP options, DNS Service Discovery, Multicast DNS and IPv6 Neighbors • Downloads installer via URL • Passes various environment variables to installer • Launches installer cumulusnetworks.com
    18. 18. Why Not Use PXE? • IPv6 and HTTP out of the box • Use existing Linux device drivers. No need to write new ones • Integrated automation cumulusnetworks.com
    19. 19. ONIE: Other Behaviors Provides a mechanism for a network OS to invoke the above behaviors. Reinstall Remove currently installed OS and return to the “out of box” provisioning state Uninstall Completely wipe out everything, except ONIE Rescue Reboot box into ONIE for repair, debug and forensics Update Install a new ONIE version Diag Run HW Vendor’s diag (optional) cumulusnetworks.com
    20. 20. Lessons Learned Hardware Platforms  Diverse mix of CPU complex designs, even within a single CPU family.  HW Vendors need freedom to customize ONIE  Need mechanism to run HW vendor diag Recurring Themes  TLV based EEPROM format widely adopted  Common HW designs reduce development time cumulusnetworks.com
    21. 21. ONIE Ongoing Development Support Multiple CPU Architectures  PowerPC – Today  x86 – VM Prototype available today  Thinking about ARM, MIPS  Maintain ONIE behaviors across architectures Testing and Compliance  Enhance and extend regression test suites  Develop compliance test suites cumulusnetworks.com
    22. 22. ONIE Ongoing Development New Features  DHCPv6  DNS Service Discovery / Multicast DNS discovery  HW Vendor diagnostic Releases  Quarterly release cadence  Ongoing maintenance  Enhancements cumulusnetworks.com
    23. 23. ONIE x86 Strategy  Use existing BIOS from hardware vendors  During manufacturing install GRUB2 and ONIE-x86 on the mass storage block device  NOS Installer adds partitions, installs software and updates GRUB2 configuration. cumulusnetworks.com
    24. 24. Block Device Partitioning  Partition using GUID Partition Table (GPT) format  GPT supported by Linux, GRUB, UEFI  Supports dual-booting multiple operating systems Partition # Name R/W Notes 1 GRUB Boot r/o Used by GRUB and GPT 2 ONIE BOOT r/o ONIE Kernel 3 ONIE CONFIG r/w Configuration 4+ For NOS use cumulusnetworks.com
    25. 25. GRUB Menu cumulusnetworks.com
    26. 26. NOS Installer Duties  Create new GPT partition(s)  Create file systems on partition(s)  Install NOS files into partition(s)  Update ONIE-CONFIG using ONIE provided CLIs  onie-boot-entry-add  onie-boot-entry-remove  onie-boot-entry-show  onie-boot-default  onie-boot-update cumulusnetworks.com
    27. 27. Resources ONIE Websites  Main Page: http://www.onie.org/  Source Code: https://github.com/onie/onie/  Documentation: http://onie.github.io/onie/ cumulusnetworks.com
    28. 28. © 2013 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. Thank You!
    29. 29. Back Up Slides – Cumulus Background cumulusnetworks.com
    30. 30. Pushing Changes Down CPU, RAM, Flash, etc. Switch Silicon Front Panel Ports lldpd Routing Tables ARP Table Devices Bridge FDB Filter Tables Bonds VLANs LinuxKernel Virtual Kernel Ports Bridging mstpd ACLRouting Suite Quagga snmpd vconfig iptable ebtable ip6tableiproute2 VXLAN Bridges Switch HAL brctl Switch Driver UserSpace Quagga daemon, Quagga.conf, and vtysh CLI and /etc/network/interfaces switchd cumulusnetworks.com
    31. 31. Cumulus Linux Hardware Compatibility List 40G Portfolio Part number Description Merchant Silicon Cumulus Linux Release Dell S6000 32 x 40G-QSFP+ Trident II 2.1 or later Edge-Core AS6700-32X 32 x 40G-QSFP+ Trident II 2.0 or later Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later Quanta QCT T5032-LY6* 32 x 40G-QSFP+ Trident II 2.0 or later 10G Portfolio Part number Description Merchant Silicon Cumulus Linux Release Agema AG-7448CU 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 1.5.0 or later Dell S4810 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 2.0.x or later** Edge-Core AS5600-52X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later Edge-Core AS5710-54X 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later** Edge-Core AS5710-96X 96 x 10G-SFP+ and 8 x 40G-QSFP+ Trident II 2.1 or later** Penguin Computing Arctica 4804X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.1 or later Quanta QCT T-3048-LY2* 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later Quanta QCT T5048-LY8* 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later** 1G Portfolio Part number Description Merchant Silicon Cumulus Linux Release Edge-Core AS4600-54T 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later Penguin Computing Artica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2 1.5.1 or later Quanta QCT T1048-LB9* 48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or later cumulusnetworks.com

    ×