Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Using Agilio SmartNICs for OpenStack Networking Acceleration

1,828 views

Published on

Johann Tönsing, Chief Architect & SVP, Software, presentation from OpenStack Days: Silicon Valley 2016: "Using Agilio SmartNICs for OpenStack Networking Acceleration."

Published in: Technology
  • Login to see the comments

Using Agilio SmartNICs for OpenStack Networking Acceleration

  1. 1. © 2016 NETRONOME Johann Tönsing August 9, 2016 Using Agilio™ SmartNICs for
 OpenStack Networking Acceleration
  2. 2. © 2016 NETRONOME 2 Agenda • Overview of Traditional OpenStack Networking Options
 OVS (with/without connection tracking), Contrail vRouter, SR-IOV, VirtIO… • Issues: Flexibility, Performance, CPU Utilization • Accelerating and Offloading OpenStack Networking using Agilio™ SmartNICs • Results: Features, Throughput, CPU Utilization • Implementation Details: Hardware and Software Architectures • Evolution of OpenStack Networking • Supporting P4, C and eBPF Programmability • Integration Activities • Conclusions
  3. 3. © 2016 NETRONOME Traditional OpenStack Networking Options 3 Forwarding / Virtual Switching Technology OpenStack 
 Control Plane Forwarding Overlays (Tunnels) Security Groups (Microsegmentation, Stateful Firewalling) SR-IOV with regular NIC Vendor specific plugin Limited
 
 E.g. MAC/VLAN
 based directing None None
  4. 4. © 2016 NETRONOME Traditional OpenStack Networking Options 3 Forwarding / Virtual Switching Technology OpenStack 
 Control Plane Forwarding Overlays (Tunnels) Security Groups (Microsegmentation, Stateful Firewalling) SR-IOV with regular NIC Vendor specific plugin Limited
 
 E.g. MAC/VLAN
 based directing None None OVS ML2 plugin 
 (optional: SDN controller) 
 OVN plugin L2 / L3 VXLAN, GRE, 
 (for OVN) GENEVE
 …
 
 Emerging: 
 NSH (for NFV) OVS 2.5+ can interface to Linux conntrack
 
 (Previously Linux bridge with iptables/nftables)
  5. 5. © 2016 NETRONOME Traditional OpenStack Networking Options 3 Forwarding / Virtual Switching Technology OpenStack 
 Control Plane Forwarding Overlays (Tunnels) Security Groups (Microsegmentation, Stateful Firewalling) SR-IOV with regular NIC Vendor specific plugin Limited
 
 E.g. MAC/VLAN
 based directing None None OVS ML2 plugin 
 (optional: SDN controller) 
 OVN plugin L2 / L3 VXLAN, GRE, 
 (for OVN) GENEVE
 …
 
 Emerging: 
 NSH (for NFV) OVS 2.5+ can interface to Linux conntrack
 
 (Previously Linux bridge with iptables/nftables) Contrail vRouter Contrail plugin (via Contrail controller) L2 / L3 MPLS in UDP, 
 MPLS in GRE,
 VXLAN
 … Built in, via flow table
  6. 6. © 2016 NETRONOME Traditional OpenStack Networking Options 3 Forwarding / Virtual Switching Technology OpenStack 
 Control Plane Forwarding Overlays (Tunnels) Security Groups (Microsegmentation, Stateful Firewalling) SR-IOV with regular NIC Vendor specific plugin Limited
 
 E.g. MAC/VLAN
 based directing None None OVS ML2 plugin 
 (optional: SDN controller) 
 OVN plugin L2 / L3 VXLAN, GRE, 
 (for OVN) GENEVE
 …
 
 Emerging: 
 NSH (for NFV) OVS 2.5+ can interface to Linux conntrack
 
 (Previously Linux bridge with iptables/nftables) Contrail vRouter Contrail plugin (via Contrail controller) L2 / L3 MPLS in UDP, 
 MPLS in GRE,
 VXLAN
 … Built in, via flow table Additional features: load balancing, NAT etc.
  7. 7. © 2016 NETRONOME OpenStack Networking Options Evaluated 4 Forwarding / Virtual Switching Technology Traditional Approach Agilio™ SmartNIC Accelerated Approach SR-IOV Limited expressiveness to direct traffic to VMs (no support for general match/action rules, tunnel termination, stateful firewalling) 
 High throughput
 
 No VM migration support Full OVS or vRouter virtual switching incl. tunnel termination, stateless/stateful firewalling

 and SR-IOV based data delivery to VMs 
 
 High throughput
 
 VirtIO integration, supporting VM migration OVS
 
 and
 
 Contrail vRouter High expressiveness - match/action, tunnels, stateless/stateful firewalling etc. 
 Limited throughput 
 High CPU utilization (e.g. 50% of cores) Same expressiveness - match/action, tunnels, stateless/stateful firewalling etc.
 Higher throughput (~5x higher) 
 Lower CPU utilization (~10x lower)
  8. 8. © 2016 NETRONOME 5 Agilio™ CX SmartNIC Family • Optimized for standard server based cloud data centers • Low Profile Half Length PCIe form factor, power < 25W • Based on Netronome’s NFP-4xxx silicon (72 C programmable cores, 8 threads each) • 2GB DRAM for lookup tables / state tables (millions of entries) • Dataplane fully implemented in software 1x 40GbE 2x 40GbE2x 10GbE New: 2x 25GbE Also available: Agilio™ LX 2x40G / 1x100G with dual PCIe interfaces, 120 cores, 8GB DRAM…
  9. 9. © 2016 NETRONOME 6 OVS Throughput vs. Packet Size OVS L2 Forward to VMs Packet Size MillionsofPacketsperSecond OVS VXLAN + L2 Forward to VMs Packet Size MillionsofPacketsperSecond
  10. 10. © 2016 NETRONOME 7 OVS Throughput vs. Number of Rules 5 10 15 20 25 30 OVS in Kernel Space OVS in User Space on DPDK 100 Wildcard Rules 1000 Wildcard Rules 10000 Wildcard Rules 64000 Wildcard Rules MillionsofPacketsperSecond 12 CPU Cores 12 CPU Cores OVS Offloaded to Agilio™ CX-4000 1 CPU Core 5X Throughput Improvement + 90% CPU Savings OVS L2/L3 Forwarding to 8 VMs with 64K Flows
  11. 11. © 2016 NETRONOME 8 Tested Scenario: Server CPU Core Allocation Unaccelerated OVS (Kernel / User Mode) Agilio™ OVS OVS
  12. 12. © 2016 NETRONOME 9 Efficiency: Throughput Per Server CPU Core Throughput with single server CPU core
 dedicated to network related processing MillionsofPacketsperSecond • 50x Efficiency vs. Kernel OVS • 20x Efficiency vs. User Mode (DPDK) OVS
 
 => Replace 3-6 racks with 1 rack! Throughput / efficiency similar for Agilio™ Contrail vRouter Benefit for your use case: 
 search for
 “netronome.com roi calculator”
  13. 13. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API (Nova, Neutron) Execute Action OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act
  14. 14. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act
  15. 15. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss
  16. 16. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss Miss
  17. 17. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API 2 2 OVS userspace agent populates kernel cache (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss Miss
  18. 18. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API 2 2 OVS userspace agent populates kernel cache (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) 3 Offload datapath: copy match tables, sync stats 3 OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss Miss
  19. 19. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API 2 2 OVS userspace agent populates kernel cache (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) 3 Offload datapath: copy match tables, sync stats 3 4 Flow tracking: per-microflow state learning 4 Self Learning Exact Match
 Flow Tracker Miss Hit OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss Miss
  20. 20. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API 2 2 OVS userspace agent populates kernel cache (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) 3 Offload datapath: copy match tables, sync stats 3 Conn track FTP SIP 4 Flow tracking: per-microflow state learning 4 Self Learning Exact Match
 Flow Tracker Miss Hit OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss Miss
  21. 21. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API 2 2 OVS userspace agent populates kernel cache (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) Conn track 3 Offload datapath: copy match tables, sync stats 3 Conn track FTP SIP 4 Flow tracking: per-microflow state learning 4 Self Learning Exact Match
 Flow Tracker Miss Hit OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss Miss
  22. 22. © 2016 NETRONOME Offload Model: Agilio™ OVS Acceleration 10 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API 2 2 OVS userspace agent populates kernel cache (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) Conn track 3 Offload datapath: copy match tables, sync stats 3 5 Offload connection tracking: synchronize state 5 Conn track FTP SIP 4 Flow tracking: per-microflow state learning 4 Self Learning Exact Match
 Flow Tracker Miss Hit OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss Miss
  23. 23. © 2016 NETRONOME Agilio™ vRouter Acceleration 11 vRouter Subsystem vRouter Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK vRouter CLI 1 Configuration via controller or CLI: BGP over XMPP (Nova, Neutron) Execute Action vRouter Flow Classify vRouter Forwarding vRouter Forwarding
  24. 24. © 2016 NETRONOME vRouter Datapath vRouter Flow Classify Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) vRouter Forwarding vRouter Forwarding Agilio™ vRouter Acceleration 11 vRouter Subsystem vRouter Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK vRouter CLI 1 Configuration via controller or CLI: BGP over XMPP (Nova, Neutron) Execute Action vRouter Flow Classify vRouter Forwarding vRouter Forwarding
  25. 25. © 2016 NETRONOME vRouter Datapath vRouter Flow Classify Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) vRouter Forwarding vRouter Forwarding Agilio™ vRouter Acceleration 11 vRouter Subsystem vRouter Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK vRouter CLI 1 Configuration via controller or CLI: BGP over XMPP (Nova, Neutron) Execute Action vRouter Flow Classify vRouter Forwarding vRouter Forwarding 2 2 Offload forwarding: pre-emptively copied
  26. 26. © 2016 NETRONOME vRouter Datapath vRouter Flow Classify Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) vRouter Forwarding vRouter Forwarding Agilio™ vRouter Acceleration 11 vRouter Subsystem vRouter Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK vRouter CLI 1 Configuration via controller or CLI: BGP over XMPP (Nova, Neutron) Execute Action vRouter Flow Classify Miss vRouter Forwarding vRouter Forwarding 2 2 Offload forwarding: pre-emptively copied
  27. 27. © 2016 NETRONOME vRouter Datapath vRouter Flow Classify Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) vRouter Forwarding vRouter Forwarding Agilio™ vRouter Acceleration 11 vRouter Subsystem vRouter Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK vRouter CLI 1 Configuration via controller or CLI: BGP over XMPP (Nova, Neutron) Execute Action vRouter Flow Classify Miss Miss vRouter Forwarding vRouter Forwarding 2 2 Offload forwarding: pre-emptively copied
  28. 28. © 2016 NETRONOME vRouter Datapath vRouter Flow Classify Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) vRouter Forwarding vRouter Forwarding Agilio™ vRouter Acceleration 11 vRouter Subsystem vRouter Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK vRouter CLI 1 Configuration via controller or CLI: BGP over XMPP 3 3 Userspace agent populates kernel tables (Nova, Neutron) Execute Action vRouter Flow Classify Miss Miss vRouter Forwarding vRouter Forwarding 2 2 Offload forwarding: pre-emptively copied
  29. 29. © 2016 NETRONOME vRouter Datapath vRouter Flow Classify Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) vRouter Forwarding vRouter Forwarding Agilio™ vRouter Acceleration 11 vRouter Subsystem vRouter Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK vRouter CLI 1 Configuration via controller or CLI: BGP over XMPP 3 3 Userspace agent populates kernel tables (Nova, Neutron) Execute Action 4 Offload flows: copy tables, sync stats 4 vRouter Flow Classify Miss Miss vRouter Forwarding vRouter Forwarding 2 2 Offload forwarding: pre-emptively copied
  30. 30. © 2016 NETRONOME vRouter Datapath vRouter Flow Classify Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) vRouter Forwarding vRouter Forwarding Agilio™ vRouter Acceleration 11 vRouter Subsystem vRouter Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK vRouter CLI 1 Configuration via controller or CLI: BGP over XMPP 3 3 Userspace agent populates kernel tables (Nova, Neutron) Execute Action 4 Offload flows: copy tables, sync stats 4 vRouter Flow Classify Miss Miss vRouter Forwarding vRouter Forwarding Notes: - Control protocol: BGP over XMPP - Tunnel protocols: VXLAN, MPLS over UDP / GRE - Distinct policy (flows - ACLs) and forwarding (IP/MPLS) - Flow table is used to implement stateful firewalling 2 2 Offload forwarding: pre-emptively copied
  31. 31. © 2016 NETRONOME SmartNIC Firmware: Pre-programmed or Custom • SmartNIC with dynamically downloadable firmware
  32. 32. © 2016 NETRONOME SmartNIC Firmware: Pre-programmed or Custom • SmartNIC with dynamically downloadable firmware Contrail OVS OpenStack ONOS ODL Linux BSD • OVS / Contrail / Linux eBPF datapath on host can be accelerated by SmartNIC D P D K eBPF
  33. 33. © 2016 NETRONOME SmartNIC Firmware: Pre-programmed or Custom Compiler Debugger Run-Time app.P4 app.C Editor • Firmware can be developed in P4 and/or C • TBD how to integrate custom programmed datapaths (implemented in P4 / C etc.) into OpenStack • SmartNIC with dynamically downloadable firmware Contrail OVS OpenStack ONOS ODL Linux BSD • OVS / Contrail / Linux eBPF datapath on host can be accelerated by SmartNIC D P D K eBPF
  34. 34. © 2016 NETRONOME 13 SmartNIC Datapath “Worker” Software Architecture • Load balancer distributes each packet to next available thread for optimum throughput • Hardware assisted reordering ensures packet order is maintained • Flow tracker statefully learns / tracks millions of sessions • Matching performed using DRAM-backed tables - capacity > 500k entries • Actions efficiently performed in on-chip memory Parse Match Act Flow Tracker Learn microflows Cache action Datapath Run to completion Load
 Balance C Plugin Re-
 order Pool of worker threads on flow processing cores net or PCIE net or PCIE = Ring / Work Queue (multi producer / consumer)
  35. 35. © 2016 NETRONOME 14 Example: P4 “main” implementing a simple NIC header_type eth_hdr { fields { dst : 48; src : 48; etype : 16; } } header eth_hdr eth; parser start { return eth_parse; } parser eth_parse { extract(eth); return ingress; } action drop_act() { drop(); } action fwd_act(port) { modify_field(standard_metadata.egress_spec, port); } table in_tbl { reads { standard_metadata.ingress_port : exact; } actions { fwd_act; drop_act; } } control ingress { apply(in_tbl); }
  36. 36. © 2016 NETRONOME SmartNIC Firmware (P4/C) Example of Fully Customized Datapath (P4 / C) 15 Run-Time Interface Server (x86 - Linux) PCIe Agilio™ SmartNIC Virtual Machine 1 VNF Kernel Mode (C) netdev P4 / C Development Environment Edit - Debug Control App Populate tables, display statistics Security µVNF (C) Timestamp µVNF (C) Latency Stats µVNF (C) Virtual Machine 2 VNF User Mode
 (C) DPDK Timestamp µVNF (C) Latency Stats µVNF (C) Match Protocol Meter Other TCP Concepts: • P4 and C running on SmartNIC 
 implements datapath - e.g.
 defines protocols, match / action
 behavior • Datapath steers traffic to VNFs
 running on x86 server and 
 on SmartNIC
  37. 37. © 2016 NETRONOME SmartNIC Firmware (P4/C) Example of Fully Customized Datapath (P4 / C) 15 Run-Time Interface Server (x86 - Linux) PCIe Agilio™ SmartNIC Virtual Machine 1 VNF Kernel Mode (C) netdev P4 / C Development Environment Edit - Debug Control App Populate tables, display statistics Security µVNF (C) Timestamp µVNF (C) Latency Stats µVNF (C) Virtual Machine 2 VNF User Mode
 (C) DPDK Timestamp µVNF (C) Latency Stats µVNF (C) Match Protocol Meter Other TCP Concepts: • P4 and C running on SmartNIC 
 implements datapath - e.g.
 defines protocols, match / action
 behavior • Datapath steers traffic to VNFs
 running on x86 server and 
 on SmartNIC To Discuss: 
 OpenStack
 Integration
  38. 38. © 2016 NETRONOME Extending OpenStack Networking - P4/C Plugins 16 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API 2 2 OVS userspace agent populates kernel cache (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) 3 Offload datapath: copy match tables, sync stats 3 OVS
 Kernel DP Match/Act OVS
 Kernel DP Match/Act Miss Miss
  39. 39. © 2016 NETRONOME Extending OpenStack Networking - P4/C Plugins 16 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API 1 Configuration via controller, CLI, or Callable API 2 2 OVS userspace agent populates kernel cache (Nova, Neutron) Execute Action Open vSwitch Datapath Execute Action (e.g. Entunnel, Deliver to VM, Send to Port) 3 Offload datapath: copy match tables, sync stats 3 OVS
 Kernel DP Match/Act Datapath Extension or Plugin P4 / C in Sandbox DP Ext. 4 Datapath extension software 4 4 OVS
 Kernel DP Match/Act Miss Miss
  40. 40. © 2016 NETRONOME OVS “on” SmartNIC P4 Datapath 17 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API (Nova, Neutron) Execute Action OVS
 Kernel DP Match/Act P4 Matching
  41. 41. © 2016 NETRONOME OVS “on” SmartNIC P4 Datapath 17 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API (Nova, Neutron) Execute Action P4 Generated Datapath Execute P4 Action OVS
 Kernel DP Match/Act P4 Matching
  42. 42. © 2016 NETRONOME OVS “on” SmartNIC P4 Datapath 17 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API (Nova, Neutron) Execute Action P4 Generated Datapath Execute P4 Action OVS
 Kernel DP Match/Act P4 Matching Fallback
  43. 43. © 2016 NETRONOME OVS “on” SmartNIC P4 Datapath 17 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API (Nova, Neutron) Execute Action P4 Generated Datapath Execute P4 Action OVS
 Kernel DP Match/Act P4 Matching Fallback Fallback
  44. 44. © 2016 NETRONOME P4 “into” OVS Datapath 18 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API (Nova, Neutron) Execute P4 / OVS Action P4 / OVS Matching P4 / OVS Matching
  45. 45. © 2016 NETRONOME P4 “into” OVS Datapath 18 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API (Nova, Neutron) P4 Generated Datapath Execute P4 / OVS Action Execute P4 / OVS Action P4 / OVS Matching P4 / OVS Matching
  46. 46. © 2016 NETRONOME P4 “into” OVS Datapath 18 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API (Nova, Neutron) P4 Generated Datapath Execute P4 / OVS Action Execute P4 / OVS Action P4 / OVS Matching P4 / OVS Matching Fallback
  47. 47. © 2016 NETRONOME P4 “into” OVS Datapath 18 Open vSwitch Subsystem OVS Agent OpenFlow Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK OVS CLI Callable API (Nova, Neutron) P4 Generated Datapath Execute P4 / OVS Action Execute P4 / OVS Action P4 / OVS Matching P4 / OVS Matching Fallback Fallback
  48. 48. © 2016 NETRONOME P4 “instead of” OVS (Datapath and Control) 19 Open vSwitch Subsystem P4 Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK (Nova, Neutron) P4 Generated Datapath Execute P4 Action Execute Action P4 or eBPF or XDP P4 Matching Fallback Fallback 1 Protocol(s) to be discussed (could become callable API)
  49. 49. © 2016 NETRONOME P4 “instead of” OVS (Datapath and Control) 19 Open vSwitch Subsystem P4 Agent Virtual Machine Virtual Machine Virtual Machine x86 Kernel x86 Userspace PCIe Virtual Machine SR-IOV / VirtIO VFs SR-IOV / VirtIO VFs Agilio™
 SmartNIC Apps Apps 1 netdev or DPDK netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK Apps netdev or DPDK (Nova, Neutron) P4 Generated Datapath Execute P4 Action Execute Action P4 or eBPF or XDP P4 Matching Fallback Fallback 1 Protocol(s) to be discussed (could become callable API) Other open issues: - Downloading programs via
 OpenStack or other systems
 - Scheduling VMs to run on
 nodes with acceleration hardware
 (Nova)
 
 Acceleration discussion started at OpenStack Spring 2016 (Austin)

  50. 50. © 2016 NETRONOME 20 Integration / Open Sourcing Activities Area Activities Linux drivers VF netdev (kernel device driver) upstreamed in kernel 4.5 Patches available for older kernels, or use VirtIO Representative netdev (for fallback processing of traffic) proposals imminent FreeBSD drivers Kernel device driver implemented DPDK drivers Poll mode driver upstreamed in DPDK 2.2 Patches available for older DPDK versions Open vSwitch acceleration
 integration Hooks for acceleration proposed in October 2014 Iteration in progress OpenStack integration Proposing plugins and agents to support virtual switching acceleration via RFE process Integration for OVS in process - Mirantis, Ericsson CEE etc. Integration for Contrail vRouter in process - Juniper etc.
  51. 51. © 2016 NETRONOME 20 Integration / Open Sourcing Activities Area Activities Linux drivers VF netdev (kernel device driver) upstreamed in kernel 4.5 Patches available for older kernels, or use VirtIO Representative netdev (for fallback processing of traffic) proposals imminent FreeBSD drivers Kernel device driver implemented DPDK drivers Poll mode driver upstreamed in DPDK 2.2 Patches available for older DPDK versions Open vSwitch acceleration
 integration Hooks for acceleration proposed in October 2014 Iteration in progress OpenStack integration Proposing plugins and agents to support virtual switching acceleration via RFE process Integration for OVS in process - Mirantis, Ericsson CEE etc. Integration for Contrail vRouter in process - Juniper etc. Also: P4 and protocol independent forwarding, switchdev, eBPF / XDP…
  52. 52. © 2016 NETRONOME 21 Next Steps • Use Agilio™ SmartNICs with existing dataplanes • Use Agilio™ OVS (with / without Conntrack) • Use Agilio™ vRouter • Future: Agilio™ eBPF/XDP, etc.
 • Program Agilio™ SmartNICs • Use APIs (on x86 servers) - with above dataplanes • Program in P4 and/or C (on SmartNIC / on x86)
 • Improve performance + free up server resources!
  53. 53. © 2016 NETRONOME More information: netronome.com and open-nfp.org Thank You! 22

×