This event will deliver technical sessions related to the acceleration of server-based networking data planes and VNFs using SmartNIC hardware within the framework of the NFVi and orchestration components in the OPNFV Danube software platform. Virtual switching data planes using Open vSwitch (OVS) and FD.IO (with VPP) will be discussed and offload architectures for such data planes into SmartNIC platforms will be presented. Extension of such data planes for VNF specific requirements utilizing micro-VNF sandbox functions utilizing P4 and/or C language programming will be presented and discussed. Early work defining VNF acceleration architectures and APIs for SmartNIC-accelerated data planes and VNF sandbox extensions will be presented and discussed, followed by a proposal to create a collaborative project to complete the definition of such an API within the frameworks of OPNFV and ETSI. In addition, a proposal for a Pharos community lab related VNF acceleration will be put forward by the Open-NFP organization.

1. © 2017 Open-NFP 1
Data Plane and VNF Acceleration Mini-Summit
OPNFV Summit Beijing
June 12, 2017

2. © 2017 Open-NFP 2
Mini-Summit Agenda
9:00 – 9:15 AM: Welcome and introduction
9:15 – 10:00 AM: NFVi acceleration models and offload architectures for OVS and VPP data planes.
The role of SmartNICs for NFVi acceleration will be discussed.
10:00 – 10:45 AM: VNF acceleration models and offload architectures for NFV. The role of SmartNICs
for VNF acceleration will be discussed.
10:45 – 11 AM: Break
11:00 – 11:30 AM: An open API model for enabling NFVi and VNF acceleration, including the use of
sandbox functions using P4 and/or C programming languages. Proposal to collaborate on the
definition of an open API within the context of an OPNFV project.
11:30AM – 12:15 PM: Proposal for developing and testing VNF acceleration utilizing resources and
support from the Open-NFP community. A brief introduction to available resources including SmartNIC
data plane development tools, and proposal for a VNF acceleration-focused Pharos community lab.
12:15 – 12:30 PM: Summary, call to action, adjourn

3. © 2017 Open-NFP 3
About Open-NFP www.open-nfp.org
Support and grow reusable code in accelerating data plane network
functions processing using SmartNICs and COTS servers
Reduce/eliminate the cost and technology barriers in this space
▪ Technologies:
• P4, SDN, OpenFlow, Open vSwitch (OVS) offload, eBPF
▪ Tools:
• Hardware, development tools, software, cloud access
▪ Community:
• Website (www.open-nfp.org): learning & training materials, active Google group https://
groups.google.com/d/forum/open-nfp, open project descriptions, code repository
▪ Learning/Education/Research support:
• Regular seminar series/webinars, conferences, tutorials, research proposal support for
proposals to the NSF and state agencies
• Present applied reusable research

4. © 2017 Open-NFP 4
Contact Us
Web: www.open-nfp.org
Google Group: https://groups.google.com/d/forum/open-nfp
Email: info@open-nfp.org
Facebook: www.facebook.com/opennfp
Twitter: https://twitter.com/Open_NFP
YouTube: Open-NFP
SlideShare: http://www.slideshare.net/Open-NFP
Become a member today!

5. © 2017 Open-NFP 5
Session 1: NFVi Acceleration Models
and Offload Architectures

6. © 2017 Open-NFP 6
Agenda
Problem statement
NFVi performance bottlenecks
Acceleration and offload of NFVi data planes
Towards an open data plane acceleration framework
Data plane extensions via programming

7. © 2017 Open-NFP 7
NFV Challenges
Server CPU resources not getting cheaper but network bandwidth continues to grow
60
50
40
30
20
10
0
2015 2018 2021
Data: mobile PCs, Tablets and mobile routers
Data: Smartphone
Voice
ExaBytes per month
Global Network Traffic Transistors per $

8. © 2017 Open-NFP 8
Networking Consuming Server CPU Cycles
SmartNIC solutions return expensive CPU cores to revenue generating VMs
CPU Cycles Used up for Networking
With increasing adoption of SDN and NFV
CPU Cycles Available for
Revenue Generating Applications
PercentofCPU
CyclesinaServer
5%
100%
80%
1990s 2000s 2010s 2020s
Cost per cycle
increasing with
slow down or
demise of
Moore’s law
Output per server
dwindling as VMs
and Apps are
starved for CPU
cycles and I/O
bandwidth

9. © 2017 Open-NFP 9
Data Plane in the NFV Infrastructure
VNF VNF VNF
Virtualization and Virtual Switching
Compute
Hardware
Storage Hardware Network Hardware
NFVi
EMS EMS EMS
OSS/BSS
VNFs
NFV Management & Orchestration
Orchestrator
VNF
Manager
Virtualized
Infrastructure
Manager
This is the focus area for
NFVi data plane
acceleration

10. © 2017 Open-NFP 10
Agenda
Problem statement
NFVi performance bottlenecks
Acceleration and offload of NFVi data planes
Towards an open data plane acceleration framework
Data plane extensions via programming

11. © 2017 Open-NFP 11
A word about Bottlenecks…
• In Fluid Dynamics, the throughput of water flowing through a pipe
is limited by the narrowest diameter opening
• In Networking, the throughput of packets flowing through a network
is limited by the slowest processing step

12. © 2017 Open-NFP 12
What about the bottlenecks in the NFV Cloud?
Network Switch/NIC Virtual Switch VNFs
• Designed in HW to operate
at 64-Byte Wire Rate
• Rarely a Bottleneck unless
there is problem
• Software implementation
limits packet rates
• Aggregation point for all
VNFs – typical bottleneck
• Software implementation
limits packet rates
• Can be starved by the
vSwitch

13. © 2017 Open-NFP 13
Example of the Server Bottleneck…
VM
NIC
Ethernet Controller
VM
VM
VM
VM
vSW
vSW
VM
VM
vSW
vSW
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
24 Core Server Compute Node
Assuming 20 VMs consuming 1Mpps each
Assuming 4 Cores for OVS at 1Mpps per core
60 Mpps
Available from
Network
(40GbE)
20 Mpps
Wanted by VMs4 Mpps
Throughput of vSwitch
Bottleneck will cause VMs to be under-utilized by 80%
40GE
Basic NIC Functions

14. © 2017 Open-NFP 14
You can use more cores for OVS but…
VM
SmartNIC
Ethernet Controller
VM
VM
VM
VM
vSW
vSW
VM
VM
vSW
vSW
VM
VM
VM
VM
VM
24 Core Server Compute Node
Assuming 12 VMs consuming 1 Mpps each
Assuming 12 Cores for OVS at 1 Mpps per core
12 Mpps
Wanted by VMs
12 Mpps
Throughput of
vSwitch
No Bottleneck, but 50% of the Server is wasted doing
networking
vSW
vSW
vSW
vSW
vSW
vSW
vSW
vSW
60 Mpps
Available from
Network
(40GbE)
Basic NIC functions40GE

15. © 2017 Open-NFP 15
Eliminating the Bottleneck with a SmartNIC
VM
SmartNIC
Ethernet Controller
VM
VM
VM
VM
VM
VM
OVS
VM
VM
VM
VM
VM
24 Core Server Compute Node
Assuming 23 VMs consuming 1 Mpps each
Assuming 1 Cores for OVS Control Plane
23 Mpps
Wanted by VMs
23 Mpps
Throughput of
vSwitch on
SmartNIC
No Bottleneck, 95% of the server available for VNFs or
revenue generating VMs
60 Mpps
Available from
Network
(40GbE)
OVS Offload Functions40GE
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM VM

16. © 2017 Open-NFP 16
Agenda
Problem statement
NFVi performance bottlenecks
Acceleration and offload of NFVi data planes
Towards an open data plane acceleration framework
Data plane extensions via programming

17. © 2017 Open-NFP 17
Data Plane Offload Models
Server
NIC
Partial Functionality Offload
Control
Agent
VM
Server
NIC
Full Flow Offload
Control
Agent
Functions c,d
Functions a,b
Functions a,b,c,d
Functions a,b,c,d
Flow Cache
VMVM VM
HitMiss

18. © 2017 Open-NFP 18
Embedded Offload Models
Server
NIC
Functions A, B, C,
D, E, F
Full Data Plane Offload
Control
Agent
Server
NIC
Functions A, B, C,
D, E, F
Full Data Plane Offload &
Full Control Plane Offload
Control
Agent
VM VM VM VM

19. © 2017 Open-NFP 19
Implementation of OVS Offload to a SmartNIC
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
virtio VFs
SR-IOV VFs
SmartNIC
Apps
Apps
1
1
netdev/DPDK
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
OVS CLI
1 Configuration via control protocol or CLI
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute
Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
3 Offload datapath: copy match tables, sync stats
3
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

20. © 2017 Open-NFP 20
Example of Significant Offload Gains
Throughput with single server CPU core
50X
Efficiency Gain vs. Kernel OVS
20X
Efficiency Gain vs. Usermode OVS
Kernel OVS
0
5
10
15
25
30
20
Million Packets Per Second
Usermode OVS SmartNIC OVS
VXLAN Tunneling
L2 Forwarding

21. © 2017 Open-NFP 21
Agenda
Problem statement
NFVi performance bottlenecks
Acceleration and offload of NFVi data planes
Towards an open data plane acceleration framework
Data plane extensions via programming

22. © 2017 Open-NFP 22
Overview
• Multiple SmartNIC vendors are working with the Linux community
(e.g. Red Hat) to define a unified/common API for data plane offload
• Approach is based on TC Flower implementation to support offload
• Currently working towards preview level implementation for OVS
• Longer term goal is to incrementally reach feature parity with OVS
• Other data planes can be offloaded with the same mechanism
• For example, additional data planes supported by the OPNFV Danube
software stack

23. © 2017 Open-NFP 23
OVS Offloaded via TC Flower
ovs-vswitchd
TC Flower Kernel
Datapath
Driver
SmartNIC
User-Space
Kernel
Hardware

24. © 2017 Open-NFP 24
What is TC Flower
• Packet classifier for Linux kernel traffic classification (TC) subsystem
• Allows match on key with a wide number of packet and metadata fields
• TC actions may be used to provide match-action behavior similar to
OVS

25. © 2017 Open-NFP 25
How to Participate
• Feedback on feature set
• OVS has many features
• OVS-TC is starting with few
• Plans for adding new features to OVS-TC is not fixed
• Discussion and code review on mailing lists
• Kernel: netdev@vger.kernel.org
• Open vSwitch: dev@openvswitch.org

26. © 2017 Open-NFP 26
User-Space Offload Hooks
• Offload hooks are present at Netdev (datapath vport) layer
• Called by DPIF (datapath flow) layer
• Translates DPIF flows to TC filters (flows) that use Flower classifier
• Communicates with kernel TC implementation using netlink
• May flag TC filters as software-only, hardware-only
• Default is software and if available hardware

27. © 2017 Open-NFP 27
Agenda
Problem statement
NFVi performance bottlenecks
Acceleration and offload of NFVi data planes
Towards an open data plane acceleration framework
Data plane extensions via programming

28. © 2017 Open-NFP 28
Extending OVS using P4/C Plugins
Open vSwitch Subsystem
OVS
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
SmartNIC
Apps
Apps
1
netdev/DPDK
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
OVS CLI
1 Configuration via control protocol or CLI
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
3
Offload datapath: copy match tables,
sync stats
3
OVS
Kernel DP
Match/Act
Datapath
Extension
or Plugin
P4 / C
DP
Ext.
4
Datapath extension
software in “sandbox”
(in P4, C, and/or eBPF)
4
4
OVS
Kernel DP
Match/Act
Miss
Miss
1

29. © 2017 Open-NFP 29
P4 Based OVS Datapath
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
virtio VFs
SR-IOV VFs
SmartNIC
Apps
Apps
netdev/DPDK
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
OVS CLI (Nova, Neutron)
P4 Generated Datapath
Execute
P4 / OVS
Action
Execute
P4 / OVS
Action
P4 / OVS
Matching
P4 / OVS
Matching
Fallback
Fallback
P4 / OVS Datapath

30. © 2017 Open-NFP 30
P4 Based “Other” Datapath
Host Code
Control
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
virtio VFs
SR-IOV VFs
SmartNIC
Apps
Apps
netdev/DPDK
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
(Nova, Neutron)
P4 Datapath Execute
Action
Execute
ActionP4
Matching
P4
Fallback
Fallback?Protocol(s) to be defined
(could become callable API)
Other open issues:
- Downloading programs via
OpenStack vs. via other systems
- Scheduling VMs to run on nodes
with acceleration hardware (Nova)

31. © 2017 Open-NFP 31
Offload Concept for VPP Acceleration
VPP Subsystem
VPP Agent
e.g.
Honeycomb
Virtual Machine
Virtual Machine
Virtual Machine
x86 Userspace
PCIe
Virtual Machine
SmartNIC
Apps
Apps
1 netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
Apps
netdev/DPDK
1 Configuration via controller (Yang)
2
2 Policy download: preemptive
(Nova, Neutron)
Offloaded VPP Datapath
VPP Datapath
XVIO
to virtio
endpoints
netdev/DPDK
4 Indirect connection (inline partial offload)
4
SR-IOV VFs
SR-IOV VFs
5 Direct connection to host/guest endpoints
5
Egress offload
e.g. entunnel /
L3 forward /
meter + QoS…
Ingress offload
e.g. L2 / L3 /
detunnel…
Packet + metadata
3
Policy download: preemptive / on demand
(Control message system)
3
Control

32. © 2017 Open-NFP 32
Session 2: VNF Acceleration
Requirements and Models

33. © 2017 Open-NFP 33
Agenda
• Representative use cases and VNFs
• Virtualized Mobile Network: virtual Evolved Packet Core (vEPC)
• Virtualized Security: Firewall / IPS / UTM
• Monitoring for Auto-Scaling
• (More: ETSI NFV use cases, OPNFV scenarios, OVS DPDK use cases…)
• Common requirements
• Packet operations: Matching + Actions, Tunnels
• Stateful operations
• Service chain related operations
• Other processing
• Acceleration models
• Location - inline / lookaside / fastpath
• Management - exposed to cloud management vs. handled by VNF

34. © 2017 Open-NFP 34
Virtualized Mobile Networks Use Case
Presented by Dejan Leskaroski - Affirmed Networks

35. AFFIRMED NETWORKS © 2016 Affirmed Networks, Inc. All rights reserved. 35
Affirmed Networks VNF Portfolio
AP/AC
MME
SGSN
eNB
Gi / SGi
Gx
S11
S6a
S1MME
PCRF
HSS
S1-U
ePDG
IPSec
Affirmed’s Mobile Content Cloud (MCC)
OFCS
Affirmed VNFs
3rd Party PNFs
Control Plane
Data Plane
Ga/Gz/Rf
GRE
InternetSGSN
Gp
OCS
S2b
UE
GiLAN
Captive
Portal
Gy
AAA
TWAG
AP/ACUE
S2a
Packet
Analyzer Event Stream
Analyzer
Active Flow
Analytics
SGW
PGW
GGSN
Automation Automation
S6b
vProbe

36. AFFIRMED NETWORKS © 2016 Affirmed Networks, Inc. All rights reserved. 36
High-Level Virtualization Architecture
Hypervisor
PHY
Server
…
PHY PHY
Top of Rack (TOR) Switch
ETH1 ETH2 ETH n
CPU1 CPU2
Kernel Space
User Space
Linux
GuestOS
vSwitch
vNICs
Kernel Space
User Space
Linux
GuestOS
vNICs
Affirmed VM #1 Affirmed VM #2
PhysicalVirtual
Data Plane Development Kit
(DPDK)
• Used for high performance network I/O
• SR-IOV or DPDK-Accelerated vSwitch
vCPUs …
• Hypervisor: KVM, VMware
• vSwitch: OVS, DPDK-OVS, SDN
vRouter, VMware vSD
• Smart NIC Offload/Acceleration

37. AFFIRMED NETWORKS © 2016 Affirmed Networks, Inc. All rights reserved. 37
U-Plane Flows with Smart NIC
MCC-C
Standard NFVi Standard NFVi + Smart NIC
MCC-U
x86
x86
MCC-C
MCC-U
x86
x86

38. AFFIRMED NETWORKS © 2016 Affirmed Networks, Inc. All rights reserved. 38
NFVi and VNF Flow Offload using Smart NICs
• Current Support – NFVi Offload
• OVS and vRouter Flow Offload/Acceleration (Flow-Tail Offload)
• Software vSwitch-agent still sees all flows (first packet(s))
• SDN-C decision enforced in Smart-NIC
• Standard capability and APIs architected into OVS for this purpose
• Future Support – VNF Offload (simultaneous with OVS Flow Offload)
• vEPC flow handling requirements
• Required synchronization between VNF & Smart-NIC (API integration)
• Encapsulate and Decapsulate various tunnels (e.g., GTP-U, GRE, L2TP/PPP, etc.)
• MAC header rewrite (L3 nexthop)
• Hierarchical QoS (e.g., QosFlow, AMBR)
• Online Charging and Granular Charging (e.g., Rating Groups)
• Subscriber Firewall (e.g., Rate Limiting, Flow Counting, etc.)
• Flow Based Analytics

39. © 2017 Open-NFP 39
Firewall / IPS / UTM Use Case
• Firewall
• Rule based policies - Access Control List (stateless)
• Stateful operation - permit forward and reverse traffic for a TCP connection / UDP
session, handle related flows (FTP control + data, SIP signaling + media, etc.)
• Intrusion Prevention - e.g. Snort
• Scans network traffic for intrusions e.g. malware / exploits
• Implies reassembly of various protocols (defragment, TCP stream, HTTP/SMTP…)
• Rules (signatures) have 5-tuple / n-tuple packet header match component
and string / regexp match component => could offload n-tuple part (match offload)
• Flow inspection depth => offload “rest of microflow” once no further intrusions possible
• Hybrids - UTM, Next Generation Firewall
• Identifies applications, users (by content - not just headers)
• Incorporates anti-virus, web filtering, anti-spam

40. © 2017 Open-NFP 40
Monitoring for App Auto-Scaling Use Case
• (Grey area - NFVi vs. VNF)
• Monitor dataplane traffic
• By routing traffic through the monitoring VNF
• By interfacing with NFVi datapath
• Detect cases requiring action
• Bandwidth above / below threshold
• Excessive latency (perhaps determined using INT)
• Take action - VNFs + traffic
• Add / delete / reconfigure application VNF instances
• Reconfigure tables (e.g. load balancing) accordingly
• Related: monitoring for DoS attack mitigation
• Reconfigure tables to drop or rate limit traffic

41. © 2017 Open-NFP 41
Agenda
• Representative use cases and VNFs
• Virtualized Mobile Network: virtual Evolved Packet Core (vEPC)
• Virtualized Security: Firewall / IPS / UTM
• Monitoring for Auto-Scaling
• (More: ETSI NFV use cases, OPNFV scenarios, OVS DPDK use cases…)
• Common requirements
• Packet operations: Matching + Actions, Tunnels
• Stateful operations
• Service chain related operations
• Other processing
• Acceleration models
• Location - inline / lookaside / fastpath
• Management - exposed to cloud management vs. handled by VNF

42. © 2017 Open-NFP 42
VNF Requirements: Packet Operations
• Matching
• Wildcard match of n-tuple to apply policy (security - ACL, QoS etc.) based on categories
• Exact match of n-tuple to apply policies to (rest of) microflow
• Fields consulted - pre-defined protocols (L2/L3/L4…) vs. program defined protocols (P4)
• Actions
• Forwarding: send to physical or virtual interface (allow), drop, load balance
• QoS: rate limit, schedule, shape
• Header modification: L3 forwarding, add / remove VLAN tags or MPLS labels, NAT
• Fields affected - pre-defined vs. program defined
• Encapsulation
• Tunnel termination / origination
• Tunnel conversion - combined termination and re-origination
• Affixing headers or options for telemetry
• Related: NFVi’s use of these operations, e.g. to build overlay network, build service chain…

43. © 2017 Open-NFP 43
VNF Requirements: Stateful Operations
• Security / QoS
• Learn microflows - connections / sessions; time out actively or passively
• Track protocols, e.g. TCP (flags/sequence numbers), HTTP (header/payload, chunks)
• Apply policy - allow / drop, rate limit / schedule / shape
• Monitoring
• Learn at required granularity - microflows (connection) vs. larger flows (IP / MAC etc.)
• Keep statistics (throughput / latency) or other aggregated information
• Mirror or log - only headers / all traffic
• State synchronization
• Peer - create distributed state table to facilitate load sharing / high availability
• External - enable related state (in accelerated VNF or other VNFs) to be stored

44. © 2017 Open-NFP 44
VNF Requirements: Service Chain / Other
• Service chain
• Unsubscribe - remove my VNF (myself) or other VNFs from chain
• Scaling and fail over - adjust chain as more / fewer / different instances are available
• Other - non-network-related
• Examples: crypto, compression, transcoding, storage related
• Refer to ETSI NFV specifications for details
• Other - VNF specific
• VNF could need to offload other classification / processing
• Potentially stateful, algorithms potentially complex - not possible to predict in general
• Therefore accommodate with flexible mechanism - plugin in datapath, running in
software datapath or in accelerated datapath
• Concerns: portability - language / RT environment, deployment, scaling

45. © 2017 Open-NFP 45
Agenda
• Representative use cases and VNFs
• Virtualized Mobile Network: virtual Evolved Packet Core (vEPC)
• Virtualized Security: Firewall / IPS / UTM
• Monitoring for Auto-Scaling
• (More: ETSI NFV use cases, OPNFV scenarios, OVS DPDK use cases…)
• Common requirements
• Packet operations: Matching + Actions, Tunnels
• Stateful operations
• Service chain related operations
• Other processing
• Acceleration models
• Location - inline / lookaside / fastpath
• Management - exposed to cloud management vs. handled by VNF

46. © 2017 Open-NFP 46
ETSI NFV IFA 002 Pass Through ModelETSI GS NFV-IFA 002 V2.1.1 (2016-03)10
• VNF must be updated
before new hardware
(yellow) can be used
• Drivers baked into VNFs

47. © 2017 Open-NFP 47
ETSI NFV IFA 002 Abstracted Model
become available on the market, the operator will update its NFVI to allow the VNF to make use of the new
hardware. An implementation independent VNF is thus based on implementation independent VNF software
that makes use of a functional abstraction of an accelerator supported by an adaptation layer in the NFVI. This
model is close to the abstracted model defined in ETSI GS NFV-INF 003 [i.1], clause 7.2.2.
Figure 4.1.1-2: Abstracted model
• Operator can update NFVi
to add support for new HW
- no VNF changes needed
• HW drivers and SW impl.
in NFV infrastructure,
e.g. host OS or hypervisor

48. © 2017 Open-NFP 48
VNF Acceleration Models
NFVI
VNFs
Datapath
VNF #1
1%
Accel. #1
99%
• Fastpath / slowpath (fallback)
- for certain traffic categories
- for part (e.g. rest) of flow
NFVI
VNFs
Datapath
VNF #1
Accel. #1
Accel. #1
• Inline
- ingress
- egress
NFVI
VNFs
Datapath
VNF #1
Accel. #1
• Lookaside
- packet related
- stream related
- other

49. © 2017 Open-NFP 49
Control Considerations
• Key axiom for SDN: logically centralized control
• One (redundant) SDN controller is typically the master for a given datapath
• Need provisions for selective delegation to grant other entities direct access
• Need “meta-policy” to govern which VNFs can control / access which resources
NFVI
VNFs
Datapath
API
VNFa
TBL1 TBL2
TBLa
exclusive
access
SDN
Controller
exclusive
access

50. © 2017 Open-NFP 50
Representation of Tables and Plugins
• VNF acceleration table representation
• Reserve table per VNF
(or VNFC, or instance thereof)
• Reserve logical switch per VNF
(or VNFC, or instance thereof)
• Not recommended: co-locate in shared table
• VNF acceleration plugin representation
• Visible to controller vs. not exposed
(e.g. bump in the wire)
• Represent as logical interface on switch
• Represent as custom action
NFVI
VNFs
Datapath
VNFa
table
entries
TBLa
packets
control /
events /
data
TBL1
Plugin
for
VNFa
API

51. © 2017 Open-NFP 51
VNF Acceleration Summary
• Use cases => distilled requirements
• Table driven - match/action tables, state tables
• Plugin
• Acceleration Platforms
• SmartNIC in server
• External physical switch
• Software datapath (fallback - when hardware not available)
• Acceleration Models
• Inline / lookaside / fastpath
• Coordination with SDN control - delegation / meta-policy

52. © 2017 Open-NFP 52
Session 3: Proposed VNF Acceleration API

53. © 2017 Open-NFP 53
Agenda
• Survey of existing initiatives and APIs
• Proposed offload models
• Proposed mapping to hardware / software platforms
• Next steps - invitation to collaborate

54. © 2017 Open-NFP 54
Existing APIs and Initiatives
• Initiatives and Projects
• ETSI NFV - requirements, management aspects
• OPNFV - DPACC project
• ONF / ON.lab - OpenFlow + SDN Evolution, models, PIF Intermediate Representation
• P4 - language, architecture, API
• Open-NFP - network functions processing for e.g. SmartNICs
• NFVi dataplanes - OVS, Contrail vRouter, fd.io VPP, ODP/OFP, DPDK, Linux/iovisor eBPF
• APIs
• DPDK rte_flow - match/action
• Linux kernel TC FLOWER - match/action (used by OVS)
• OpenFlow - match/action
• Enhanced by BEBA project / OpenState proposal for statefulness
• P4 run-time APIs (generic vs. generated)

55. © 2017 Open-NFP 55
OPNFV - DPACC Project
Presented by Tapio Tallgren - Nokia

56. OpenDataplane.org

57. DPDK

58. VPP

59. VPP on ODP

60. 7
DPACC Virtio Inline Accelerator
2017-06-11
SAL inhost
userspaceSW Accelerator
SAL inuser space
sioin Kernel
virtio-inline
hio
HW Accelerator
Hostuserspaceguestdevice
VirtioInline
BackendvHost-user
VNFApplication
g-API
VirtioInline UserFrontendDriver
Commands/
re-injected
packets
Status/
Exception
Packets
Accelerated
Traffic
Non-accelerated
Traffic
Incoming
Traffic
g-accel-driver
Virtio-netUser
FrontendDriver
Vhost-net
Backend
Physical ports
vRings vRings
Virtio-netUser
FrontendDriver
Vhost-net
Backend
SRL
g-net-driver g-net-driver
Physical ports

61. © 2017 Open-NFP 56
Agenda
• Survey of existing initiatives and APIs
• Proposed offload models
• Proposed mapping to hardware / software platforms
• Next steps - invitation to collaborate

62. © 2017 Open-NFP 57
VNF Offload - Table Model
• VNF can delegate processing to NFVi
datapath
• Match-action: e.g. match dest IP 10.*
dest port 80, action drop
• Match only: e.g. match VLAN 1 port 22,
mark 123, send to VNFa
• Stateful: e.g. match 5-tuple (a,b,c,d,e),
action forward to port 1, count packets,
timeout 30s
• Table entries populated by VNFs
or SDN controller
NFVI
Connect.
Table
VNFs
Datapath
VNFa
match
table
entries
API
VNFb
API
TBLaTBL1 TBL2
packets
TBLb
state
table
entries
packets

63. © 2017 Open-NFP 58
Table Model - Mapping to Hardware
Datapath in SmartNIC
(e.g. accelerated vSwitch)
Datapath in vSwitch
with traditional NIC
Datapath in
physical switch
Server
VMb
vSwitch
VMa
Datapath
VNFa
API
VNFb
API
TBLaTBL1 TBL2 TBLb
NIC
ToR Switch
Server
VMbVMa
Datapath
VNFa
API
VNFb
API
TBLaTBL1 TBL2 TBLb
SmartNIC
Server
VMbVMa
Datapath
VNFa
API
VNFb
API
TBLaTBL1 TBL2 TBLb

64. © 2017 Open-NFP 59
VNF Offload - Table + Plugin Model
• Embedded Network Function (eNF) =
plugin in NFVi datapath
• “Satellite” to a VNFC (or part thereof)
• Each eNF offloads and accelerates the
corresponding VNF
• API enables VNF to control the eNF and
receive events (e.g. security /telemetry)
• Traffic can be flow between eNFs and VNFs
• Match tables can direct traffic to eNFs
• Table entries populated by VNFs or
SDN controller
• Table entries can have associated state
NFVI
VNFs
Datapath
VNFa
table
entries
TBLa
packets
control /
events /
data
Datapath
plugin
offloading
VNFa
VNFb
control /
events /
data
TBL1
eNFa
eNFb
APIAPI

65. © 2017 Open-NFP 60
Plugin Model - Mapping to Datapath + Hardware
Physical switchSmartNIC in server
(e.g. OVS, vRouter, VPP)
Software vSwitch in server
(e.g. OVS, vRouter, VPP)
eNF in P4 / C / eBPF
Kernel: eNF in eBPF / P4
Usermode: eNF in C / P4
eNF in P4
Server
Host
VMb
NIC
VMa
vSwitch
VNFa
TBLa
VNFb
TBL1
eNFa
eNFb
APIAPI
Server
VMb
ToR
Switch
VMa
Datapath
VNFa
TBLa
VNFb
TBL1
eNFa
eNFb
APIAPI
SmartNIC
Server
VMbVMa
Datapath
VNFa
TBLa
VNFb
TBL1
eNFa
eNFb
APIAPI

66. © 2017 Open-NFP 61
Agenda
• Survey of existing initiatives and APIs
• Proposed offload models
• Proposed mapping to hardware / software platforms
• Next steps - invitation to collaborate

67. © 2017 Open-NFP 62
VNF Acceleration API Evolution
• Requirements
• API must be vendor + OS + platform + hardware independent…
… but need to start with implementing specific instances, then expand
• Leverage existing APIs, e.g. P4 RT API, DPDK flow API, Linux TC FLOWER API
• Consider existing specifications / initiatives - e.g. ETSI NFV, OPNFV DPACC
• Proposing activity to further evolve VNF acceleration APIs and mechanisms
• Primarily hosted at OPNFV — name of project TBD
• Coordinate with other relevant groups, e.g. ETSI NFV, P4, fd.io VPP, OVS…
• Collaborators - VNF and NFVi vendors (hardware/software), end users (operators)

68. © 2017 Open-NFP 63
Potential First Prototype (to discuss)
• Selected Platforms
• Linux guest and host OS, KVM hypervisor, x86_64 server, C language binding
• P4-capable SmartNIC, e.g. Netronome Agilio
• P4-capable software datapath, e.g. fd.io VPP with P4 to VPP translator,
and/or P4 for Linux kernel eBPF
• P4-capable physical switch, e.g. Barefoot Tofino powered devices
• Selected Use cases (applications / VNFs)
• Firewall
• vEPC
• Features
• API exposing match-action and stateful table operations in chosen datapath
• P4 plugin implementing additional capabilities for the use case
Note: other hardware and software platforms, use cases, languages for plugins to follow

69. © 2017 Open-NFP 64
Session 4: Developing and Testing VNF Acceleration with
the Open-NFP Community

70. © 2017 Open-NFP 65
Agenda
Problem Statement
Open-NFP Community Application Testing
Towards an Open Data Plane Acceleration Test Framework

71. © 2017 Open-NFP 66
NFV Current PoC / Testing Process
Each PoC/Test Cycle requires different tools, different criteria and different test environment.

72. © 2017 Open-NFP 67
NFVi Test Requirements with SmartNIC
• Need test environment that allows for SmartNIC-based testing of NFVi
• Test Cases change based on functionality enabled by SmartNIC
OpenStack
Neutron
OVS ML2
Compute Node
VM VM VMVM
. . .OpenStack Nova
Agent
OVSDB
OpenFlow
SmartNIC
OVS Datapath
Actions
Match
Tables
OpenStack
Nova
Open Daylight
Controller (ODL)
Tunnels
Deliver to Host
Update Statistics
SR-IOV & Virtio
ovs-vswitchd
ovs-dbserver
OVS Datapath
Actions
Match
Tables
Tunnels

73. © 2017 Open-NFP 68
VNF Offload – Table Model Requirements
• Need ability to offload VNF API to SmartNIC
• Need test environment that allows for SmartNIC-based testing of VNF API

74. © 2017 Open-NFP 69
VNF Offload – Plugin Model Requirements
• Need Ability to Offload VNF API to SmartNIC
• Need tools to develop and deploy eNF in P4 / C / eBPF
• Need Test Environment that allows for SmartNIC-based testing of VNF

75. © 2017 Open-NFP 70
Summary Of Test Requirements
Need A Community-Based Approach to address all components and requirements.
• NFVi should have ability to offload NFVi data path to SmartNIC
• VNF should have ability to offload VNF API to SmartNIC
• Developers need tools to develop and deploy eNF on SmartNIC
• End Users need standard test cases to exercise SmartNIC functionality
• Test framework should allow testing of NFVi, VNF, eNF and NFV System with SmartNIC

76. © 2017 Open-NFP 71
Agenda
Problem Statement
Open-NFP Community Application Testing
Towards an Open Data Plane Acceleration Test Framework

77. © 2017 Open-NFP 72
Open-NFP Infrastructure
• Growing community for Data Plane Developers -- www.open-nfp.org.
• ~40 contributing organizations
• SDK, Working Code examples, Tutorials, Webinars etc.
• Wide range of P4 / C projects on server-based networking
• Ability to engage with over 200 community members through the open-nfp Google Group.
• Annual DXDD event for developer community
• Hardware
• Netronome Agilio Platform – 10 GbE – 100GbE SmartNICs
• Remote access with the cloud lab infrastructure
• Software
• OVS acceleration software, P4 / C SDK
• Lots of example code at https://github.com/open-nfpsw
Infrastructure with Hardware, Software and Community Support for Data Plane Apps

78. © 2017 Open-NFP 73
P4 / C SDK Tool Chain
SDK-6 - Integrated Development Environment (IDE)
Simple_Router.p4 Packet_filter.c
Editor with language highlight and breakpoint support
P4 front end compiler
Simple_Router.IR
P4 back end compiler
Compiler Linker
Loader
Simulator
C scripting
Assembler (NFAS)
Debugger

79. © 2017 Open-NFP 74
P4 / C Application Work Flow
Native code compiler
Sandbox C

80. © 2017 Open-NFP 75
NFVi Test Case Examples
SmartNIC
Test / Traffic Generator
(e.g. IXIA, Spirent)
External Traffic Generator
with SR-IOV
Test / Traffic
Generator
Test / Traffic
Generator
VM-VM - Virtio Configuration
SmartNICSmartNIC

81. © 2017 Open-NFP 76
VNF Test Case Examples

82. © 2017 Open-NFP 77
Agenda
Problem Statement
Open-NFP Community Application Testing
Towards Open Data Plane Acceleration Testing

83. © 2017 Open-NFP 78
Towards Open Data Plane Acceleration Testing
• Cloud-Based Test Framework Access
• Enable OPNFV Community to access Open-NFP Remote Access Cloud Lab Infrastructure
• Working with Tier 1 Cloud Providers to setup additional Community VNF Acceleration Test Lab
• Define Acceleration POD
• Working with Tier 1 Server vendors to define Acceleration POD(s)
• Defining VNF, VNF Models and Test Tools
• Open Source: iPerf, DPDK pktgen, Moongen, vsperf, trafgen etc.
• Commercial: Spirent, IXIA – Both VM-to-VM and Physical-to-VM Test Models.
• Define repeatable process
• Leverage Pharos, Lab-As-A-Service and OPNFV Infrastructure to define an repeatable process
Join us to define, refine and execute test cases.

84. © 2017 Open-NFP 79
Enable TCO Analysis Of Acceleration OptionsIncreasingTCOBenefits
Fewer number of CPU cores per VM App
6X
3X
1X
8 cores/VM 1 core/VM4 cores/VM
High VM Workload
vEPC, vRAN
Med VM Workload
vCDN, vCPE
Low VM Workloads
IT Apps
TCO Improves for:
VM Apps that consume fewer CPU cores
and higher PPS
More services enabled in the data path –
policies, per flow statistics, mirroring for
analytics, service chains, per flow load
balancing etc.
Increased connections per second
Bring your VNF to Offload and/or Test.

85. © 2017 Open-NFP 80
Summary and Call to Action

86. © 2017 Open-NFP 81
Summary
● NFVi Data Plane Acceleration with SmartNICs is moving forward
• Acceleration and Offload needed to make NFV efficient
• Many acceleration models and data planes exist
• Need for HW and SW independence is key to success of offload solutions
• Movement towards common/unified API already underway in Linux community
● VNF Acceleration with SmartNICs will bring the next set of gains
• VNFs become the bottleneck once the NFVi layer is accelerated
• VNF requirements and models for acceleration are being discussed
• Open API for VNF acceleration is critical to success
• Need to agree on the proper framework for the Open API definition (OPNFV, ETSI, etc)
● Important to have a common framework for VNF acceleration and VNF
testing and API compliance (OPNFV Danube, Pharos Labs, etc.)

87. © 2017 Open-NFP 82
Call To Action
● Get involved in Linux community and other activities around NFVi offload with common/unified
approaches
● Join the discussion and contribute to establishing a framework to define an Open API for VNF
acceleration (OPNFV Project, ETSI NFV, etc.)
● Provide input around VNF interoperability and performance testing and infrastructure requirements:
help to define test cases, and supply VNFs to be tested
Contact us to provide feedback and discuss how you can get involved:
VNF acceleration API: email vnfapi@open-nfp.org
VNF testing: email vnftest@open-nfp.org
Enjoy the rest of the OPNFV Summit!

88. © 2017 Open-NFP 83
Thank You!