Recommended
More Related Content
What's hot
What's hot (20)
Similar to Unleashing the Power of Fabric Orchestrating New Performance Features for SR-IOV VNFs
Similar to Unleashing the Power of Fabric Orchestrating New Performance Features for SR-IOV VNFs (20)
More from Liz Warner
More from Liz Warner (20)
Recently uploaded
Recently uploaded (20)
Unleashing the Power of Fabric Orchestrating New Performance Features for SR-IOV VNFs
- 1. Unleashing Network hardware to Cloud Manjeet Singh Bhatia (Intel) Munish Mehan (AT&T) Deepak Tiwari (AT&T)
- 2. Outline Introduction Background on Networking Technologies in cloud Custom NIC Features Software layers Sriov and Port Mirroring Upstream changes in OpenStack Tap as a Service for SRIOV Development Progress: Completed CI testing using zuul v3 Demo
- 3. 3 • SRIOV • Neutron Integration • DPDK • OpenVSwitch or OVS + Dpdk • Linux Bridge Background on the Networking Solutions in Cloud
- 4. 4 • /sys/class/net/<interface-name>/device/sriov*1 • | +-- qos • +-- [TC, 0-7] # TC • | +-- priority # list of PCP values mapped to this TC • | +-- lsp # link strict priority • | +-- max_bw # max bandwidth for this class • | +-- min_bw # min bandwidth for this class • | +-- egress_mirror # mirror traffic from this PF to specified VF • | +-- ingress_mirror Custom NIC Features
- 5. 5 • +-- [VF-id, 0 ... 127]*2 • | +-- vlan_mirror # list of VLANs to mirror to this VF • | +-- trunk # list of VLANs to filter on (802.1Q trunk) • | +-- tpid # TPID of outer (s-tag) 0x8100 | 0x88A8 • | +-- egress_mirror # mirror traffic from this VF to specified VF • | +-- ingress_mirror • | +-- mac_anti_spoof # enable/disable MAC anti spoofing • | +-- vlan_anti_spoof # enable/disable VLAN anti spoofing • | +-- loopback # enable/disable local traffic loopback (VEB/VEPA) • | +-- default_mac # default MAC, if not set use random • | +-- mac_list # list of additional MACs (00:11:22:33:44:55, aa:bb:cc:dd:ee:ff) • | +-- ucast_promisc # unicast promiscuous • | +-- mcast_promisc # multicast promiscuous • | +-- allow_bcast # allow/not allow bcast • | +-- strip_stag # strip outer tag (s-tag) • | +-- enable # enable/disable VF Custom NIC Features
- 6. 6 • Expose to cloud api via Kernel driver • (Intel I40 e drivers expose features to cloud via sysinterface) • Might need kernel changes • Cloud Api changes (OpenStack Tap-as-a-service) • Add OpenStack CLI for feature usage Software Stack changes Hardware Hardware DriversKernel
- 7. 7 • Single Root IO virtualization SRIOV
- 8. 8 SRIOV Traffic Mirroring: VLANs to VF P1P1 (PF 1) VF1 VF2 VF3 VF4 VF5 VF6 VF7 VF8 VF9 VF10 VF11 VF12 VF62 VF63 Service VM – 1 eth1 eth2 eth0 eth3 A Probe VM eth1 eth2 eth0 Service VM – 2 eth1 eth2 eth0 eth3 eth4 P3P1 (PF 2) VF1 VF2 VF3 VF4 VF5 VF6 VF7 VF8 VF9 VF10 VF11 VF12 VF62 VF63 B1 2 1 Tap Service bound to NIC P1P1 2 Tap Service bound to NIC P3P1 A Tap Flow bound to port “eth1” of Service VM-1 B Tap Flow bound to port “eth2” of Service VM-1 A VF3 1 B VF5 2 VLANs – 163, 172, 181-198 VLANs – 154 VLANs – 163, 172 VLANs – 154 VLANs – 163, 172 Direction: BOTH VLANs – 154 Direction: BOTH VLAN (s) – 163, 172 VLAN (s) – 154
- 9. 9 TaaS • An extension to the OpenStack network service (Neutron) • Provides an API to enable port mirroring (PM) capability for tenant virtualized workload Packet Mirroring (PM) • Network packets entering (and/or leaving) one port (or vLAN) are copied and sent to another port, where the packets can be analyzed. Tap as a Service (TaaS) • Use-cases – Network Debugging: Analyze and debug data or diagnose errors on a network – Passive IDS: Mirrored packets sent to IDS that monitors a network for malicious activity or policy violation – Network Forensics • Security: Investigate security incidents, monitor anomalous traffic • Law enforcements: reassembling transferred files, searching for keywords, parsing human communications.
- 10. 10 • What is Zuul V3 ? • In-repo configuration • Native support for multi-node jobs • Ansible job content • Integration with more systems CI/CD with Zuul V3
- 11. 11 Zuul CI Architecture Gerrit (review.openstack.org) Tap-as-a-service Launcher nodepool Executor Executes test scripts Log service Post logs Zuul Scheduler Gerrit Stream Zuul Web Service
- 14. 14 Test Pipeline - Define zuul config - Add devices to nodepool - Define test pipeline - Define jobs - Docker-compose up
- 15. 15 Demo ?
Editor's Notes
- Hello, I’m Manjeet I work for intel as a Cloud Software Engineer, I’ve worked on openstack networking in the past and was also core contributor to networking-odl, neutron-lib and networking-omnipath projects, I’ll be talking about enabling network hardware in the openstack cloud, I’d like like to give credit to two engineers from At&T who I worked with in the past to enable some networking features in the cloud and we presented some of these features at Open Infra summit this year in Denver.
- So I’ll start with background on networking technologies in the cloud, these are mostly layer 2 technologies, and then will talk a bit about custom NIC card features and one major feature we worked on last openstack release, then I’ll give little intro sriov in comparison to openvswitch, Software layers that are required to be modified to expose these features
- SRIOV – Single root I/o virtualization a pci pass through, DPDK data plane development kit, a user space fast packet processing which bypass kernel for packet processor can be used along with difftent switcihing technologies like vswitches, linux bridge etc,
- There are many custom nic features available in the network cards some of them are quality of service like min bandwidth ingress/egress. These features can be exposed to application running on top of operating system or cloud services via system interfaces
- VFd can be thought of as the NIC hypervisor inasmuch as it provides the policy enforcement through both configuration and real-time validation of guest[2] requests. Vlan mirror is the feature we worked on last cycle of openstack release, which I’ll talk about more in detail about how it was implemented and now I’ll give high level overview of the feature, its basically probling traffic packets based on vlan filters to a remote machine or a port. To enable these features it require lot of changes at many levels in system software stack.
- Of course first we needed a hardware (NIC card capable of this feature) and this feature was exposed to cloud api’s vis kernel module which I40e ethernet driver, it also required changing cloud network services like neutron and tap-as-a-service to utilitize this feature which will make calls to sysfs interface exposed by i40 Ethernet driver which then would write vlans values to registers.
- As many of you may know single root I/o virtualization which provides a direct device as network interface to VM without any vswitch/v bridge in between, there are use cases where you want to use sriov for near native performance where as in switching can also be provided by vswitches in the hypervisor.
- Lets talk a bit more details about the feature sriov, so you see have three VMS on same server and two nic ports, the port mirroring works only within same NIC, vlans mirrored on a vf can be probed at different within the same nic card, eth1 is basically a flat network having many vlans 162, 172, 1810198 and eth2 is basically another network with single vlan 154, the way this works is the vlans to be mirrored have to be written into particular VF’s registers
- Tap as a service is project in openstack that provides api tp enable port mirroring over different layer 2 backends like ovs, linux bridge, it provides packet mirroring based on vlans, ports etc, There are many use cases for which it can be used, packet copy to remote devices is really helpful in packet analysis for debugging, and it can also be used for network forensics investigating malicious activity, for example the packets probed to diferent vm can be used by some application running on that vm, it could be ml model using that packet data to detect malicious activity.
- Though these features looks very cool from users perspective and have many valuable use cases but it comes with challenges as well, the main challenge was not enabling the feature in cloud service but was indeed automated testing integration especially when there are so many dependencies and configuration changes required at many levels. For just this feature it required a custom syfs driver that exposed feature via syinterface and then cloud api and client side changes spaning across multiple openstack projects and writing a Continuous testing pipeline was a challenge but we had zuul which made things bit simpler.
- As you see the architecture here for zuul, there are multiple services running in containers, first we need to monitor gerrit stream of patch submissions which zuul-scheduler will do, it will monitor projects based on configuration, you can monitor any number of projects, then it will ask launcher to provide a machine, nodepool is the service which can be used to provide baremetals, virtual machines or even cloud endpoints to the use as endpoints, once machine is ready executor will start running test jobs (like deploying all the dependencies and then running some integration testing) it also captures logs at all the levels which are posted at the end to gerrit
- As you see we have two connections, configuration for gerrit endpoint to monitor, in this case its openstack and we have main.yaml we our pipeline is deifned
- This is nodepool configuration, nodepool service basically runs as part of zuul service to keep track of what machines are in use or are available for next iteration of testing, we are using two baremetal machines to schedule our jobs for now, this is just an example, a cloud endpoint can also be configured, names are defined for each pool of machines and then those names will be used in jobs configuration to pin specific type of machines to jobs based on requirements