Ocpeu14

Accelerate networking innovation through programmable data plane
Removing switches from datacenters with TRILL/VNT and smartNIC
Ahmed Amamou, ahmed@gandi.net
Benoît Ganne, bganne@kalray.eu

•Gandi is a domain name registrars since 1999 and a cloud provider since 2008
•We provide both
–IaaS: Infrastructure As A Service
–PaaS: Platform As A Service
•We support open source community:
–Provide open source code : https://github.com/Gandi
–Support open source project: VLC, Debian, … *
* Check http://www.gandi.net/supports/ for exhaustive list
Who is Gandi?
2

IaaS new network’s challenges
3
•Cisco Forecast report*:
–Cloud traffic was about 3.3 zetta (1021) Bytes in 2013
–Cloud traffic will reach 6.6 zetta Bytes in 2016
–76% of cloud traffic are East-West (within the same datacenter)
 A high density of links within a datacenter is needed
•Customer need a full network access
–Should be isolated
– VM network configuration should not be restrictive
Overlaying tenant traffic should be considered
* Cisco Global Cloud Index Forecast and Methodology, 2011-2016.

•New protocols are proposed to solve these problems (TRILL , VXLAN, 802.1 ad STT …) but:
– Hardware integration is slow
– Protocol extensions are hard to integrate
•We believe the OpenCompute community can help us
–To define an open, vendor-neutral API for programmable data plane
–Bring open hardware fulfilling those needs
Why OpenCompute?
4

•Switch from classic datacenter architecture to a full-mesh one
•Upgrade hardware to improve performances
New datacenter architecture
5

TRILL @Gandi
6
•Gandi uses commodity hardware as TRILL Rbridges since 2013
•We did not yet found hardware that suits our needs.

•Layer 2 Routing Protocol
•Uses a control and a data plane
•Control plane : based on IS-IS that computes all Routing information
•Data plane : forward packets using provided information from control plane
•Uses Mac-in-Mac encapsulation
TRILL: TRansparent Interconnection of Lot of Links
7
Original payload
TRILL Header

TRILL benefits
8
Commutation(L2)
Routing (L3)
TRILL
Configuration
Minimal
Intense
Minimal
Plug & play
Yes
No
Yes
Discovery
Automatic
Configured
Automatic
Learning
Automatic
Configured
Automatic
Multi path
No
Yes
Yes
Convergence
Slow
Fast
Fast
Connectivity
Inflexible
Flexible
Flexible
Scale
Limited
Important
Important

Control Plane: Forwarding database
9

Multitenancy: Virtual Network over TRILL (VNT)
10
New cloud architecture have to take into consideration Multitenancy
Trill does not provide Multitenancy handling mechanisms
→ We need to extend it

•Update Both control and data planes
–Control plane : Prune multicast tree to limit multicast traffic
–Data plane : Forwarding is conditioned by VNI support
VNT vs TRILL
11
VNT Encapsulation
Original Ethernet Frame
Outer Destination
Mac Address
Outer Source
Mac Address
Optional
Outer IEEE 802.1Q
TRILL Header
VNT Header
Extensions
Original
Packet Payload
Egress Rbridge Nickname
Ingress Rbridge Nickname
Options description
TLV
VNI Tag (24 bits)
L2 Routing information
Tenant
identification
Publication:
Amamou, A., Haddadou, K., & Pujolle, G. (2014).
A TRILL-based multi-tenant data center network. Computer Networks.

VNT: Multicast tree pruning
12
n3
n4
n5
n8
n7
n6
n1
n2
i1
i1
i2
i2
i1
i2
i1
i2
i1
i2
i1
i3
i3
i3
i2
i2
i1
i2
i1
i3
n3
n4
n5
n8
n7
n6
n1
n2
i1
i1
i2
i2
i1
i2
i1
i2
i1
i2
i1
i3
i3
i3
i2
i2
i1
i2
i1
i3
n5
n2
n8
n1
n7
n6
n4
n3
n5
n2
n1
n6
A –Vni1
A –Vni1
B –Vni1
Topology
Multicast tree

Current VNT implementation on Linux
13
Control plane : Quagga daemon
Data plane:
Linux Bridge Module

Current VNT implementation on Linux
14
Control plane : Quagga daemon
Data plane:
Linux Bridge Module
https://github.com/Gandi/

•Throughput is affected by the addition processing operation
•Processing for a single packet is not affected
Data plane: performance
15
Throughput
Delay

•Shift data plane from host to smartNIC
–Increase performance
–Offload x86 for other usages
•eg. Customers workload
Improving performance
16
Host
Host
NIC
smartNIC
Control plane
Data plane
Control plane
Data plane

•Founded in 2008, fabless semiconductor company
•Kalray has developed the disruptive MPPA® (Multi-Purpose Processing Array) programmable architecture
–Leading Performance / Energy Ratio Worldwide
–Time predictability and low latency
–Heterogeneous applications on the same chip
–High programmability
•Working with industry-leading partners and customers
•55 employees
•Offices in France and US
KALRAY deterministic supercomputing on a chip
17
First MPPA®-256 Chips with CMOS 28nm TSMC Leading Performance / Energy Ratio Worldwide

Software Defined NIC
Smart packet classification/dispatching
256 cores for packets processing
Standard C/C++ with GCC-4.9
Advanced debugging and profiling
Low latency
Zero-copy Ethernet  PCIe
< 1μs port-to-port transparent mode
< 1μs port to system memory
System integration
Linux support
Virtualization support
Low power
High throughput / Line rate
80 Gbps full-duplex line-rate (2x120MPPS)
3400 instructions per packet @64B
AES, SHA-1, SHA-2,CRC accelerators
2 x PCIe Gen3 8-lanes
MPPA®-256 Bostan Networking Strengths
18

MPPA®-256 Bostan
•64-bit processor
•Up to 800MHz
•High Performance
–845 GFLOPS SP / 422 GFLOPS DP
–1 TOPS
•High Bandwidth Network On a Chip
–2 x 12.8 GB/s
•High Speed Ethernet
–Up to 2x40 Gbps / 2x120 MPPS @ 64B
•DDR3 Memory interfaces
–2 x 64-bit + ECC @2133MT/s / 2 x 17GB/s
•PCIe Gen3 interface
–2 x 8-lanes / 2 x 8 GB/s full duplex
–End Point / Root Complex
•NoCX extension
–2 x 40 Gbps + 2 x 80 Gbps ILK
•Flash controller, GPIOs…
19

MPPA®-256 Processor Hierarchical Architecture 256 Processing Engine cores + 32 Resource Management cores
20
Manycore Processor
Compute Cluster
VLIW Core
Instruction Level Parallelism
Thread Level Parallelism
Process Level Parallelism

High Speed Ethernet Packet processing
•Ethernet Rx dispatcher
–8 classification tables
•Classify
•Extract fields
•Smart Dispatch
–Round Robin way
–Flexible cores allocation
•Round Robin vs. classification
•Per 10G Ports
• Ethernet Tx
–64 Tx FIFOs
–QoS between the FIFOs
–Flow Control between clusters and Tx FIFOs
21
Patent pending

VNT on a programmable data plane Multicast forwarding example
22
MPPA Linux ethernet driver
Linux networking stack
TRILL controller
Kalray Bostan smartNIC
x86
Hypervisor
Userspace application
•On-going work between Gandi and Kalray
–Explore programmable data plane opportunities
–Study a VNT smartNIC feasibility and architecture
•Multicast forwarding put a high load on each node
IO ethernet driver
8x10GbE

23
TRILL controller
x86
Hypervisor
•Dispatch the packet based on Egress Rbridge
–In case of multicast, Egress RBridge is set to the tree root
–Each cluster “owns” a subset of the possible Egress RBridge (ie. a FIB subset)
8x10GbE
IO ethernet driver
if (Packet[Ethertype] == TRILL) {
send to cluster #HASH(Egress RBridge)
}
<Ethertype=TRILL, Egress=DTROOT, VNI=VNI-1>

24
TRILL controller
x86
Hypervisor
8x10GbE
IO ethernet driver
•Dispatch the packet based on Egress Rbridge
–In case of multicast, Egress RBridge is set to the tree root
–Each cluster “owns” a subset of the possible Egress RBridge (ie. a FIB subset)

25
TRILL controller
x86
Hypervisor
•Lookup the list of next-hop RBridges for this multicast tree
–RBridge owner clusters can be local or remote
•Lookup the LIB for local ports if any
8x10GbE
IO ethernet driver
FIB[Egress RBridge] = {
Egress RBridge MAC;
Egress RBridge Interface;
MCTree = [ RBx, RBy, … ];
VNI = [ VNI-1, VNI-2, … ];
}
LIB = {
(Local MACx, Local Portx, VNI-1);
…
}

26
TRILL controller
x86
Hypervisor
•Forward the frame
–Remote
•Forward to clusters owning the next-hop RBridge
–Local
•Decapsulte inner frame
•Forward it the local VM
8x10GbE
IO ethernet driver

27
TRILL controller
x86
Hypervisor
•Check if the RBridge support the appropriate VNI
–If yes forward to Rbridge
–If not, stop here
8x10GbE
IO ethernet driver
FIB[Egress RBridge] = {
Egress RBridge MAC;
Egress RBridge Interface;
MCTree = [ RBx, RBy, … ];
VNI = [ VNI-1, VNI-2, … ];
}

28
TRILL controller
x86
Hypervisor
•Check if the RBridge support the appropriate VNI
–If yes forward to Rbridge
–If not, stop here
8x10GbE
IO ethernet driver

•Solving SDN and network virtualization challenges requires new protocols
–eg. VXLAN, NVGRE, TRILL/VNT…
•Efficiency generally means hardware support
…But hardware development cannot keep up with software and slow down innovation
•Gandi and Kalray think a programmable data plane can reconcile efficiency and innovation
…But we need open ecosystems, standards and API
Innovation and efficiency
29

Thank you for your attention!
Questions?
Ahmed Amamou, ahmed@gandi.net
Benoît Ganne, bganne@kalray.eu

Ocpeu14

More Related Content

What's hot

Similar to Ocpeu14

Recently uploaded

Ocpeu14