by Pino De Candia of Midokura at MidoDay Tokyo 2015

1. MidoNet Deep Dive
Pino de Candia

2. Agenda
1. Virtual Network Topology
2. Physical-Virtual Boundary
3. Cluster nodes (aka Network State DB)
4. Compute nodes
5. Flow Switch concept
6. Gateway nodes
7. How tunneling works
8. Change propagation and flow invalidation
9. L4 Flow State

3. Bare Metal
Server
Bare Metal
Server
MidoNet transforms this...
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
IP Fabric

4. Bare Metal
Server
Bare Metal
Server
into this...
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM VM
VM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VM
VM
VM
FW
LB
FW
LB
Internet/
WAN
FW
LB
LB
then moves packets...

5. “Port-Interface Bindings”
● Vport1 => Compute1, tap12345
● Vport2 => Compute2, tap67890
● Uplink1 => Gateway1, eth1
Virtual-Physical Boundary
Bindings (and the virtual network topology) are stored in
MidoNet’s cluster and propagated to the MidoNet
Agents.

6. Bare Metal
Server
Bare Metal
Server
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
Cluster stores and propagates topology
midonet
cluster 2
midonet
cluster 3
midonet
cluster 1
IP FabricIP Fabric

7. Bare Metal
Server
Bare Metal
Server
Port-Interface Bindings
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM VM
VM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VM
VM
VM
FW
LB
FW
LB
Internet/
WAN
FW
3
LB
LB
Vport1 => Compute1, tap12345
Uplink1 => Gateway1, eth1
VM1
VM
2
Vport2 => Compute2, tap67890

8. Bare Metal
Server
Bare Metal
Server
Back to the physical view...
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
Compute 1 Compute 2
midonet
cluster 2
midonet
cluster 3
midonet
cluster 1
IP Fabric

9. Port-Interface Bindings in the Physical View
Compute 1
Flow Switch (in-kernel OVS)
Compute 2
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
Flow Switch (in-kernel OVS)
IP1 IP2
VXLAN
Tunnel Port
VXLAN
Tunnel Port
eth0 eth0
port5, tap12345 port6, tap678902
Vport2 => Compute2, tap67890
1
Vport1 => Compute1, tap12345
The compute hosts in a little more detail

10. Compute 1
Flow Switch (in-kernel OVS)
What is a flow switch?
VM
VM
VM
VM VM
VM VM
VM
IP1
VXLAN
Tunnel Port
eth0
10.0.0.4->10.0.0.5
10.0.0.3->200.0.0.5
port6 port8
port1
MidoNet Agent
(Java Daemon)
10.0.0.3->10.10.0.2
Miss packets go
to user-space via
Netlink channel
Rule1: Match: in=6, srcIP=10.0.0.4
➔ Actions: []
Rule2: Match: in=8, srcIP=10.0.0.3,
dstIP=200.0.0.5, proto=TCP, srcPort=23109,
dstPort=22
➔ Actions: [srcIP=111.0.0.4, tunnel=[src=192.
168.0.3, dst=192.168.0.4, key=100], out=1]
MidoNet can:
1. ignore it
2. send it back with actions
3. install a new flow rule
4. do both #3 and #4
port2 Rule3: Match: in=8, srcIP=10.0.0.3, dstIP=10.
10.0.2, proto=ICMP
➔ Actions: [srcMAC=M1, dstMAC=M2, out=2]

11. Bare Metal
Server
Bare Metal
Server
Port-Interface Bindings
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM VM
VM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VM
VM
VM
FW
LB
FW
LB
Internet/
WAN
FW
3
LB
LBUplink1 => Gateway1, eth1
VM
VM

12. Gateway 1
Detail of the Gateway Node
Compute 1
VM
VM
VM
VM VM
VM VM
VM
Quagga,
bgpd
IP Fabric
Flow Switch (in-kernel OVS) Flow Switch (in-kernel OVS)
IP1 IP3
VXLAN
Tunnel Port
eth0 eth0 eth1
VXLAN
Tunnel Port
3
Uplink1 => Gateway1, eth1
Internet/WAN/DC
port5, tap123451
Vport1 => Compute1, tap12345

13. Bare Metal
Server
Bare Metal
Server
Back to the physical view...
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
midonet
cluster 2
midonet
cluster 3
midonet
cluster 1
midonet
gateway
2
midonet
gateway
3
midonet
gateway
1
IP FabricIP Fabric
Internet/
WAN/DC

14. Gateway 1
Detail of the Gateway Node - pre-installed flows
Quagga,
bgpd
Flow Switch (in-kernel OVS)
IP3
eth0 eth1
VXLAN
Tunnel Port
Internet/WAN/DC
Compute 1
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
Flow Switch (in-kernel OVS)
IP1
VXLAN
Tunnel Port
eth0 3
Uplink1 => Gateway1, eth1
port5, tap123451
Vport1 => Compute1, tap12345
port1 port2
port3, veth0
veth1
Rule1: Match: in=2, srcIP=<Uplink1 Peer’s IP>,
dstIP=<Uplink1’s IP>, proto=TCP, dstPort=BGP
➔ Actions: [out=3]
Rule2: Match: in=2, srcIP=<Uplink1 Peer’s IP>,
dstIP=<Uplink1’s IP>, proto=TCP, srcPort=BGP
➔ Actions: [out=3]
Rule3: Match: in=3
➔ Actions: [out=2]
Rule4: Match: in=2, ethertype=ARP, op=BOTH,
srcIP=<Uplink1 Peer’s IP>
➔ Actions: [out=3, to-user-space]
Internet/
WAN
Uplink1 => Gateway1, eth1
MidoNet Agent
(Java Daemon)

15. ● Flow rules are computed at the ingress host
● by simulating a packet’s path through the virtual topology
● without fetching any information off-box (~99% of the time)
● if the egress port is on a different host, then the packet is
tunneled
● and the tunnel key encodes the egress port
● so that no computation is needed at the egress
MidoNet uses VNIs to encode Vports - NOT network
segments.
Flow rule computation and tunneling

16. Compute 1
Flow Switch (in-kernel OVS)
VM
VM
VM
VM VM
VM VM
VM
IP1
VXLAN
Tunnel Port
eth0
Compute 2
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
Flow Switch (in-kernel OVS)
IP2
VXLAN
Tunnel Port
eth0
Pre-installed flows on the compute hosts
Rule1: Match: in=1, tunKey=<VNI of VM1>
➔ Actions: [out=2]
Rule2: Match: in=1, tunKey=<VNI of VM2>
➔ Actions: [out=3]
Rule3: Match: in=1, tunKey=<VNI of VM3>
➔ Actions: [out=4]
… and so on...
port1
ExtIP->VM1
IP3 -> IP1
VNI of VM1
ExtIP->VM1

17. Bare Metal
Server
Bare Metal
Server
A flow between two VMs...
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM VM
VM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VM
VM
VM
FW
LB
FW
LB
Internet/
WAN
FW
LB
LBVM1->FIP1
VIP1->VM2
FIP2->FIP1
FIP2->VIP1

18. is tunneled C1 to C2 (no middle compute nodes)
Compute 2Compute 1
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
Flow Switch (in-kernel OVS) Flow Switch (in-kernel OVS)
IP1 IP2
VXLAN
Tunnel Port
VXLAN
Tunnel Port
VM1->FIP1
VIP1->VM2
IP1 -> IP2
VNI of VM2
VIP1->VM2
Host network stack
performs encapsulation Host network stack
performs decapsulation
New Rule: Match: in=5, srcIP=VM1, dstIP=FIP1,
proto=TCP
➔ Actions: [srcIP=VIP1, dstIP=VM2, tunnel=
[src=IP1, dst=IP2, key=<VNI of VM2],
out=1]
port5, tap12345

19. Bare Metal
Server
Bare Metal
Server
A flow that exits an uplink...
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM VM
VM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VM
VM
VM
FW
LB
FW
LB
Internet/
WAN
FW
LB
LB
VM1->ExtIP1
FIP1->ExtIP1

20. Gateway 1
...is tunneled C1 to L3GW node
Compute 1
VM
VM
VM
VM VM
VM VM
VM
Quagga,
bgpd
IP Fabric
Flow Switch (in-kernel OVS) Flow Switch (in-kernel OVS)
IP1 IP3
VXLAN
Tunnel Port
eth0 eth0 eth1
VXLAN
Tunnel Port
VM1->ExtIP1
FIP1->ExtIP1
IP1 -> IP2
Uplink1 VNI
FIP1->ExtIP1
Internet/WAN/DC
port5, tap12345
New Rule: Match: in=5, srcIP=VM1, dstIP=ExtIP1,
proto=TCP
➔ Actions: [srcIP=FIP1, dstIP=ExtIP1, tunnel=
[src=IP1, dst=IP3, key=<VNI of Uplink1],
out=1]

21. Bare Metal
Server
Bare Metal
Server
If an uplink fails...
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM VM
VM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VM
VM
VM
FW
LB
FW
LB
Internet/
WAN
FW
LB
LB

22. Bare Metal
Server
Bare Metal
Server
notify whomever needs to know
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
midonet
cluster 2
midonet
cluster 3
midonet
cluster 1
midonet
gateway
2
midonet
gateway
3
midonet
gateway
1
IP FabricIP Fabric
Internet/
WAN/DC

23. Compute 1
Flow Switch (in-kernel OVS)
The receiving Agent invalidates related rules
VM
VM
VM
VM VM
VM VM
VM
IP1
VXLAN
Tunnel Port
eth0
port1
MidoNet Agent
(Java Daemon)
New Rule: Match: in=5, srcIP=VM1, dstIP=ExtIP1,
proto=TCP
➔ Actions: [srcIP=FIP1, dstIP=ExtIP1, tunnel=
[src=IP1, dst=IP3, key=<VNI of Uplink1],
out=1]
port5, tap12345
VM1->ExtIP1
If the flow is still active, a miss packet
will be sent to the MN Agent via Netlink
and a new flow rule can be recomputed
that doesn’t use the failed uplink.
Uplink1 is Down

24. Bare Metal
Server
Bare Metal
Server
If a flow had L4 state (SNAT)...
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM VM
VM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VM
VM
VM
FW
LB
FW
LB
Internet/
WAN
FW
LB
LB
VM1->ExtIP1
FIP1->ExtIP1

25. Bare Metal
Server
Bare Metal
Server
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
midonet
cluster 2
midonet
cluster 3
midonet
cluster 1
midonet
gateway
2
midonet
gateway
3
midonet
gateway
1
IP FabricIP Fabric
Internet/
WAN/DC
The state is shared with return flow ingress(es)

26. ...is tunneled C1 to L3GW node
Compute 1
VM
VM
VM
VM VM
VM VM
VM
IP Fabric
Flow Switch (in-kernel OVS)
IP1 IP3
VXLAN
Tunnel Port
eth0
FIP1->ExtIP1
IP1 -> IP2
Uplink1 VNI
Internet/WAN/DC
port5, tap12345
VM1->ExtIP1
Gateway 1
Quagga,
bgpd
Flow
Switch
(in-kernel
OVS)
eth0
Tunnel
Port
eth1
Gateway 2
Quagga,
bgpd
Flow
Switch
(in-kernel
OVS)
eth0
Tunnel
Port
eth1
Gateway 3
Quagga,
bgpd
Flow
Switch
(in-kernel
OVS)
eth0
Tunnel
Port
eth1
IP5 IP6
Flow State
IP1 -> IP2
Special VNI

27. Port’s packet pipeline in MN 5.0
Port
Mirroring
from wire
Service
Redirection
Chain
Filtering
Chain
into
device
Filtering
Chain
from
device Service
Redirection
Chain
Port
Mirroring
onto wire to next
port or end
simulation

28. Bridge packet pipeline in MN 5.0
Pre-
forwarding
Chain
from port Forwarding
Table
Post-
forwarding
Chain
to one or
more ports

29. Router packet pipeline in MN 5.0
Pre-
forwarding
Chain
from
port
Routing
Table
Post-
forwarding
Chain
to one or
more ports
L4 LBaaS

30. Security Groups are translated to Chains and Rules

31. New in MN 5.0: L2 SFC API Objects
L2Insertion:
● inspected vm port UUID
● inspected vm MAC
● service port UUID
● vlan tag
● fail-open (true/false)
● position (relative to other insertions for the same inspected vm port)
L2Service
● service port UUID

32. 1 protected VM, 1 SF
VM1
(protected)
VM2 SF1
1 protected VM, SF down, fail-close1 protected VM, SF down, fail-open