This document discusses the requirements and challenges of networking in an Infrastructure as a Service (IaaS) cloud environment. It proposes using distributed overlay-based network virtualization as a solution. Key points include: - Traditional networking devices do not scale well for IaaS clouds with high churn and micro-granularity needs. - An overlay-based approach using encapsulation can build a virtual network that decouples from the physical network and handles network intelligence at the edge. - The Midokura solution presented uses this approach with a distributed control plane, centralized database, and packet processing at ingress points. - It integrates with popular cloud stacks like OpenStack and CloudStack to provide scalable L

1. Making a case for
distributed overlay-based
network virtualization
Ben Cherian
Chief Strategy Officer
@bencherian
Midokura

2. So, you’re building a
cloud?

3. Requirements

4. 1 2 3 4 5
vs
1 New
1
Horizontal scaling

5. Building blocks of an
IaaS cloud

6. Cloud management
system

7. Compute

8. Storage

9. Networking

10. Traditional networking
devices scale up

11. Service interruptions

12. High churn, micro
granularity

13. Limitations of
VLANs

14. Traffic trombones

15. Human costs don’t scale

16. Additional
Requirements

17. IaaS Cloud Networking Requirements
• Multi-tenancy • ACLs
• L2 isolation • Stateful (L4) Firewall
 Security Groups
• L3 routing isolation
 VPC • VPN
 Like VRF (virtual  IPSec
routing and forwarding) • BGP gateway
• Scalable control • REST API
plane • Integration with CMS
 ARP, DHCP, ICMP  CloudStack
• NAT (Floating IP)  OpenStack

18. IaaS Cloud Networking Requirements
Typical Network Topology
uplink
- Creat e one provider rout er upon deployment - Link to uplink
- Creat e a rout er f or a t enant - BGP multi-homing
- M ap a bridge f or a quant um net work - Global NAT/route settings,
e.g. for floating ip
Provider Virtual
Router (L3)
- Tenant router for
FW, LB, DHCP and NAT
Tenant/Project A Tenant/Project B
Tenant B
Tenant A
Virtual Router
Virtual Router
Network A1 Network A2 Network B1
TenantB office
Virtual L2 Virtual L2 Virtual L2
Switch A1 Switch A2 Switch B1 Tenant B
VPN Router
VM1 VM3 VM5 VM2 VM4 VM6
Office
Network

19. Solution: Distributed overlay-based network
virtualization

20. Use encapsulation to
build a virtual network

21. Handle network intelligence /
network state at the edge

22. Require less of the
physical network

23. Edge to Edge IP Overlays
• Isolation not using VLANs
 IP encapsulation
• Decouple from physical network
• Provisioning VM doesn’t change underlay state
• Underlay delivers to destination host IP
• Use scalable IGP (iBGP, OSPF) to build multi-path
underlay
• Inspired by VL2 from MSR

24. Market trends supporting overlay model
• Packet processing on x86 CPUs (at edge)
– Intel DPDK facilitates packet processing
– Number of cores in servers increasing fast
• Clos Networks (for underlay)
– Spine and Leaf architecture with IP
– Economical and high E-W bandwidth
• Merchant silicon (cheap IP switches)
– Broadcom, Intel (Fulcrum Micro), Marvell
– ODMs (Quanta, Accton) starting to sell directly
– Switches are becoming just like Linux servers
• Optical intra-DC Networks

25. The MidoNet Solution
• Virtual L2 Distributed Switching
• Virtual L2 Isolation
• Virtual L3 Distributed Routing
• Virtual L3 Isolation
• L4 Services (Load Balancing, Firewall)
• NAT
• Access Control Lists (ACLs)
• Virtual port and device monitoring
• Restful API
• Web based management control panel

26. The MidoNet Solution
Logical Topology
vPort Virtual
Tenant A
Switch A1
Virtual vPort
Router
vPort Provider Virtual
Virtual Switch A2
vPort
Router
Tenant B
vPort Virtual Virtual
Router Switch B1
vPort
VM
MN MN VM
BGP BGP
Multi To ISP1
Homing
Internet Private IP VM
MN Network MN VM
BGP
To ISP2 Tunnel
BGP
To ISP3
VM
MN MN VM
MN MN MN
Network State Database
Physical Topology

27. The MidoNet Solution
• Distributed and scalable control plane
 Handle all control packets at local MidoNet agent adjacent to
VM
• Scalable and fault tolerant central database
 Stores virtual network configuration
 Dynamic network state
 MAC learning, ARP cache, etc
 Cached at edges on demand
• All packet modifications at ingress Packet Tunnel
Ingress
 One virtual hop MN
 No travel through middle boxes Encapsulated
 Drop at ingress Drop/Block

28. Scale out model

29. The MidoNet Solution
• Scalable edge gateway interface to external networks
– Multihomed BGP to ISP
• REST API and GUI
• Integration with popular open source cloud stacks
– OpenStack
• Removes SPOF of network node
• Scalable and fault tolerant NAT for floating IP
• Implements security groups efficiently
– CloudStack (in progress)

30. CloudStack integration
• Currently have L2 integration
• Full integration is slated for Q1, 2013
– L3 isolation (without VM / appliance)
– Security groups (stateful firewall)
– Floating IP (NAT)
– Load balancing (L4)

31. Questions?
Slides: http://www.slideshare.net/midokura

32. Backup slides

33. Candidate Models
• Traditional network
• Centrally controlled OpenFlow based hop-
by-hop switching fabric
• Edge to edge overlays

34. Traditional Netowrk
• Ethernet VLANs for L2 isolation
 4096 limit
 VLANs will have large spanning trees terminating on many hosts
 High churn in switch control planes doing MAC learning non-stop
 Need MLAG for L2 multi-path
 Vendor specific
• MPLS VPN?
• VRFs for L3 isolation
 Not scalable to cloud scale
 Expensive hardware
 Not fault tolerant

35. OpenFlow Fabric
• State in switches
 Proportional to virtual network state
 Need to update all switches in path when provisioning
 Not scalable, not fast enough to update, no atomicity of
updates
• Not good for IaaS cloud virtual networking

36. Spine and Leaf Network Architecture

37. Deep OpenStack Integration
• Quantum Plugin
– L2 isolation, of course
• Also…
– L3 isolation (without VM / appliance)
– Security groups (stateful firewall)
– Floating IP (NAT)
– Load balancing (L4)
37