About overlay network in CORD(Central Office Re-architected as a Datacenter), overlay network is driven by VTN application in ONOS controller, and this talk will deep into how VTN use OpenFlow rules to construct Service Networking in CORD.

1. CORDVTN
Services Networking inside CORD architecture
1
Speaker: Wei-Yu Chen
Date: September 18, 2018

2. 2About ME
陳威祐 Wei-Yu Chen
q Pythonist
q ONF CORD Ambassador
q 很常出沒於各 conference
q COSCUP
q PyConTW
q HITCON
q ONF Intern
aweimeow
wychen@cord-ambassadors.org

3. 3Outline
q Introduction
q SDN & NFV
q ONOS
q Central Office Re-architecture as a Datacenter
q XOS
q Service, Tenant, Slices
q Service Dependency and Service Instance Link
q Virtual Tenant Network (VTN)
q VTN components
q VTN OVS pipeline
q Service Dependency relationship

4. 4SDN/NFV
Control Plane
Data Plane
Ref: An introduction to SDN

5. 5ONOS Distributed Architecture

6. 6Central Office Re-architect as a Datacenter (CORD)
SDN + NFV +
Cloud
Open Source Software
Commodity Hardware
(Servers, White-Box Switches, I/O Blades)
Large number of COs
Evolved over 40-50 years
300+ Types of equipment
Huge source of CAPEX/OPEX

7. 7CORD Network Architecture
Underlay Networking
Overlay Networking
ONOS Fabric
Underlay Controller
ONOS CORD
Overlay Controller
CORD composed by:
• Fabric Network
• A management head node
• Underlay controller
• Overlay controller
• Service orchestrator
• 2+ compute nodes

8. 8Mobile CORD (M-CORD)
Disaggregated
/Virtualized
RAN
Disaggregated
/Virtualized
EPC
Mobile Edge
Services
BBU, RRU
HSS, MME,
SPGW-C,
SPGW-U
Caching, SON,
Billing
SDN Fabric
Commodity servers, switches, access
SDN Control Plane
ONOS
NFV Orchestration
XOS
Cloud Solution
OpenStack

9. 9
Internet
Virtualized Mobile Services Architecture
Control Platform (XOS)
BBU MME HSS SPGW-C SPGW-U
Virtualized
BBU
Virtualized
BBU
Virtualized
MME
Virtualized
HSS
Virtualized
SPGW-C
Virtualized
SPGW-U

10. 10XOS
q XOS is an orchestrator to manage CORD architecture.
q XOS can manage networks of VM
(e.g. create a network using DHCP 10.0.0.0/24 by calling Openstack Neutron)
q XOS can manage VMs
(e.g. create a 2 CPU VM with HSS image by calling Openstack Nova)
q XOS can connect to VMs to perform user defined operations
(e.g. connect to HSS VM and enable HSS service with specific HSS config via SSH)
NFV Orchestration
XOS
CORD
Infrastructure

11. 11Orchestration behavior as module
q XOS provides a framework for implementing orchestration behavior as module.
NFV Orchestration
XOS
CORD
Infrastructure
XOS Developer
XOS User
Create Module
Request Orchestrate
Module
User intends to initialize VM
=> XOS calls Nova to initialize VM

12. 12Orchestration behavior as module
q Each installed module(include synchronizer) is a XOS Service
q Sychronizer manages the operation of module
XOS GUI
XOS User
Request
XOS RESTful API
XOS Web Socket
XOS Core
REDIS DB XOS DB
synchronizer
Module - X
xos service
synchronizer
Module - Y
xos service
CORD infrastructure services (ONOS, OpenStack, …)

13. 13
Service Slice
VM related XOS Models (illustrated)
Service VM
Service VM
Service VM
Multiple VM
(Service Instance)
Service VM
Service VM
Service VM
Service VM
Container
Service Slice
Service VM
Service VM
Service VM
Service
Controller
Controller +
Container
Tenant Service Slice Service
Service Slice
Service VM
Service VM
Service VM
Service Controller
(Synchronizer)
Service Slice
Service VM
Service VM
Service VM
Service with
multiple Slices
Controller +
2 Containers

14. 14VM related XOS Models
q Service: First Class in XOS, the model owned “service related objects”.
q Service composed by Service Slices and Service Contoller.
q Slice: Service Slice, Container to contain several Tenants.
q Tenant: Child model of Service, represent Service VM in XOS.
q Vendor: Service VM’s vendor image model.
q Instance: Model to save OpenStack VM state.
Service Slice
Service VM
Service VM
Service VM
Service VM
Service VM
Service VM
Service Slice
Service VM
Service VM
Service VM
Service
Controller
Tenant Service Slice Service
Service Slice
Service VM
Service VM
Service VM
Service Controller
(Synchronizer)
Service Slice
Service VM
Service VM
Service VM
Service with
multiple Slices

15. 15Service Dependency Model
q ServiceDependency: Create dependency relationship between Services.
q Dependency used in calculate Service process order and Network Connectivity.
q ServiceInstanceLink: Create dependency relationship between Tenants.
q Link used in calculate Tenant process order and Synchronizer sync sequence.
Service Slice
Tenant
Service
Controller
Service A
Service Slice
Tenant
Service
Controller
Service B
simplify simplify
Service Instance Link
Service Dependency
Subscriber Provider

16. 16CORDVTN: Virtual Tenant Network architecture
XOS
OVS
VTN @ ONOS-CORD
Service Network
Manager
Dependency
Handler
Instance
Handler
Node Manager
OpenStack
Neutron
CORDVTN enables network functionality
on overlay networking.
1. Build OVS pipeline with 7 tables.
2. Build Service private network flows.
3. Build Service Dependency network
flows.
4. Trasmit Tunneling packets to other
compute nodes.
5. VLAN tagged packets handling.

17. 17CORDVTN: OVS pipeline
Service
Chaining
Table
Access
Table
VNI
Table
Input
Port
Table
Ingress
Table
VLAN
Table
DstIP
Table
SELECT
Group
vSG
Port
Tunnel
Port
Service
Port
0
1 2
3
4
5
6

18. 18CORDVTN: Table introduction
3
2
5
1
0
6
4
SEL
ECT
vSG
Port
Tunnel
Port
Service
Port
• Ingress (Table 0): Check if a packet is VLAN tagged or not.
• Input Port (Table 1): Check packet’s source, and send to corresponding table.
• Access (Table 2): Check source IP to determine if packet is valid.
• Service Chaining (Table 3): Check if source net and destination net in service dependency relationship.
• Destination IP (Table 4): send to VM port by matching IP address.
• VNI (Table 5): VxLAN Network ID Table for Tunnel use.
• VLAN (Table 6): (depressed) Check VLAN tag and send to vSG port.
• SELECT Group: Service Dependency Load Balance use.

19. 19CORDVTN: Service Network Manager
XOS
OVS
VTN @ ONOS-CORD
Service Network
Manager
Dependency
Handler
Instance
Handler
Node Manager
OpenStack
Neutron
Service Network Manager offers RESTful API and is
responsible for update information from XOS.
1. RESTful API for CRUD an network.
• using ML2 driver to interact with Neutron
• update overlay network via Neutron
2. Inform Instance Handler of network state change.
3. Inform Dependency Handler of dependencies info.

20. 20CORDVTN: Node Manager
XOS
OVS
VTN @ ONOS-CORD
Service Network
Manager
Dependency
Handler
Instance
Handler
Node Manager
OpenStack
Neutron
Node Manager is responsible to manage overlay OVS,
namely create and install bridge.
Steps:
1. Connnect to OVSDB.
2. Create a br-int as and set its controller.
3. Create Management, VxLAN, Fabric Ports.
4. Set management IP and Fabric IP to bridge br-int.

21. 21CORDVTN: Instance Handler
XOS
OVS
VTN @ ONOS-CORD
Service Network
Manager
Dependency
Handler
Instance
Handler
Node Manager
OpenStack
Neutron
Instance Handler provides Tenant’s network
Connectivity and maintains Tenant’s port on OVS.
1. Provides several network types.
• Private: private network
• Public: external connectivity network
• Flat: external accessibled network
• Management: accessible from
head/compute for only management usage
2. Detect / Configure port for Tenant on OVS.

22. 22CORDVTN: Dependency Handler
XOS
OVS
VTN @ ONOS-CORD
Service Network
Manager
Dependency
Handler
Instance
Handler
Node Manager
OpenStack
Neutron
Dependency Handler provides network connectivity
from subscriber to provider.
Dependency Handler create flows about service
dependency, flows have following limit:
1. subscriber can access providers’ tenants by
providers’ service gateway.
i.e. subscriber can’t access providers‘ tenants by
IP directly.

23. 23CORDVTN: Network Architecture
Fabric
vHSS Tenant
vMME Tenant
vSPGW-C Tenant
vSPGW-U Tenant
management
hss_net
management
management
management
mme_net
spgwc_net
spgwu_net
public
Compute Node
Data Plane
In OAI M-CORD Network Architecture,
have use 4 different VTN network types.
MANAGEMENT: management usage.
PRIVATE: private network.
FLAT: external accessibled private network.
PUBLIC: external connectivity network.
eNodeB

24. 24
vMME Slice
MME
10.0.6.2/24
eNB
10.0.5.2/24
vHSS Slice
HSS
10.0.7.2/24
vSPGW Slice
SPGW
10.0.8.2/24
SPGW
10.0.8.3/24
MME: 10.0.6.1/24
HSS: 10.0.7.1/24
SPGW: 10.0.8.1/24
Services communicate with each other by Service Gateway IP.
eNB learned SPGW’s IP from MME.
Virtual Service Gateway
Virtualized Gateway
(Done by OVS Group Table)

25. 25
vMME Service
MME
10.0.6.2/24
eNB
10.0.6.100/24
10.0.9.100/24
vHSS Service
HSS
10.0.7.2/24
vSPGW-C Service
SPGW-C
10.0.8.2/24
vSPGW-U Service
SPGW-U
10.0.9.3/24
SPGW-U
10.0.9.2/24
SPGW-U
10.0.9.4/24
M-CORD Services Dependency

26. 26
vMME
10.0.6.2/24
Virtual Gateway
10.0.6.1 / 10.0.7.1
Service Gateway Mac
in NetConf
ONOS CORDVTN
Using OVS group to
Load Balance
vHSS
10.0.7.2/24
Service Gateway Design

27. 27
vMME
(10.0.6.2)
vHSS GW
(10.0.7.1)
vHSS
(10.0.7.2)
Change L2/L3 Header
Dst Addr:
aa:bb:cc:dd:ee:ff
Dst IP: 10.0.7.2
Why is 10.0.7.2 ?
3-ways handshake fail in Service Dependency Design

28. 28
vMME
(10.0.6.2)
vHSS GW
(10.0.7.1)
vHSS
(10.0.7.2) Change L2/L3 Header
Dst Addr:
aa:bb:cc:dd:ee:ff
Dst IP: 10.0.7.2
10.0.7.1 is correct.
Change L3 Header
Src IP: 10.0.7.1
3-ways handshake fail solution

29. 29Traffic from Subscriber to Provider
Subscriber
10.0.6.2/24
Provider
10.0.7.2/24
Provider
10.0.7.3/24
Provider
10.0.7.4/24
10.0.6.1
10.0.7.1
1. ARP Broadcast
Who has 10.0.6.1
2. ARP Reply
10.0.6.1 is at
aa:bb:cc:dd:ee:ff
3. ICMP echo
10.0.6.2 to 10.0.7.1
4. ICMP echo
10.0.6.2 to 10.0.7.2
5. ICMP reply
10.0.7.2 to 10.0.6.2
6. ICMP reply
10.0.7.1 to 10.0.6.2
Patch: https://gerrit.opencord.org/#/c/8626/

30. 30Traffic from Provider to Subscriber
Subscriber
10.0.6.2/24
Provider
10.0.7.2/24
Provider
10.0.7.3/24
Provider
10.0.7.4/24
10.0.6.1
10.0.7.1
1. ICMP echo
10.0.7.2 to 10.0.6.2
4. ICMP reply
10.0.6.2 to 10.0.7.4
2. ICMP echo
10.0.7.1 to 10.0.6.2
3. ICMP reply
10.0.6.2 to 10.0.7.1

31. 31
Table Priority Criteria Treatment
0 0 Table: 1
1 5000 IN_PORT=73, IPV4_SRC=10.0.6.2/32 Table: 2
2 5000 IPV4_SRC=10.0.6.0/24, IPV4_DST=10.0.7.1/32 Group: 0xa85da774
vMME
10.0.6.2/24
Group APP ID Bucket Action
0xa85da774 vtn [ETH_DST: FA:16:3E:3E:3F:5C, IPV4_DST: 10.0.7.2, OUTPUT: 78]
[ETH_DST: FA:16:3E:3E:3F:6B, IPV4_DST: 10.0.7.3, OUTPUT: 79]
vHSS
10.0.7.2/24
Virtual Gateway
10.0.7.1
CORDVTN FlowTable: ICMP echo from subscriber
73 78

32. 32
Table Priority Criteria Treatment
0 0 Table: 1
1 5000 IN_PORT=78, IPV4_SRC=10.0.7.2/32 Table: 2
2 5000 IPV4_SRC=10.0.7.0/24, IPV4_DST=10.0.6.0/24 Table: 4
IPV4_SRC=10.0.7.1
4 5000 IPV4_DST=10.0.6.2/24 ETH_DST:
FA:16:3E:3E:3F:6B
OUTPUT: 73
vMME
10.0.6.2/24
vHSS
10.0.7.2/24
Virtual Gateway
10.0.7.1
CORDVTN FlowTable: ICMP reply from provider
73 78

33. 33Conclusion
q VTN’s design concept
q Categorized flows into different tables in VTN OVS pipeline
q Service Dependencies: Subscriber to Provider
q Considered packet transmission between 2 compute nodes
q Service Dependency concept
q Subscriber can only communicate with Provider by Service Gateway IP
q When provider reply to subscriber, source IP will be updated to Gateway IP

34. Q & A
34
Any Question?
aweimeow
wychen@cord-ambassadors.org