The document discusses network architectures in OpenStack. It provides diagrams to illustrate the networking components including compute nodes, virtual machines, linux bridges, agents, and routers. MPLS is introduced as a solution to address issues with tenant network separation and performance challenges with other approaches like VxLAN. MPLS uses label switching to encapsulate and forward packets instead of relying on IP routing and overlays, improving east-west traffic performance between tenants.

1. kakao

2. kakao
•
•
•
•
•
•
•
•
•
•
•
Supervise
d,
Korean
edition
Korean
Edition.

3. kakao

4. kakao
Tunneling, without Tunnel
2018 What?
LoadBalancing, without LoadBalancer
2017 What is LoadBalancer?
SDN, without SDN
2015 What is SDN?

5. kakao
kakao
eth0
Compute node
nova-compute
neutron-
linuxbridge-
agent
neutron-dhcp-
agent
Gateway
10.10.100.1
linux bridge
vm
IP:10.10.100.2/32
Routing Table
1 10.10.100.2/32 via 192.1.1.201
BGP
192.1.1.202 BGP
Virtual Switch block
Process block
vlan.bgp
vlan.0
Virtual Router
Service Route Table
1
192.1.1.201

6. kakao
eth1
Compute node1
linux bridge
vm
IP:10.10.100.2/32
192.1.1.201
Routing Table
Default GW 192.168.1.1 eth1
Host Route dest 10.10.100.2/32
to 10.10.100.1
connected dest 192.168.100.2
Routing Table
1 10.10.100.2/32 via 192.1.1.201
2 10.10.100.3/32 via 192.168.1.202
3 192.168.100.2/32 via 192.168.1.201, 192.1
68.1.202
192.1.1.202
Switch
Namespace
global name
space
IPTable
DNAT Dest 192.168.100.2 is for
warded to 10.10.100.2
Compute Node Router
Veth pair
Gateway 10.10.100.1
neutron-
linuxbridge-agent
neutron-dhcp-agent
neutron-l3-agent
Host Rout
e
dest 10.10.100.2/32
to 10.10.100.1
New IP 192.168.100.2
connected dest 192.168.100.2
Neutron Floating IP
eth1
Compute node1
linux bridge
vm
IP:10.10.100.3/32
Routing Table
Default GW 192.168.1.1 eth1
Host Route dest 10.10.100.3/32
to 10.10.100.1
connected dest 192.168.100.2
Switch
Namespace
global name
space
IPTable
DNAT Dest 192.168.100.2 is for
warded to 10.10.100.2
Compute Node Router
Veth pair
Gateway 10.10.100.1
neutron-
linuxbridge-agent
neutron-dhcp-agent
neutron-l3-agent
Host Rout
e
dest 10.10.100.3/32
to 10.10.100.1
New IP 192.168.100.2
connected dest 192.168.100.2
Neutron Floating IP
192.1.1.202

7. kakao
•
•
• process
• dhcp agent, ml2 agent, l3 agent, metadata agent and nova compute
• state
• linux bridge connection stage
• routing table list
• iptable
• bgp state
kakao
eth0
Compute node
nova-compute
neutron-
linuxbridge-
agent
neutron-dhcp-
agent
Gateway
10.10.100.1
linux bridge
vm
IP:10.10.100.2/32
Routing Table
1 10.10.100.2/32 via 192.1.1.201
BGP
192.1.1.202 BGP
Virtual Switch block
Process block
vlan.bgp
vlan.0
Virtual Router
Service Route Table
1
192.1.1.201

8. kakao
•
•
• Consul is hashcorp’s product
• Designed for Datacenter level coordination and service discovery
consul
agent
consul server
hostname
•nova process
•neutron process
•routing state
•etc...
process check
state check
fail?
yes
Alrams
-kakaotalk
-URL

9. kakao

10. kakao
broadcast
domain2
•
Application
TCP
IPv4
ethernet
driver
broadcast domain
ARP Table
SRC IP mac eth0
Router
IP
mac eth0
Application
TCP
IPv4
ethernet
driver
ARP Table
dest IP mac eth0
Router
IP
mac eth0
broadcast termination
A.K.A Router
client destination

11. kakao
Application
or
VM
TCP
IPv4
ethernet
driver
broadcast domain
ARP Table
SRC IP mac eth0
Router
IP
mac eth0
Application
or
VM
TCP
IPv4
ethernet
driver
ARP Table
dest IP mac eth0
Router
IP
mac eth0
tun
nel
broadcast domaintun
nel

12. kakao
It solves a problem
• Tenant network link layer(East-West) separation issue.
It creates problems
• Tennant Network Performance
• Latency due to the geographical separation
• full mesh between the nodes
• Retransmission due to fat(+50bytes) L2 frames
• Jumbo frame is not the right solutions if you try to cover datacenter,
actually you can’t
• Vxlan offloading is also do-able, but needs money
• NAT bottleneck
• e.g. openstack neutron network node bottle neck issue
• DVR can distribute the NAT network but it needs router

13. kakao
It solves a problem
• Tenant network link layer(East-West) separation issue.
It creates problems
• Tennant Network Performance
• Latency due to the geographical separation
• Retransmission due to fat(+50bytes) L2 frames
• Jumbo frame is not the right solutions if you try to cover datacenter,
actually you can’t
• Vxlan offloading is also do-able, but needs money
• NAT bottleneck (North-South)
• e.g. openstack neutron network node bottle neck issue
• DVR can distribute the NAT network but it needs routers

14. kakao
•
•
•

15. kakao
•

16. kakao
•
•
•
•

17. kakao
•
•
•
•
•
comments by Qumulus

18. kakao
•
• IP routing lookup
• Attaches labels
• Forwards based on label
•
• Use label to route ( This determined by RR)
•
• Removes label
• Packet is delivered using normal routing
Pic. by Qumulus

19. kakao
•
•
•
à
kakao
eth0
Compute node
nova-compute
neutron-
linuxbridge-
agent
neutron-dhcp-
agent
Gateway
10.10.100.1
linux bridge
vm
IP:10.10.100.2/32
Routing Table
1 nexthop as to 200 via inet 192.1.1.201
BGP
192.1.1.202 BGP
Virtual Switch block
Process block
vlan.bgp
vlan.0
Virtual Router
Label Route Table
1
10.10.100.2/32 encap mpls 200 via 192.1.201
192.1.1.201
Routing Table
1 10.10.100.2/32 via 192.1.1.201
Add
mpls
info

20. kakao
•
• The important thing is that Doesn’t touch Packet in L2
• no fat packet
• it add label at L3
•
• Just thinks about that I didn’t use overlay network IP, only I add label. So, don’’t
need NAT at all
• Use the legacy IP , So the existing routing and network modeling is just working
fine.

21. kakao
• ßà
0
5
10
15
20
25
router to vm
Network Performance
L3 routing L3 routing with MPLS Tunnel

22. kakao

23. kakao