4. kakao
Andrew’s ‘Without’ Series
Tunneling, without Tunnel
2018 What?
Load Balancing, without Load Balancer
2017 What is Load Balancer?
SDN, without SDN
2015 What is SDN?
6. kakao
eth1
Compute node1
linux bridge
vm
IP:10.10.100.2/32
192.1.1.201
Routing Table
Default GW 192.168.1.1 eth1
Host Route dest 10.10.100.2/32
to 10.10.100.1
connected dest 192.168.100.2
Routing Table
1 10.10.100.2/32 via 192.1.1.201
2 10.10.100.3/32 via 192.168.1.202
3 192.168.100.2/32 via 192.168.1.201, 192.1
68.1.202
192.1.1.202
Switch
Namespace
global name
space
IPTable
DNAT Dest 192.168.100.2 is for
warded to 10.10.100.2
Compute Node Router
Veth pair
Gateway 10.10.100.1
neutron-
linuxbridge-agent
neutron-dhcp-agent
neutron-l3-agent
Host Rout
e
dest 10.10.100.2/32
to 10.10.100.1
New IP 192.168.100.2
connected dest 192.168.100.2
Neutron Floating IP
eth1
Compute node1
linux bridge
vm
IP:10.10.100.3/32
Routing Table
Default GW 192.168.1.1 eth1
Host Route dest 10.10.100.3/32
to 10.10.100.1
connected dest 192.168.100.2
Switch
Namespace
global name
space
IPTable
DNAT Dest 192.168.100.2 is for
warded to 10.10.100.2
Compute Node Router
Veth pair
Gateway 10.10.100.1
neutron-
linuxbridge-agent
neutron-dhcp-agent
neutron-l3-agent
Host Rout
e
dest 10.10.100.3/32
to 10.10.100.1
New IP 192.168.100.2
connected dest 192.168.100.2
Neutron Floating IP
192.1.1.202
7. kakao
•
•
• process
• dhcp agent, ml2 agent, l3 agent, metadata agent and nova compute
• state
• linux bridge connection stage
• routing table list
• iptable
• bgp state
kakao
eth0
Compute node
nova-compute
neutron-
linuxbridge-
agent
neutron-dhcp-
agent
Gateway
10.10.100.1
linux bridge
vm
IP:10.10.100.2/32
Routing Table
1 10.10.100.2/32 via 192.1.1.201
BGP
192.1.1.202 BGP
Virtual Switch block
Process block
vlan.bgp
vlan.0
Virtual Router
Service Route Table
1
192.1.1.201
8. kakao
•
•
• Consul is hashcorp’s product
• Designed for Datacenter level coordination and service discovery
consul
agent
consul server
hostname
•nova process
•neutron process
•routing state
•etc...
process check
state check
fail?
yes
Alrams
-kakaotalk
-URL
12. kakao
It solves a problem
• Tenant network link layer(East-West) separation issue.
It creates problems
• Tennant Network Performance
• Latency due to the geographical separation
• full mesh between the nodes
• Retransmission due to fat(+50bytes) L2 frames
• Jumbo frame is not the right solutions if you try to cover datacenter,
actually you can’t
• Vxlan offloading is also do-able, but needs money
• NAT bottleneck
• e.g. openstack neutron network node bottle neck issue
• DVR can distribute the NAT network but it needs router
13. kakao
It solves a problem
• Tenant network link layer(East-West) separation issue.
It creates problems
• Tennant Network Performance
• Latency due to the geographical separation
• Retransmission due to fat(+50bytes) L2 frames
• Jumbo frame is not the right solutions if you try to cover datacenter,
actually you can’t
• Vxlan offloading is also do-able, but needs money
• NAT bottleneck (North-South)
• e.g. openstack neutron network node bottle neck issue
• DVR can distribute the NAT network but it needs routers
18. kakao
•
• IP routing lookup
• Attaches labels
• Forwards based on label
•
• Use label to route ( This determined by RR)
•
• Removes label
• Packet is delivered using normal routing
Pic. by Qumulus
20. kakao
•
• The important thing is that Doesn’t touch Packet in L2
• no fat packet
• it add label at L3
•
• Just thinks about that I didn’t use overlay network IP, only I add label. So, don’’t
need NAT at all
• Use the legacy IP , So the existing routing and network modeling is just working
fine.