This document summarizes a LINE Developer meetup in Kyoto on load balancing techniques. It discusses LINE's private cloud "Verda" and the load balancing architecture used including L3, L4, and L7 load balancers. It also covers programming load balancers using eBPF and XDP to improve performance by handling traffic directly on the NIC. Load testing is discussed using tools like TRex to generate and analyze traffic.

1. L D
C a M Y e NK I LEC
194 5 28 72
0 9 9 2 598
0 9 / 58 9 9 # #

2. d
a pn / 7
O
N N A l
V Lo ( ekB
2 - /04 L S 2 1/ O
7/1 )/ / 02 / / -2 02 )//- icB BekB
2018-09-27 LINE Developer meetup #45 in Kyoto 2
• LINE Engineer Insights vol.4 OpenStack Private cloud "Verda" (2017-07)
https://engineering.linecorp.com/ja/blog/detail/158
• Verda Cloud Family (2017-09)
https://www.slideshare.net/linecorp/verda-cloud-family
• Introduction to our private cloud Verda at a glance (2018-08)
https://www.slideshare.net/linecorp/introduction-to-our-private-cloud-verda-at-a-glance

3. 1 ( ) 2 A ( B
1 ( B : E
2018-09-27 LINE Developer meetup #45 in Kyoto 3

4. 2018-09-27 LINE Developer meetup #45 in Kyoto 4

5. + ) + B S 7
• + ) D N 6
• 7 E6N
• +2 6I
• S SX
• V 6 12
• B0( 6 LP6
2018-09-27 LINE Developer meetup #45 in Kyoto 5

6. ) ( I
aL P V S
B T
2018-09-27 LINE Developer meetup #45 in Kyoto 6
Real Server (RS)LBClient
VIP RSIPi
• RS
• IP
• TLS (TLS Offload)
• …

7. 2018-09-27 LINE Developer meetup #45 in Kyoto 7
Layer 4 LB (L4LB) Layer 7 LB (L7LB)
TCP TCP
• IP/TCP/UDP
•
• (L7LB )
HTTP(S) HTTP(S)
•
• HTTP
• (L4LB )

8. 2018-09-27 LINE Developer meetup #45 in Kyoto 8
OpenStack Octavia
HAProxy
LVS (IPVS)
LBaaS Frontend LBaaS Backend
LB
API / Web UI API
API / Web UI / CLI

9. - I m
vi cDb df g
eg n l
) A7 1 3 4B ( t a
n +1 1 3 7 ; 3;3 7 l
y LR 2 1ou l
E T R S N
2018-09-27 LINE Developer meetup #45 in Kyoto 9

10. ( )
( ) D R ( 3 3 3 ) 3
? R
( e S g :<fB L ?n
L lia r R S =ct
2018-09-27 LINE Developer meetup #45 in Kyoto 10
RS 1
RS 2
Client 1
Client 2
Source Destination Real Server
Client 1:43210 VIP 1:443 RS 1:443
Client 2:34567 VIP 1:443 RS 2:443
VIP 1
Session Table

11. T
+ B . E 2
3 HR
B T L
S
x E + 2
+ x 3 2
.
1 D C
2018-09-27 LINE Developer meetup #45 in Kyoto 11

12. 1 L + 1 1
1 & 1 4B M F G N , -
2018-09-27 LINE Developer meetup #45 in Kyoto 12
1 + 1 N + 1 Multi-tier
Client
L4LB
(Hardware)
Real Server
: In service
: Not in service
L4LB
(Software)
Real ServerL7LB
(Software)
L3 Switch
(Hardware)
Client
Stateful L3DSR
TCP/HTTP(S) Proxy
Stateless L3DSR
ECMP
[1] http://yunazuno.hatenablog.com/entry/2016/02/29/090001

13. +
? N
+ M 1 :- N
2018-09-27 LINE Developer meetup #45 in Kyoto 13
L3 Switch L4LB L7LB
Whitebox Switch
100G x 32 ports
JPY 2,000,000
x86 Linux Server
25G x 1 ports
JPY 800,000
x86 Linux Server
25G x 1 ports
JPY 800,000

14. B
- 4 41 :
2018-09-27 LINE Developer meetup #45 in Kyoto 14
Equal Cost Multi Path (ECMP)
VIP: 192.0.2.1/32
L4LB-1
Hash(5-tuple)
-> L4LB-[1-3]
L3 Switch
VIP: 192.0.2.1/32
L4LB-2
VIP: 192.0.2.1/32
L4LB-3
eBGP peer
Flow X
Flow Y
5-tuple = (Src IP, Dst IP,
Src Port, Dst Port, Proto)

15. C 2 2
2 2 B 3 :7
2018-09-27 LINE Developer meetup #45 in Kyoto 15
L3 Switch L4LB-1
L7LB-1
L7LB-2
L4LB-2
L7LB-3
L7LB-4
ConsistentHash(Hash(5-tuple))
-> L7LB-[1-4]
Flow X
Flow Y

16. 2018-09-27 LINE Developer meetup #45 in Kyoto 16

17. L4LB Program
) 4
4 B 4 (
2018-09-27 LINE Developer meetup #45 in Kyoto 17
Rx
Tx
Match
Lookup
Rewrite
Table
VIP-Backend
table
Statistics
table
From
L3 Switch
To L7 LB
NIC
1. Match
Dst. IP/Port
2. Lookup
Hash Real Server
3. Rewrite
IP Dst. IP, DSCP

18. ( ( ) D
( ( ( BF L
2018-09-27 LINE Developer meetup #45 in Kyoto
eBPF Map
VIP-Backend
table
Statistics
table
C eBPF
C code
(Data Plane)
Clang/LLVM
eBPF
Bytecode
Python
Control Plane
eBPF JIT
eBPF VM
API
VIP->Backend
lookup
Statistics
KernelUser
eBPF
Verifier
18
BGP daemon
XDP Program From
L3 Switch
To L7LB
Kernel NIC
Advertise VIP
Compiler
1. Lookup VIP
2. Hash(5-tuple)
3. Modify IP header
Maintain map

19. DP
D ? ? D X P
K
2018-09-27 LINE Developer meetup #45 in Kyoto 19
XDP
NICKernelKernelUser
NIC
Driver
EthernetIPv4TCPApp Socket
NIC
Driver
XDP
App
Ether
net
IPv4TCPApp Socket

20. ? 6
I D 1
2018-09-27 LINE Developer meetup #45 in Kyoto 20
Cloudflare
[1] How to drop 10 million packets per second - The Cloudflare Blog https://blog.cloudflare.com/how-to-drop-10-million-packets/

21. AN. P ?
X A . ?
DC Q
AI ? U
2018-09-27 LINE Developer meetup #45 in Kyoto 21

22. ) MN
I A C (P (
(P (
2018-09-27 LINE Developer meetup #45 in Kyoto 22
NIC CPU Memory
PCIe
Interconect
Fast
Slow

23. NIC CPU
&) F UC B
A &) F U P
H C - ) )- &) / - ( C
2018-09-27 LINE Developer meetup #45 in Kyoto 23
Queue 0
Queue 1
Core 0 L2 Cache
Core 1 L2 Cache
L3
Cache
+
Memory

24. 1
& & &
2018-09-27 LINE Developer meetup #45 in Kyoto 24
[1] Cisco's TRex https://trex-tgn.cisco.com/
[2] GitHub - opcm/pcm https://github.com/opcm/pcm
[3] GitHub - cloudflare/ebpf_exporter https://github.com/cloudflare/ebpf_exporter
TRex Server
(Traffic Generator)
L4LB
Generate packets
Collect statistics
• Python
•
• Exporter L4LB NIC
• Grafana
• PCM[2] CPU
• ebpf_exporter[3]

25. .
?A D B ?A
2018-09-27 LINE Developer meetup #45 in Kyoto 25
OpenStack Octavia
HAProxy
LVS (IPVS)
LBaaS Frontend LBaaS Backend
LB
API / Web UI
API
API / Web UI / CLI
オンプレミス環境での典型的なLBサービス構成
(D-Plane)
(C-Plane)
Verda LBaaS
( , BGPd )
800 LoC (in C)14,000 LoC (in Python)

26. :
- 3 DC
2018-09-27 LINE Developer meetup #45 in Kyoto 26
Health Check Agent
on L4LB
Real Server
Real Server
•
VIP
• DSCP
Dst. IP VIP
4
4
4
4
Python Scapy

27. 2018-09-27 LINE Developer meetup #45 in Kyoto 27

28. ! (
E L B
I ( ( (
N ()
!
! N
2018-09-27 LINE Developer meetup #45 in Kyoto 28

29. K F FI
4 5:1/ / 9 6. /. 11 5 5 :
7 4 5:1/ / 9 6. /. 11 5 5 :
7 7. 4 5:12 7 7. / 6 6. /. 11 5 1: 5:11 5:2 . / 1 1 1 91: :1 7
: 1 : 4 5:1/ / 9 6. /. 11 5 5 :
2018-09-27 LINE Developer meetup #45 in Kyoto 29