1. Load Balancing 101
HungWei Chiu

2. Who Am I
• Hung-Wei Chiu (hwchiu)
• Member of Technical Staff at
Open Networking Foundation
• https://www.hwchiu.com
• Co-Organizer of SDNDS-TW/CNTUG

3. Outline
• What’s Load-Balancing
• How it works
• Kubernetes

4. Load Balancing
• Efficiently distribute incoming network traffic
across a server pool
• Distribute Algorithm

5. Functionality
• Ensure high availability and reliability
• Only send requests to servers that are online
• Provides the flexibility to add or subtract servers
as demand dictates
• Distributes client requests or network load
efficiently across multiple servers.
https://www.nginx.com/resources/glossary/load-balancing/

6. Distribute Algorithm
• Round Robin
• Header Hash
• Layer 3
• Layer 4
• Layer 7
• Server Status
• Connection number
• …etc

7. OSI / TCP-IP
OSI – TCP/IP
https://techdifferences.com/difference-between-tcp-ip-and-osi-model.html
Layer1
Layer2
Layer3
Layer4
Layer5
Layer6
Layer7

8. Layer 7
https://www.javatpoint.com/computer-network-tcp-ip-model

9. Transaction
Apps
TCP/
UDP
IP
Layer2
Layer1
Apps
TCP/
UDP
IP
Layer2
Layer1
Layer2
Header
Layer2
Data
Layer2
Footer
Layer3
Header
Layer3
Data
Layer4
Header
Layer4
Data
Layer7
Data
01010101010101010101`10111010101010101010111111110100011111111110

10. Client
Client
Client
Client
Server
Server
Server
Magic
Load Balancing

11. True/False ?
• Kubernetes Service (Layer 4)
• ClusterIP
• NodePort
• Kubernetes Ingress (Layer 7)
• Different Implementation
• TCP/UDP/HTTP

12. How Load-Balancing
Works

13. Client
Client
Client
Client
Server
Server
Server
Load
Balancer
Typical Implementation
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
client_ip 1.2.3.4 1.2.3.4 192.168.1.2
NGINX
1.2.3.4 client_ip 192.168.1.2 1.2.3.4
src dest src dest

14. Client
Client
Client
Client
Server
Server
Server
Load
Balancer
Typical Implementation
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
client_ip 1.2.3.4 1.2.3.4 192.168.1.2
NGINX
1.2.3.4 client_ip 192.168.1.2 1.2.3.4
src dest src dest
HTTPS HTTPCertificate

15. How it works
• Type
• Client-Side
• Middleware
• Connection Statue
• Proxy
• Transparency

16. Client-Side
• Client directly sends request to backend server.
• Query all possible addresses of backend
servers
• Apply distributed algorithm
• Send request to backend server
• Someone should provide address list of all
backend server

17. Client
Client
Client
Client
Server
Server
Server
Address list
Provider
Client-Side LB
192.168.1.2
192.168.1.3
192.168.1.4
Maintain list

18. Client
Client
Client
Client
Server
Server
Server
Address list
Provider
Client-Side LB
192.168.1.2
192.168.1.3
192.168.1.4
Query

19. Client
Client
Client
Client
Server
Server
Server
Address list
Provider
Client-Side LB
192.168.1.2
192.168.1.3
192.168.1.4
Reply
192.168.1.{2,3,4}

20. Client
Client
Client
Client
Server
Server
Server
Address list
Provider
Client-Side LB
192.168.1.2
192.168.1.3
192.168.1.4
Send to 192.168.1.3
client_ip 192.168.1.3
192.168.1.3 client_ip

21. Real Case
• gRPC is a layer 7 protocol, multiple gRPC request
rely on the same TCP connection.
• gRPC in Kubernetes environment
• Server type: Headless
• Return a set of PodIPs
• https://medium.com/google-cloud/loadbalancing-
grpc-for-kubernetes-cluster-services-3ba9a8d8fc03

22. Middleware
• Client sends packets to a middleware
• Middleware handles the traffic load-balancing
• Client doesn’t need to know the IP address of
backend servers, only the middleware.
• Different implementation
• Proxy
• Transparency

23. Proxy
• Usually, a demand daemon to handle the
incoming traffic.
• Establish a new network connection.
• One to one
• One to many
• SSL-terminator

24. Client
Client
Client
Client
Server
Server
Server
Load
Balancer
Proxy
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
client_ip 1.2.3.4 1.2.3.4 192.168.1.2
NGINX
1.2.3.4 client_ip 192.168.1.2 1.2.3.4
Different TCP connections

25. Real Case
• gRPC is a layer 7 protocol, multiple gRPC request rely on the
same TCP connection.
• Layer 4 proxy can’t identify gRPC connection.
• All gRPC request will be forward to same backend server.
• Layer 7 proxy if supports gRPC
• One to Many
• Identify gRPC format
• Envoy/LinkerD
https://www.bugsnag.com/blog/envoy

26. Client
Client
Client
Client
Server
Server
Server
Load
Balancer
Proxy
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
gRPC L7

27. Client
Client
Client
Client
Server
Server
Server
Load
Balancer
Proxy
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
gRPC L7

28. Client
Client
Client
Client
Server
Server
Server
Load
Balancer
Proxy
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
gRPC L7

29. Client
Client
Client
Client
Server
Server
Server
Load
Balancer
Proxy
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
gRPC L7

30. Transparency
• Modify packets to make it be routed to backend
server.
• Keep the same connection, no additional one.
• Client still doesn’t know the IP address of
backend servers.

31. Client
Client
Client
Client
Server
Server
Server
Transparency
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
client_ip 1.2.3.4 client_ip 192.168.1.2
1.2.3.4 client_ip 192.168.1.2 client_ip
Same tcp connection

32. Client
Client
Client
Client
Server
Server
Server
Transparency
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
client_ip 1.2.3.4 client_ip 192.168.1.2
1.2.3.4 client_ip 192.168.1.2 client_ip
Same tcp connection

33. Real Case
• gRPC is a layer 7 protocol, multiple gRPC
request rely on the same TCP connection.
• gRPC load-balancing can’t work correctly in this
mode.
• Only single TCP connection.
• TCP is a connection based protocol (3 way
handshake)

34. Client
Client
Client
Client
Server
Server
Server
Transparency
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
client_ip 1.2.3.4 client_ip 192.168.1.2
1.2.3.4 client_ip 192.168.1.2 client_ip
Same tcp connection
Clone packets
TCP needs connection status.

35. Client
Client
Client
Client
Server
Server
Server
Transparency
1.2.3.4
192.168.1.2
192.168.1.3
192.168.1.4
client_ip 1.2.3.4 client_ip 192.168.1.2
1.2.3.4 client_ip 192.168.1.2 client_ip
Same tcp connection
TCP needs connection status.

36. Example
• Kubernetes Service (by default)
• ClusterIP/NodeIP
• Packet are modified by iptables
• Connection persistence are maintained by
kernel conntrack.

37. Node1 Node2 Node3
BackendBackend Backend
clusterIP

38. Node1 Node2 Node3
BackendBackend Backend
clusterIP
Kube-proxy Kube-proxy Kube-proxy
Insert iptables
rules
Insert iptables
rules
Insert iptables
rules

39. Node1 Node2 Node3
BackendBackend Backend
clusterIP
Client
1. Choose the backend server
2. Change the packet destination IP address
3. Forward packet
Iptables..

40. Node1 Node2 Node3
BackendBackend Backend
clusterIP
10.104.8.48
10.244.0.4 10.244.0.5 10.244.0.6

41. Node3
Backend
Client
Iptables.. PacketsPacketsPackets
Match ClusterIP
Choose backend
server
Change Packet IP
ip/protocol
Distributed algo
Action

42. Node3
Backend
Client
Iptables.. PacketsPacketsPackets
Match ClusterIP
Choose backend
server
Change Packet IP
ip/protocol
Distributed algo
Action

44. EP1 EP2 EP3
P < 0.2
P < 0.25
EP1
EP2
EP3
EP4 EP5
P < 0.33
P < 0.5
EP4 EP5
P = 4/5 * 1/4
= 1/5
P = 4/5 * 3/4 * 1/3
= 1/5

45. Question
• How to make the connection consistent?
https://medium.com/@chirag.singla/tcp-3-way-handshake-and-latency-aae5a2a6a68b

46. Conntrack
Linux Kernel helps you

47. Answer
• Only the first packet meets NAT rules
• Kernel modifies the packets for you
• You need the conntrack tool to control it.
• You should choose the destination before you
see the real data packets.

48. Kubernetes service
• People always said kubernetes only supports
layer 4 load-balancing
• Is it possible to provide layer 7 load-balancing in
?
• Without modifying kubernetes source code

49. I think
• For normal TCP implementation, No.
• Should be Yes for customized TCP protocol
• Real zero-RTT TCP.
• How about other protocols ?
• UDP ?

50. DEMO
• Source code
• https://github.com/cloud-native-taiwan/k8s-
course/pull/5/files
• Modify the iptables linux kernel module
• Statistic module
• Parse the UDP payload and try to match