Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Load Balancing 101

179 views

Published on

Introduce the basic concept of load-balancing, common implementations of load-balancing and the detail fo kubernetes service. In the last, demonstrate how to modify the linux iptable kernel module to fulfill the layer-7 load-balcning for kubernetes

Published in: Software
  • Be the first to comment

Load Balancing 101

  1. 1. Load Balancing 101 HungWei Chiu
  2. 2. Who Am I • Hung-Wei Chiu (hwchiu) • Member of Technical Staff at 
 Open Networking Foundation • https://www.hwchiu.com • Co-Organizer of SDNDS-TW/CNTUG
  3. 3. Outline • What’s Load-Balancing • How it works • Kubernetes
  4. 4. Load Balancing • Efficiently distribute incoming network traffic across a server pool • Distribute Algorithm
  5. 5. Functionality • Ensure high availability and reliability • Only send requests to servers that are online • Provides the flexibility to add or subtract servers as demand dictates • Distributes client requests or network load efficiently across multiple servers. https://www.nginx.com/resources/glossary/load-balancing/
  6. 6. Distribute Algorithm • Round Robin • Header Hash • Layer 3 • Layer 4 • Layer 7 • Server Status • Connection number • …etc
  7. 7. OSI / TCP-IP OSI – TCP/IP https://techdifferences.com/difference-between-tcp-ip-and-osi-model.html Layer1 Layer2 Layer3 Layer4 Layer5 Layer6 Layer7
  8. 8. Layer 7 https://www.javatpoint.com/computer-network-tcp-ip-model
  9. 9. Transaction Apps TCP/ UDP IP Layer2 Layer1 Apps TCP/ UDP IP Layer2 Layer1 Layer2 Header Layer2 Data Layer2 Footer Layer3 Header Layer3 Data Layer4 Header Layer4 Data Layer7 Data 01010101010101010101`10111010101010101010111111110100011111111110
  10. 10. Client Client Client Client Server Server Server Magic Load Balancing
  11. 11. True/False ? • Kubernetes Service (Layer 4) • ClusterIP • NodePort • Kubernetes Ingress (Layer 7) • Different Implementation • TCP/UDP/HTTP
  12. 12. How Load-Balancing Works
  13. 13. Client Client Client Client Server Server Server Load Balancer Typical Implementation 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 client_ip 1.2.3.4 1.2.3.4 192.168.1.2 NGINX 1.2.3.4 client_ip 192.168.1.2 1.2.3.4 src dest src dest
  14. 14. Client Client Client Client Server Server Server Load Balancer Typical Implementation 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 client_ip 1.2.3.4 1.2.3.4 192.168.1.2 NGINX 1.2.3.4 client_ip 192.168.1.2 1.2.3.4 src dest src dest HTTPS HTTPCertificate
  15. 15. How it works • Type • Client-Side • Middleware • Connection Statue • Proxy • Transparency
  16. 16. Client-Side • Client directly sends request to backend server. • Query all possible addresses of backend servers • Apply distributed algorithm • Send request to backend server • Someone should provide address list of all backend server
  17. 17. Client Client Client Client Server Server Server Address list Provider Client-Side LB 192.168.1.2 192.168.1.3 192.168.1.4 Maintain list
  18. 18. Client Client Client Client Server Server Server Address list Provider Client-Side LB 192.168.1.2 192.168.1.3 192.168.1.4 Query
  19. 19. Client Client Client Client Server Server Server Address list Provider Client-Side LB 192.168.1.2 192.168.1.3 192.168.1.4 Reply
 192.168.1.{2,3,4}
  20. 20. Client Client Client Client Server Server Server Address list Provider Client-Side LB 192.168.1.2 192.168.1.3 192.168.1.4 Send to 192.168.1.3 client_ip 192.168.1.3 192.168.1.3 client_ip
  21. 21. Real Case • gRPC is a layer 7 protocol, multiple gRPC request rely on the same TCP connection. • gRPC in Kubernetes environment • Server type: Headless • Return a set of PodIPs • https://medium.com/google-cloud/loadbalancing- grpc-for-kubernetes-cluster-services-3ba9a8d8fc03
  22. 22. Middleware • Client sends packets to a middleware • Middleware handles the traffic load-balancing • Client doesn’t need to know the IP address of backend servers, only the middleware. • Different implementation • Proxy • Transparency
  23. 23. Proxy • Usually, a demand daemon to handle the incoming traffic. • Establish a new network connection. • One to one • One to many • SSL-terminator
  24. 24. Client Client Client Client Server Server Server Load Balancer Proxy 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 client_ip 1.2.3.4 1.2.3.4 192.168.1.2 NGINX 1.2.3.4 client_ip 192.168.1.2 1.2.3.4 Different TCP connections
  25. 25. Real Case • gRPC is a layer 7 protocol, multiple gRPC request rely on the same TCP connection. • Layer 4 proxy can’t identify gRPC connection. • All gRPC request will be forward to same backend server. • Layer 7 proxy if supports gRPC • One to Many • Identify gRPC format • Envoy/LinkerD https://www.bugsnag.com/blog/envoy
  26. 26. Client Client Client Client Server Server Server Load Balancer Proxy 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 gRPC L7
  27. 27. Client Client Client Client Server Server Server Load Balancer Proxy 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 gRPC L7
  28. 28. Client Client Client Client Server Server Server Load Balancer Proxy 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 gRPC L7
  29. 29. Client Client Client Client Server Server Server Load Balancer Proxy 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 gRPC L7
  30. 30. Transparency • Modify packets to make it be routed to backend server. • Keep the same connection, no additional one. • Client still doesn’t know the IP address of backend servers.
  31. 31. Client Client Client Client Server Server Server Transparency 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 client_ip 1.2.3.4 client_ip 192.168.1.2 1.2.3.4 client_ip 192.168.1.2 client_ip Same tcp connection
  32. 32. Client Client Client Client Server Server Server Transparency 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 client_ip 1.2.3.4 client_ip 192.168.1.2 1.2.3.4 client_ip 192.168.1.2 client_ip Same tcp connection
  33. 33. Real Case • gRPC is a layer 7 protocol, multiple gRPC request rely on the same TCP connection. • gRPC load-balancing can’t work correctly in this mode. • Only single TCP connection. • TCP is a connection based protocol (3 way handshake)
  34. 34. Client Client Client Client Server Server Server Transparency 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 client_ip 1.2.3.4 client_ip 192.168.1.2 1.2.3.4 client_ip 192.168.1.2 client_ip Same tcp connection Clone packets TCP needs connection status.
  35. 35. Client Client Client Client Server Server Server Transparency 1.2.3.4 192.168.1.2 192.168.1.3 192.168.1.4 client_ip 1.2.3.4 client_ip 192.168.1.2 1.2.3.4 client_ip 192.168.1.2 client_ip Same tcp connection TCP needs connection status.
  36. 36. Example • Kubernetes Service (by default) • ClusterIP/NodeIP • Packet are modified by iptables • Connection persistence are maintained by kernel conntrack.
  37. 37. Node1 Node2 Node3 BackendBackend Backend clusterIP
  38. 38. Node1 Node2 Node3 BackendBackend Backend clusterIP Kube-proxy Kube-proxy Kube-proxy Insert iptables rules Insert iptables rules Insert iptables rules
  39. 39. Node1 Node2 Node3 BackendBackend Backend clusterIP Client 1. Choose the backend server 2. Change the packet destination IP address 3. Forward packet Iptables..
  40. 40. Node1 Node2 Node3 BackendBackend Backend clusterIP 10.104.8.48 10.244.0.4 10.244.0.5 10.244.0.6
  41. 41. Node3 Backend Client Iptables.. PacketsPacketsPackets Match ClusterIP Choose backend server Change Packet IP ip/protocol Distributed algo Action
  42. 42. Node3 Backend Client Iptables.. PacketsPacketsPackets Match ClusterIP Choose backend server Change Packet IP ip/protocol Distributed algo Action
  43. 43. EP1 EP2 EP3 P < 0.2 P < 0.25 EP1 EP2 EP3 EP4 EP5 P < 0.33 P < 0.5 EP4 EP5 P = 4/5 * 1/4 = 1/5 P = 4/5 * 3/4 * 1/3 = 1/5
  44. 44. Question • How to make the connection consistent? https://medium.com/@chirag.singla/tcp-3-way-handshake-and-latency-aae5a2a6a68b
  45. 45. Conntrack Linux Kernel helps you
  46. 46. Answer • Only the first packet meets NAT rules • Kernel modifies the packets for you • You need the conntrack tool to control it. • You should choose the destination before you see the real data packets.
  47. 47. Kubernetes service • People always said kubernetes only supports layer 4 load-balancing • Is it possible to provide layer 7 load-balancing in ? • Without modifying kubernetes source code
  48. 48. I think • For normal TCP implementation, No. • Should be Yes for customized TCP protocol • Real zero-RTT TCP. • How about other protocols ? • UDP ?
  49. 49. DEMO • Source code • https://github.com/cloud-native-taiwan/k8s- course/pull/5/files • Modify the iptables linux kernel module • Statistic module • Parse the UDP payload and try to match

×