Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주

7,960 views

Published on

제3회 네이버 오픈소스 세미나
2018.09.04

Published in: Engineering
  • Dating for everyone is here: ♥♥♥ http://bit.ly/2ZDZFYj ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ❤❤❤ http://bit.ly/2ZDZFYj ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주

  1. 1. Maglev Hashing Scheduler in IPVS, Linux Kernel Inju Song Software Engineer, Container Platform Team NAVER
  2. 2. CONTENTS Load balancing problems in Container Cluster Maglev Consistent Hashing IPVS Maglev Hashing Scheduler Build A Efficient and Reliable Load Balancer Q & A
  3. 3. Load balancing Containers
  4. 4. Internet L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4 Load Balancing
  5. 5. Container Host Container Host Container Host Container Host Container Cluster There are many containers...
  6. 6. Load balancing Problems in Container Cluster
  7. 7. Static Routing L4-LB L4-LB L4-LB L4-LB
  8. 8. Container Host Container Host Container Host Container Host Container Cluster Added / Removed Frequently
  9. 9. Static Routing Problem Container Host(Past) Container Host(Current) L4-LB L4-LB L4-LB L4-LB
  10. 10. Container Cluster N Too many containers . . . . . . Container Cluster 5 Container Cluster 4 Container Cluster 3 Container Cluster 2 Container Cluster 1
  11. 11. Guarantee Load Balancing L4-LB Container Host Container Host Container Host Container Host L4-LB
  12. 12. Internet L4-LB L4-LB L4-LB Load Balancer Down L4-LB L4-LB L4-LB L4-LB
  13. 13. Load Balancer Down Container Host Container Host Container Host Container Host L4-LB
  14. 14. Efficient and Reliable Load Balancing
  15. 15. Large Traffic Load Balancing
  16. 16. Large Traffic Balancing L4-LB Container Host Container Host Container Host Container Host L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB
  17. 17. BGP/ECMP L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB 1.1.1.1 IP Advertise
  18. 18. Single Path L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB
  19. 19. ECMP L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB Equal Cost Multi Path
  20. 20. Hash Based Routing L4-LB L4-LB 1.1.1.1 L4-LB L4-LB 1.1.1.2 PATH= Hashing( IP.Src, IP.Dst, IP.Protocol, Port.Src, Port.Dst ) Mod N
  21. 21. Efficient and Reliable Load Balancing BGP/ ECMP
  22. 22. Internet L4-LB L4-LB L4-LB If a load balancer down L4-LB L4-LB L4-LB L4-LB
  23. 23. Hashing Disruption 192.68.0.1 Hashing(IP) % 2 Container[0] Container[1] 192.68.0.2 192.68.0.3 192.68.0.4 Hashing(IP) % 2 Hashing(IP) % 2 Hashing(IP) % 2
  24. 24. Hashing Disruption 192.68.0.1 Hashing(IP) % 4 192.68.0.2 192.68.0.3 192.68.0.4 Hashing(IP) % 4 Hashing(IP) % 4 Hashing(IP) % 4 Container[0] Container[1] Container[2] Container[3]
  25. 25. ECMP is A Good Load Balancing, but.. Not Reliable Not scalable High Availability
  26. 26. Consistent Hashing for High Availability
  27. 27. Consistent Hash Ring 192.68 .0.2 Container[0] 192.68 .0.3 Container[2] 192.68 .0.4 Container[0] Container[3] Container[2] 192.68 .0.1 Container[1] Container[3] Container[1] Guarantee to remap K/n only
  28. 28. Load Balancing 192.68 .0.2 Container[0] 192.68 .0.3 Container[2] 192.68 .0.4 Container[0] Container[3] Container[2] 192.68 .0.1 Container[1] Container[3] Container[1] Efficient Load Balancing
  29. 29. Google’s Network Load Balancer Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf
  30. 30. Google’s Maglev Features Consistent Hashing Backend Selection Packet Processing Packet Forwarding Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf
  31. 31. Permutation B0 B1 B2 3 0 3 0 2 4 4 4 5 1 6 6 5 1 0 2 3 1 6 5 2 Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Backend = 3 Table Size = 7 Permutate Hashing Permutationtable ( 3x 7 ) Preference list of Backends Lookup_table ( size =7,empty )
  32. 32. Population Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Population Permutationtable ( 3x 7 ) B1 B0 B1 B0 B2 B2 B0 Lookup_table ( size =7 ) B0 B1 B2 3 0 3 0 2 4 4 4 5 1 6 6 5 1 0 2 3 1 6 5 2 [0] [1] [2] [3] [4] [5] [6] Assign backends by preference list
  33. 33. Load Balancing Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf B1 B0 B1 B0 B2 B2 B0 each backend will receive an almost equal numberof connections. [0] [1] [2] [3] [4] [5] [6] L4-LB L4-LB B0 =3 * connection B1 =2 * connection B2 =2 * connection
  34. 34. Minimal disruption Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Population Permutationtable ( 2x 7 ) B0 B2 3 3 0 4 4 5 1 6 5 0 2 1 6 2 Assign backends by preference list Before B1 B0 B1 B0 B2 B2 B2 B1 is Removed Re-Permutate Hashing Lookup_table ( size =7 ) After B1 -> B0 B0 B1 -> B0 B0 B2 B2 B2 Same Value By hashing Lookup_table ( size =7 )
  35. 35. Hard to implement Google’s Maglev Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Backend SelectionPacket ProcessingPacket Forwarding
  36. 36. IPVS Maglev Hashing Scheduler
  37. 37. Internet L4-LB L4-LB L4-LB L4-LB Network Software L4 Load Balancer
  38. 38. Using Netfilter in Linux Kernel Nftables – Wikipedia| https://en.wikipedia.org/wiki/Nftables
  39. 39. Replace Maglev With IPVS Consistent Hashing Backend Selection: FUNCTION Packet Processing: NETFILTER Packet Forwarding: NAT, DR, IP Tunneling
  40. 40. Pluggable Scheduler Module Weighted Round Robin Round Robin Source Hashing least connection
  41. 41. Module Compile Makefile Maglev Hashing Scheduling Module
  42. 42. Maglev Scheduling Module Maglev Hashing Scheduling Round Robin Source Hashing least connection Weighted Round Robin
  43. 43. Maglev Hashing Table Size Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf ... Lookup_table Size=251 ... Lookup_table Size=131071 Disruption % VS Memory usage
  44. 44. Optimize your Memory Usage Kernel Configuration -> Networking support -> Networking options -> Netfilter -> IPVS -> Maglev hashing table size IPVS MaglevHashing SchedulerKConfig | https://github.com/torvalds/linux/blob/master/net/netfilter/ipvs/Kconfig#L290
  45. 45. L4-LB L4-LB L4-LB L4-LB Canary Release Strategy New Service Old Service 5% 95%
  46. 46. Weight Based Balancing Container Host Container Host Container Host Container Host L4-LB W: 10 W: 40 W: 80
  47. 47. Destination Fallback Container Host Container Host Container Host Container Host L4-LB W: 10 W: 40 -> 0 W: 80
  48. 48. 5-Tuples Hash 192.68.0.2 IPVS IPVS MH IPVS IPVS MH IPVS IPVS MH Container[0] Container[1] Container[2] Container[3] IPVS MH is no need to track connection table for forwarding packets to same destination.
  49. 49. High Availability
  50. 50. Internet L4-LB L4-LB L4-LB A load balancer down L4-LB L4-LB L4-LB L4-LB
  51. 51. ECMP Hashing Disruption L4-LB L4-LB 1.1.1.2 L4-LB L4-LB 1.1.1.3 PATH= Hashing() Mod ( N - 1 ) L4-LB L4-LB 1.1.1.4 L4-LB L4-LB 1.1.1.1
  52. 52. TCP Connection Loss 192.68.0.2 Container[0] Container[1] Container[2] Container[3] L4-LB 1. Connection Established With Container[2] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 4. Container[0] has no idea about connection with container[2] 5. Container[0] Send RST. Connection Closed
  53. 53. TCP Connection Loss 2 192.68.0.3 Container[0] Container[1] Container[2] Container[3] L4-LB L4-LB 1. Connection Established With Container[3] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 although the serving LB is alive. 4. Container[1] has no idea about connection with container[3] 5. Container[1] Send RST. Connection Closed L4-LB
  54. 54. Internet L4-LB L4-LB L4-LB High Availability L4-LB L4-LB L4-LB L4-LB
  55. 55. High Availability Container Host Container Host Container Host Container Host L4-LB
  56. 56. Build A Efficient and Reliable Load Balancer
  57. 57. Efficient and Reliable Load Balancer IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
  58. 58. No Loss TCP Connection 192.68.0.2 Container[0] Container[1] Container[2] Container[3] L4-LB IPVS 1. Connection Established With Container[2] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 4. IPVS MH can forward the packet to same destination by hashing. 5. Continue the Established connection. IPVS MH
  59. 59. No Loss TCP Connection 2 192.68.0.3 Container[0] Container[1] Container[2] Container[3] L4-LB 1. Connection Established With Container[3] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 although the serving LB is alive. 5. Continue the Established connection. IPVS IPVS IPVS MH 4. IPVS MH can forward the packet to same destination by hashing With no connection info.
  60. 60. At least 1 Load Balancer up, IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
  61. 61. Can Hash Consistently With No Connection Loss IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
  62. 62. Efficient and Reliable Load Balancing BGP/ ECMP IPVS Maglev Hashing Scheduler
  63. 63. Required Linux Kernel >= 4.18 See Also: https://github.com/torvalds/linux/blob/master/net/netfilter/ipvs Choose M in Kernel menuconfig to use IP_VS_MH
  64. 64. Required Kernel Parameters echo 1 > /proc/sys/net/ipv4/vs/sloppy_tcp See Also: https://github.com/torvalds/linux/blob/master/Documentation/networking/ipvs-sysctl.txt echo 2 > /proc/sys/net/ipv4/vs/conn_reuse_mode
  65. 65. Q & A

×