Layer 2 Load Balancing in a Layer 3 World

L3DSRLayer 2 Load Balancing in a Layer 3 World SF Bay AreaLarge Scale Production Engineering Jan 19, 2012 Carl Stanley Load Balancer Architect cstan@yahoo-inc.com

Hardware Load Balancing Hardware Load Balancing works on a LAN type environment (within a colo), as opposed to GSLB which works with DNS. Network configurations are either Inline or Direct Server Return. Inline LB has both inbound and outbound traffic go thru the load balancer. DSR only has inbound traffic go thru load balancer. 2

Inline, L4, L7 Load Balancing Inline Load Balancing is where both inbound and outbound traffic pass through the LB In Layer 4 Load Balancing the initial SYN is passed to the server. Layer 4 Load Balancing can make LB decisions based on info in the SYN packet (source IP, port) or data it has on the servers (connection count). In Layer 7 Load Balancing the clients TCPsession is terminated at the LB; it then opens a separate but sometimes very similar TCP session with the server and passes data in between them. Layer 7 Load Balancing can make LB decisions based on packet data – http url, cookies, uri, headers, etc. 3

L2 Direct Server Return L2 DSR has been around >12 years, almost every LB vendor supports it. Also called 1 arm LB, Direct Routing, SwitchBack, nPath. Client send SYN packet to VIP, which lives on LB. LB figure out what server should handle request, changes destination MAC address to server (rest of packet kept intact, including Dest IP=VIP). Switched network delivers packet to server. Server has VIP IP configured as loopback alias, so it can accept packet. Application on server has to listen to VIP IP address. Server replies directly to client with Source IP=VIP 4

L2 DSR 5

Advantages of DSR Title LB only sees inbound part of connection, so LB only (unit)handles inbound packets. Most traffic at content providers is HTTP, which hassmall inbound bandwidth (requests) and large outboundbandwidth (content). Not limited to outbound bandwidth of LB as with inlineVIPs. LB can do more VIPs (about 8x more) then in-line –typically 100’s of VIPs, 1000s’ of servers, gigs of traffic. Doesn’t have the return-path problem – no need to useSNAT or default route as with inline VIPs. 6

Disadvantage of L2DSR Not able to do L7 inspection of packet for load balancingdecisions Only works in L2 environments (within a singleaggregation layer) Since LB only sees inbound part of the connection theLB doesn’t really know when a connection is closed. Canrun into problems if small set of clients have large numberof connections. Requires host configuration of VIP IP address asloopback alias Port Address Translation (PAT) not possible. Susceptible to SYN attacks 7

Scaling Beyond Aggregation Switches Aggregation Switch – or backplane – is a chassis based10g switch to which multiple rack switches connect to. Within Data Centers there can be multiple AggregationSwitches. As servers are added to VIPs over time these serverseventually will be on separate Aggregation Switches(which have no L2 connectivity). Aggregation Switches can be a Layer 2 design usingspanning tree or Layer 3 design using routing. A Layer 2 DSR VIP can only load balance to serverswhich are on the same Aggregation Switch. 8

Layer 3 Load Balancing Because of these scalability issues there needs to beLoad Balancing that can support hosts that are routerhops away. Traditional way to do load balancing in Layer 3environments is to either use SNAT or Inline. SNAT (Source NAT) – load balancer replaces Client IPwith IP that lives on LB, the server replies to that IPaddress getting packet back to LB. In Inline the Load Balancer is physically between serversand egress – either using VLANs or having all traffic routethrough the Load Balancer. 9

Disadvantages of SNAT, Inline In SNAT the server does not know what the clients IP address is.  The LB can insert a HTTP header with the client’s IP address  Since this is a L7 function it costs CPU. HTTPS (ssl) traffic must be terminated at the LB. This limits the throughput to the SSL capability of the LB  Non-http/https traffic has no way of knowing clients IP address. In Inline all traffic must flow through the Load Balancer  This limits the total traffic – load balanced traffic or otherwise - to bandwidth and switching capability of that LB. 10

Layer 3 DSR Layer 3 DSR is a way of implementing DSR across L3 boundaries. Supported by 3 vendors – A10, Brocade (Foundry), Citrix (Netscaler) Needs kernel modules installed on host, currently the supported OS’s are Linux Rhel 4.x, Rhel 5.x, Rhel 6.x or FreeBSD 6.x or 7.x. Kernel modules are open sourced – at https://github.com/yahoo/l3dsr - maintained by Jan Schaumann (jans@netmeister.org) L3DSR is a Yahoo! invention, created by Dave Temkin, Dave Barrow, and Igor Gashinsky from Network Architecture team. Kernel modules written by Quentin Barnes, John Baldwin, David Discher, Jan Schaumann. 11

Disadvantages of L3DSR As with L2DSR, no L7 load balancing capabilities. Server config requires kernel module, iptable setup, and loopback configuration. Use of DSCP precludes using QOS within colo. In lab our stock server shows .04 ms added latency. Limited OS’s currently supported. Limited Load Balancing vendors support. Susceptible to SYN attacks DSCPVIP mappings must be tracked. 12

L3 DSR – How it works Load Balancer sets the DSCP (differentiated services code point) bits, which are the 6 most significant bits in byte 2 of the IP header and are normally used for QoS, to a value (0-63). The server has a kernel module that when seeing an inbound packet with this value in the DSCP field changes the destination IP of the inbound packet to the corresponding VIP IP(which is also configured as a loopback alias). From this point it acts like a L2 DSR would – the application listens to the VIP IP, and replies directly to the client sourcing the packet with the VIP IP. It discards the DSCP setting in the reply packet. 13

L3 DSR 14

L3 DSR packet flow Description Source IP Dest IP CommentsClient 144.100.10.1 198.18.0.250 (VIP) Going to VIPLoad Balancer 144.100.10.1 74.80.1.1 (Server) LB sets DSCP=4chooses ServerServer iptables/ 144.100.10.1 198.18.0.250 VIP config’d as lo0kernel moduleServer Reply 198.18.0.250 144.100.10.1 15

L3 DSR HealthcheckHealthchecks of real servers need to test if kernel module is in place & working.When the LB healthchecks it will send the following HTTP query: SourceIP: <LB Management IP> DestIP: Client IP Healthcheck URL “GET /status.html” DSCP: 10Server matches DSCP=10, changes packet coming into the server to: SourceIP: <LB Management IP> DestIP: VIP IPAssuming kernel module, loopback alias, and healthcheck URL are in the server sends a reply: SourceIP: VIP IP Dest IP: <LB Management IP> Status code “200 OK”Note that DestIP of query != SourceIP of return packet, so LB cannot use its regular TCP stack. 16

Linux/FreeBSD Server Confighttps://github.com/yahoo/l3dsrYVIPAgent is a tool that is installed on servers behind a L3DSR VIP. It provides system startup services and and a command line tool to configure the host for participation in one or more VIPs according to a flat configuration file.Configuration files are in /usr/local/etc/yvpi/vips/, and look like:[cstan@lab8 ~]$ cat /usr/local/etc/yvip/vips/10.128.8.45ip=10.128.8.45type=l3-dsrdscp=27To start the L3DSR service you run:[cstan@lab8 ~]$ yvipagent startTo stop the L3DSR service you run:[cstan@lab8 ~]$ yvipagent stop 17

Debugging on HostTo see if yvipagent is working, trying pinging the host with the –z option (-q if using Linux), which sets the DSCP byte, if its working you will see a response from the vip IP:cstan@lab1$ ping -c 2 -z 8 10.128.9.30PING 10.128.9.30 (10.128.9.30): 56 data bytes64 bytes from 10.128.8.45: icmp_seq=0 ttl=64 time=0.253 ms64 bytes from 10.128.8.45: icmp_seq=1 ttl=64 time=0.221 ms--- 10.128.9.30 ping statistics ---2 packets transmitted, 2 packets received, 0% packet lossround-trip min/avg/max/stddev = 0.221/0.237/0.253/0.016 msNote that DSCP is the 6 most significant bits of the byte – 10.128.9.30 is actually setup with DSCP=2, but with the –z you need to specify the entire byte, so you use –z 8. 10.128.8.45 is the VIP ip address.To see if modules are loaded on FreeBsd you can usesysctl -a | grep dscpon Linux use[cstan@lab8 ~]$ sudo iptables -t mangle -LChain PREROUTING (policy ACCEPT)target prot opt source destinationDADDR all -- anywhere anywhere DSCP match 0x02 DADDR set 10.128.8.45 18

Implementing L3DSR in Large Scale Networks Need to have source of truth for allocating DSCP values. For each VIP that shares servers a unique DSCP value is required. Automation of yvipagent (or similar) is necessary. Careful of network gear that strips (or listens to by default) QOS/DSCP bits. 19

Future Enhancements to L3DSR 20

L3DSR Port TranslationIn DSR the real server port = VIP port, if you want to do port translation (example; vip listens on port 80, but reals listen to port 8080) you need to have the VIP Inline.In L3 DSR Port Translation the kernel module has to change the port on packet outbound to the VIP port in addition to the inbound DSCP/IP address change.Yahoo’s FreeBSD & Linux Kernel teams are working on kernel module.A10 and Brocade have this in their roadmap in 2012. 21

L3DSR Port TranslationConfig file looks like:ip=198.19.0.151type=l3-dsrdscp=16remap1=80:81remap2=80:82remap3=443:444remap4=443:445Outbound iptable rules look something like:if <src ip>=198.19.0.151 AND <src port>=81 then set <src port>=80if <src ip>=198.19.0.151 AND <src port>=82 then set <src port>=80if <src ip>=198.19.0.151 AND <src port>=444 then set <src port>=443if <src ip>=198.19.0.151 AND <src port>=445 then set <src port>=443 22

L3DSR PAT HealthcheckHealthchecking real servers would need to see if this PAT is working - the destination port that the healthcheck packets go out on would be different from the source port that replies came back on.The load balancer would send the healthcheck request with:<source_ip>=<lb ip><dest_ip>=<real ip><dest_port>=81<dscp>=16and the reply would come back<dest_ip>=<lb ip><source_ip>=<vip ip><source_port>=80<dscp>=0 (cleared) 23

IPv6 L3DSR• L3DSR for IPv6 is implemented in a similar fashion to IPv4 – it uses the DSCP field (bits 4-7 of 1st octet & bits 0-1 in 2nd octet) in the IPv6 header (RFC2474)• Yahoo Linux & FreeBSD kernel teams are developing the kernel modules for FreeBSD7, RHEL5.• Brocade, A10, and Citrix are all implementing IPv6 L3DSR in future code revisions, should be available 2012.• Long term plans (2014) to develop with vendors IPv6 header extension instead of using DSCP field – might include VIP IP, ports translated, other info. 24

Appendix 25

Brocade/Foundry L3DSR configL3DSR is supported on the ADX (12.1d or above) and the si450 with M7/Jetcore only (M6 will not work) running 10.2.01*.server remote-name realserver1.test 10.128.9.31 port http port http max-conn 100000 port http url "GET /status.html"!server virtual vip1.test 10.128.8.44 tos-marking 27 hc-l3-dsr port default disable port http bind http realserver1.test http 26

A10 L3DSR configL3DSR with healthchecking fully supported on 2.2.5 or above.Ability to assign dscp templates at virtual level on 2.6.1 or above.slb template port dscp_port_27dscp 27!slb server realserver1.test 10.128.9.31 port 80 tcp template port dscp_port_27!slb service-group vip1.test:80 tcp health-check status.html-ssl member realserver1.test:80!slb virtual vip1.test 10.128.8.44 port 80 tcp service-group vip1.test:80 no-dest-nat 27

Citrix Netscaler L3DSR configadd lb monitor index.html HTTP -respCode 200 -httpRequest "GET /index.html" - interval 7 -retries 2add server realserver1.test 10.128.9.31add service realserver1.test-80 realserver1.test ANY 80 -usip YES -cip DISABLEDbind lb monitor index.html hostname.yahoo.com-80add lb vserver vip1.test ANY 10.128.8.44 80 -lbmethod ROUNDROBIN -m TOS -tosId 27bind lb vserver vip1.test realserver1.test-80 28

http://coderwall.com	20
https://si0.twimg.com	1
https://twitter.com	1

Layer 2 Load Balancing in a Layer 3 World

by Chris Westin

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 22

Statistics

Layer 2 Load Balancing in a Layer 3 World Presentation Transcript