This document discusses various load balancing techniques. It begins with an introduction to load balancing and outlines strategies from using multiple web servers with a software load balancer to scaling load balancers using DNS. Lower level techniques like n-tuple hashing and anycast routing are covered. Optimizing for locality with techniques like Geo DNS and BGP routing are also summarized. Emerging techniques involving service meshes and direct container return are briefly outlined. Finally, several Go projects for implementing various load balancing functions are listed.

1. SHANE HANSENSHANE HANSEN
Created: 2018-01-02 Tue 18:28

2. TABLE OF CONTENTSTABLE OF CONTENTS
1. Load balancing
1.1. Who Am I?
1.2. Why talk about load balancing:
1.3. What is load balancing?
1.4. Caveats
2. Load balancing by construction
2.1. One webserver
2.2. Patterns for N webservers
2.2.1. Web server selection
2.2.2. Health checks
2.2.3. Anatomy of HTTP check
2.2.4. N web server architecture review:
2.3. Scaling load balancers

3. 2.3.1. Exploring DNS load balancing
2.3.2. DNS load balancing
3. Lower level strategies
3.1. Sub-TCP load balancing
3.2. n-tuple hashing
3.3. 5-tuple hashing bene ts
3.4. Equal Cost Multi Path
3.5. Direct Server Return
4. Optimizing for locality/performance
4.1. Simpli ed model of BGP
4.2. Anycast
4.3. Using BGP for anycast load balancing
4.4. Geo DNS
4.5. Put it all together
5. Next-gen techniques at walmartlabs edge
5.1. Direct Container Return
5.2. Service Mesh

4. 5.3. Beyond service mesh
6. Projects using Go to do all the load balancing
things

5. 11 LOAD BALANCINGLOAD BALANCING
A survey of techniques for scaling busyish websites for non-
experts.

6. 1.11.1 WHO AM I?WHO AM I?
Shane Hansen
(@shanemhansen)
Walmartlabs Edge Engineering
Performance connoisseur

7. 1.21.2 WHY TALK ABOUT LOAD BALANCING:WHY TALK ABOUT LOAD BALANCING:
Engineers need to understand options to make good
choices.
Load balancing is rapidly changing and adapting to evolving
protocols
It's interesting

8. 1.31.3 WHAT IS LOAD BALANCING?WHAT IS LOAD BALANCING?
Distributing requests over multiple endpoints to increase
the reliability or performance of a service.
Inline load balancers lter and transform traf c, so they
often quickly grow features around
observability
routing
OOB load balancers either make a decision and leave the
request ow

9. 1.41.4 CAVEATSCAVEATS
Some companies do these things
Nobody would learn anything if I just said: "use dyn" "use
cloud are" or "use elb"
This is utah go, not utah people-who-write-checks-to-
vendors
Many links to go libraries will be left until the end

10. 22 LOAD BALANCING BY CONSTRUCTIONLOAD BALANCING BY CONSTRUCTION
Making a website scale, step by step.

11. 2.12.1 ONE WEBSERVERONE WEBSERVER
Let's say one day that you build a website. What if it starts
making money or getting popular?

12. 2.22.2 PATTERNS FOR N WEBSERVERSPATTERNS FOR N WEBSERVERS

13. 2.2.12.2.1 WEB SERVER SELECTIONWEB SERVER SELECTION
If your web app is truly stateless, round robin is a solid choice
that's hard to mess up.
If it's not stateless, make it stateless.

14. 2.2.22.2.2 HEALTH CHECKSHEALTH CHECKS
Two major types: inline and oob
oob: app speci c health endpoint hit on an interval
Inline: utilize observed info from real requests to select
backends
Bene ts and drawbacks to both.

15. 2.2.32.2.3 ANATOMY OF HTTP CHECKANATOMY OF HTTP CHECK
Typically HTTP request to endpoint with 2xx status code
Extended validation of body supported
Endpoint can return a health "number" for advanced
algos
Timeouts, con gured retry policies

16. 2.2.42.2.4 N WEB SERVER ARCHITECTURE REVIEW:N WEB SERVER ARCHITECTURE REVIEW:
Pros:
Rolling app upgrades possible
Capacity can be easily
increased
Cons:
Limited by performance of software load balancer. ballpark
$X0,000 requests/s
Your site is still bottlenecked by uptime of a single vm.

17. 2.32.3 SCALING LOAD BALANCERSSCALING LOAD BALANCERS
What to do when your company is relying on a single nginx
instance.

18. 2.3.12.3.1 EXPLORING DNS LOAD BALANCINGEXPLORING DNS LOAD BALANCING
Pros:
DNS infrastructure is quite scalable
Possible to scale out and upgrade load
balancers
Cons:
DNS resolvers are quirky, hard to guarantee
distribution
Broken software with bad caches
Quirky protocol: hard to do priorities, weighting, etc.

19. 2.3.22.3.2 DNS LOAD BALANCINGDNS LOAD BALANCING

20. 33 LOWER LEVEL STRATEGIESLOWER LEVEL STRATEGIES

21. 3.13.1 SUB-TCP LOAD BALANCINGSUB-TCP LOAD BALANCING
Strategies
Can be implemented in hardware
Can be implemented in kernel
Order of magnitude more throughput in appliances that do
so
Can be implemented by pushing state/routing to the edge

22. 3.23.2 N-TUPLE HASHINGN-TUPLE HASHING

23. 3.33.3 5-TUPLE HASHING BENEFITS5-TUPLE HASHING BENEFITS
Extremely fast, can be implemented in hardware
persistence: ow always goes to same node
Avoids hot-spots due to "natted university
effect"

24. 3.43.4 EQUAL COST MULTI PATHEQUAL COST MULTI PATH
Implementation of n-tuple hashing
Provide multiple routes to
destination.
Deterministic route selection

25. 3.53.5 DIRECT SERVER RETURNDIRECT SERVER RETURN
Optimize by only using lb to select backend, not to
proxy.

26. 44 OPTIMIZING FOR LOCALITY/PERFORMANCEOPTIMIZING FOR LOCALITY/PERFORMANCE
Examples so far mostly spray traf c
everywhere
How to build truly global systems

27. 4.14.1 SIMPLIFIED MODEL OF BGPSIMPLIFIED MODEL OF BGP
Border Gateway Protocol
Allows autonomous systems to advertise ip
space

28. 4.24.2 ANYCASTANYCAST
Allow multiple pops to advertise same ip space
Routers choose "best route" where best is a function
of:
network hops
whether or not packet will leave ISP's network
stuff

29. 4.34.3 USING BGP FOR ANYCAST LOAD BALANCINGUSING BGP FOR ANYCAST LOAD BALANCING

30. 4.44.4 GEO DNSGEO DNS
Attempt to direct clients to closest physical server
May not follow network topology
Relies on datasets of questionable accuracy
First hop latency unless anycasting DNS
Bene t: DNS is safest proto to anycast
Good idea if you want unicast TCP for long lived
connections

31. 4.54.5 PUT IT ALL TOGETHERPUT IT ALL TOGETHER
Example setup for multi-pop anycast

32. 55 NEXT-GEN TECHNIQUES AT WALMARTLABSNEXT-GEN TECHNIQUES AT WALMARTLABS
EDGEEDGE

33. 5.15.1 DIRECT CONTAINER RETURNDIRECT CONTAINER RETURN
zero copy resource passing (technically one packet is
copied)
Approach used internally at Walmartlabs/cloud are (based
on gophercon talk)
HTTP/1.1 and TLS/SNI
Assumes shared kernel

34. 5.25.2 SERVICE MESHSERVICE MESH
Utilize Edge capabilities internally via
sidecar
service discovery
yada yada

35. 5.35.3 BEYOND SERVICE MESHBEYOND SERVICE MESH
What comes next?
Protocol optimizations: upgrade to http2? QUIC?
Eliminate moving parts
Turn logical microservices into functions at runtime where
possible
Use REST concepts to facilitate pervasive caching.
Make networked services the fallback, calling local
functions is the optimization.

36. 66 PROJECTS USING GO TO DO ALL THE LOADPROJECTS USING GO TO DO ALL THE LOAD
BALANCING THINGSBALANCING THINGS
Seesaw (LVS):
metallb (BGP):
BGP:
DNS:
GoBPF:
cilium
OSS repo for fd passing load balancer: TBD
https://github.com/google/seesaw
https://github.com/google/metallb
https://osrg.github.io/gobgp/
https://github.com/miekg/dns
https://github.com/iovisor/gobpf
https://github.com/cilium/cilium