The document discusses strategies for load balancing to scale to 80,000 transactions per second (TPS) while maintaining low latency. It evaluates different load balancing approaches like DNS-based, software/hardware-based layer 3/4 proxying, layer 7 proxying, and routing at layer 4. It finds that routing using the Linux Virtual Server (LVS) and IPVS kernel modules is the most scalable approach and allows routing workloads across data centers using IPIP tunnels. Benchmarking shows LVS can handle the target workload with less than half the resources of proxy-based approaches. Key considerations with LVS are CPU affinity of interrupts, RP filter bypass, manageability, high availability, and IP tunnel extensibility.

1. In-Depth Study to scale @ 80K TPS
Load Balancing

2. Starting
❖ About Me : Engineering @ Paytm . Working on this
problem for 2 months
❖ Problem : Identifying Entry Solution for 80K TPS, 20K
active transacting connections , while keeping latency loss
< 2 ms
❖ Outline : Evaluation and Perf test of all sorts of LB,
Routers and classify them
❖ Not Covering : After Every solution, things which are not
covered

3. Evaluation criteria
▸High Availability ( HA ) : Unaffected service during any
predefined number of simultaneous failures
▸Balancing strategies : Round robin, least connection,
weighted .
▸Health Checks
▸Extensibility : C/Lua Lib support
▸Monitoring and Manageability
▸Perf

4. Categories of LB
❖ DNS Based
❖ Software & Hardware Based
❖ Layer 3/4 Proxying
❖ Layer 7 Proxying
❖ Routing at L4
cue 5

5. DNS Based
❖ Multiple IPs : Round
Robin
❖ No Concept of HA,
Monitoring, health
checks
❖ Health Checks, Routing
policies are available via
custom solutions

6. Layer 3/4 Load Balancing
❖ Hardware Based LBs mostly.
❖ No well known Prog. which runs in Kernel Space.
❖ Software Based User Space Proxy based LBs examples
are Haproxy and Nginx

7. Haproxy Monitoring
❖ Socket Based Stats are available with ~60 CSV
❖ Web Interface

8. Benchmarking Env
cue 10

9. Issues with Haproxy L4
❖ Scale Constraint
❖ Only CPU. Cores 100% with Load(1 min) as 64
❖ Benchmark
❖ 20K TPS , keep-alive off and 100ms backend latency.

10. Layer 7 load balancing
❖ Hardware based Lb : F5, Fortinet.
❖ Protocol rigidness
❖ No well known Prog. which runs in Kernel Space.
❖ Software Based : Nginx and HaProxy are popular ones.
❖ Benchmarking Issues with Nginx as L7
❖ Even more CPU Constraint than L4 : 18-20K TPS in
same Env

11. Not covering these for
Haproxy
❖ Security Aspects : IPTables, WAF, Selinux
❖ Bare Metal Machines Detailed Specs and Part Numbers
❖ Decision on choice of Machine.
❖ Networking Details
❖ NIC Bonding Specs
❖ Benchmark Tools Detailing : GOR Detailing
cue 15

12. Routing L3/4
❖ What is routing
❖ Routing scales , less than half resources are required than
proxying.

13. Types of routing
❖ Natting : Works like proxy
❖ Direct Route : Spoof MAC address and send it back.
❖ IP Tunneling : Most Scalable, works on IPIP Tunnel ( across different DCs
)

14. Routers
❖ Hardware routers : Not designed to be horizontally
scalable
❖ No Well-Known Horizontally scalable Hw Routers.
❖ We needed a Software Router : LVS/IPVS
cue 20

15. Software Router : LVS
❖ LVS : Linux Virtual server , 20 years old,
both Layer 4 and 7
❖ IPVS : IP Virtual Server, merged in
Kernel 2.4
❖ KTCPVS : App LB , in dev for last 8
years.
❖ Runs in Kernel Space
❖ Supports different distribution methods : RR,
Least connection, Weighted LC

16. LVS Issues
❖ CPU Affinity of Interrupts
❖ RP Filter Bypass
❖ Manageability and Monitoring
❖ HA
❖ IP Tunnel Extensibility

17. LVS : CPU Affinity
❖ CPU Affinity of Interrupts
❖ Kernel tries to load balance IRQ ( Interrupt Request Line ) across
cores.
❖ irqbalance service is responsible.
❖ cat /proc/interrupts will help see which core will max out.
❖ Balance (1) : echo fff > /sys/class/net/eth0/queues/rx-0/rps_cpus
❖ Balance (2) : echo 'fff' > /proc/irq/14/smp_affinity
❖ Balance (3) : echo '0-3' > /proc/irq/28/smp_affinity_list

18. LVS : RP Filter
❖ RP Filter : To Avoid Spoofing and DDOS
❖ Kernel checks whether the source of the received
packet is reachable through the route it came in.
❖ To Disable : net.ipv4.conf.tun.rp_filter = 0 in
/etc/sysctl.conf ( and sysctl -p )

19. LVS :Monitoring &
management❖ Managed by System Calls , No config ( use Consul Template )
❖ Logging : No Logs in user Space, Kernel messages for Errors
❖ Monitoring : Telegraf plugin available ( internals : ipvsadm —list —numeric /—connection /—
stats /—rate )

20. LVS : HA
❖ KeepAlive(d)
+ VIP
❖ Connection
Sync Service
❖ ipvsadm —start/stop-
daemon=master/backu
p --mcast-interface=<> -
-syncid <>

21. ❖ KeepAlive(d) for own Health Check
❖ Consul Template for Real Server Healtch Check
LVS : HealthCheck
cue 30

22. LVS IPIP Debugging
❖ IPIP Tunnel and VIP extension to multiple machines :Painful
❖ IPIP Tunnel Issues and recovery across DC
❖ Setup Probes and Packet Capture

23. Final Load Test

24. Final Arch
cue 35

25. Willy Tarreau : Haproxy
❖ Creator of Haproxy
❖ wtarreau.blogspot.com/2006/11/making-applications-
scalable-with-load.html
❖ The PPT structure is based on the article.

26. Shrey Agarwal
in.linkedin.com/in/shreyagarwal
❖ wtarreau.blogspot.com/2006/11/making-applications-scalable-with-load.html
❖ opensourceforu.com/2009/05/balancing-traffic-across-data-centres-using-lvs/
❖ www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html
❖ linux.die.net/man/8/ipvsadm
❖ serverfault.com/questions/723786/udp-packets-seen-on-interface-level-but-not-delivered-to-application-on-redhat
❖ serverfault.com/questions/163244/linux-kernel-not-passing-through-multicast-udp-packets
References