Geoff Ballinger discussed how DeltaDNA used HAProxy load balancers to handle traffic to their analytics APIs and customer portal. As traffic grew, they implemented a two-layer HAProxy configuration to better utilize modern multi-core servers. However, the newer experimental multi-threaded mode in HAProxy v1.8 provided a better solution by removing bottlenecks. Ballinger concluded the load balancer acts as the "front door" where systems meet the world, and is worth understanding in detail to optimize services and keep up to date.

1. Geoff Ballinger
System Architect
The Load Balancer:
War Stories with HAProxy

2. Agenda
● Software Engineer & Live Ops
● Technical scaling of startups
● Entertainment, Mobile and
Connected Systems
● How real people interact with
technology
Geoff

3. AgendaDeltaDNA

4. Agenda
● DeltaDNA has been providing fast
and powerful analytics tools and
insight to the games sector’s leading
publishers and developers since
2010
● Including Take-Two Interactive,
Square Enix Montreal, Wargaming
and Bandai Namco Entertainment
Europe.
DeltaDNA

5. AgendaThe Load Balancer in devops
● Why am I talking about Load
Balancers at a devops
conference?

6. Agenda
● Assemble your individual
(micro)services into a coherent
whole
● Individual and appropriate
configuration of connectivity to all
your services in one place.
The Load Balancer in devops

7. AgendaThe Load Balancer in devops
● Analyse activity at the all traffic
level and drop traffic from
problematic sources

8. Agenda
• L7 edge proxy for APIs + small webapp
• Keepalived (VRRP) to manage VIPs over multiple instances for availability
HAProxy (v1.5-v1.7) at deltaDNA

9. AgendaHAProxy (v1.5-v1.7) at deltaDNA
• Compose multiple groups of paths requiring different properties to our customer portal
webapp

10. AgendaHAProxy (v1.5-v1.7) at deltaDNA
• 10k/s++ hits to our data collection and campaign APIs running 24/7/365
• Block unintentional DDOS from problem clients

11. AgendaProblems
• Fundamentally single threaded by
design
• vs modern multi-core servers!

12. AgendaProblems
• Can be multi-process but state
tables, reporting, and health
checks are local per-process so
not very practical

13. Agenda
• Steady traffic growth
• Increasing proportion of compute
intensive HTTPS API traffic
• Need to scale!
Problems

14. Agenda
• Cascade 2 layers of HAproxy
processes
• n-1 in outer layer accept SSL traffic
from the VIPs and sends to inner
using proxy protocol
• 1 in inner layer does the load
balancing
A solution!

15. AgendaA solution!
• Single core doing health checks,
state, and reporting
• The rest handle the SSL overhead
• Great utilisation of a modern multi-
core server

16. Agenda
• BUT HAProxy v1.8 has an
“EXPERIMENTAL” multi-threaded
mode…
A (better) solution!

17. Agenda
● Deploy in docker
● Kubernettes ingress controller
● API driven
The “Modern Devops Load Balancer”

18. Agenda
● Where your systems meet the world,
your “front door”
● The place to merge and optimise your
services into a coherent whole
● Worth understanding in detail
To wrap up the Load Balancer is...

19. Agenda
● Where your systems meet the world,
your “front door”
● The place to merge and optimise your
services into a coherent whole
● Worth understanding in detail
● Worth keeping up to date with!
To wrap up the Load Balancer is...

20. Geoff Ballinger
System Architect
Thank you!
(We are hiring...)
info@deltadna.comwww.deltadna.com