2. Some numbers
• +12M users.
• 40 billion pageviews a month.
• 40k req/s in core site at peak time (1.8 gbps).
• 10k req/s in image routing layer (2gbps).
• +500 frontend servers
3. Past
• Linux boxes running LVS and ldirectord.
• DSR strategy for load balancing.
• Frontends used to have a external public IP.
• Double investment in networking gear and its
redundancy.
• SSL balanced across all the frontends.
4. The (old) big picture
HTTP request
client
External
API
HTTP response
LVS
External network
f01 f02 fN
Internal network
5. Present
• New hardware. 4+1 LB instead of 10 LB (5+5)
• New load balancing strategy using HAProxy layer 7
capabilities.
• SSL terminated in the load balancers.
6. The big picture
HTTP request
External client
API
HTTP response
HTTP External network
HAProxy
proxy Internal network
HTTP response
f01 f02 fN
8. Networking
• 4 links for internal and 4 for external
• Connected to different stack member units
• 4gbps theorical capacity limit per node.
member unit 0
member unit 1
load balancer
member unit 0
member unit 1
9. Networking
• We tune IRQ SMP affinity for sharding IRQs across multiple
cores that share the same L2 cache [1]
• We do ECMP (Equal Cost Multi Path) [2] in our edge routers for
sharding traffic across the load balancers.
ip route 95.131.168.x/32 x.x.x.2
ip route 95.131.168.x/32 x.x.x.1
ip route 95.131.168.x/32 x.x.x.3
ip route 95.131.168.x/32 x.x.x.4
router
lb lb lb lb
10. HAProxy: Why?
• Layer7 load balancing: Content inspection,
persistence, slow start, throttling, anti-DoS features,
supervision, content switching, keep-alive, etc.
• Very robust and reliable.
• Designed to be a load balancer.
• Offers high control over HTTP delivery and status:
response codes, connections per frontend, queued
request, etc.
11. HAProxy: Concepts
• Frontend: Section where we listen() for incoming
connections.
• Backend: Pool of servers. We define algorithm,
configure healthy checks, etc.
• Listen section: frontend+backend. Useful for TCP.
• Connection != request: One connection can hold
multiple requests (keep-alive). Only the first one is
analyzed, logged and processed.
12. HAProxy: Health checks
• Standard health check
# Backend section
backend www_farm
mode http
balance roundrobin
option httpchk GET /server_health
# Servers
server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight
100
server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight
100
13. HAProxy: Health checks
• Observe mode
# Backend section
backend www_farm
mode http
balance roundrobin
option httpchk GET /server_health
observe layer7
# Servers
server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight
100
server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight
100
14. HAProxy: Persistence
• Cookie
• URI & URI parameter
• Source IP
• Header (i.e. Host header)
• RDP cookie (Anyone using MS Terminal Server?)
15. HAProxy: Cookie persistence
• Map requests between cookie value and backend
server. You can issue these cookies from the code and
play with them.
• Ideal for deploying code by stages, or caching locally
user data.
• If the server becomes unreachable the traffic will be
directed to other server within the same pool.
17. HAProxy: URL persistence
• Specially interesting for balancing HTTP caching servers
(i.e.Varnish). Without this feature the cache pool will be inefficient.
• The URLs are hashed and assigned to a server in the pool
(using a modulo operation). A server will serve always the same
object regardless of the load balancer that attends the request.
• Adding/removing/losing servers to the pool is not harmful thanks
to consistent hashing.
25. HAProxy: URL persistence
consistent hashing
A 1 7
B 2 8
C 3 9
D 4
1/6 misses =
E ~17% miss
5
F 6
26. HAProxy: URL persistence
Our images URLs always look like:
http://img3.tuenti.net/HyUdrohQQAFnCyjMJ2ekAA
We can choose the first block from the URI and use it for persistence decisions.
# balance roundrobin
balance uri depth 1
hash-type consistent
27. HAProxy: URL persistence
Our images URLs always look like:
http://img3.tuenti.net/MdlIdrAOilul8ldcRwD7AdzwAeAdB4AMtgAy
We can choose the first block from the URI and use it for persistence decisions.
# balance roundrobin
balance uri depth 1
hash-type consistent
28. HAProxy: Content switching and ACLs
• Same frontend, different backend.
• Take decisions about which backend will attend the connection
based on:
• Layer 7 information (HTTP headers, methods, URI, version,
status)
• Layer4 information (source IP, destination IP, port)
• Internal HAProxy information (amount of backend
connections, active servers in the backend, etc)
• Too much options for showing all on this presentation. [1]
30. HAProxy: Content switching and ACLs
# Backend section
backend www_farm
mode http
balance roundrobin
# Servers
server fe01 x.x.x.1:80 weight 100
server fe02 x.x.x.2:80 weight 100
backend mobile_farm
mode http
balance roundrobin
# Servers
server mfe01 x.x.x.1:80 weight 100
31. HAProxy: Content switching and ACLs
# Another example using internal HAProxy information
frontend http
bind x.x.x.x:80
mode http
option forwardfor except 127.0.0.1/8 header X-Forwarded-For
# Insert 250ms delay if the session rate is over 35k req/s
acl too_fast fe_sess_rate ge 35000
tcp-request inspect-delay 250ms
tcp-request content accept if ! too_fast
tcp-request content accept if WAIT_END
32. HAProxy: Content blocking
# Another example using internal HAProxy information
frontend http
bind x.x.x.x:80
mode http
option forwardfor except 127.0.0.1/8 header X-Forwarded-For
# Block requests with negative Content-Length value
acl invalid-cl hdr_val(content-length) le 0
block if invalid-cl
33. HAProxy: Slow start
# Backend section
backend www_farm
mode http
balance roundrobin
option httpchk GET /server_health
# Servers
server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise
2 fall 3 weight 100
server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise
2 fall 3 weight 100
34. HAProxy: Graceful shutdown
# Backend section
backend www_farm
mode http
balance roundrobin
option httpchk GET /server_health
http-check disable-on-404
# Servers
server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise
2 fall 3 weight 100
server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise
2 fall 3 weight 100
35. HAProxy: Monitoring
•Traffic through different frontend interfaces. Easy to
aggregate incoming/outgoing traffic.
• Amount of different HTTP response codes
• /proc/net/sockstat
37. Client-side load balancing
• When user logs into the site the browser loads a
javascript API. Browser talks to it.
• Browser communicates with the API and this one
uses EasyXDM.
• Using application logic we control user request to a
defined farm.
• A/B testing based in any criteria.
• Where are from?
• How old are you?
39. SSL
• TCP load balancing is not useful for us.
• We deployed stunnel and it worked fine for a while.
• Then we started to suffer contention when accepting new
connections.
• We are currently using stud [2] for terminating SSL in our load
balancers.
40. SSL: Legal issues
• You can’t use this strategy of SSL termination in your PCI
compliant platform.
• We transport client IP information into X-Forwarded-For headers
in order to log users IPs because law enforcements.
• We terminate SSL in the load balancer because balancing TCP
(SSL) you can’t inform the backend about the client IP.
41. stud: The Scalable TLS Unwrapping
Daemon
• Supports both SSL and TLS using OpenSSL.
• Uses a process-per-core model.
• Asynchronous I/O using libev.
• Very little overhead per connection.
• Designed for long-living connections.
• Supports PROXY protocol.
• Recently they added inter-process communication [5].
42. PROXY protocol
• Created by HAProxy [5] author for safely transport connection
information across multiple layers of NAT or TCP proxies.
• Native support in stud. Patches available for stunnel4.
• We use it for stud informing to HAProxy about the real IP of the
client, converting this information to X-Forwarded-For header that
we can read and store in our application.
![Networking
• We tune IRQ SMP affinity for sharding IRQs across multiple
cores that share the same L2 cache [1]
• We do ECMP (Equal Cost Multi Path) [2] in our edge routers for
sharding traffic across the load balancers.
ip route 95.131.168.x/32 x.x.x.2
ip route 95.131.168.x/32 x.x.x.1
ip route 95.131.168.x/32 x.x.x.3
ip route 95.131.168.x/32 x.x.x.4
router
lb lb lb lb](https://image.slidesharecdn.com/loadbalancingattuentiexternal-111118102642-phpapp01/75/load-balancing-at-tuenti-9-2048.jpg?cb=1666628535)
![HAProxy: Content switching and ACLs
• Same frontend, different backend.
• Take decisions about which backend will attend the connection
based on:
• Layer 7 information (HTTP headers, methods, URI, version,
status)
• Layer4 information (source IP, destination IP, port)
• Internal HAProxy information (amount of backend
connections, active servers in the backend, etc)
• Too much options for showing all on this presentation. [1]](https://image.slidesharecdn.com/loadbalancingattuentiexternal-111118102642-phpapp01/75/load-balancing-at-tuenti-28-2048.jpg?cb=1666628535)
![SSL
• TCP load balancing is not useful for us.
• We deployed stunnel and it worked fine for a while.
• Then we started to suffer contention when accepting new
connections.
• We are currently using stud [2] for terminating SSL in our load
balancers.](https://image.slidesharecdn.com/loadbalancingattuentiexternal-111118102642-phpapp01/75/load-balancing-at-tuenti-39-2048.jpg?cb=1666628535)
![stud: The Scalable TLS Unwrapping
Daemon
• Supports both SSL and TLS using OpenSSL.
• Uses a process-per-core model.
• Asynchronous I/O using libev.
• Very little overhead per connection.
• Designed for long-living connections.
• Supports PROXY protocol.
• Recently they added inter-process communication [5].](https://image.slidesharecdn.com/loadbalancingattuentiexternal-111118102642-phpapp01/75/load-balancing-at-tuenti-41-2048.jpg?cb=1666628535)
![PROXY protocol
• Created by HAProxy [5] author for safely transport connection
information across multiple layers of NAT or TCP proxies.
• Native support in stud. Patches available for stunnel4.
• We use it for stud informing to HAProxy about the real IP of the
client, converting this information to X-Forwarded-For header that
we can read and store in our application.](https://image.slidesharecdn.com/loadbalancingattuentiexternal-111118102642-phpapp01/75/load-balancing-at-tuenti-42-2048.jpg?cb=1666628535)
![REST API
• Not official feature (yet) [6]
• You can easily communicate to the server via HTTP.
• Awesome for orchestrating your web tier.](https://image.slidesharecdn.com/loadbalancingattuentiexternal-111118102642-phpapp01/75/load-balancing-at-tuenti-44-2048.jpg?cb=1666628535)
![Related links
http://software.intel.com/en-us/articles/improved-linux-smp-scaling-
• [1]
user-directed-processor-affinity/
• [2] http://en.wikipedia.org/wiki/Equal-cost_multi-path_routing
• [3] stud repo: https://github.com/bumptech/stud
• [4] Scaling SSL: http://blog.exceliance.fr/2011/11/07/scaling-out-ssl/
PROXY protocol: http://haproxy.1wt.eu/download/1.5/doc/proxy-
• [5]
protocol.txt
• [6] REST API patch: https://github.com/jbuchbinder/haproxy-forked
• HAProxy configuration doc:
http://haproxy.1wt.eu/download/1.5/doc/configuration.txt](https://image.slidesharecdn.com/loadbalancingattuentiexternal-111118102642-phpapp01/75/load-balancing-at-tuenti-46-2048.jpg?cb=1666628535)
