On October 23rd, 2014, we updated our
By continuing to use LinkedIn’s SlideShare service, you agree to the revised terms, so please take a few minutes to review them.
Load balancing @Tuenti Ricardo Bartolomé, Senior Systems Engineer
Some numbers• +12M users.• 40 billion pageviews a month.• 40k req/s in core site at peak time (1.8 gbps).• 10k req/s in image routing layer (2gbps).• +500 frontend servers
Past• Linux boxes running LVS and ldirectord.• DSR strategy for load balancing.• Frontends used to have a external public IP.• Double investment in networking gear and itsredundancy.• SSL balanced across all the frontends.
The (old) big picture HTTP request clientExternal API HTTP response LVS External network f01 f02 fN Internal network
Present• New hardware. 4+1 LB instead of 10 LB (5+5)• New load balancing strategy using HAProxy layer 7capabilities.• SSL terminated in the load balancers.
The big picture HTTP request External client API HTTP responseHTTP External network HAProxyproxy Internal network HTTP response f01 f02 fN
Networking• 4 links for internal and 4 for external• Connected to different stack member units• 4gbps theorical capacity limit per node. member unit 0 member unit 1 load balancer member unit 0 member unit 1
Networking• We tune IRQ SMP affinity for sharding IRQs across multiplecores that share the same L2 cache • We do ECMP (Equal Cost Multi Path)  in our edge routers forsharding traffic across the load balancers. ip route 95.131.168.x/32 x.x.x.2 ip route 95.131.168.x/32 x.x.x.1 ip route 95.131.168.x/32 x.x.x.3 ip route 95.131.168.x/32 x.x.x.4 router lb lb lb lb
HAProxy: Why?• Layer7 load balancing: Content inspection,persistence, slow start, throttling, anti-DoS features,supervision, content switching, keep-alive, etc.• Very robust and reliable.• Designed to be a load balancer.• Offers high control over HTTP delivery and status:response codes, connections per frontend, queuedrequest, etc.
HAProxy: Concepts• Frontend: Section where we listen() for incomingconnections.• Backend: Pool of servers. We define algorithm,configure healthy checks, etc.• Listen section: frontend+backend. Useful for TCP.• Connection != request: One connection can holdmultiple requests (keep-alive). Only the first one isanalyzed, logged and processed.
HAProxy: Health checks• Standard health check# Backend sectionbackend www_farm mode http balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight100
HAProxy: Health checks• Observe mode# Backend sectionbackend www_farm mode http balance roundrobin option httpchk GET /server_health observe layer7 # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight100
HAProxy: Persistence• Cookie• URI & URI parameter• Source IP• Header (i.e. Host header)• RDP cookie (Anyone using MS Terminal Server?)
HAProxy: Cookie persistence• Map requests between cookie value and backendserver. You can issue these cookies from the code andplay with them.• Ideal for deploying code by stages, or caching locallyuser data.• If the server becomes unreachable the traffic will bedirected to other server within the same pool.
HAProxy: URL persistence• Specially interesting for balancing HTTP caching servers(i.e.Varnish). Without this feature the cache pool will be inefficient.• The URLs are hashed and assigned to a server in the pool(using a modulo operation). A server will serve always the sameobject regardless of the load balancer that attends the request.• Adding/removing/losing servers to the pool is not harmful thanksto consistent hashing.
HAProxy: URL persistenceOur images URLs always look like: http://img3.tuenti.net/HyUdrohQQAFnCyjMJ2ekAAWe can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent
HAProxy: URL persistenceOur images URLs always look like: http://img3.tuenti.net/MdlIdrAOilul8ldcRwD7AdzwAeAdB4AMtgAyWe can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent
HAProxy: Content switching and ACLs• Same frontend, different backend.• Take decisions about which backend will attend the connectionbased on: • Layer 7 information (HTTP headers, methods, URI, version, status) • Layer4 information (source IP, destination IP, port) • Internal HAProxy information (amount of backend connections, active servers in the backend, etc)• Too much options for showing all on this presentation. 
HAProxy: Content switching and ACLs# Backend sectionbackend www_farm mode http balance roundrobin # Servers server fe01 x.x.x.1:80 weight 100 server fe02 x.x.x.2:80 weight 100backend mobile_farm mode http balance roundrobin # Servers server mfe01 x.x.x.1:80 weight 100
HAProxy: Content switching and ACLs# Another example using internal HAProxy informationfrontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Insert 250ms delay if the session rate is over 35k req/s acl too_fast fe_sess_rate ge 35000 tcp-request inspect-delay 250ms tcp-request content accept if ! too_fast tcp-request content accept if WAIT_END
HAProxy: Content blocking# Another example using internal HAProxy informationfrontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Block requests with negative Content-Length value acl invalid-cl hdr_val(content-length) le 0 block if invalid-cl
HAProxy: Slow start# Backend sectionbackend www_farm mode http balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise2 fall 3 weight 100
HAProxy: Graceful shutdown# Backend sectionbackend www_farm mode http balance roundrobin option httpchk GET /server_health http-check disable-on-404 # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise2 fall 3 weight 100
HAProxy: Monitoring•Traffic through different frontend interfaces. Easy toaggregate incoming/outgoing traffic.• Amount of different HTTP response codes• /proc/net/sockstat
SSL• TCP load balancing is not useful for us.• We deployed stunnel and it worked fine for a while.• Then we started to suffer contention when accepting newconnections.• We are currently using stud  for terminating SSL in our loadbalancers.
SSL: Legal issues• You can’t use this strategy of SSL termination in your PCIcompliant platform.• We transport client IP information into X-Forwarded-For headersin order to log users IPs because law enforcements.• We terminate SSL in the load balancer because balancing TCP(SSL) you can’t inform the backend about the client IP.
stud: The Scalable TLS Unwrapping Daemon• Supports both SSL and TLS using OpenSSL.• Uses a process-per-core model.• Asynchronous I/O using libev.• Very little overhead per connection.• Designed for long-living connections.• Supports PROXY protocol.• Recently they added inter-process communication .
PROXY protocol• Created by HAProxy  author for safely transport connectioninformation across multiple layers of NAT or TCP proxies.• Native support in stud. Patches available for stunnel4.• We use it for stud informing to HAProxy about the real IP of theclient, converting this information to X-Forwarded-For header thatwe can read and store in our application.