Load balancing at tuenti

Load balancing @Tuenti Ricardo Bartolomé, Senior Systems Engineer

Some numbers• +12M users.• 40 billion pageviews a month.• 40k req/s in core site at peak time (1.8 gbps).• 10k req/s in image routing layer (2gbps).• +500 frontend servers

Past• Linux boxes running LVS and ldirectord.• DSR strategy for load balancing.• Frontends used to have a external public IP.• Double investment in networking gear and itsredundancy.• SSL balanced across all the frontends.

The (old) big picture HTTP request clientExternal API HTTP response LVS External network f01 f02 fN Internal network

Present• New hardware. 4+1 LB instead of 10 LB (5+5)• New load balancing strategy using HAProxy layer 7capabilities.• SSL terminated in the load balancers.

The big picture HTTP request External client API HTTP responseHTTP External network HAProxyproxy Internal network HTTP response f01 f02 fN

Hardware• Intel Xeon X5677 (4 core, 8 threads @ 3.47GHz)• 8 gigabit network interfaces (Broadcon NextExtreme5702 w/ multiqueue support)• 16 GB of memory

Networking• 4 links for internal and 4 for external• Connected to different stack member units• 4gbps theorical capacity limit per node. member unit 0 member unit 1 load balancer member unit 0 member unit 1

Networking• We tune IRQ SMP affinity for sharding IRQs across multiplecores that share the same L2 cache [1]• We do ECMP (Equal Cost Multi Path) [2] in our edge routers forsharding traffic across the load balancers. ip route 95.131.168.x/32 x.x.x.2 ip route 95.131.168.x/32 x.x.x.1 ip route 95.131.168.x/32 x.x.x.3 ip route 95.131.168.x/32 x.x.x.4 router lb lb lb lb

HAProxy: Why?• Layer7 load balancing: Content inspection,persistence, slow start, throttling, anti-DoS features,supervision, content switching, keep-alive, etc.• Very robust and reliable.• Designed to be a load balancer.• Offers high control over HTTP delivery and status:response codes, connections per frontend, queuedrequest, etc.

HAProxy: Concepts• Frontend: Section where we listen() for incomingconnections.• Backend: Pool of servers. We define algorithm,configure healthy checks, etc.• Listen section: frontend+backend. Useful for TCP.• Connection != request: One connection can holdmultiple requests (keep-alive). Only the first one isanalyzed, logged and processed.

HAProxy: Health checks• Standard health check# Backend sectionbackend www_farm mode http balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight100

HAProxy: Health checks• Observe mode# Backend sectionbackend www_farm mode http balance roundrobin option httpchk GET /server_health observe layer7 # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight100

HAProxy: Persistence• Cookie• URI & URI parameter• Source IP• Header (i.e. Host header)• RDP cookie (Anyone using MS Terminal Server?)

HAProxy: Cookie persistence• Map requests between cookie value and backendserver. You can issue these cookies from the code andplay with them.• Ideal for deploying code by stages, or caching locallyuser data.• If the server becomes unreachable the traffic will bedirected to other server within the same pool.

HAProxy: Cookie persistencebackend www mode http balance roundrobin option redispatch cookie mycookie insert maxidle 120 maxlife 900 indirect preservedomain .tuenti.com server fe01 1.1.1.1:80 weight 100 cookie 1111 server fe02 1.1.1.2:80 weight 100 cookie 1112 server fe03 1.1.1.3:80 weight 100 cookie 1113

HAProxy: URL persistence• Specially interesting for balancing HTTP caching servers(i.e.Varnish). Without this feature the cache pool will be inefficient.• The URLs are hashed and assigned to a server in the pool(using a modulo operation). A server will serve always the sameobject regardless of the load balancer that attends the request.• Adding/removing/losing servers to the pool is not harmful thanksto consistent hashing.

HAProxy: URL persistence map-based hashingA 1 7B 2 8C 3 9D 4E 5F 6

HAProxy: URL persistence map-based hashingA 1 7 1 6B 2 8 2 7C 3 9 3 8D 4 10 4 9E 5 5 10F 6

HAProxy: URL persistence map-based hashingA 1 7 1 6 High miss rate. #FAILB 2 8 2 7C 3 9 3 8D 4 10 4 9E 5 5 10F 6

HAProxy: URL persistence consistent hashingA 1 7B 2 8C 3 9D 4E 5F 6

HAProxy: URL persistence consistent hashingA 1 7B 2 8C 3 9D 4 1/6 misses =E ~17% miss 5F 6

HAProxy: URL persistenceOur images URLs always look like: http://img3.tuenti.net/HyUdrohQQAFnCyjMJ2ekAAWe can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent

HAProxy: URL persistenceOur images URLs always look like: http://img3.tuenti.net/MdlIdrAOilul8ldcRwD7AdzwAeAdB4AMtgAyWe can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent

HAProxy: Content switching and ACLs• Same frontend, different backend.• Take decisions about which backend will attend the connectionbased on: • Layer 7 information (HTTP headers, methods, URI, version, status) • Layer4 information (source IP, destination IP, port) • Internal HAProxy information (amount of backend connections, active servers in the backend, etc)• Too much options for showing all on this presentation. [1]

HAProxy: Content switching and ACLs# Frontend sectionfrontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Farm content switching acl acl-api-uri path /api acl acl-mobile-site hdr(host) -i m.tuenti.com acl acl-cdn-service hdr(host) -i cdn.tuenti.net use_backend mobile_farm if acl-mobile-site use_backend api_farm if acl-api-uri use_backend cdn_farm if acl-cdn-service default_backend www_farm

HAProxy: Content switching and ACLs# Backend sectionbackend www_farm mode http balance roundrobin # Servers server fe01 x.x.x.1:80 weight 100 server fe02 x.x.x.2:80 weight 100backend mobile_farm mode http balance roundrobin # Servers server mfe01 x.x.x.1:80 weight 100

HAProxy: Content switching and ACLs# Another example using internal HAProxy informationfrontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Insert 250ms delay if the session rate is over 35k req/s acl too_fast fe_sess_rate ge 35000 tcp-request inspect-delay 250ms tcp-request content accept if ! too_fast tcp-request content accept if WAIT_END

HAProxy: Content blocking# Another example using internal HAProxy informationfrontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Block requests with negative Content-Length value acl invalid-cl hdr_val(content-length) le 0 block if invalid-cl

HAProxy: Slow start# Backend sectionbackend www_farm mode http balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise2 fall 3 weight 100

HAProxy: Graceful shutdown# Backend sectionbackend www_farm mode http balance roundrobin option httpchk GET /server_health http-check disable-on-404 # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise2 fall 3 weight 100

HAProxy: Monitoring•Traffic through different frontend interfaces. Easy toaggregate incoming/outgoing traffic.• Amount of different HTTP response codes• /proc/net/sockstat

HAProxy: Monitoringfrontend stats1 mode http bind-process 1 bind :8081 default_backend haproxy-stats1backend haproxy-stats1 bind-process 1 mode http stats enable stats refresh 60s stats uri / stats auth mgmt:password

Client-side load balancing• When user logs into the site the browser loads ajavascript API. Browser talks to it.• Browser communicates with the API and this oneuses EasyXDM.• Using application logic we control user request to adefined farm. • A/B testing based in any criteria. • Where are from? • How old are you?

Client-side load balancing‘frontend_farm_map‘ => array( 1 => www1, // x% (Alava) 2 => www4, // y% (Albacete) 3 => www4, // z% (Alicante) …)‘users_using_staging => array( ‘level’ => ‘limited’, ‘percent’ => 10,)

SSL• TCP load balancing is not useful for us.• We deployed stunnel and it worked fine for a while.• Then we started to suffer contention when accepting newconnections.• We are currently using stud [2] for terminating SSL in our loadbalancers.

SSL: Legal issues• You can’t use this strategy of SSL termination in your PCIcompliant platform.• We transport client IP information into X-Forwarded-For headersin order to log users IPs because law enforcements.• We terminate SSL in the load balancer because balancing TCP(SSL) you can’t inform the backend about the client IP.

stud: The Scalable TLS Unwrapping Daemon• Supports both SSL and TLS using OpenSSL.• Uses a process-per-core model.• Asynchronous I/O using libev.• Very little overhead per connection.• Designed for long-living connections.• Supports PROXY protocol.• Recently they added inter-process communication [5].

PROXY protocol• Created by HAProxy [5] author for safely transport connectioninformation across multiple layers of NAT or TCP proxies.• Native support in stud. Patches available for stunnel4.• We use it for stud informing to HAProxy about the real IP of theclient, converting this information to X-Forwarded-For header thatwe can read and store in our application.

PROXY protocol# stud --ssl -c OPENSSL_CIPHERS -b 127.0.0.1 8888 -f x.x.x.x 443 -n 2-u stud --write-proxy certificate.pemfrontend http-localhost-proxy-443 bind 127.0.0.1:8888 accept-proxy mode http reqadd X-Protocol: SSL reqadd X-Port: 443 default_backend www_farm

REST API• Not official feature (yet) [6]• You can easily communicate to the server via HTTP.• Awesome for orchestrating your web tier.

Questions?

Related links http://software.intel.com/en-us/articles/improved-linux-smp-scaling-• [1]user-directed-processor-affinity/• [2] http://en.wikipedia.org/wiki/Equal-cost_multi-path_routing• [3] stud repo: https://github.com/bumptech/stud• [4] Scaling SSL: http://blog.exceliance.fr/2011/11/07/scaling-out-ssl/ PROXY protocol: http://haproxy.1wt.eu/download/1.5/doc/proxy-• [5]protocol.txt• [6] REST API patch: https://github.com/jbuchbinder/haproxy-forked• HAProxy configuration doc:http://haproxy.1wt.eu/download/1.5/doc/configuration.txt

https://twitter.com	3
http://a0.twimg.com	2
http://wordpress.com	1
https://si0.twimg.com	1

Load balancing at tuenti

by Ricardo Bartolomé

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 7

Statistics

Load balancing at tuenti Presentation Transcript