This document summarizes different techniques for scaling HTTP infrastructure, including DNS load balancing, IP load balancing, and HTTP load balancing. It discusses how DNS maps hostnames to IP addresses and the resolution process. It then explains how TCP and HTTP work on top of IP, and some of their limitations for large-scale services. The document evaluates the benefits and drawbacks of different load balancing approaches and discusses optimizations like SPDY and TCP termination proxies to improve performance over long distances.

1. HTTP at your local BigCo:How the internet sausage gets made Peter Griess @pgriess

2. Goals and non-goals Basics of TCP/IP, DNS and HTTP and how they work together; pitfalls and optimizations A 1,000 foot view of scaling out HTTP infrastructure All manner of load balancing / traffic shaping Living on the edge Not: how to make a fast application (database access, rendering performance, etc)

3. Background: DNS Map hostnames to IP(s) www.facebook.com 69.171.229.12, 69.171.228.40 Resolution process Recursion (and what does the DNS server see?) Caching Latencies: on-host, cached in LAN, cached at ISP, miss

4. Background: TCP Stateful protocol Negotiated by a synchronous 3-way handshake: 2xRTT before first byte is sent! e.g. USA => South America ~250ms RTT Seamless failover is hard (but not impossible) Load balancing must be aware of flows

5. Background: HTTP Layered on top of TCP/TLS Has some useful bits Compression Connection re-use Pipelining Caching Kind of sucks Headers on all requests/responses Compression on bodies only Pipelining has to be disabled most of the time Pipelining suffers from head-of-line blocking

6. mycutekittens.tv 68.193.17.4 Big bad internet HTTP

7. Problem?

8. Problem Availability Server goes down (kernel panic?) Network goes down (cable cut?) Datacenter goes down (EC2?) Overload Shed load (good, can be transparent) Get infinitely slow (not good)

9. mycutekittens.tv: multi-server Big bad internet ???

10. We have options DNS load balancing IP load balancing HTTP load balancing

11. DNS load balancing mycutekittens.tv resolves to IPs: A, B, C, D Add new IPs to scale out Remove IPs when hosts go down Benefits Don’t need extra hardware to do load balancing Can span datacenters DNS servers are cheap / fast Drawbacks Hotspots due to caching Hotspots due to ordering in result list Hotspots due to resolver size TTL / flexibility trade-off

12. mycutekittens.tv: DNS Big bad internet DNS Server DNS 68.193.17.4 68.193.17.5 68.193.17.6

13. IP load balancing (1) mycutekittens.tv resolves to 1 public IP owned by an IP load balancer Add new backend hosts w/ private IPs to scale out Load balancer health-checks hosts actively or passively to avoid dead hosts Scheduling policies vs. failover DSR

14. IP load balancing (2) Benefits Only 1 public IP (high DNS TTL) Backend network capacity/membership transparent to the internet Cheap-ish Failover is possible, not insanely difficult Drawbacks Can’t do what you can with HTTP

15. mycutekittens.tv: IP 10.0.0.1 Big bad internet 10.0.0.2 GW 68.193.17.4 10.0.0.3 LB

16. HTTP load balancing (1) mycutekittens.tv resolves to 1 public IP owned by an HTTP load balancer Largely same as IP load balancing Terminates TCP connections (sees all bytes) Can make routing decisions based on HTTP Can autonomously serve requests (caching, access control, etc) Examples: Send requests for /foo/* to pool A 401 requests without cookie Q

17. HTTP load balancing (2) Benefits Largely the same as IP More flexible rules Can terminate TLS (security+, cost+) Drawbacks No DSR Failover difficult Not as performant as IP

18. mycutekittens.tv: HTTP 10.0.0.1 Big bad internet 10.0.0.2 68.193.17.4 LB HTTP(S) 10.0.0.3

19. mycutekittens.tv: MOAR Eventually a single LB is going to be a problem Not enough capacity Availability Turtles all the day way down LB of LBs! DNS load balancing between datacenters …

20. HTTPS: myths and reality Too computationally expensive Only a few percent (imperialviolet.org); is your webserver actually CPU bound? doubt it SSL acceleration cards, GPUs, etc Too much latency Handshaking is 5-7xRTT Session resume False start Snap start Caching breaks

21. My latency is huge in Japan RTT to USA is (or any single DC) can be huge Re-use connections (connection: keep-alive) Send work in parallel (pipelining) Use compression (content-encoding) Lots of tricks for static resources (bundling, CDNs, caching, etc) Pre-fetch data

22. Let’s get crazy: SPDY Don’t limit yourself to HTTP; use a different protocol SPDY developed by Google, supported by Chrome, google.com (and soon facebook.com) Connection re-use w/o head-of-line blocking Headers always compressed Always SSL (but breaks caching)

23. Let’s get crazy: TCP termination Synchronous RTTs: the silent killer Opening new TCP connections is very costly Run proxies close to users and proxy traffic back to core using optimized protocol Low RTT to proxy Do SPDY-like tricks between edge + core Potentially faster network to core than public internet Advertise these proxies via DNS Geo-targetting AS-adjacency Akamai CDN does this, sort of

24. Let’s get crazy: DNS anycast Remember how DNS resolutions were slow? DNS servers could be far away from a user Advertise multiple network routes for the same DNS IP, let the IP stack pick the closest one