Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Varnish at the BBC


Published on

Presentation on how the BBC uses Varnish, from VUG6 (

  • Be the first to comment

  • Be the first to like this

Varnish at the BBC

  1. 1. Varnish at the BBCWinning Gold in the London 2012 Olympic Games Graham Lyons
  2. 2. Varnish at the BBC● First deployed in 2009 ○ Specifically caching layer for iPlayer ○ New dynamic Platform● Platform has grown to 100s of applicationsHow do we scale Varnish across the Platform?(It served LOTS of traffic during the Olympics)
  3. 3. In the BBC Infrastructure● is made up of lots of applications● Load balancer in front● Sends request to Varnish● Varnish sends request to another load balancer● Second layer of load balancer distributes load across application servers ○ All applications installed on all servers
  4. 4. How it looks
  5. 5. Routing● First load balancer adds header with name of a pool of servers● Varnish forwards it on● Second load balancer knows what to do with the header to route the request
  6. 6. How do we use Varnish● General HTTP cache● Make use of header manipulation for more efficient caching, e.g. ○ GeoIP ○ Device detection ○ Cookie decomposition
  7. 7. In 2009...● Application logic in VCL● Very small number of applications so it was manageable
  8. 8. Where should we take it?● BBC Platform HTTP cache● Platform-wide features● Different requirements to application- specific Varnish
  9. 9. ...2012 (What we changed)● Removed application logic (mostly)● Added features to be used generally ○ e.g. GeoIP, Device detection● Features on by default - no special configuration● Try to stay vanilla and RFC2616(ish)
  10. 10. Features? What features?● GeoIP lookup● Device meta information● Cookie decomposition ○ Signed in headerAll exposed as headers added to the requestCompanion PHP libraries to manage headeraccess and Vary header on response
  11. 11. Geo and Device Information● Looked up via an HTTP call to respective services● Logic in C library● Cached locally (in process, in memory cache) ○ 70% hit for geoip ○ >95% hit for device data
  12. 12. Cookies?● Incoming Cookie header split into a header for each value● e.g. Cookie: UID=4321... ○ ...becomes: X-Cookie-UID: 4321Actually only operates on cookie values withparticular prefixes (introduced for the Great EUCookie Debacle)
  13. 13. Signed in header● Boolean ○ Signed in ○ Not signed in● Allows caching of page for not signed in state
  14. 14. Cache VariationsAll these features allow more efficient cachevariations.Can cache variations based on:● where the user is● what type of device theyre using● any personalisationse.g. Norwegian Android user who lovesEastenders gets served straight from the cache
  15. 15. Response to outside world● External caches dont know about request headers Varnish adds● Responses have to be reduced to being privately cacheable● GeoIP exception ○ lookup is done on the last step outside our infrastructure
  16. 16. Vary: Cookie?● Originally planned to send this out for responses using X-Cookie-...● Analytics cookie on site● Changes on each page...● Send responses out as uncacheable
  17. 17. Setting a Unique Cookie● Previously sent from backend● Generate unique ID cookie in Varnish● Allows cookie to be set and content served from cache
  18. 18. Feedback features...● How well is the cache being used?● Record per application hit/miss ratios
  19. 19. Big Sporting Event, 2012
  20. 20. Big Sporting Event, 2012"Dont f*** up the Olympics..."
  21. 21. Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applications
  22. 22. Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applicationsI think we can handle this...
  23. 23. Special preparations?
  24. 24. Special preparations?Reduced number of Varnish servers
  25. 25. Crosstown TrafficOlympics Daily Peak:● 10.4 million browsers to● 8 million UK● 2.4 million International● (Record numbers)
  26. 26. Device SplitMainly mobile and desktop
  27. 27. So that went wellWhat didnt work for us?
  28. 28. Varnish and HD Streaming● 24 HD streams● Planned to use Varnish at the front● Cached very, very well● Needed to be highly available● HA layer didnt hold up● Had to use a load balancer instead and use the cache there
  29. 29. What else has hurt?ESI● Increase in complexity● Working out best practice● Seg faults! ○ Overflow of sess_workspace
  30. 30. However...● Synthetic end point generated in Varnish● Included as ESI● Very good performance... ○ Almost 4 times previous load
  31. 31. Other pains● No Saint mode ○ Load balancing behind and multiple apps● Network bandwidth ○ As few boxes as possible
  32. 32. Next?● Everywhere! ○ Ubiquitous caching layer ○ Already have most big players● More monitoring● Version 3 ○ VMODs?● Make it simpler ○ Remove anything we can
  33. 33. tl;drTook Varnish from being an application-specific component to a Platform-wide essential
  34. 34. Questions? Graham Lyons
  35. 35. Questions?(Yes, were hiring...) Graham Lyons