Varnish at the BBC


Published on

Presentation on how the BBC uses Varnish, from VUG6 (

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Varnish at the BBC

  1. 1. Varnish at the BBCWinning Gold in the London 2012 Olympic Games Graham Lyons
  2. 2. Varnish at the BBC● First deployed in 2009 ○ Specifically caching layer for iPlayer ○ New dynamic Platform● Platform has grown to 100s of applicationsHow do we scale Varnish across the Platform?(It served LOTS of traffic during the Olympics)
  3. 3. In the BBC Infrastructure● is made up of lots of applications● Load balancer in front● Sends request to Varnish● Varnish sends request to another load balancer● Second layer of load balancer distributes load across application servers ○ All applications installed on all servers
  4. 4. How it looks
  5. 5. Routing● First load balancer adds header with name of a pool of servers● Varnish forwards it on● Second load balancer knows what to do with the header to route the request
  6. 6. How do we use Varnish● General HTTP cache● Make use of header manipulation for more efficient caching, e.g. ○ GeoIP ○ Device detection ○ Cookie decomposition
  7. 7. In 2009...● Application logic in VCL● Very small number of applications so it was manageable
  8. 8. Where should we take it?● BBC Platform HTTP cache● Platform-wide features● Different requirements to application- specific Varnish
  9. 9. ...2012 (What we changed)● Removed application logic (mostly)● Added features to be used generally ○ e.g. GeoIP, Device detection● Features on by default - no special configuration● Try to stay vanilla and RFC2616(ish)
  10. 10. Features? What features?● GeoIP lookup● Device meta information● Cookie decomposition ○ Signed in headerAll exposed as headers added to the requestCompanion PHP libraries to manage headeraccess and Vary header on response
  11. 11. Geo and Device Information● Looked up via an HTTP call to respective services● Logic in C library● Cached locally (in process, in memory cache) ○ 70% hit for geoip ○ >95% hit for device data
  12. 12. Cookies?● Incoming Cookie header split into a header for each value● e.g. Cookie: UID=4321... ○ ...becomes: X-Cookie-UID: 4321Actually only operates on cookie values withparticular prefixes (introduced for the Great EUCookie Debacle)
  13. 13. Signed in header● Boolean ○ Signed in ○ Not signed in● Allows caching of page for not signed in state
  14. 14. Cache VariationsAll these features allow more efficient cachevariations.Can cache variations based on:● where the user is● what type of device theyre using● any personalisationse.g. Norwegian Android user who lovesEastenders gets served straight from the cache
  15. 15. Response to outside world● External caches dont know about request headers Varnish adds● Responses have to be reduced to being privately cacheable● GeoIP exception ○ lookup is done on the last step outside our infrastructure
  16. 16. Vary: Cookie?● Originally planned to send this out for responses using X-Cookie-...● Analytics cookie on site● Changes on each page...● Send responses out as uncacheable
  17. 17. Setting a Unique Cookie● Previously sent from backend● Generate unique ID cookie in Varnish● Allows cookie to be set and content served from cache
  18. 18. Feedback features...● How well is the cache being used?● Record per application hit/miss ratios
  19. 19. Big Sporting Event, 2012
  20. 20. Big Sporting Event, 2012"Dont f*** up the Olympics..."
  21. 21. Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applications
  22. 22. Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applicationsI think we can handle this...
  23. 23. Special preparations?
  24. 24. Special preparations?Reduced number of Varnish servers
  25. 25. Crosstown TrafficOlympics Daily Peak:● 10.4 million browsers to● 8 million UK● 2.4 million International● (Record numbers)
  26. 26. Device SplitMainly mobile and desktop
  27. 27. So that went wellWhat didnt work for us?
  28. 28. Varnish and HD Streaming● 24 HD streams● Planned to use Varnish at the front● Cached very, very well● Needed to be highly available● HA layer didnt hold up● Had to use a load balancer instead and use the cache there
  29. 29. What else has hurt?ESI● Increase in complexity● Working out best practice● Seg faults! ○ Overflow of sess_workspace
  30. 30. However...● Synthetic end point generated in Varnish● Included as ESI● Very good performance... ○ Almost 4 times previous load
  31. 31. Other pains● No Saint mode ○ Load balancing behind and multiple apps● Network bandwidth ○ As few boxes as possible
  32. 32. Next?● Everywhere! ○ Ubiquitous caching layer ○ Already have most big players● More monitoring● Version 3 ○ VMODs?● Make it simpler ○ Remove anything we can
  33. 33. tl;drTook Varnish from being an application-specific component to a Platform-wide essential
  34. 34. Questions? Graham Lyons
  35. 35. Questions?(Yes, were hiring...) Graham Lyons