• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Varnish at the BBC

Varnish at the BBC



Presentation on how the BBC uses Varnish, from VUG6 (https://www.varnish-cache.org/vug6)

Presentation on how the BBC uses Varnish, from VUG6 (https://www.varnish-cache.org/vug6)



Total Views
Views on SlideShare
Embed Views



3 Embeds 6

https://twitter.com 4
https://si0.twimg.com 1
https://twimg0-a.akamaihd.net 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Varnish at the BBC Varnish at the BBC Presentation Transcript

    • Varnish at the BBCWinning Gold in the London 2012 Olympic Games Graham Lyons
    • Varnish at the BBC● First deployed in 2009 ○ Specifically caching layer for iPlayer ○ New dynamic Platform● Platform has grown to 100s of applicationsHow do we scale Varnish across the Platform?(It served LOTS of traffic during the Olympics)
    • In the BBC Infrastructure● bbc.co.uk is made up of lots of applications● Load balancer in front● Sends request to Varnish● Varnish sends request to another load balancer● Second layer of load balancer distributes load across application servers ○ All applications installed on all servers
    • How it looks
    • Routing● First load balancer adds header with name of a pool of servers● Varnish forwards it on● Second load balancer knows what to do with the header to route the request
    • How do we use Varnish● General HTTP cache● Make use of header manipulation for more efficient caching, e.g. ○ GeoIP ○ Device detection ○ Cookie decomposition
    • In 2009...● Application logic in VCL● Very small number of applications so it was manageable
    • Where should we take it?● BBC Platform HTTP cache● Platform-wide features● Different requirements to application- specific Varnish
    • ...2012 (What we changed)● Removed application logic (mostly)● Added features to be used generally ○ e.g. GeoIP, Device detection● Features on by default - no special configuration● Try to stay vanilla and RFC2616(ish)
    • Features? What features?● GeoIP lookup● Device meta information● Cookie decomposition ○ Signed in headerAll exposed as headers added to the requestCompanion PHP libraries to manage headeraccess and Vary header on response
    • Geo and Device Information● Looked up via an HTTP call to respective services● Logic in C library● Cached locally (in process, in memory cache) ○ 70% hit for geoip ○ >95% hit for device data
    • Cookies?● Incoming Cookie header split into a header for each value● e.g. Cookie: UID=4321... ○ ...becomes: X-Cookie-UID: 4321Actually only operates on cookie values withparticular prefixes (introduced for the Great EUCookie Debacle)
    • Signed in header● Boolean ○ Signed in ○ Not signed in● Allows caching of page for not signed in state
    • Cache VariationsAll these features allow more efficient cachevariations.Can cache variations based on:● where the user is● what type of device theyre using● any personalisationse.g. Norwegian Android user who lovesEastenders gets served straight from the cache
    • Response to outside world● External caches dont know about request headers Varnish adds● Responses have to be reduced to being privately cacheable● GeoIP exception ○ lookup is done on the last step outside our infrastructure
    • Vary: Cookie?● Originally planned to send this out for responses using X-Cookie-...● Analytics cookie on site● Changes on each page...● Send responses out as uncacheable
    • Setting a Unique Cookie● Previously sent from backend● Generate unique ID cookie in Varnish● Allows cookie to be set and content served from cache
    • Feedback features...● How well is the cache being used?● Record per application hit/miss ratios
    • Big Sporting Event, 2012
    • Big Sporting Event, 2012"Dont f*** up the Olympics..."
    • Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applications
    • Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applicationsI think we can handle this...
    • Special preparations?
    • Special preparations?Reduced number of Varnish servers
    • Crosstown TrafficOlympics Daily Peak:● 10.4 million browsers to bbc.co.uk/sport● 8 million UK● 2.4 million International● (Record numbers)
    • Device SplitMainly mobile and desktop
    • So that went wellWhat didnt work for us?
    • Varnish and HD Streaming● 24 HD streams● Planned to use Varnish at the front● Cached very, very well● Needed to be highly available● HA layer didnt hold up● Had to use a load balancer instead and use the cache there
    • What else has hurt?ESI● Increase in complexity● Working out best practice● Seg faults! ○ Overflow of sess_workspace
    • However...● Synthetic end point generated in Varnish● Included as ESI● Very good performance... ○ Almost 4 times previous load
    • Other pains● No Saint mode ○ Load balancing behind and multiple apps● Network bandwidth ○ As few boxes as possible
    • Next?● Everywhere! ○ Ubiquitous caching layer ○ Already have most big players● More monitoring● Version 3 ○ VMODs?● Make it simpler ○ Remove anything we can
    • tl;drTook Varnish from being an application-specific component to a Platform-wide essential
    • Questions? Graham Lyons
    • Questions?(Yes, were hiring...) Graham Lyons