Varnish at the BBCWinning Gold in the London 2012 Olympic Games                                  Graham Lyons
Varnish at the BBC● First deployed in 2009   ○ Specifically caching layer for iPlayer   ○ New dynamic Platform● Platform h...
In the BBC Infrastructure● bbc.co.uk is made up of lots of applications● Load balancer in front● Sends request to Varnish●...
How it looks
Routing● First load balancer adds header with name of  a pool of servers● Varnish forwards it on● Second load balancer kno...
How do we use Varnish● General HTTP cache● Make use of header manipulation for more  efficient caching, e.g.  ○ GeoIP  ○ D...
In 2009...● Application logic in VCL● Very small number of applications so it was  manageable
Where should we take it?● BBC Platform HTTP cache● Platform-wide features● Different requirements to application-  specifi...
...2012 (What we changed)● Removed application logic (mostly)● Added features to be used generally  ○ e.g. GeoIP, Device d...
Features? What features?● GeoIP lookup● Device meta information● Cookie decomposition  ○ Signed in headerAll exposed as he...
Geo and Device Information● Looked up via an HTTP call to respective  services● Logic in C library● Cached locally (in pro...
Cookies?● Incoming Cookie header split into a header  for each value● e.g. Cookie: UID=4321...   ○ ...becomes: X-Cookie-UI...
Signed in header● Boolean   ○ Signed in   ○ Not signed in● Allows caching of page for not signed in  state
Cache VariationsAll these features allow more efficient cachevariations.Can cache variations based on:● where the user is●...
Response to outside world● External caches dont know about request  headers Varnish adds● Responses have to be reduced to ...
Vary: Cookie?● Originally planned to send this out for  responses using X-Cookie-...● Analytics cookie on site● Changes on...
Setting a Unique Cookie● Previously sent from backend● Generate unique ID cookie in Varnish● Allows cookie to be set and c...
Feedback features...● How well is the cache being used?● Record per application hit/miss ratios
Big Sporting Event, 2012
Big Sporting Event, 2012"Dont f*** up the Olympics..."
Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applications
Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applicationsI think ...
Special preparations?
Special preparations?Reduced number of Varnish servers
Crosstown TrafficOlympics Daily Peak:● 10.4 million browsers to bbc.co.uk/sport● 8 million UK● 2.4 million International● ...
Device SplitMainly mobile and desktop
So that went wellWhat didnt work for us?
Varnish and HD Streaming●   24 HD streams●   Planned to use Varnish at the front●   Cached very, very well●   Needed to be...
What else has hurt?ESI● Increase in complexity● Working out best practice● Seg faults!  ○ Overflow of sess_workspace
However...● Synthetic end point generated in Varnish● Included as ESI● Very good performance...  ○ Almost 4 times previous...
Other pains● No Saint mode  ○ Load balancing behind and multiple apps● Network bandwidth  ○ As few boxes as possible
Next?● Everywhere!  ○ Ubiquitous caching layer  ○ Already have most big players● More monitoring● Version 3  ○ VMODs?● Mak...
tl;drTook Varnish from being an application-specific component to a Platform-wide essential
Questions?             Graham Lyons
Questions?(Yes, were hiring...)                         Graham Lyons
Upcoming SlideShare
Loading in...5
×

Varnish at the BBC

1,204

Published on

Presentation on how the BBC uses Varnish, from VUG6 (https://www.varnish-cache.org/vug6)

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,204
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Varnish at the BBC

  1. 1. Varnish at the BBCWinning Gold in the London 2012 Olympic Games Graham Lyons
  2. 2. Varnish at the BBC● First deployed in 2009 ○ Specifically caching layer for iPlayer ○ New dynamic Platform● Platform has grown to 100s of applicationsHow do we scale Varnish across the Platform?(It served LOTS of traffic during the Olympics)
  3. 3. In the BBC Infrastructure● bbc.co.uk is made up of lots of applications● Load balancer in front● Sends request to Varnish● Varnish sends request to another load balancer● Second layer of load balancer distributes load across application servers ○ All applications installed on all servers
  4. 4. How it looks
  5. 5. Routing● First load balancer adds header with name of a pool of servers● Varnish forwards it on● Second load balancer knows what to do with the header to route the request
  6. 6. How do we use Varnish● General HTTP cache● Make use of header manipulation for more efficient caching, e.g. ○ GeoIP ○ Device detection ○ Cookie decomposition
  7. 7. In 2009...● Application logic in VCL● Very small number of applications so it was manageable
  8. 8. Where should we take it?● BBC Platform HTTP cache● Platform-wide features● Different requirements to application- specific Varnish
  9. 9. ...2012 (What we changed)● Removed application logic (mostly)● Added features to be used generally ○ e.g. GeoIP, Device detection● Features on by default - no special configuration● Try to stay vanilla and RFC2616(ish)
  10. 10. Features? What features?● GeoIP lookup● Device meta information● Cookie decomposition ○ Signed in headerAll exposed as headers added to the requestCompanion PHP libraries to manage headeraccess and Vary header on response
  11. 11. Geo and Device Information● Looked up via an HTTP call to respective services● Logic in C library● Cached locally (in process, in memory cache) ○ 70% hit for geoip ○ >95% hit for device data
  12. 12. Cookies?● Incoming Cookie header split into a header for each value● e.g. Cookie: UID=4321... ○ ...becomes: X-Cookie-UID: 4321Actually only operates on cookie values withparticular prefixes (introduced for the Great EUCookie Debacle)
  13. 13. Signed in header● Boolean ○ Signed in ○ Not signed in● Allows caching of page for not signed in state
  14. 14. Cache VariationsAll these features allow more efficient cachevariations.Can cache variations based on:● where the user is● what type of device theyre using● any personalisationse.g. Norwegian Android user who lovesEastenders gets served straight from the cache
  15. 15. Response to outside world● External caches dont know about request headers Varnish adds● Responses have to be reduced to being privately cacheable● GeoIP exception ○ lookup is done on the last step outside our infrastructure
  16. 16. Vary: Cookie?● Originally planned to send this out for responses using X-Cookie-...● Analytics cookie on site● Changes on each page...● Send responses out as uncacheable
  17. 17. Setting a Unique Cookie● Previously sent from backend● Generate unique ID cookie in Varnish● Allows cookie to be set and content served from cache
  18. 18. Feedback features...● How well is the cache being used?● Record per application hit/miss ratios
  19. 19. Big Sporting Event, 2012
  20. 20. Big Sporting Event, 2012"Dont f*** up the Olympics..."
  21. 21. Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applications
  22. 22. Olympic Requirements● UK and non-UK versions● Mobile and Desktop versions● Traffic served by multiple applicationsI think we can handle this...
  23. 23. Special preparations?
  24. 24. Special preparations?Reduced number of Varnish servers
  25. 25. Crosstown TrafficOlympics Daily Peak:● 10.4 million browsers to bbc.co.uk/sport● 8 million UK● 2.4 million International● (Record numbers)
  26. 26. Device SplitMainly mobile and desktop
  27. 27. So that went wellWhat didnt work for us?
  28. 28. Varnish and HD Streaming● 24 HD streams● Planned to use Varnish at the front● Cached very, very well● Needed to be highly available● HA layer didnt hold up● Had to use a load balancer instead and use the cache there
  29. 29. What else has hurt?ESI● Increase in complexity● Working out best practice● Seg faults! ○ Overflow of sess_workspace
  30. 30. However...● Synthetic end point generated in Varnish● Included as ESI● Very good performance... ○ Almost 4 times previous load
  31. 31. Other pains● No Saint mode ○ Load balancing behind and multiple apps● Network bandwidth ○ As few boxes as possible
  32. 32. Next?● Everywhere! ○ Ubiquitous caching layer ○ Already have most big players● More monitoring● Version 3 ○ VMODs?● Make it simpler ○ Remove anything we can
  33. 33. tl;drTook Varnish from being an application-specific component to a Platform-wide essential
  34. 34. Questions? Graham Lyons
  35. 35. Questions?(Yes, were hiring...) Graham Lyons

×