My Opera meets Varnish, Dec 2009

My Opera meets Varnish varnish high performance web caching cosimo@opera.com

What is Varnish? varnish ● Caching reverse proxy, like Squid backends ● Delegates memory mgmt to OS cache ● Mainly developed at Linpro in Oslo

Two typical Varnish setups incoming requests frontends varnish backends

VCL - Varnish Config Language • man vcl • VCL is compiled to C code • Injected into the running instance, without restart • Must define a backend or a director • VCL gives you several hooks: vcl_recv() vcl_hash() vcl_fetch() vcl_hit() vcl_miss() vcl_deliver()

Varnish deployment in My Opera • In production beginning of October 2009 • 1 old recycled machine, 2 Gb of disk allocated • Started serving avatars 1M+ requests per day before Unite http://my.opera.com/<username>/avatar.pl • Soon after, added Desktop Team RSS (very popular!) • then user pictures, hundreds of thousands req/day • then Unite/ASD API requests - friends of a user - groups of a user • In total, 13,25% of all My Opera requests are «varnished» • Around 7,2M req/day

Varnish deployment in My Opera Problems /1 • Still using Debian Etch? First Varnish instance was running v1.x from Etch. several years old, not good • Experienced VIPs – ”Very Interesting Problems” – User X getting User Y's session – Random users getting admin powers. Nightmare! • Theory: Varnish was caching response bodies that contained Set-Cookie: opera_session=<session_id>

Varnish deployment in My Opera Problems /2 • There wasn't any obvious configuration problem. Same config worked with 2.0.x from Backports. • v2.0.{4,5} is highly recommended!

Varnish deployment in My Opera Problems /3 • We tried caching the frontpage of My Opera, but had to revert the change due to too many different custom layouts for Opera Mobile, Mini, IE, Firefox, etc... • Maybe using clever vcl_hash() tricks we can achieve that too.

My Opera configuration

Backends and Directors • Backend single backend machine, or load-balanced virtual server • Director – simple round-robin or random weighted “balancing” logic – has basic connection retries mechanism – has basic backend health check • If you already have an LVS, define a single Backend Otherwise, go for the Director

Backends and Directors Define a backend # Only hit the upload servers backend myopera { .host = "upload.my.opera.com"; .port = "80"; }

Backends and Directors Define a director director myopera round-robin { .backend { .host = "b1.opera.com"; .port = "80"; } .backend { .host = "b2.opera.com"; .port = "80"; } ... }

Backends and Directors ...and then use them sub vcl_recv { ... set req.backend = myopera; ... }

vcl_recv() / 1 sub vcl_recv { set req.backend = myopera; set req.grace = 3m; # URL patterns based cache. # Avoid possible mixups. if(req.http.host !~ "^my.opera.com$") { pass; }

vcl_recv() / 2 if (req.url ~ "^/community/users/avatar.pl/[0-9]+$" || req.url ~ "^/.+/avatar.pl$" || req.url ~ "^/.+/picture.pl?xscale=100$" || req.url ~ "^/desktopteam/xml/atom/blog/?$" || req.url ~ "^/desktopteam/xml/rss/blog/?$" || req.url ~ "^/community/api/users/friends.pl?user=.+$" || req.url ~ "^/community/api/users/groups.pl?user=.+$" ) { unset req.http.Cookie; unset req.http.Authorization; lookup; }

vcl_recv() / 3 ... # Check for cookie only after always-cache URLs if (req.http.Cookie ~ "(opera_session|opera_persistent_)") { pass; } # DANGER, Will Robinson! Caching the front-page # At this point, lots of Google Analytics cookies will go in. # No problem. It's stuff used by Javascript if (req.url ~ "^/community/$") { lookup; } pass; }

vcl_fetch() / 1 sub vcl_fetch { set obj.http.X-Varnish-URL = req.url; set obj.grace = 3m; if (obj.http.Set-Cookie) { set obj.http.X-Varnish-Cacheable = "no, set-cookie"; pass; } if (req.request != "GET") { set obj.http.X-Varnish-Cacheable = "no, !GET"; pass; }

vcl_fetch() / 2 if (req.http.host !~ "^my.opera.com$") { set obj.http.X-Varnish-Cacheable = "no, !my.opera.com"; pass; } if (req.url ~ "^/community/users/avatar.pl/[0-9]+$" || req.url ~ "^/[A-Za-z0-9]+/avatar.pl$" || ... ) { unset obj.http.Set-Cookie; set obj.http.X-Varnish-Cacheable = "yes, url"; set obj.ttl = 24h; deliver; }

vcl_hash() sub vcl_hash { # Default Varnish behavior set req.hash += req.url; set req.hash += req.http.host; # Have a different cached frontpage per language if (req.url ~ "^/community/$") { set req.http.X-FrontPage-Language = regsub( req.http.Cookie, "^.*?language=([^;]*?);*.*$", "1" ); set req.hash += "lang:"; set req.hash += req.http.X-FrontPage-Language; } hash; }

Testing Varnish how to avoid nightmares... • Developed a testing tool (varnish-test) – outputs a TAP stream and some debug info – works best if varnish is specially tuned • Can quickly check if a test/production instance is performing correctly or having problems • Invoked as a simple script: va rnis h-tes t --profile=tes ts .url --hos t=b1

Testing Varnish caching test list # Fro ntpa g e / N O _C O O K I E S V A R N I S H _C A C H E D / N O _C O O K I E S V A R N I S H _N O T _C A C H E D H o s t: m y.c n.o pera .c o m / N O _C O O K I E S V A R N I S H _C A C H E D C o o k ie:la ng ua g e=it # B lo g s /des k to ptea m /blo g / N O _C O O K I E S V A R N I S H _N O T _C A C H E D # A va ta rs /c o m m unity/us ers /a va ta r/817271 N O _C O O K I E S V A R N I S H _C A C H E D /c o m m unity/us ers /a va ta r/442 N O _C O O K I E S V A R N I S H _C A C H E D /g ra phic s /a va ta r.g if N O _C O O K I E S V A R N I S H _N O T _C A C H E D

Testing Varnish caching test list • We can specify exactly how the varnish instance should behave. – Production acceptance tests – Test new varnish versions, new OS distributions – Fine tune config changes quickly with no impact on production • Midway through there's a request that logs in as a test user. From then on, we can verify what resources are cached when a user is logged in. Some resources should be cached in any case.

Testing Varnish sample run ... ok 289 - Got response from backend for /community/ (from ...) ok 290 - Correct status line # Adding header [Cookie] => [language=it] # ---------- # GET http://cache01.my.opera.com:6081/community/ # Host: my.opera.com # ------------ ok 291 - 2nd request: got response from backend for /community/ (from...) ok 292 - Correct status line X-Varnish: 1211283813 1211283812 # X-Varnish: 1211283813 1211283812 X-Varnish-Status: hit # X-Varnish-Status: hit # X-Varnish-Cacheable: yes, language cookie X-Varnish-Cacheable: yes, language cookie # X-Varnish-URL: /community/ X-Varnish-URL: /community/ ok 293 - URL '/community/' was handled correctly by varnish # cookie_header: ok 294 - URL '/community/' has correct cookies (or no cookies) 1..294 All tests successful.

Monitoring Varnish built-in tools • varnishlog – Reads shared memory log info and displays it – Full instance log, on My Opera, 1 day is about 15 Gb – You can get an emulated Apache-style access.log from it • varnishncsa – Displays requests to Varnish as Apache access logs – Can read from an archived log by varnishlog • varnishstat – Displays realtime stats (hit ratio, space allocated, connections,...)

Monitoring Varnish external tools • Munin plugins – Hit ratio – Requests rate – Backend traffic • Nagios plugins – Nothing special, TCP connection to port 6081

Monitoring Varnish

Next steps • My Opera front page caching • My Opera files server? • Working on a prototype thumbnail server

References and more information • Redpill-Linpro website – http://varnish.projects.linpro.no – Bug tracking, documentation and community support – Users and developers mailing lists • Commercial support and training – http://www.varnish-cache.com

Questions? • At Opera, there's several teams using Varnish in production • If you want to know more, contact me: cosimo@opera.com

http://axonflux.com	212
http://www.slideshare.net	8
http://feeds.feedburner.com	6
http://posterous.com	2

My Opera meets Varnish, Dec 2009

by Cosimo Streppone on Dec 30, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 228

Statistics

My Opera meets Varnish, Dec 2009 — Presentation Transcript