My Opera meets Varnish, Dec 2009

2,951 views

Published on

Slide for a talk I presented internally at Opera in December 2009 about the deployment of varnish in our production environment at my.opera.com, the social network community.

Published in: Technology
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total views
2,951
On SlideShare
0
From Embeds
0
Number of Embeds
322
Actions
Shares
0
Downloads
16
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

My Opera meets Varnish, Dec 2009

  1. 1. My Opera meets Varnish varnish high performance web caching cosimo@opera.com
  2. 2. What is Varnish? varnish ● Caching reverse proxy, like Squid backends ● Delegates memory mgmt to OS cache ● Mainly developed at Linpro in Oslo
  3. 3. Two typical Varnish setups incoming requests frontends varnish backends
  4. 4. VCL - Varnish Config Language • man vcl • VCL is compiled to C code • Injected into the running instance, without restart • Must define a backend or a director • VCL gives you several hooks: vcl_recv() vcl_hash() vcl_fetch() vcl_hit() vcl_miss() vcl_deliver()
  5. 5. Varnish deployment in My Opera • In production beginning of October 2009 • 1 old recycled machine, 2 Gb of disk allocated • Started serving avatars 1M+ requests per day before Unite http://my.opera.com/<username>/avatar.pl • Soon after, added Desktop Team RSS (very popular!) • then user pictures, hundreds of thousands req/day • then Unite/ASD API requests - friends of a user - groups of a user • In total, 13,25% of all My Opera requests are «varnished» • Around 7,2M req/day
  6. 6. Varnish deployment in My Opera Problems /1 • Still using Debian Etch? First Varnish instance was running v1.x from Etch. several years old, not good • Experienced VIPs – ”Very Interesting Problems” – User X getting User Y's session – Random users getting admin powers. Nightmare! • Theory: Varnish was caching response bodies that contained Set-Cookie: opera_session=<session_id>
  7. 7. Varnish deployment in My Opera Problems /2 • There wasn't any obvious configuration problem. Same config worked with 2.0.x from Backports. • v2.0.{4,5} is highly recommended!
  8. 8. Varnish deployment in My Opera Problems /3 • We tried caching the frontpage of My Opera, but had to revert the change due to too many different custom layouts for Opera Mobile, Mini, IE, Firefox, etc... • Maybe using clever vcl_hash() tricks we can achieve that too.
  9. 9. My Opera configuration
  10. 10. Backends and Directors • Backend single backend machine, or load-balanced virtual server • Director – simple round-robin or random weighted “balancing” logic – has basic connection retries mechanism – has basic backend health check • If you already have an LVS, define a single Backend Otherwise, go for the Director
  11. 11. Backends and Directors Define a backend # Only hit the upload servers backend myopera { .host = "upload.my.opera.com"; .port = "80"; }
  12. 12. Backends and Directors Define a director director myopera round-robin { .backend { .host = "b1.opera.com"; .port = "80"; } .backend { .host = "b2.opera.com"; .port = "80"; } ... }
  13. 13. Backends and Directors ...and then use them sub vcl_recv { ... set req.backend = myopera; ... }
  14. 14. vcl_recv() / 1 sub vcl_recv { set req.backend = myopera; set req.grace = 3m; # URL patterns based cache. # Avoid possible mixups. if(req.http.host !~ "^my.opera.com$") { pass; }
  15. 15. vcl_recv() / 2 if (req.url ~ "^/community/users/avatar.pl/[0-9]+$" || req.url ~ "^/.+/avatar.pl$" || req.url ~ "^/.+/picture.pl?xscale=100$" || req.url ~ "^/desktopteam/xml/atom/blog/?$" || req.url ~ "^/desktopteam/xml/rss/blog/?$" || req.url ~ "^/community/api/users/friends.pl?user=.+$" || req.url ~ "^/community/api/users/groups.pl?user=.+$" ) { unset req.http.Cookie; unset req.http.Authorization; lookup; }
  16. 16. vcl_recv() / 3 ... # Check for cookie only after always-cache URLs if (req.http.Cookie ~ "(opera_session|opera_persistent_)") { pass; } # DANGER, Will Robinson! Caching the front-page # At this point, lots of Google Analytics cookies will go in. # No problem. It's stuff used by Javascript if (req.url ~ "^/community/$") { lookup; } pass; }
  17. 17. vcl_fetch() / 1 sub vcl_fetch { set obj.http.X-Varnish-URL = req.url; set obj.grace = 3m; if (obj.http.Set-Cookie) { set obj.http.X-Varnish-Cacheable = "no, set-cookie"; pass; } if (req.request != "GET") { set obj.http.X-Varnish-Cacheable = "no, !GET"; pass; }
  18. 18. vcl_fetch() / 2 if (req.http.host !~ "^my.opera.com$") { set obj.http.X-Varnish-Cacheable = "no, !my.opera.com"; pass; } if (req.url ~ "^/community/users/avatar.pl/[0-9]+$" || req.url ~ "^/[A-Za-z0-9]+/avatar.pl$" || ... ) { unset obj.http.Set-Cookie; set obj.http.X-Varnish-Cacheable = "yes, url"; set obj.ttl = 24h; deliver; }
  19. 19. vcl_hash() sub vcl_hash { # Default Varnish behavior set req.hash += req.url; set req.hash += req.http.host; # Have a different cached frontpage per language if (req.url ~ "^/community/$") { set req.http.X-FrontPage-Language = regsub( req.http.Cookie, "^.*?language=([^;]*?);*.*$", "1" ); set req.hash += "lang:"; set req.hash += req.http.X-FrontPage-Language; } hash; }
  20. 20. Testing Varnish how to avoid nightmares... • Developed a testing tool (varnish-test) – outputs a TAP stream and some debug info – works best if varnish is specially tuned • Can quickly check if a test/production instance is performing correctly or having problems • Invoked as a simple script: va rnis h-tes t --profile=tes ts .url --hos t=b1
  21. 21. Testing Varnish caching test list # Fro ntpa g e / N O _C O O K I E S V A R N I S H _C A C H E D / N O _C O O K I E S V A R N I S H _N O T _C A C H E D H o s t: m y.c n.o pera .c o m / N O _C O O K I E S V A R N I S H _C A C H E D C o o k ie:la ng ua g e=it # B lo g s /des k to ptea m /blo g / N O _C O O K I E S V A R N I S H _N O T _C A C H E D # A va ta rs /c o m m unity/us ers /a va ta r/817271 N O _C O O K I E S V A R N I S H _C A C H E D /c o m m unity/us ers /a va ta r/442 N O _C O O K I E S V A R N I S H _C A C H E D /g ra phic s /a va ta r.g if N O _C O O K I E S V A R N I S H _N O T _C A C H E D
  22. 22. Testing Varnish caching test list • We can specify exactly how the varnish instance should behave. – Production acceptance tests – Test new varnish versions, new OS distributions – Fine tune config changes quickly with no impact on production • Midway through there's a request that logs in as a test user. From then on, we can verify what resources are cached when a user is logged in. Some resources should be cached in any case.
  23. 23. Testing Varnish sample run ... ok 289 - Got response from backend for /community/ (from ...) ok 290 - Correct status line # Adding header [Cookie] => [language=it] # ---------- # GET http://cache01.my.opera.com:6081/community/ # Host: my.opera.com # ------------ ok 291 - 2nd request: got response from backend for /community/ (from...) ok 292 - Correct status line X-Varnish: 1211283813 1211283812 # X-Varnish: 1211283813 1211283812 X-Varnish-Status: hit # X-Varnish-Status: hit # X-Varnish-Cacheable: yes, language cookie X-Varnish-Cacheable: yes, language cookie # X-Varnish-URL: /community/ X-Varnish-URL: /community/ ok 293 - URL '/community/' was handled correctly by varnish # cookie_header: ok 294 - URL '/community/' has correct cookies (or no cookies) 1..294 All tests successful.
  24. 24. Monitoring Varnish built-in tools • varnishlog – Reads shared memory log info and displays it – Full instance log, on My Opera, 1 day is about 15 Gb – You can get an emulated Apache-style access.log from it • varnishncsa – Displays requests to Varnish as Apache access logs – Can read from an archived log by varnishlog • varnishstat – Displays realtime stats (hit ratio, space allocated, connections,...)
  25. 25. Monitoring Varnish external tools • Munin plugins – Hit ratio – Requests rate – Backend traffic • Nagios plugins – Nothing special, TCP connection to port 6081
  26. 26. Monitoring Varnish
  27. 27. Monitoring Varnish
  28. 28. Monitoring Varnish
  29. 29. Next steps • My Opera front page caching • My Opera files server? • Working on a prototype thumbnail server
  30. 30. References and more information • Redpill-Linpro website – http://varnish.projects.linpro.no – Bug tracking, documentation and community support – Users and developers mailing lists • Commercial support and training – http://www.varnish-cache.com
  31. 31. Questions? • At Opera, there's several teams using Varnish in production • If you want to know more, contact me: cosimo@opera.com

×