0
VARNISH-CACHE *                         Lessons Learnt from Performance Testing* https://www.varnish-cache.orgTuesday, 25 ...
WHY? FAST!    • If  your throughput is currently measured in the hundreds per        second, think of varnish in the thous...
GET OUT OF JAIL. FREE.    • Crappy             cache control?    • Expensive            CDNs? 20¢/GB?    • Not           e...
IT WON’T    • Do   HTTPS... terminate SSL with something else, hopefully        your load balancers.Tuesday, 25 October 11
INSTALL THE LATEST    • RHEL               / CentOS    rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3...
USE 64BIT*                         “It’s all about virtual memory: Things like                         stack size suddenly...
SYSTEM CONFIG    • The           minimum.          echo fs.file-max=1000000 >> /etc/sysctl.conf          echo net.ipv4.ip_...
THREADS    • Adjust             default thread* settings in /etc/sysconfig/varnish        thread_pool_add_delay=2        t...
STORAGE    • Memory             is fast, duh.    • Got    enough RAM to store the cache?        Yes! then use -s malloc   ...
VCL    • Varnish            Configuration Language    • Looks like C and compiled into C but       ...      ~ “has regex ti...
VCL    • Default VCL is defined in      /etc/varnish/default.vcl    • Then create your custom VCL                  in      ...
LOOKUP AND MISS   • request              received from client            recv   • hash           request                  ...
LOOKUP AND HIT   • request              received from client            recv   • hash           request                   ...
PASS AND DELIVER   • request              received from client            recv                                            ...
PIPE   • request             received from client            recv   • pipe  to backend until client or             hash   ...
PROBES • Define    common health          probe health {                                     .url = "/healthCheck?onFailure...
BACKENDS    • Define    backend host, port        and health checks inline.         { .backend = { .host = "10.10.172.1"; ....
DIRECTORS    • Group      backends together using different algorithms.          Random          Client (based on identity...
YMMV    •I    found it better to balance        in front of Varnish using a        hardware load balancer ...    • Random ...
DESIGN DECISIONS    • Share   nothing.        Easy to scale horizontally.        Great if your site is not ‘transactional’...
DESIGN DECISIONS    • Share   everything.        Varnish loves RAM, scale vertically.        Varnish cache becomes the sha...
SHARE NOTHING. FALLBACK.         director varnishA fallback {           { .backend = { .host = "a"; } }                  L...
TERMINATE SSL.    • Get   the LB to set a Port header.        Useful when user switches from HTTP to HTTPS so varnish     ...
VCL TRICKS   • Rewrite headers to/from production.com.au                                       to     testdomain.com   •  ...
VCL TRICKS    • Add some debug               headers for hit/miss in firebug in      vcl_deliver        if (obj.hits > 0) {...
VCL TRICKS    • Normalize the Accept-Encoding                       headers to reduce cache      usage in vcl_recv       i...
VCL TRICKS    • Remove             Google Analytics cookies in vcl_recv      set req.http.Cookie = regsuball(req.http.Cook...
VCL TRICKS    • Pass  on the client IP to the backend servers using the X-        Forwarded-For header in vcl_recv        ...
VCL TRICKS    • Let varnish keep           objects for 2 hours beyond their TTL in      vcl_fetch        set beresp.grace ...
VCL TRICKS    • Force  a grace on error. Restart keyword can be called any        where and will return request all the wa...
TTL    • Varnish            will obey cache control headers from backend        Cache-Control: s-maxage        Cache-Contr...
REDUCING CACHE SIZE • Normalize                  / sanitize request headers • Set         Content-Length on backends • Ser...
MONITORING VARNISH    • Handy command          line tools      varnishstat      varnishlog      varnishtop      varnishadm...
VARNISHSTAT    •   varnishstat        Hit rate % (x100)        Average per second        Average/sec since boot        Sum...
VARNISHSTAT    • Log          all statistics to disk ...         #!/bin/bash         logdir=/var/log/perf/`hostname`/lates...
VARNISHLOG    •   varnishlog debugging (-d to dump shared memory)          12   BackendOpen    b   xx_80[0] x.x.x.x 45546 ...
VARNISHTOP    • Monitor top URLs hitting   backend      varnishtop -i txurl    • Monitor top received URLs      varnishtop...
VARNISHADM  •   varnishadm for administration   # varnishadm   CLI connected to 127.0.0.1 6082   200   -------------------...
MORE HELP?    • Talk   to the devs, read their blogs.        http://kristianlyng.wordpress.com        http://ingvar.blog.r...
Upcoming SlideShare
Loading in...5
×

Caching with varnish

12,607

Published on

Published in: Technology
0 Comments
20 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
12,607
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
189
Comments
0
Likes
20
Embeds 0
No embeds

No notes for slide

Transcript of "Caching with varnish"

  1. 1. VARNISH-CACHE * Lessons Learnt from Performance Testing* https://www.varnish-cache.orgTuesday, 25 October 11
  2. 2. WHY? FAST! • If your throughput is currently measured in the hundreds per second, think of varnish in the thousands per second. • Theoretical limits in hundreds of thousands per second.** http://kristianlyng.wordpress.com/2010/10/23/275k-req/Tuesday, 25 October 11
  3. 3. GET OUT OF JAIL. FREE. • Crappy cache control? • Expensive CDNs? 20¢/GB? • Not enough hardware? • Backend struggling?Tuesday, 25 October 11
  4. 4. IT WON’T • Do HTTPS... terminate SSL with something else, hopefully your load balancers.Tuesday, 25 October 11
  5. 5. INSTALL THE LATEST • RHEL / CentOS rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish- release-3.0-1.noarch.rpm yum install -y varnish • Debian apt-get install varnishTuesday, 25 October 11
  6. 6. USE 64BIT* “It’s all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, you’re somewhat on your own. However, try to fit it within 2GB. I wouldn’t recommend a cache larger than 1GB, and no more than a few hundred threads… (Why are you on 32bit again?)”* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/Tuesday, 25 October 11
  7. 7. SYSTEM CONFIG • The minimum. echo fs.file-max=1000000 >> /etc/sysctl.conf echo net.ipv4.ip_local_port_range = 1024 65535 >> /etc/sysctl.conf sysctl -p echo * soft nofile 1000000 >> /etc/security/limits.conf echo * hard nofile 1000000 >> /etc/security/limits.conf echo session required pam_limits.so >> /etc/pam.d/login ulimit -n 1000000 ulimit -n -H • Do not set “tw_reuse” to 1 (sysctl). It will cause problems for NAT based clients. • Enable keepalives on load balancers.Tuesday, 25 October 11
  8. 8. THREADS • Adjust default thread* settings in /etc/sysconfig/varnish thread_pool_add_delay=2 thread_pools = <Number of cpu cores> thread_pool_min = <800/number of cpu cores> thread_pool_max = 4000 session_linger = 50 sess_workspace = <16k to 5m> “The only thing that made a real difference while tuning Varnish was the number of threads. And making sure it actually caches. Beyond that, it really doesn’t matter much. Our defaults are good.”* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/Tuesday, 25 October 11
  9. 9. STORAGE • Memory is fast, duh. • Got enough RAM to store the cache? Yes! then use -s malloc Unsure? then use -s fileTuesday, 25 October 11
  10. 10. VCL • Varnish Configuration Language • Looks like C and compiled into C but ... ~ “has regex tilda operator” Assignments use set keyword. Only has conditional if tests. Has a bunch of variables and some built in functions. • Defined by backends, probes, ACLs and subroutines which hook into the varnish workflow.Tuesday, 25 October 11
  11. 11. VCL • Default VCL is defined in /etc/varnish/default.vcl • Then create your custom VCL in -f /var/custom.vcl so varnish can fall through to the default.vcl Can also include "/var/other.vcl"; • Keep it simple within the VCL workflow* Do most of the work in vcl_recv and vcl_fetch* https://www.varnish-cache.org/trac/wiki/VCLExampleDefaultTuesday, 25 October 11
  12. 12. LOOKUP AND MISS • request received from client recv • hash request hash pipe • lookup object and miss hit miss pass • fetch response from backend fetch • deliver to client deliverTuesday, 25 October 11
  13. 13. LOOKUP AND HIT • request received from client recv • hash request hash pipe • lookup object and hit hit miss pass fetch • deliver to client deliverTuesday, 25 October 11
  14. 14. PASS AND DELIVER • request received from client recv hash pipe • pass through to backend hit miss pass • fetch response fetch • deliver to client deliverTuesday, 25 October 11
  15. 15. PIPE • request received from client recv • pipe to backend until client or hash pipe backend closes hit miss pass fetch deliverTuesday, 25 October 11
  16. 16. PROBES • Define common health probe health {   .url = "/healthCheck?onFailure=true"; checks for multiple   .expected_response = 200;   .interval  = 15s;         backends.   .timeout   = 5s;             .window    = 4;             .threshold = 4;           } • window is how many of the latest polls are probed for health. • threshold is how many polls must succeed to be healthy.Tuesday, 25 October 11
  17. 17. BACKENDS • Define backend host, port and health checks inline. { .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } } • Or named. backend mysite {   .host = "www.example.com";   .port = "http"; .probe = "health"; }Tuesday, 25 October 11
  18. 18. DIRECTORS • Group backends together using different algorithms. Random Client (based on identity) Hash (based on url) Round robin FallbackTuesday, 25 October 11
  19. 19. YMMV •I found it better to balance in front of Varnish using a hardware load balancer ... • Random seemed a little too random using Varnish directors however was probably an artefact of the load generation.Tuesday, 25 October 11
  20. 20. DESIGN DECISIONS • Share nothing. Easy to scale horizontally. Great if your site is not ‘transactional’. Backend session affinity via LB set cookie. LB Set-Cookie=BackendID varnish varnish varnish app app appTuesday, 25 October 11
  21. 21. DESIGN DECISIONS • Share everything. Varnish loves RAM, scale vertically. Varnish cache becomes the shared source of truth. LB varnish app app appTuesday, 25 October 11
  22. 22. SHARE NOTHING. FALLBACK. director varnishA fallback {   { .backend = { .host = "a"; } } LB Set-Cookie=VarnishID   { .backend = { .host = "b"; } }   { .backend = { .host = "c"; } }    } director varnishB fallback { varnish varnish varnish   { .backend = { .host = "b"; } }   { .backend = { .host = "a"; } }   { .backend = { .host = "c"; } }    } director varnishC fallback {   { .backend = { .host = "c"; } }   { .backend = { .host = "b"; } } app app app   { .backend = { .host = "a"; } }    }Tuesday, 25 October 11
  23. 23. TERMINATE SSL. • Get the LB to set a Port header. Useful when user switches from HTTP to HTTPS so varnish knows which backend port to route to. LB Set-Cookie=VarnishID Port=https varnish app :80 :443 if (req.http.Port ~ "https") { set req.backend = appa_80; } else {   set req.backend = appa_443; }Tuesday, 25 October 11
  24. 24. VCL TRICKS • Rewrite headers to/from production.com.au to testdomain.com • vcl_recv set req.http.Host = regsuball(req.http.Host, "testdomain.com", "com.au"); set req.http.Referer = regsuball(req.http.Host, "testdomain.com", "com.au"); • vcl_deliver set resp.http.Host = regsuball(resp.http.Host, ".com.au", ".testdomain.com"); set resp.http.Referer = regsuball(resp.http.Host, ".com.au", ".testdomain.com");   set resp.http.Location = regsuball(resp.http.Location, ".com.au", ".testdomain.com"); set resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");Tuesday, 25 October 11
  25. 25. VCL TRICKS • Add some debug headers for hit/miss in firebug in vcl_deliver if (obj.hits > 0) {   set resp.http.X-Varnish-Hit = "++";   set resp.http.X-Varnish-Hits = obj.hits;   set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(d+).*", "1"); } else {   set resp.http.X-Varnish-Hit = "--";   set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(d+).*", "1"); }Tuesday, 25 October 11
  26. 26. VCL TRICKS • Normalize the Accept-Encoding headers to reduce cache usage in vcl_recv if (req.http.Accept-Encoding) {   if (req.url ~ ".(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {     remove req.http.Accept-Encoding;   } elsif (req.http.Accept-Encoding ~ "gzip") {     set req.http.Accept-Encoding = "gzip";   } elsif (req.http.Accept-Encoding ~ "deflate") {     set req.http.Accept-Encoding = "deflate";   } else {     remove req.http.Accept-Encoding;   } }Tuesday, 25 October 11
  27. 27. VCL TRICKS • Remove Google Analytics cookies in vcl_recv set req.http.Cookie = regsuball(req.http.Cookie, "(^|;s*)(__[a-z]+|has_js)=[^;]*", ""); set req.http.Cookie = regsub(req.http.Cookie, "^;s*", ""); • Remember, varnish will pass any request with cookies by default. You will need to over-ride this behaviour if your web app uses cookies. • Allow lookup of requests with basic auth (in test) or cookies in vcl_recv but beware of creating a per-user-session store. if (req.http.Authorization || req.http.Cookie) { return (lookup); }Tuesday, 25 October 11
  28. 28. VCL TRICKS • Pass on the client IP to the backend servers using the X- Forwarded-For header in vcl_recv remove req.http.X-Forwarded-For; set    req.http.X-Forwarded-For = client.ip;Tuesday, 25 October 11
  29. 29. VCL TRICKS • Let varnish keep objects for 2 hours beyond their TTL in vcl_fetch set beresp.grace = 2h; • Let varnish serve stale objects up to 30 secs old for healthy servers, otherwise 2 hours in vcl_recv   if (! req.backend.healthy) {     set req.grace = 2h;   } else {     set req.grace = 30s;   } • Can also use saint mode to disable backends for a period of time, otherwise if all backends unavailable, use grace.Tuesday, 25 October 11
  30. 30. VCL TRICKS • Force a grace on error. Restart keyword can be called any where and will return request all the way back to vcl_recv sub vcl_error {   if (req.restarts == 0) {     set req.http.X-Serve-Graced = "1";     return(restart);   } }Tuesday, 25 October 11
  31. 31. TTL • Varnish will obey cache control headers from backend Cache-Control: s-maxage Cache-Control: max-age • Can also override TTL control in VCL, but best do it from the backend set obj.ttl = xTuesday, 25 October 11
  32. 32. REDUCING CACHE SIZE • Normalize / sanitize request headers • Set Content-Length on backends • Serve (static) digital assets from VFS cache • Pipe large objects like videosTuesday, 25 October 11
  33. 33. MONITORING VARNISH • Handy command line tools varnishstat varnishlog varnishtop varnishadmTuesday, 25 October 11
  34. 34. VARNISHSTAT • varnishstat Hit rate % (x100) Average per second Average/sec since boot Sum total • Filtered varnishstat -f client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_failTuesday, 25 October 11
  35. 35. VARNISHSTAT • Log all statistics to disk ... #!/bin/bash logdir=/var/log/perf/`hostname`/latest mkdir -p $logdir   while true do   varnishstat -x >> "$logdir/varnishstat.xml"   sleep 60 done • Then graph as you see fit ...Tuesday, 25 October 11
  36. 36. VARNISHLOG • varnishlog debugging (-d to dump shared memory) 12 BackendOpen b xx_80[0] x.x.x.x 45546 x.x.x.x 80 12 TxRequest b GET 12 TxURL b / 12 TxProtocol b HTTP/1.1 12 TxHeader b Host: 12 TxHeader b Referer: 12 TxHeader b Cookie: 12 TxHeader b X-Forwarded-For: 1.2.3.4 12 TxHeader b X-Varnish: 2092287889 12 TxHeader b Accept-Encoding: gzip 12 RxProtocol b HTTP/1.1 12 RxStatus b 401 12 RxResponse b Authorization Required 12 RxHeader b Date: Mon, 24 Oct 2011 05:46:18 GMT 12 RxHeader b WWW-Authenticate: Basic realm="XX Authorised Personel Only" 12 RxHeader b Vary: Accept-Encoding 12 RxHeader b Content-Encoding: gzip 12 RxHeader b Content-Length: 336 12 RxHeader b Connection: close 12 RxHeader b Content-Type: text/html; charset=iso-8859-1 12 Fetch_Body b 4 0 1 12 Length b 336 12 BackendClose b xx_80[0]Tuesday, 25 October 11
  37. 37. VARNISHTOP • Monitor top URLs hitting backend varnishtop -i txurl • Monitor top received URLs varnishtop -i rxurlTuesday, 25 October 11
  38. 38. VARNISHADM • varnishadm for administration # varnishadm CLI connected to 127.0.0.1 6082 200 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit Type help for command list. Type quit to close CLI session. varnish> help 200 help [command] ping [timestamp] auth response quit banner status start stop vcl.load <configname> <filename> vcl.inline <configname> <quoted_VCLstring> vcl.use <configname> vcl.discard <configname> vcl.listTuesday, 25 October 11
  39. 39. MORE HELP? • Talk to the devs, read their blogs. http://kristianlyng.wordpress.com http://ingvar.blog.redpill-linpro.com/ • Get some online training. https://www.varnish-software.com/products-services/training/ varnish-administration-course • Consider the Varnish ‘Moral’ License. http://phk.freebsd.dk/VML/Tuesday, 25 October 11
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×