Caching with varnish
Upcoming SlideShare
Loading in...5
×
 

Caching with varnish

on

  • 12,132 views

 

Statistics

Views

Total Views
12,132
Views on SlideShare
12,131
Embed Views
1

Actions

Likes
16
Downloads
149
Comments
0

1 Embed 1

http://redmine.advertiser.tech 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Caching with varnish Caching with varnish Presentation Transcript

  • VARNISH-CACHE * Lessons Learnt from Performance Testing* https://www.varnish-cache.orgTuesday, 25 October 11
  • WHY? FAST! • If your throughput is currently measured in the hundreds per second, think of varnish in the thousands per second. • Theoretical limits in hundreds of thousands per second.** http://kristianlyng.wordpress.com/2010/10/23/275k-req/Tuesday, 25 October 11
  • GET OUT OF JAIL. FREE. • Crappy cache control? • Expensive CDNs? 20¢/GB? • Not enough hardware? • Backend struggling?Tuesday, 25 October 11
  • IT WON’T • Do HTTPS... terminate SSL with something else, hopefully your load balancers.Tuesday, 25 October 11
  • INSTALL THE LATEST • RHEL / CentOS rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish- release-3.0-1.noarch.rpm yum install -y varnish • Debian apt-get install varnishTuesday, 25 October 11
  • USE 64BIT* “It’s all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, you’re somewhat on your own. However, try to fit it within 2GB. I wouldn’t recommend a cache larger than 1GB, and no more than a few hundred threads… (Why are you on 32bit again?)”* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/Tuesday, 25 October 11
  • SYSTEM CONFIG • The minimum. echo fs.file-max=1000000 >> /etc/sysctl.conf echo net.ipv4.ip_local_port_range = 1024 65535 >> /etc/sysctl.conf sysctl -p echo * soft nofile 1000000 >> /etc/security/limits.conf echo * hard nofile 1000000 >> /etc/security/limits.conf echo session required pam_limits.so >> /etc/pam.d/login ulimit -n 1000000 ulimit -n -H • Do not set “tw_reuse” to 1 (sysctl). It will cause problems for NAT based clients. • Enable keepalives on load balancers.Tuesday, 25 October 11
  • THREADS • Adjust default thread* settings in /etc/sysconfig/varnish thread_pool_add_delay=2 thread_pools = <Number of cpu cores> thread_pool_min = <800/number of cpu cores> thread_pool_max = 4000 session_linger = 50 sess_workspace = <16k to 5m> “The only thing that made a real difference while tuning Varnish was the number of threads. And making sure it actually caches. Beyond that, it really doesn’t matter much. Our defaults are good.”* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/Tuesday, 25 October 11
  • STORAGE • Memory is fast, duh. • Got enough RAM to store the cache? Yes! then use -s malloc Unsure? then use -s fileTuesday, 25 October 11
  • VCL • Varnish Configuration Language • Looks like C and compiled into C but ... ~ “has regex tilda operator” Assignments use set keyword. Only has conditional if tests. Has a bunch of variables and some built in functions. • Defined by backends, probes, ACLs and subroutines which hook into the varnish workflow.Tuesday, 25 October 11
  • VCL • Default VCL is defined in /etc/varnish/default.vcl • Then create your custom VCL in -f /var/custom.vcl so varnish can fall through to the default.vcl Can also include "/var/other.vcl"; • Keep it simple within the VCL workflow* Do most of the work in vcl_recv and vcl_fetch* https://www.varnish-cache.org/trac/wiki/VCLExampleDefaultTuesday, 25 October 11
  • LOOKUP AND MISS • request received from client recv • hash request hash pipe • lookup object and miss hit miss pass • fetch response from backend fetch • deliver to client deliverTuesday, 25 October 11
  • LOOKUP AND HIT • request received from client recv • hash request hash pipe • lookup object and hit hit miss pass fetch • deliver to client deliverTuesday, 25 October 11
  • PASS AND DELIVER • request received from client recv hash pipe • pass through to backend hit miss pass • fetch response fetch • deliver to client deliverTuesday, 25 October 11
  • PIPE • request received from client recv • pipe to backend until client or hash pipe backend closes hit miss pass fetch deliverTuesday, 25 October 11
  • PROBES • Define common health probe health {   .url = "/healthCheck?onFailure=true"; checks for multiple   .expected_response = 200;   .interval  = 15s;         backends.   .timeout   = 5s;             .window    = 4;             .threshold = 4;           } • window is how many of the latest polls are probed for health. • threshold is how many polls must succeed to be healthy.Tuesday, 25 October 11
  • BACKENDS • Define backend host, port and health checks inline. { .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } } • Or named. backend mysite {   .host = "www.example.com";   .port = "http"; .probe = "health"; }Tuesday, 25 October 11
  • DIRECTORS • Group backends together using different algorithms. Random Client (based on identity) Hash (based on url) Round robin FallbackTuesday, 25 October 11
  • YMMV •I found it better to balance in front of Varnish using a hardware load balancer ... • Random seemed a little too random using Varnish directors however was probably an artefact of the load generation.Tuesday, 25 October 11
  • DESIGN DECISIONS • Share nothing. Easy to scale horizontally. Great if your site is not ‘transactional’. Backend session affinity via LB set cookie. LB Set-Cookie=BackendID varnish varnish varnish app app appTuesday, 25 October 11
  • DESIGN DECISIONS • Share everything. Varnish loves RAM, scale vertically. Varnish cache becomes the shared source of truth. LB varnish app app appTuesday, 25 October 11
  • SHARE NOTHING. FALLBACK. director varnishA fallback {   { .backend = { .host = "a"; } } LB Set-Cookie=VarnishID   { .backend = { .host = "b"; } }   { .backend = { .host = "c"; } }    } director varnishB fallback { varnish varnish varnish   { .backend = { .host = "b"; } }   { .backend = { .host = "a"; } }   { .backend = { .host = "c"; } }    } director varnishC fallback {   { .backend = { .host = "c"; } }   { .backend = { .host = "b"; } } app app app   { .backend = { .host = "a"; } }    }Tuesday, 25 October 11
  • TERMINATE SSL. • Get the LB to set a Port header. Useful when user switches from HTTP to HTTPS so varnish knows which backend port to route to. LB Set-Cookie=VarnishID Port=https varnish app :80 :443 if (req.http.Port ~ "https") { set req.backend = appa_80; } else {   set req.backend = appa_443; }Tuesday, 25 October 11
  • VCL TRICKS • Rewrite headers to/from production.com.au to testdomain.com • vcl_recv set req.http.Host = regsuball(req.http.Host, "testdomain.com", "com.au"); set req.http.Referer = regsuball(req.http.Host, "testdomain.com", "com.au"); • vcl_deliver set resp.http.Host = regsuball(resp.http.Host, ".com.au", ".testdomain.com"); set resp.http.Referer = regsuball(resp.http.Host, ".com.au", ".testdomain.com");   set resp.http.Location = regsuball(resp.http.Location, ".com.au", ".testdomain.com"); set resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");Tuesday, 25 October 11
  • VCL TRICKS • Add some debug headers for hit/miss in firebug in vcl_deliver if (obj.hits > 0) {   set resp.http.X-Varnish-Hit = "++";   set resp.http.X-Varnish-Hits = obj.hits;   set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(d+).*", "1"); } else {   set resp.http.X-Varnish-Hit = "--";   set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(d+).*", "1"); }Tuesday, 25 October 11
  • VCL TRICKS • Normalize the Accept-Encoding headers to reduce cache usage in vcl_recv if (req.http.Accept-Encoding) {   if (req.url ~ ".(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {     remove req.http.Accept-Encoding;   } elsif (req.http.Accept-Encoding ~ "gzip") {     set req.http.Accept-Encoding = "gzip";   } elsif (req.http.Accept-Encoding ~ "deflate") {     set req.http.Accept-Encoding = "deflate";   } else {     remove req.http.Accept-Encoding;   } }Tuesday, 25 October 11
  • VCL TRICKS • Remove Google Analytics cookies in vcl_recv set req.http.Cookie = regsuball(req.http.Cookie, "(^|;s*)(__[a-z]+|has_js)=[^;]*", ""); set req.http.Cookie = regsub(req.http.Cookie, "^;s*", ""); • Remember, varnish will pass any request with cookies by default. You will need to over-ride this behaviour if your web app uses cookies. • Allow lookup of requests with basic auth (in test) or cookies in vcl_recv but beware of creating a per-user-session store. if (req.http.Authorization || req.http.Cookie) { return (lookup); }Tuesday, 25 October 11
  • VCL TRICKS • Pass on the client IP to the backend servers using the X- Forwarded-For header in vcl_recv remove req.http.X-Forwarded-For; set    req.http.X-Forwarded-For = client.ip;Tuesday, 25 October 11
  • VCL TRICKS • Let varnish keep objects for 2 hours beyond their TTL in vcl_fetch set beresp.grace = 2h; • Let varnish serve stale objects up to 30 secs old for healthy servers, otherwise 2 hours in vcl_recv   if (! req.backend.healthy) {     set req.grace = 2h;   } else {     set req.grace = 30s;   } • Can also use saint mode to disable backends for a period of time, otherwise if all backends unavailable, use grace.Tuesday, 25 October 11
  • VCL TRICKS • Force a grace on error. Restart keyword can be called any where and will return request all the way back to vcl_recv sub vcl_error {   if (req.restarts == 0) {     set req.http.X-Serve-Graced = "1";     return(restart);   } }Tuesday, 25 October 11
  • TTL • Varnish will obey cache control headers from backend Cache-Control: s-maxage Cache-Control: max-age • Can also override TTL control in VCL, but best do it from the backend set obj.ttl = xTuesday, 25 October 11
  • REDUCING CACHE SIZE • Normalize / sanitize request headers • Set Content-Length on backends • Serve (static) digital assets from VFS cache • Pipe large objects like videosTuesday, 25 October 11
  • MONITORING VARNISH • Handy command line tools varnishstat varnishlog varnishtop varnishadmTuesday, 25 October 11
  • VARNISHSTAT • varnishstat Hit rate % (x100) Average per second Average/sec since boot Sum total • Filtered varnishstat -f client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_failTuesday, 25 October 11
  • VARNISHSTAT • Log all statistics to disk ... #!/bin/bash logdir=/var/log/perf/`hostname`/latest mkdir -p $logdir   while true do   varnishstat -x >> "$logdir/varnishstat.xml"   sleep 60 done • Then graph as you see fit ...Tuesday, 25 October 11
  • VARNISHLOG • varnishlog debugging (-d to dump shared memory) 12 BackendOpen b xx_80[0] x.x.x.x 45546 x.x.x.x 80 12 TxRequest b GET 12 TxURL b / 12 TxProtocol b HTTP/1.1 12 TxHeader b Host: 12 TxHeader b Referer: 12 TxHeader b Cookie: 12 TxHeader b X-Forwarded-For: 1.2.3.4 12 TxHeader b X-Varnish: 2092287889 12 TxHeader b Accept-Encoding: gzip 12 RxProtocol b HTTP/1.1 12 RxStatus b 401 12 RxResponse b Authorization Required 12 RxHeader b Date: Mon, 24 Oct 2011 05:46:18 GMT 12 RxHeader b WWW-Authenticate: Basic realm="XX Authorised Personel Only" 12 RxHeader b Vary: Accept-Encoding 12 RxHeader b Content-Encoding: gzip 12 RxHeader b Content-Length: 336 12 RxHeader b Connection: close 12 RxHeader b Content-Type: text/html; charset=iso-8859-1 12 Fetch_Body b 4 0 1 12 Length b 336 12 BackendClose b xx_80[0]Tuesday, 25 October 11
  • VARNISHTOP • Monitor top URLs hitting backend varnishtop -i txurl • Monitor top received URLs varnishtop -i rxurlTuesday, 25 October 11
  • VARNISHADM • varnishadm for administration # varnishadm CLI connected to 127.0.0.1 6082 200 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit Type help for command list. Type quit to close CLI session. varnish> help 200 help [command] ping [timestamp] auth response quit banner status start stop vcl.load <configname> <filename> vcl.inline <configname> <quoted_VCLstring> vcl.use <configname> vcl.discard <configname> vcl.listTuesday, 25 October 11
  • MORE HELP? • Talk to the devs, read their blogs. http://kristianlyng.wordpress.com http://ingvar.blog.redpill-linpro.com/ • Get some online training. https://www.varnish-software.com/products-services/training/ varnish-administration-course • Consider the Varnish ‘Moral’ License. http://phk.freebsd.dk/VML/Tuesday, 25 October 11