Caching with varnish

VARNISH-CACHE *
Lessons Learnt from Performance Testing

* https://www.varnish-cache.org
Tuesday, 25 October 11

WHY? FAST!

• If your throughput is currently measured in the hundreds per
second, think of varnish in the thousands per second.

• Theoretical limits in hundreds of thousands per second.*

* http://kristianlyng.wordpress.com/2010/10/23/275k-req/

GET OUT OF JAIL. FREE.

• Crappy cache control?

• Expensive CDNs? 20¢/GB?

• Not enough hardware?

• Backend struggling?


IT WON’T

• Do HTTPS... terminate SSL with something else, hopefully
your load balancers.


INSTALL THE LATEST

• RHEL / CentOS
rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-
release-3.0-1.noarch.rpm
yum install -y varnish

• Debian
apt-get install varnish


USE 64BIT*

“It’s all about virtual memory: Things like
stack size suddenly matter on 32bit. If you
must use Varnish on 32-bit, you’re somewhat
on your own. However, try to ﬁt it within
2GB. I wouldn’t recommend a cache larger
than 1GB, and no more than a few hundred
threads… (Why are you on 32bit again?)”

* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/

SYSTEM CONFIG

• The minimum.
echo 'fs.file-max=1000000' >> /etc/sysctl.conf
echo 'net.ipv4.ip_local_port_range = 1024 65535' >> /etc/sysctl.conf
sysctl -p
echo '* soft nofile 1000000' >> /etc/security/limits.conf
echo '* hard nofile 1000000' >> /etc/security/limits.conf
echo 'session required pam_limits.so' >> /etc/pam.d/login
ulimit -n 1000000
ulimit -n -H

• Do not set “tw_reuse” to 1 (sysctl). It will cause problems for
NAT based clients.

• Enable keepalives on load balancers.

THREADS

• Adjust default thread* settings in /etc/sysconfig/varnish

thread_pool_add_delay=2
thread_pools = <Number of cpu cores>
thread_pool_min = <800/number of cpu cores>
thread_pool_max = 4000
session_linger = 50
sess_workspace = <16k to 5m>

“The only thing that made a real difference while tuning Varnish
was the number of threads. And making sure it actually caches.
Beyond that, it really doesn’t matter much. Our defaults are
good.”

* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/

STORAGE

• Memory is fast, duh.

• Got enough RAM to store the cache?
Yes! then use -s malloc
Unsure? then use -s file


VCL

• Varnish Configuration Language

• Looks like C and compiled into C but ...
~ “has regex tilda operator”
Assignments use set keyword.
Only has conditional if tests.
Has a bunch of variables and some built in functions.

• Defined by backends, probes, ACLs and subroutines which
hook into the varnish workflow.


VCL

• Default VCL is deﬁned in
/etc/varnish/default.vcl

• Then create your custom VCL in
-f /var/custom.vcl
so varnish can fall through to the default.vcl
Can also include "/var/other.vcl";

• Keep it simple within the VCL workﬂow*
Do most of the work in vcl_recv and vcl_fetch

* https://www.varnish-cache.org/trac/wiki/VCLExampleDefault

LOOKUP AND MISS

• request received from client recv

• hash request hash pipe

• lookup object and miss hit miss pass

• fetch response from backend fetch

• deliver to client deliver


LOOKUP AND HIT


• hash request hash pipe

• lookup object and hit hit miss pass

fetch



PASS AND DELIVER


hash pipe

• pass through to backend hit miss pass

• fetch response fetch



PIPE
• pipe to backend until client or hash pipe
backend closes
hit miss pass

fetch

deliver


PROBES
• Deﬁne common health probe health {
  .url = "/healthCheck?onFailure=true";
checks for multiple   .expected_response = 200;
  .interval  = 15s;
backends.   .timeout   = 5s;
  .window    = 4;
  .threshold = 4;
}
• window is how many of the
latest polls are probed for
health.

• threshold is how many
polls must succeed to be
healthy.


BACKENDS

• Deﬁne backend host, port
and health checks inline.
{ .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } }

• Or named.
backend mysite {
.host = "www.example.com";
.port = "http";
.probe = "health";
}


DIRECTORS

• Group backends together using different algorithms.
Random
Client (based on identity)
Hash (based on url)
Round robin
Fallback


YMMV

•I found it better to balance
in front of Varnish using a
hardware load balancer ...

• Random seemed a little too
random using Varnish
directors however was
probably an artefact of the
load generation.


DESIGN DECISIONS

• Share nothing.
Easy to scale horizontally.
Great if your site is not ‘transactional’.
Backend session afﬁnity via LB set cookie.

LB Set-Cookie=BackendID

varnish varnish varnish
app app app


DESIGN DECISIONS

• Share everything.
Varnish loves RAM, scale vertically.
Varnish cache becomes the shared source of truth.

LB

varnish

app app app


SHARE NOTHING. FALLBACK.

director varnishA fallback {
  { .backend = { .host = "a"; } } LB Set-Cookie=VarnishID
  { .backend = { .host = "b"; } }
  { .backend = { .host = "c"; } }
}

director varnishB fallback {
varnish varnish varnish
  { .backend = { .host = "b"; } }
  { .backend = { .host = "a"; } }
  { .backend = { .host = "c"; } }
}

director varnishC fallback {
  { .backend = { .host = "c"; } }
  { .backend = { .host = "b"; } }
app app app
  { .backend = { .host = "a"; } }
}


TERMINATE SSL.

• Get the LB to set a Port header.
Useful when user switches from HTTP to HTTPS so varnish
knows which backend port to route to.

LB Set-Cookie=VarnishID
Port=https
varnish app
:80
:443

if (req.http.Port ~ "https") {
set req.backend = appa_80;
} else {
set req.backend = appa_443;
}


VCL TRICKS

• Rewrite headers to/from production.com.au to
testdomain.com

• vcl_recv
set req.http.Host = regsuball(req.http.Host, "testdomain.com", "com.au");
set req.http.Referer = regsuball(req.http.Host, "testdomain.com", "com.au");

• vcl_deliver
set resp.http.Host = regsuball(resp.http.Host, ".com.au", ".testdomain.com");
set resp.http.Referer = regsuball(resp.http.Host, ".com.au", ".testdomain.com");
set resp.http.Location = regsuball(resp.http.Location, ".com.au", ".testdomain.com");
set resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");


VCL TRICKS

• Add some debug headers for hit/miss in ﬁrebug in
vcl_deliver
if (obj.hits > 0) {
  set resp.http.X-Varnish-Hit = "++";
  set resp.http.X-Varnish-Hits = obj.hits;
  set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(d+).*", "1");
} else {
  set resp.http.X-Varnish-Hit = "--";
  set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(d+).*", "1");
}


VCL TRICKS

• Normalize the Accept-Encoding headers to reduce cache
usage in vcl_recv

if (req.http.Accept-Encoding) {
  if (req.url ~ ".(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
    remove req.http.Accept-Encoding;
  } elsif (req.http.Accept-Encoding ~ "gzip") {
    set req.http.Accept-Encoding = "gzip";
  } elsif (req.http.Accept-Encoding ~ "deflate") {
   set req.http.Accept-Encoding = "deflate";
  } else {
    remove req.http.Accept-Encoding;
  }
}


VCL TRICKS

• Remove Google Analytics cookies in vcl_recv
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;s*)(__[a-z]+|has_js)=[^;]*", "");
set req.http.Cookie = regsub(req.http.Cookie, "^;s*", "");

• Remember, varnish will pass any request with cookies by
default. You will need to over-ride this behaviour if your web
app uses cookies.

• Allow lookup of requests with basic auth (in test) or cookies
in vcl_recv but beware of creating a per-user-session store.
if (req.http.Authorization || req.http.Cookie) { return (lookup); }


VCL TRICKS

• Pass on the client IP to the backend servers using the X-
Forwarded-For header in vcl_recv
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;


VCL TRICKS

• Let varnish keep objects for 2 hours beyond their TTL in
vcl_fetch
set beresp.grace = 2h;

• Let varnish serve stale objects up to 30 secs old for healthy
servers, otherwise 2 hours in vcl_recv
  if (! req.backend.healthy) {
    set req.grace = 2h;
  } else {
    set req.grace = 30s;
  }

• Can also use saint mode to disable backends for a period of
time, otherwise if all backends unavailable, use grace.


VCL TRICKS

• Force a grace on error. Restart keyword can be called any
where and will return request all the way back to vcl_recv
sub vcl_error {
  if (req.restarts == 0) {
    set req.http.X-Serve-Graced = "1";
    return(restart);
  }
}


TTL

• Varnish will obey cache control headers from backend
Cache-Control: s-maxage
Cache-Control: max-age

• Can also override TTL control in VCL, but best do it from the
backend
set obj.ttl = x


REDUCING CACHE SIZE

• Normalize / sanitize request headers

• Set Content-Length on backends

• Serve (static) digital assets from VFS cache

• Pipe large objects like videos


MONITORING VARNISH

• Handy command line tools
varnishstat
varnishlog
varnishtop
varnishadm


VARNISHSTAT

• varnishstat
Hit rate % (x100)
Average per second
Average/sec since boot
Sum total

• Filtered
varnishstat -f
client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail


VARNISHSTAT

• Log all statistics to disk ...
#!/bin/bash
logdir=/var/log/perf/`hostname`/latest
mkdir -p $logdir

while true
do
varnishstat -x >> "$logdir/varnishstat.xml"
sleep 60
done

• Then graph as you see ﬁt ...


VARNISHLOG

• varnishlog debugging (-d to dump shared memory)
12 BackendOpen b xx_80[0] x.x.x.x 45546 x.x.x.x 80
12 TxRequest b GET
12 TxURL b /
12 TxProtocol b HTTP/1.1
12 TxHeader b Host:
12 TxHeader b Referer:
12 TxHeader b Cookie:
12 TxHeader b X-Forwarded-For: 1.2.3.4
12 TxHeader b X-Varnish: 2092287889
12 TxHeader b Accept-Encoding: gzip
12 RxProtocol b HTTP/1.1
12 RxStatus b 401
12 RxResponse b Authorization Required
12 RxHeader b Date: Mon, 24 Oct 2011 05:46:18 GMT
12 RxHeader b WWW-Authenticate: Basic realm="XX Authorised Personel Only"
12 RxHeader b Vary: Accept-Encoding
12 RxHeader b Content-Encoding: gzip
12 RxHeader b Content-Length: 336
12 RxHeader b Connection: close
12 RxHeader b Content-Type: text/html; charset=iso-8859-1
12 Fetch_Body b 4 0 1
12 Length b 336
12 BackendClose b xx_80[0]


VARNISHTOP

• Monitor top URLs hitting backend
varnishtop -i txurl

• Monitor top received URLs
varnishtop -i rxurl


VARNISHADM

• varnishadm for administration
# varnishadm
CLI connected to 127.0.0.1 6082
200
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit

Type 'help' for command list.
Type 'quit' to close CLI session.

varnish> help
200
help [command]
ping [timestamp]
auth response
quit
banner
status
start
stop
vcl.load <configname> <filename>
vcl.inline <configname> <quoted_VCLstring>
vcl.use <configname>
vcl.discard <configname>
vcl.list

MORE HELP?

• Talk to the devs, read their blogs.
http://kristianlyng.wordpress.com
http://ingvar.blog.redpill-linpro.com/

• Get some online training.
https://www.varnish-software.com/products-services/training/
varnish-administration-course

• Consider the Varnish ‘Moral’ License.
http://phk.freebsd.dk/VML/


Caching with varnish

More Related Content

What's hot

Similar to Caching with varnish

Recently uploaded

Caching with varnish