Varnish Summit 2016

2. Debugging your Varnish Instance
Varnish Summit, Los Angeles

3. Federico G. Schwindt
Technical Presales & Engineer
London, UK
fgsch@varnish-software.com

4. Problems
● Stability
● Functional
● Operational
● Capacity
● Not my problem™

5. Getting the data
● https://github.com/varnish/varnishgather was born
● Gathers OS,Varnish and companion tools details
● One file per command

6. Stability
● Crash, panic or something else?
● Check varnishadm panic.show and syslog
● Start from command line
● Install debugging symbols, allow coredumps
● Get your hands dirty

7. Functional
● Code not reached, wrong logic or expectations
● printf debugging (std.log / std.syslog)
● Enable tracing (param.set vsl_mask +VCL_trace)
● Reproduce with varnishtest
● Check varnishlog

8. Anatomy of a .vtc
varnishtest "Hello world"
server s1 {
rxreq
expect req.http.msg == "hello"
txresp -body "world"
} -start
varnish v1 -vcl+backend {
sub vcl_recv {
set req.http.msg = "hello";
}
} -start
client c1 {
txreq
rxresp
expect resp.status == 200
expect resp.body == "world"
} -run

9. varnishlog
$ varnishlog -g request
- Begin req 459063 rxreq
- Timestamp Start: 1474493939.499338 0.000000 0.000000
- Timestamp Req: 1474493939.499338 0.000000 0.000000
- ReqStart 1.2.3.4 38780
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: example.com
- ReqHeader Accept: */*
- ReqHeader Accept-Encoding: gzip
- ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) ...
- ReqHeader Connection: close
- ReqHeader X-Forwarded-For: 1.2.3.4
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
...

10. Operational
● Low hitrate or high response times
● Inspect headers on the client side
● Use varnishtop
● Increase your EXP with varnishlog
● Understand the waitinglist

11. Request coalescing
● Automatically enabled for misses
● Only one request at a time goes to the backend
● Other requests for the same object will wait
● Avoids overloading the backend

12. Capacity
● Inadequate number of threads
● Dataset too big / cache too small
● Monitor varnishstat output
● Did I say varnishlog?

13. Not my problem
● “It must be the cache” axiom
● Test bypassing Varnish
● Check timeouts
● The truth is on the wire (aka look at tcpdump)

14. Some examples
● [5-9]xx responses:
○ varnishlog -q 'RespStatus >= 500'
● Misses:
○ varnishlog -q 'VCL_call ~ MISS'
● Timing details:
○ varnishlog -i Timestamp
● Requests taking more than 5 seconds:
○ varnishlog -q 'Timestamp:Fetch[2] > 5.’
● Top requested URLs:
○ varnishtop -i ReqURL

15. The bottom line
● It is not always the cache
● Not all errors are the same
● Become familiarised with the tools
● varnishlog is the swiss army knife of varnish

16. Thank you!