SlideShare a Scribd company logo
1 of 125
HTTP LOGGING MET VARNISHLOG
Thijs Feryn
256
LOGS
LOGS
LOHS
172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45
172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45
NCSA COMMON LOH FORMOAT
172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45
HOST IDENT AUTH
USER
DATE REQUEST STATUS BYTES
NCSA COMMON LOH FORMOAT
NATIONAL CENTER FOR
SUPERCOMPUTING
APPLICATIONS
172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"
172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"
NCSA COMBINED LOH FORMOAT
172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"
NCSA COMBINED LOH FORMOAT
REFERER
172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"
NCSA COMBINED LOH FORMOAT
USER-AGENT
CACHE
GEBRUKER VARNISH SERVER
* << Request >> 10
- Begin req 9 rxreq
- Timestamp Start: 1651671623.893211 0.000000 0.000000
- Timestamp Req: 1651671623.893211 0.000000 0.000000
- ReqStart 172.21.0.1 62448 http
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost
- ReqHeader User-Agent: curl/7.79.1
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 172.21.0.1
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- ExpBan 6 banned lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 11 fetch
- Timestamp Fetch: 1651671623.896113 0.002902 0.002902
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Content-Type: application/json; charset=utf-8
- RespHeader Content-Length: 562
- RespHeader ETag: W/"232-+/Gw91fMkC3FNffhpz0wpLeOy1o"
- RespHeader Date: Wed, 04 May 2022 13:40:23 GMT
- RespHeader X-Varnish: 10
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.0)
YO, IK ZIN THIJS
IK WEUNEN IN DIKSMUIDE
BACHTEN DE KUPE?
IK ZIN DE TECH
EVANHELIST
BIE
9,000,000 WEBSITES


21% VAN DE TOP 10K WEBSITES
IK ZIN @THIJSFERYN
GEBRUKER VARNISH SERVER
OJ CACHT EJ GIN LOHS


IP JE WEB SERVER
GEBRUKER VARNISH SERVER
VARNISH SHARED
MEMORY LOGS (VSL)
CIRCULAR MEMORY BUFFER
vsl_space


Value is: 80M [bytes] (default)


Minimum is: 1M


Maximum is: 4G


The amount of space to allocate for the VSL fifo buffer in the


VSM memory segment. If you make this too small,


varnish{ncsa|log} etc will not be able to keep up. Making it


too large just costs memory resources.


NB: This parameter will not take any effect until the child


process has been restarted.
varnishd -p vsl_space=150M
varnishd -l 150M
$ varnishadm param.set vsl_space 150M


$ varnishadm stop


$ varnishadm start
VARNISHLOG
VARNISHLOG
VARNISHLOH
bash: varnishloh: command not found
Usage: /usr/bin/varnishlog <options>


Options:


[-a] Append to file


[-A] Text output


[-b] Only display backend records


[-c] Only display client records


[-C] Caseless regular expressions


[-d] Process old log entries and exit


[-D] Daemonize


[-g <session|request|vxid|raw>] Grouping mode (default: vxid)


[-h] Usage help


[-i <taglist>] Include tags


[-I <[taglist:]regex>] Include by regex


[-k <num>] Limit transactions


[-L <limit>] Incomplete transaction limit


[-n <dir>] varnishd working directory


[-P <file>] PID file


[-q <query>] VSL query


[-r <filename>] Binary file input


[-R <limit[/duration]>] Output rate limit


[-t <seconds|off>] VSM connection timeout


[-T <seconds>] Transaction end timeout


[-v] Verbose record printing


[-V] Version


[-w <filename>] Output filename


[-x <taglist>] Exclude tags


[-X <[taglist:]regex>] Exclude by regex
* << Session >> 1752689
- Begin sess 0 HTTP/1
- SessOpen 172.21.0.1 55962 http 172.21.0.3 80 1651675446.691148 29
- Link req 1752690 rxreq
- SessClose REM_CLOSE 0.004
- End
varnishd -F -f /etc/varnish/default.vcl 


-a http=:80,HTTP -a proxy=:8443,PROXY 


-p feature=+http2 -s malloc,100M
VARNISH RUNTIME PARAMETERS
** << Request >> 1752690
-- Begin req 1752689 rxreq
-- Timestamp Start: 1651675446.691433 0.000000 0.000000
-- Timestamp Req: 1651675446.691433 0.000000 0.000000
-- ReqStart 172.21.0.1 55962 http
-- ReqMethod GET
-- ReqURL /
-- ReqProtocol HTTP/1.1
-- ReqHeader Host: localhost
-- ReqHeader User-Agent: curl/7.79.1
-- ReqHeader Accept: */*
-- ReqHeader X-Forwarded-For: 172.21.0.1
-- VCL_call RECV
-- VCL_return hash
-- VCL_call HASH
-- VCL_return lookup
-- ExpBan 1679159 banned lookup
-- VCL_call MISS
-- VCL_return fetch
-- Link bereq 1752691 fetch
-- Timestamp Fetch: 1651675446.693975 0.002542 0.002542
-- RespProtocol HTTP/1.1
-- RespStatus 200
-- RespReason OK
-- RespHeader Content-Type: application/json; charset=utf-8
-- RespHeader Content-Length: 567
-- RespHeader ETag: W/"237-ulyqMICHeGkCSm+psitDJ2K3qqg"
-- RespHeader Date: Wed, 04 May 2022 14:44:06 GMT
-- RespHeader X-Varnish: 1752690
-- RespHeader Age: 0
-- RespHeader Via: 1.1 varnish (Varnish/6.0)
-- VCL_call DELIVER
-- VCL_return deliver
-- Timestamp Process: 1651675446.693981 0.002548 0.000006
-- RespHeader Accept-Ranges: bytes
-- RespHeader Connection: keep-alive
-- Timestamp Resp: 1651675446.694030 0.002597 0.000049
-- ReqAcct 73 0 73 273 567 840
-- End
*** << BeReq >> 1752691
--- Begin bereq 1752690 fetch
--- VCL_use boot
--- Timestamp Start: 1651675446.691561 0.000000 0.000000
--- BereqMethod GET
--- BereqURL /
--- BereqProtocol HTTP/1.1
--- BereqHeader Host: localhost
--- BereqHeader User-Agent: curl/7.79.1
--- BereqHeader Accept: */*
--- BereqHeader X-Forwarded-For: 172.21.0.1
--- BereqHeader Accept-Encoding: gzip
--- BereqHeader X-Varnish: 1752691
--- VCL_call BACKEND_FETCH
--- VCL_return fetch
--- BackendOpen 32 boot.default 172.21.0.2 8080 172.21.0.3 36102
--- BackendStart 172.21.0.2 8080
--- Timestamp Bereq: 1651675446.691704 0.000143 0.000143
--- Timestamp Beresp: 1651675446.693860 0.002299 0.002156
--- BerespProtocol HTTP/1.1
--- BerespStatus 200
--- BerespReason OK
--- BerespHeader Content-Type: application/json; charset=utf-8
--- BerespHeader Content-Length: 567
--- BerespHeader ETag: W/"237-ulyqMICHeGkCSm+psitDJ2K3qqg"
--- BerespHeader Date: Wed, 04 May 2022 14:44:06 GMT
--- BerespHeader Connection: keep-alive
--- BerespHeader Keep-Alive: timeout=5
--- TTL RFC 120 10 0 1651675447 1651675447 1651675446 0 0 cacheable
--- VCL_call BACKEND_RESPONSE
--- VCL_return deliver
--- Storage malloc s0
--- Fetch_Body 3 length stream
--- BackendReuse 32 boot.default
--- Timestamp BerespBody: 1651675446.693945 0.002384 0.000085
--- Length 567
--- BereqAcct 145 0 145 214 567 781
--- End
<esi:include src="/header" />
$ varnishlog -i Begin
* << BeReq >> 3
- Begin bereq 2 fetch
* << BeReq >> 5
- Begin bereq 4 fetch
* << Request >> 4
- Begin req 2 esi
* << Request >> 2
- Begin req 1 rxreq
* << Session >> 1
- Begin sess 0 HTTP/1
$ varnishlog -i Begin
* << BeReq >> 3
- Begin bereq 2 fetch
* << Request >> 2
- Begin req 1 rxreq
* << Session >> 1
- Begin sess 0 HTTP/1
EEN SLAGSJE SIMPELER
TRANSACTIES GROEPEREN
✓ VXID (DEFAULT)


✓ SESSION


✓ REQUEST


✓ RAW
$ varnishlog -i Begin -g session
* << Session >> 1
- Begin sess 0 HTTP/1
** << Request >> 2
-- Begin req 1 rxreq
*** << BeReq >> 3
--- Begin bereq 2 fetch
*** << Request >> 4
--- Begin req 2 esi
*4* << BeReq >> 5
-4- Begin bereq 4 fetch
$ varnishlog -i Begin -g request
* << Request >> 2
- Begin req 1 rxreq
** << BeReq >> 3
-- Begin bereq 2 fetch
** << Request >> 4
-- Begin req 2 esi
*** << BeReq >> 5
--- Begin bereq 4 fetch
$ varnishlog -i Begin -g raw
1 Begin c sess 0 HTTP/1
3 Begin b bereq 2 fetch
5 Begin b bereq 4 fetch
4 Begin c req 2 esi
2 Begin c req 1 rxreq
varnishlog -g request
varnishlog -g session
varnishlog -g vxid
varnishlog -g raw
BITJE MINDER OUTPUT
RATE LIMITING
varnishlog -R 1/5S
TAG FILTERING
varnishlog -g request -c -i "Req*" -i Timestamp
varnishlog -g request -b -i "BeReq*" -i Timestamp
* << Request >> 1659437
- Timestamp Start: 1651676801.915256 0.000000 0.000000
- Timestamp Req: 1651676801.915256 0.000000 0.000000
- ReqUnset Upgrade: h2c
- ReqUnset HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA
- ReqProtocol HTTP/2.0
- ReqStart 172.21.0.1 57084 http
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/2.0
- ReqHeader Host: localhost
- ReqHeader User-Agent: curl/7.79.1
- ReqHeader Accept: */*
- ReqHeader Connection: Upgrade, HTTP2-Settings
- ReqHeader X-Forwarded-For: 172.21.0.1
- Timestamp Process: 1651676801.916362 0.001106 0.001106
- Timestamp Resp: 1651676801.916672 0.001416 0.000311
- ReqAcct 166 0 166 178 566 744
varnishlog -g request -c -i "Req*,Resp*" -i Timestamp
varnishlog -g request -c -i "Req*" -x ReqHeader
varnishlog -g request -i ReqUrl 


-I ReqHeader:Accept-Language
varnishlog -g request -i ReqUrl 


-i RespHeader -X "RespHeader:(X|x)-"
VSL QUERIES
<record selection criteria> <operator> <operand>
ReqUrl eq '/'
RECORD SELECTIE CRITERIA
{level}taglist:record-prefix[field]
{2+}Timestamp:Resp[2]
%s: %f %f %f
| | | |
| | | +- Time since last timestamp
| | +---- Time since start of work unit
| +------- Absolute time of event
+----------- Event label
TIMESTAMP FORMOAT
* << Request >> 11
- Timestamp Start: 1606398588.811189 0.000000 0.000000
- Timestamp Req: 1606398588.811189 0.000000 0.000000
- Timestamp Fetch: 1606398588.818399 0.007210 0.007210
- Timestamp Process: 1606398588.818432 0.007243 0.000032
- Timestamp Resp: 1606398588.818609 0.007421 0.000178
varnishlog -c -i ReqUrl -I Timestamp:Resp 


-g request -q "{2+}Time*:Resp[2] > 2.0"
varnishlog -g request -q "BerespStatus == 500"
varnishlog -g request -q "BerespStatus >= 500"


varnishlog -g request -q "BerespStatus > 500"


varnishlog -g request -q "BerespStatus <= 500"


varnishlog -g request -q "BerespStatus < 500"
varnishlog -g request -q "ReqUrl ~ '^/contact'"
varnishlog -g request -q "not ReqHeader:Accept-Language"
varnishlog -g request -q "ReqUrl eq '/'"


varnishlog -g request -q "ReqUrl ne '/contact'"
varnishlog -c -i ReqUrl -I RespHeader:Content-Type 


-i reqacct -g request 


-q "RespHeader:Content-Type ~ '^image/' and
ReqAcct[5] >= 2000000"
%d %d %d %d %d %d
| | | | | |
| | | | | +- Total bytes transmitted
| | | | +---- Body bytes transmitted
| | | +------- Header bytes transmitted
| | +---------- Total bytes received
| +------------- Body bytes received
+---------------- Header bytes received
REQACCT FORMOAT
varnishlog -c -g request -i ReqUrl 


-I VCL_call:PASS -I VCL_call:MISS 


-q "VCL_call eq 'MISS' or VCL_call eq 'PASS'"
varnishlog -c -g request -i ReqUrl 


-q "TTL[6] eq 'uncacheable' and (BerespHeader:Set-
Cookie or BerespHeader:Cache-Control ~ '(private|no-
cache|no-store)')"
DE BUFFER DUMP'N
varnishlog -i ReqUrl -d
varnishlog -g request -i requrl -d 


-q "Timestamp:Start[1] >= $(date -d '10/05/2022
11:00:00' +%s.0)"
LEZEN/SCHRIEV'N
varnishlog -a -w vsl.log


varnishlog -r vsl.log


varnishlog -A -a -w vsl.log
VARNISHLOG
VARNISHLOG
VARNISHNCSA
172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200
45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
100.0.4896.127 Safari/537.36"
varnishncsa -a -w /var/log/varnish/access.log
%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
%b Size of the response body in bytes
%D Time taken to serve the request in microseconds
%H The request protocol
%h The hostname or IP address of the remote host
%I Total bytes received
%{X}i Contents of request header X
%l Remote log name
%m Request method
%{X}o Contents of response header X
%O In client mode, total bytes sent to client. In backend mode, total bytes received from the backend
%q The query string
%r First line of the HTTP request composed using other formatters
%s HTTP status code of the response
%t Time when the request was received
%{X}t Time when request was received in the strftime time specification format
%T Time taken to serve the request
%U Request URL without query string
%u Remote authenticated user
%{X}x Extended Varnish & VCL variables
varnishncsa -c -b -F '%h %l %u %t "%r" %s %b "%
{Referer}i" "%{User-agent}i" "%{Varnish:side}x"'
172.18.0.3 - - [02/Dec/2020:10:18:38 +0000] "GET http://localhost/
HTTP/1.1" 200 6 "-" "curl" "b"
127.0.0.1 - - [02/Dec/2020:10:18:38 +0000] "GET http://localhost/
HTTP/1.1" 200 6 "-" "curl" "c"
varnishncsa -F '%U %{Varnish:handling}x %{Varnish:side}x' -b -c
/test - b
/test miss c
/test hit c
/test - b
/test pass c
varnishncsa -b -F '%{VSL:Timestamp:BerespBody[2]}x %
{Host}i %U%q' -q 'Timestamp:BerespBody[2] > 1.0'
2.007142 localhost /
varnishncsa -a -D -w /var/log/varnish/access.log 
-P /var/run/varnishncsa.pid
VARNISHNCSA DAEMONIZEN
[Unit]
Description=Varnish Cache HTTP accelerator NCSA logging daemon
After=varnish.service
[Service]
RuntimeDirectory=varnishncsa
Type=forking
User=varnishlog
Group=varnish
ExecStart=/usr/bin/varnishncsa -a -w /var/log/varnish/
varnishncsa.log -D
ExecReload=/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target
SYSTEMD
/var/log/varnish/varnishncsa.log {
daily
rotate 7
compress
delaycompress
missingok
postrotate
systemctl -q is-active varnishncsa.service || exit 0
systemctl reload varnishncsa.service
endscript
}
LOH ROTOASJE
/var/log/varnish/varnishncsa.log
/var/log/varnish/varnishncsa.log.1
/var/log/varnish/varnishncsa.log.2.gz
/var/log/varnish/varnishncsa.log.3.gz
/var/log/varnish/varnishncsa.log.4.gz
/var/log/varnish/varnishncsa.log.5.gz
/var/log/varnish/varnishncsa.log.6.gz
/var/log/varnish/varnishncsa.log.7.gz
LOH ROTOASJE
VARNISHTOP
varnishtop -i requrl
list length 3
31.67 ReqURL /contact
24.33 ReqURL /products
8.50 ReqURL /
varnishtop -i ReqUrl -q "VCL_call eq 'MISS'"
varnishtop -i ReqUrl -q "VCL_return eq 'pass'"
varnishtop -i BeReqUrl -q "TTL[6] eq 'uncacheable'"
varnishtop -i BeReqUrl -q "Timestamp:Beresp[3] > 2.0"
varnishtop -i RespProtocol
list length 2
310.29 RespProtocol HTTP/1.1
19.56 RespProtocol HTTP/2.0
varnishtop -I Begin:sess
list length 2
234.25 Begin sess 0 HTTP/1
49.89 Begin sess 0 PROXY
VSL TAHS IP MOATE
VARNISH CONFIGURATION LANGUAGE
STRING std.log(STRING s)
vcl 4.1;


import std;


import vsthrottle;


backend default {


.host = "localhost";


.port = "8080";


}


sub vcl_recv {


if (vsthrottle.is_denied(client.identity, 15, 10s, 30s)) {


std.log("Ratelimit: " + client.identity +


" blocked for another " + vsthrottle.blocked(client.identity, 15, 10s, 30s)


+ " seconds");


return (synth(429, "Too Many Requests"));


}


}
vcl 4.1;


import std;


import vsthrottle;


backend default {


.host = "localhost";


.port = "8080";


}


sub vcl_recv {


if (vsthrottle.is_denied(client.identity, 15, 10s, 30s)) {


std.log("Ratelimit: " + client.identity +


" blocked for another " + vsthrottle.blocked(client.identity, 15, 10s, 30s)


+ " seconds");


return (synth(429, "Too Many Requests"));


}


}
varnishlog -g request -i requrl -I VCL_Log:Ratelimit
* << Request >> 557828
- ReqURL /
- VCL_Log Ratelimit: 172.24.0.1 blocked for
another 29.151 seconds
DEMOOTJE?
HTTP logging met Varnishlog - PHPWVL 2022
HTTP logging met Varnishlog - PHPWVL 2022

More Related Content

Similar to HTTP logging met Varnishlog - PHPWVL 2022

Ibm tivoli access manager for e business tracing http connections redp4622
Ibm tivoli access manager for e business tracing http connections redp4622Ibm tivoli access manager for e business tracing http connections redp4622
Ibm tivoli access manager for e business tracing http connections redp4622
Banking at Ho Chi Minh city
 
Apache web server installation/configuration, Virtual Hosting
Apache web server installation/configuration, Virtual HostingApache web server installation/configuration, Virtual Hosting
Apache web server installation/configuration, Virtual Hosting
webhostingguy
 
Стек Linux HTTPS/TCP/IP для защиты от HTTP-DDoS-атак
Стек Linux HTTPS/TCP/IP для защиты от HTTP-DDoS-атакСтек Linux HTTPS/TCP/IP для защиты от HTTP-DDoS-атак
Стек Linux HTTPS/TCP/IP для защиты от HTTP-DDoS-атак
Positive Hack Days
 

Similar to HTTP logging met Varnishlog - PHPWVL 2022 (20)

Debugging linux issues with eBPF
Debugging linux issues with eBPFDebugging linux issues with eBPF
Debugging linux issues with eBPF
 
DeveloperWeek 2015: A Practical Introduction to Docker
DeveloperWeek 2015: A Practical Introduction to DockerDeveloperWeek 2015: A Practical Introduction to Docker
DeveloperWeek 2015: A Practical Introduction to Docker
 
Caching and tuning fun for high scalability
Caching and tuning fun for high scalabilityCaching and tuning fun for high scalability
Caching and tuning fun for high scalability
 
cache concepts and varnish-cache
cache concepts and varnish-cachecache concepts and varnish-cache
cache concepts and varnish-cache
 
Introduction to HTTP protocol
Introduction to HTTP protocolIntroduction to HTTP protocol
Introduction to HTTP protocol
 
HTTP Caching in Web Application
HTTP Caching in Web ApplicationHTTP Caching in Web Application
HTTP Caching in Web Application
 
Ibm tivoli access manager for e business tracing http connections redp4622
Ibm tivoli access manager for e business tracing http connections redp4622Ibm tivoli access manager for e business tracing http connections redp4622
Ibm tivoli access manager for e business tracing http connections redp4622
 
Http2 kotlin
Http2   kotlinHttp2   kotlin
Http2 kotlin
 
AtlasCamp 2015 Docker continuous integration training
AtlasCamp 2015 Docker continuous integration trainingAtlasCamp 2015 Docker continuous integration training
AtlasCamp 2015 Docker continuous integration training
 
Apache web server installation/configuration, Virtual Hosting
Apache web server installation/configuration, Virtual HostingApache web server installation/configuration, Virtual Hosting
Apache web server installation/configuration, Virtual Hosting
 
03_clere-HTTP2 HTTP3 the State of the Art in Our Servers.pdf
03_clere-HTTP2 HTTP3 the State of the Art in Our Servers.pdf03_clere-HTTP2 HTTP3 the State of the Art in Our Servers.pdf
03_clere-HTTP2 HTTP3 the State of the Art in Our Servers.pdf
 
VoxxedDays Minsk - Building scalable WebSocket backend
VoxxedDays Minsk - Building scalable WebSocket backendVoxxedDays Minsk - Building scalable WebSocket backend
VoxxedDays Minsk - Building scalable WebSocket backend
 
TDC2016SP - Vamos falar sobre o futuro da web: HTTP/2
TDC2016SP - Vamos falar sobre o futuro da web: HTTP/2TDC2016SP - Vamos falar sobre o futuro da web: HTTP/2
TDC2016SP - Vamos falar sobre o futuro da web: HTTP/2
 
Nginx, PHP, Apache and Spelix
Nginx, PHP, Apache and SpelixNginx, PHP, Apache and Spelix
Nginx, PHP, Apache and Spelix
 
K8s上の containerized cloud foundryとcontainerized open stackをprometheusで監視してみる
K8s上の containerized cloud foundryとcontainerized open stackをprometheusで監視してみるK8s上の containerized cloud foundryとcontainerized open stackをprometheusで監視してみる
K8s上の containerized cloud foundryとcontainerized open stackをprometheusで監視してみる
 
Http capturing
Http capturingHttp capturing
Http capturing
 
Caching and tuning fun for high scalability @ FOSDEM 2012
Caching and tuning fun for high scalability @ FOSDEM 2012Caching and tuning fun for high scalability @ FOSDEM 2012
Caching and tuning fun for high scalability @ FOSDEM 2012
 
IETF 100: Surviving IPv6 fragmentation
IETF 100: Surviving IPv6 fragmentationIETF 100: Surviving IPv6 fragmentation
IETF 100: Surviving IPv6 fragmentation
 
Стек Linux HTTPS/TCP/IP для защиты от HTTP-DDoS-атак
Стек Linux HTTPS/TCP/IP для защиты от HTTP-DDoS-атакСтек Linux HTTPS/TCP/IP для защиты от HTTP-DDoS-атак
Стек Linux HTTPS/TCP/IP для защиты от HTTP-DDoS-атак
 
AWS re:Invent 2016: Making Every Packet Count (NET404)
AWS re:Invent 2016: Making Every Packet Count (NET404)AWS re:Invent 2016: Making Every Packet Count (NET404)
AWS re:Invent 2016: Making Every Packet Count (NET404)
 

More from Thijs Feryn

More from Thijs Feryn (17)

10 things that helped me advance my career - PHP UK Conference 2024
10 things that helped me advance my career - PHP UK Conference 202410 things that helped me advance my career - PHP UK Conference 2024
10 things that helped me advance my career - PHP UK Conference 2024
 
Distributed load testing with K6 - NDC London 2024
Distributed load testing with K6 - NDC London 2024Distributed load testing with K6 - NDC London 2024
Distributed load testing with K6 - NDC London 2024
 
HTTP headers that make your website go faster - devs.gent November 2023
HTTP headers that make your website go faster - devs.gent November 2023HTTP headers that make your website go faster - devs.gent November 2023
HTTP headers that make your website go faster - devs.gent November 2023
 
Living on the edge - EBU Horizons 2023
Living on the edge - EBU Horizons 2023Living on the edge - EBU Horizons 2023
Living on the edge - EBU Horizons 2023
 
Distributed Load Testing with k6 - DevOps Barcelona
Distributed Load Testing with k6 - DevOps BarcelonaDistributed Load Testing with k6 - DevOps Barcelona
Distributed Load Testing with k6 - DevOps Barcelona
 
Core web vitals meten om je site sneller te maken - Combell Partner Day 2023
Core web vitals meten om je site sneller te maken - Combell Partner Day 2023Core web vitals meten om je site sneller te maken - Combell Partner Day 2023
Core web vitals meten om je site sneller te maken - Combell Partner Day 2023
 
HTTP headers that make your website go faster
HTTP headers that make your website go fasterHTTP headers that make your website go faster
HTTP headers that make your website go faster
 
HTTP headers that will make your website go faster
HTTP headers that will make your website go fasterHTTP headers that will make your website go faster
HTTP headers that will make your website go faster
 
Distributed load testing with k6
Distributed load testing with k6Distributed load testing with k6
Distributed load testing with k6
 
Taking Laravel to the edge with HTTP caching and Varnish
Taking Laravel to the edge with HTTP caching and VarnishTaking Laravel to the edge with HTTP caching and Varnish
Taking Laravel to the edge with HTTP caching and Varnish
 
Build your own CDN with Varnish - Confoo 2022
Build your own CDN with Varnish - Confoo 2022Build your own CDN with Varnish - Confoo 2022
Build your own CDN with Varnish - Confoo 2022
 
Caching the uncacheable with Varnish - DevDays 2021
Caching the uncacheable with Varnish - DevDays 2021Caching the uncacheable with Varnish - DevDays 2021
Caching the uncacheable with Varnish - DevDays 2021
 
Developing cacheable backend applications - Appdevcon 2019
Developing cacheable backend applications - Appdevcon 2019Developing cacheable backend applications - Appdevcon 2019
Developing cacheable backend applications - Appdevcon 2019
 
How Cloud addresses the needs of todays internet - Korazon 2018
How Cloud addresses the needs of todays internet - Korazon 2018How Cloud addresses the needs of todays internet - Korazon 2018
How Cloud addresses the needs of todays internet - Korazon 2018
 
Developing cacheable PHP applications - PHPLimburgBE 2018
Developing cacheable PHP applications - PHPLimburgBE 2018Developing cacheable PHP applications - PHPLimburgBE 2018
Developing cacheable PHP applications - PHPLimburgBE 2018
 
Leverage HTTP to deliver cacheable websites - Codemotion Rome 2018
Leverage HTTP to deliver cacheable websites - Codemotion Rome 2018Leverage HTTP to deliver cacheable websites - Codemotion Rome 2018
Leverage HTTP to deliver cacheable websites - Codemotion Rome 2018
 
Developing cacheable PHP applications - Confoo 2018
Developing cacheable PHP applications - Confoo 2018Developing cacheable PHP applications - Confoo 2018
Developing cacheable PHP applications - Confoo 2018
 

Recently uploaded

Recently uploaded (20)

Buy Epson EcoTank L3210 Colour Printer Online.pdf
Buy Epson EcoTank L3210 Colour Printer Online.pdfBuy Epson EcoTank L3210 Colour Printer Online.pdf
Buy Epson EcoTank L3210 Colour Printer Online.pdf
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 

HTTP logging met Varnishlog - PHPWVL 2022

  • 1. HTTP LOGGING MET VARNISHLOG Thijs Feryn
  • 2. 256
  • 6.
  • 7.
  • 8. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45
  • 9. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45 NCSA COMMON LOH FORMOAT
  • 10. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45 HOST IDENT AUTH USER DATE REQUEST STATUS BYTES NCSA COMMON LOH FORMOAT
  • 12. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 100.0.4896.127 Safari/537.36"
  • 13. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 100.0.4896.127 Safari/537.36" NCSA COMBINED LOH FORMOAT
  • 14. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 100.0.4896.127 Safari/537.36" NCSA COMBINED LOH FORMOAT REFERER
  • 15. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 100.0.4896.127 Safari/537.36" NCSA COMBINED LOH FORMOAT USER-AGENT
  • 16.
  • 17. CACHE
  • 19. * << Request >> 10 - Begin req 9 rxreq - Timestamp Start: 1651671623.893211 0.000000 0.000000 - Timestamp Req: 1651671623.893211 0.000000 0.000000 - ReqStart 172.21.0.1 62448 http - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.79.1 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: 172.21.0.1 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - ExpBan 6 banned lookup - VCL_call MISS - VCL_return fetch - Link bereq 11 fetch - Timestamp Fetch: 1651671623.896113 0.002902 0.002902 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Content-Type: application/json; charset=utf-8 - RespHeader Content-Length: 562 - RespHeader ETag: W/"232-+/Gw91fMkC3FNffhpz0wpLeOy1o" - RespHeader Date: Wed, 04 May 2022 13:40:23 GMT - RespHeader X-Varnish: 10 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0)
  • 20.
  • 21.
  • 22. YO, IK ZIN THIJS
  • 23. IK WEUNEN IN DIKSMUIDE
  • 24.
  • 26. IK ZIN DE TECH EVANHELIST BIE
  • 27.
  • 28.
  • 29. 9,000,000 WEBSITES 21% VAN DE TOP 10K WEBSITES
  • 31.
  • 32.
  • 34. OJ CACHT EJ GIN LOHS IP JE WEB SERVER
  • 38. vsl_space Value is: 80M [bytes] (default) Minimum is: 1M Maximum is: 4G The amount of space to allocate for the VSL fifo buffer in the VSM memory segment. If you make this too small, varnish{ncsa|log} etc will not be able to keep up. Making it too large just costs memory resources. NB: This parameter will not take any effect until the child process has been restarted.
  • 41. $ varnishadm param.set vsl_space 150M $ varnishadm stop $ varnishadm start
  • 46. Usage: /usr/bin/varnishlog <options> Options: [-a] Append to file [-A] Text output [-b] Only display backend records [-c] Only display client records [-C] Caseless regular expressions [-d] Process old log entries and exit [-D] Daemonize [-g <session|request|vxid|raw>] Grouping mode (default: vxid) [-h] Usage help [-i <taglist>] Include tags [-I <[taglist:]regex>] Include by regex [-k <num>] Limit transactions [-L <limit>] Incomplete transaction limit [-n <dir>] varnishd working directory [-P <file>] PID file [-q <query>] VSL query [-r <filename>] Binary file input [-R <limit[/duration]>] Output rate limit [-t <seconds|off>] VSM connection timeout [-T <seconds>] Transaction end timeout [-v] Verbose record printing [-V] Version [-w <filename>] Output filename [-x <taglist>] Exclude tags [-X <[taglist:]regex>] Exclude by regex
  • 47. * << Session >> 1752689 - Begin sess 0 HTTP/1 - SessOpen 172.21.0.1 55962 http 172.21.0.3 80 1651675446.691148 29 - Link req 1752690 rxreq - SessClose REM_CLOSE 0.004 - End
  • 48. varnishd -F -f /etc/varnish/default.vcl -a http=:80,HTTP -a proxy=:8443,PROXY -p feature=+http2 -s malloc,100M VARNISH RUNTIME PARAMETERS
  • 49. ** << Request >> 1752690 -- Begin req 1752689 rxreq -- Timestamp Start: 1651675446.691433 0.000000 0.000000 -- Timestamp Req: 1651675446.691433 0.000000 0.000000 -- ReqStart 172.21.0.1 55962 http -- ReqMethod GET -- ReqURL / -- ReqProtocol HTTP/1.1 -- ReqHeader Host: localhost -- ReqHeader User-Agent: curl/7.79.1 -- ReqHeader Accept: */* -- ReqHeader X-Forwarded-For: 172.21.0.1 -- VCL_call RECV -- VCL_return hash -- VCL_call HASH -- VCL_return lookup -- ExpBan 1679159 banned lookup -- VCL_call MISS -- VCL_return fetch -- Link bereq 1752691 fetch -- Timestamp Fetch: 1651675446.693975 0.002542 0.002542
  • 50. -- RespProtocol HTTP/1.1 -- RespStatus 200 -- RespReason OK -- RespHeader Content-Type: application/json; charset=utf-8 -- RespHeader Content-Length: 567 -- RespHeader ETag: W/"237-ulyqMICHeGkCSm+psitDJ2K3qqg" -- RespHeader Date: Wed, 04 May 2022 14:44:06 GMT -- RespHeader X-Varnish: 1752690 -- RespHeader Age: 0 -- RespHeader Via: 1.1 varnish (Varnish/6.0) -- VCL_call DELIVER -- VCL_return deliver -- Timestamp Process: 1651675446.693981 0.002548 0.000006 -- RespHeader Accept-Ranges: bytes -- RespHeader Connection: keep-alive -- Timestamp Resp: 1651675446.694030 0.002597 0.000049 -- ReqAcct 73 0 73 273 567 840 -- End
  • 51. *** << BeReq >> 1752691 --- Begin bereq 1752690 fetch --- VCL_use boot --- Timestamp Start: 1651675446.691561 0.000000 0.000000 --- BereqMethod GET --- BereqURL / --- BereqProtocol HTTP/1.1 --- BereqHeader Host: localhost --- BereqHeader User-Agent: curl/7.79.1 --- BereqHeader Accept: */* --- BereqHeader X-Forwarded-For: 172.21.0.1 --- BereqHeader Accept-Encoding: gzip --- BereqHeader X-Varnish: 1752691 --- VCL_call BACKEND_FETCH --- VCL_return fetch --- BackendOpen 32 boot.default 172.21.0.2 8080 172.21.0.3 36102 --- BackendStart 172.21.0.2 8080 --- Timestamp Bereq: 1651675446.691704 0.000143 0.000143 --- Timestamp Beresp: 1651675446.693860 0.002299 0.002156
  • 52. --- BerespProtocol HTTP/1.1 --- BerespStatus 200 --- BerespReason OK --- BerespHeader Content-Type: application/json; charset=utf-8 --- BerespHeader Content-Length: 567 --- BerespHeader ETag: W/"237-ulyqMICHeGkCSm+psitDJ2K3qqg" --- BerespHeader Date: Wed, 04 May 2022 14:44:06 GMT --- BerespHeader Connection: keep-alive --- BerespHeader Keep-Alive: timeout=5 --- TTL RFC 120 10 0 1651675447 1651675447 1651675446 0 0 cacheable --- VCL_call BACKEND_RESPONSE --- VCL_return deliver --- Storage malloc s0 --- Fetch_Body 3 length stream --- BackendReuse 32 boot.default --- Timestamp BerespBody: 1651675446.693945 0.002384 0.000085 --- Length 567 --- BereqAcct 145 0 145 214 567 781 --- End
  • 53.
  • 55. $ varnishlog -i Begin * << BeReq >> 3 - Begin bereq 2 fetch * << BeReq >> 5 - Begin bereq 4 fetch * << Request >> 4 - Begin req 2 esi * << Request >> 2 - Begin req 1 rxreq * << Session >> 1 - Begin sess 0 HTTP/1
  • 56. $ varnishlog -i Begin * << BeReq >> 3 - Begin bereq 2 fetch * << Request >> 2 - Begin req 1 rxreq * << Session >> 1 - Begin sess 0 HTTP/1 EEN SLAGSJE SIMPELER
  • 57.
  • 58. TRANSACTIES GROEPEREN ✓ VXID (DEFAULT) ✓ SESSION ✓ REQUEST ✓ RAW
  • 59. $ varnishlog -i Begin -g session * << Session >> 1 - Begin sess 0 HTTP/1 ** << Request >> 2 -- Begin req 1 rxreq *** << BeReq >> 3 --- Begin bereq 2 fetch *** << Request >> 4 --- Begin req 2 esi *4* << BeReq >> 5 -4- Begin bereq 4 fetch
  • 60. $ varnishlog -i Begin -g request * << Request >> 2 - Begin req 1 rxreq ** << BeReq >> 3 -- Begin bereq 2 fetch ** << Request >> 4 -- Begin req 2 esi *** << BeReq >> 5 --- Begin bereq 4 fetch
  • 61. $ varnishlog -i Begin -g raw 1 Begin c sess 0 HTTP/1 3 Begin b bereq 2 fetch 5 Begin b bereq 4 fetch 4 Begin c req 2 esi 2 Begin c req 1 rxreq
  • 62. varnishlog -g request varnishlog -g session varnishlog -g vxid varnishlog -g raw
  • 67. varnishlog -g request -c -i "Req*" -i Timestamp
  • 68. varnishlog -g request -b -i "BeReq*" -i Timestamp
  • 69. * << Request >> 1659437 - Timestamp Start: 1651676801.915256 0.000000 0.000000 - Timestamp Req: 1651676801.915256 0.000000 0.000000 - ReqUnset Upgrade: h2c - ReqUnset HTTP2-Settings: AAMAAABkAAQCAAAAAAIAAAAA - ReqProtocol HTTP/2.0 - ReqStart 172.21.0.1 57084 http - ReqMethod GET - ReqURL / - ReqProtocol HTTP/2.0 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.79.1 - ReqHeader Accept: */* - ReqHeader Connection: Upgrade, HTTP2-Settings - ReqHeader X-Forwarded-For: 172.21.0.1 - Timestamp Process: 1651676801.916362 0.001106 0.001106 - Timestamp Resp: 1651676801.916672 0.001416 0.000311 - ReqAcct 166 0 166 178 566 744
  • 70. varnishlog -g request -c -i "Req*,Resp*" -i Timestamp
  • 71. varnishlog -g request -c -i "Req*" -x ReqHeader
  • 72. varnishlog -g request -i ReqUrl -I ReqHeader:Accept-Language
  • 73. varnishlog -g request -i ReqUrl -i RespHeader -X "RespHeader:(X|x)-"
  • 75. <record selection criteria> <operator> <operand>
  • 80. %s: %f %f %f | | | | | | | +- Time since last timestamp | | +---- Time since start of work unit | +------- Absolute time of event +----------- Event label TIMESTAMP FORMOAT
  • 81. * << Request >> 11 - Timestamp Start: 1606398588.811189 0.000000 0.000000 - Timestamp Req: 1606398588.811189 0.000000 0.000000 - Timestamp Fetch: 1606398588.818399 0.007210 0.007210 - Timestamp Process: 1606398588.818432 0.007243 0.000032 - Timestamp Resp: 1606398588.818609 0.007421 0.000178
  • 82. varnishlog -c -i ReqUrl -I Timestamp:Resp -g request -q "{2+}Time*:Resp[2] > 2.0"
  • 83. varnishlog -g request -q "BerespStatus == 500"
  • 84. varnishlog -g request -q "BerespStatus >= 500" varnishlog -g request -q "BerespStatus > 500" varnishlog -g request -q "BerespStatus <= 500" varnishlog -g request -q "BerespStatus < 500"
  • 85. varnishlog -g request -q "ReqUrl ~ '^/contact'"
  • 86. varnishlog -g request -q "not ReqHeader:Accept-Language"
  • 87. varnishlog -g request -q "ReqUrl eq '/'" varnishlog -g request -q "ReqUrl ne '/contact'"
  • 88. varnishlog -c -i ReqUrl -I RespHeader:Content-Type -i reqacct -g request -q "RespHeader:Content-Type ~ '^image/' and ReqAcct[5] >= 2000000"
  • 89. %d %d %d %d %d %d | | | | | | | | | | | +- Total bytes transmitted | | | | +---- Body bytes transmitted | | | +------- Header bytes transmitted | | +---------- Total bytes received | +------------- Body bytes received +---------------- Header bytes received REQACCT FORMOAT
  • 90. varnishlog -c -g request -i ReqUrl -I VCL_call:PASS -I VCL_call:MISS -q "VCL_call eq 'MISS' or VCL_call eq 'PASS'"
  • 91. varnishlog -c -g request -i ReqUrl -q "TTL[6] eq 'uncacheable' and (BerespHeader:Set- Cookie or BerespHeader:Cache-Control ~ '(private|no- cache|no-store)')"
  • 94. varnishlog -g request -i requrl -d -q "Timestamp:Start[1] >= $(date -d '10/05/2022 11:00:00' +%s.0)"
  • 96. varnishlog -a -w vsl.log varnishlog -r vsl.log varnishlog -A -a -w vsl.log
  • 100. 172.17.0.1 - - [04/May/2022:11:57:56 +0000] "GET / HTTP/1.1" 200 45 "http://localhost/test.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 100.0.4896.127 Safari/537.36"
  • 101. varnishncsa -a -w /var/log/varnish/access.log
  • 102. %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
  • 103. %b Size of the response body in bytes %D Time taken to serve the request in microseconds %H The request protocol %h The hostname or IP address of the remote host %I Total bytes received %{X}i Contents of request header X %l Remote log name %m Request method %{X}o Contents of response header X %O In client mode, total bytes sent to client. In backend mode, total bytes received from the backend %q The query string %r First line of the HTTP request composed using other formatters %s HTTP status code of the response %t Time when the request was received %{X}t Time when request was received in the strftime time specification format %T Time taken to serve the request %U Request URL without query string %u Remote authenticated user %{X}x Extended Varnish & VCL variables
  • 104. varnishncsa -c -b -F '%h %l %u %t "%r" %s %b "% {Referer}i" "%{User-agent}i" "%{Varnish:side}x"'
  • 105. 172.18.0.3 - - [02/Dec/2020:10:18:38 +0000] "GET http://localhost/ HTTP/1.1" 200 6 "-" "curl" "b" 127.0.0.1 - - [02/Dec/2020:10:18:38 +0000] "GET http://localhost/ HTTP/1.1" 200 6 "-" "curl" "c"
  • 106. varnishncsa -F '%U %{Varnish:handling}x %{Varnish:side}x' -b -c /test - b /test miss c /test hit c /test - b /test pass c
  • 107. varnishncsa -b -F '%{VSL:Timestamp:BerespBody[2]}x % {Host}i %U%q' -q 'Timestamp:BerespBody[2] > 1.0' 2.007142 localhost /
  • 108. varnishncsa -a -D -w /var/log/varnish/access.log -P /var/run/varnishncsa.pid VARNISHNCSA DAEMONIZEN
  • 109. [Unit] Description=Varnish Cache HTTP accelerator NCSA logging daemon After=varnish.service [Service] RuntimeDirectory=varnishncsa Type=forking User=varnishlog Group=varnish ExecStart=/usr/bin/varnishncsa -a -w /var/log/varnish/ varnishncsa.log -D ExecReload=/bin/kill -HUP $MAINPID [Install] WantedBy=multi-user.target SYSTEMD
  • 110. /var/log/varnish/varnishncsa.log { daily rotate 7 compress delaycompress missingok postrotate systemctl -q is-active varnishncsa.service || exit 0 systemctl reload varnishncsa.service endscript } LOH ROTOASJE
  • 113. varnishtop -i requrl list length 3 31.67 ReqURL /contact 24.33 ReqURL /products 8.50 ReqURL /
  • 114. varnishtop -i ReqUrl -q "VCL_call eq 'MISS'" varnishtop -i ReqUrl -q "VCL_return eq 'pass'" varnishtop -i BeReqUrl -q "TTL[6] eq 'uncacheable'" varnishtop -i BeReqUrl -q "Timestamp:Beresp[3] > 2.0"
  • 115. varnishtop -i RespProtocol list length 2 310.29 RespProtocol HTTP/1.1 19.56 RespProtocol HTTP/2.0
  • 116. varnishtop -I Begin:sess list length 2 234.25 Begin sess 0 HTTP/1 49.89 Begin sess 0 PROXY
  • 117. VSL TAHS IP MOATE
  • 120. vcl 4.1; import std; import vsthrottle; backend default { .host = "localhost"; .port = "8080"; } sub vcl_recv { if (vsthrottle.is_denied(client.identity, 15, 10s, 30s)) { std.log("Ratelimit: " + client.identity + " blocked for another " + vsthrottle.blocked(client.identity, 15, 10s, 30s) + " seconds"); return (synth(429, "Too Many Requests")); } }
  • 121. vcl 4.1; import std; import vsthrottle; backend default { .host = "localhost"; .port = "8080"; } sub vcl_recv { if (vsthrottle.is_denied(client.identity, 15, 10s, 30s)) { std.log("Ratelimit: " + client.identity + " blocked for another " + vsthrottle.blocked(client.identity, 15, 10s, 30s) + " seconds"); return (synth(429, "Too Many Requests")); } }
  • 122. varnishlog -g request -i requrl -I VCL_Log:Ratelimit * << Request >> 557828 - ReqURL / - VCL_Log Ratelimit: 172.24.0.1 blocked for another 29.151 seconds