Fastly delivers more than a million log events per second. Setting it is up is easy, but there are many features you might not be using to their full extent. This workshop will cover setting up logging to various endpoints, dealing with structured data, and getting real-time insights into your customers’ behavior.

1. Logging at the Edge
Chris Jackel
Solutions Architect | Fastly
Mehran Nazir | Google

2. • How Fastly’s logs work What to log
• Set up a service and log endpoint
• What to Log
• JSON logging
• Fun with logs
• Serverless Applicationless Edge Logging
Agenda

3. • Log endpoint configured
• Customized Fastly log string
• JSON formatted
• BigQuery data import
Expectations at the end

4. Under the Hood

10. 159847 TxHeader c Connection: keep-alive
159847 TxHeader c X-Served-By: cache-jfk8125-JFK
159847 TxHeader c X-Cache: HIT
159847 TxHeader c X-Cache-Hits: 2
159847 TxHeader c X-Timer: S1489171719.109755,VS0,VE0
159847 VCL_call c log deliver
159847 VCL_call c log
159847 VCL_Log c syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test ::
{"client.ip":"207.38.219.230","req.request":"GET","req.http.host":"altitude-fastly.s3-website-us-
east-1.amazonaws.com","req.request":"GET","req.url":"/","req.bytes_read":94,"resp.status":
200,"resp.bytes_written":545,"resp.http.X-Cache":"HIT","fastly_info.state":"HIT","time.start.usec":
1489171719109755,"time.start.iso8601":"2017-03-10 18:48:39","time.end.usec":
1489171719109876,"time.elapsed.usec":121}
159847 VCL_return c deliver

11. UI/API Control plane
Varnish Logging Aggregator

12. 2 million LPS

13. Accessing Workshop
Accounts

14. www.loggingworkshop.com

15. Demo: Create a Service

16. Demo: Logentries

17. What to Log

18. https://docs.fastly.com/guides/streaming-logs/custom-log-formats#version-2-log-format
resp.http.Set-Cookie
server.datacenter
server.identity
server.region
if(req.http.Fastly-FF, "1", "0")
resp.http.X-Cache
fastly_info.state
Useful things to log

19. ERROR
HIT
HIT-CLUSTER
HIT-REFRESH
HIT-STALE
HIT-STALE-CLUSTER
HIT-STALE-WAIT-CLUSTER
HIT-SYNTH
HIT-WAIT
HIT-WAIT-CLUSTER
HITPASS
fastly_info.state
MISS
MISS-CLUSTER
MISS-WAIT
MISS-WAIT-CLUSTER
PASS
PASS-WAIT

20. regsub(fastly_info.state, "-.*", "")
obj.hits
geoip.continent_code
geoip.country_code
geoip.region (code for region within country)
req.restarts
Useful things to log

21. JSON Logging

22. Somewhat limited support for Apache log format variables
VCL variables can just dropped in
No (good) support for string literals, let alone things like JSON
Log format v1

23. Full support for Apache log format variables
Some will never return a value
VCL variables (or code) through Fastly specific extension: %{...}V
Supports string literals, and thus JSON is much easier to do
Default for all newly created endpoints
Legacy endpoints can be upgraded to v2
Log format v2

24. Log format v2 in UI

25. • <134>2017-03-19T12:47:40Z cache-jfk8127 foobarlogs2[39694]:
{"client_ip":"207.237.138.218","req_url":"/"}
Results

26. Originally hardcoded syslog prefix
Now 4 options:
syslog classic syslog prefix, RFC 3164 (default)
loggly modern syslog structured prefix, RFC 5424
logplex Heroku style prefix
blank no prefix, good for JSON and CSV
Currently available through API — and through the Portal TODAY
See https://docs.fastly.com/guides/streaming-logs/changing-log-line-formats
Prefixes

28. Log to GCS

29. <134>2017-03-19T13:13:33Z cache-jfk8147 foobarlogs2[453796]:
{"client_ip":"207.237.138.218","req_url":"/"}
<134>1 2017-03-19T13:13:33Z cache-jfk8147 - 453796 foobarlogs2 -
{"client_ip":"207.237.138.218","req_url":"/"}
114 <134>1 2017-03-19T13:13:33+00:00 cache-jfk8147 - 453796
foobarlogs2 - {"client_ip":"207.237.138.218","req_url":"/"}
{"client_ip":"207.237.138.218","req_url":"/"}
Prefixes

30. Back to JSON logging

31. log {"syslog 60idOs66l4AbzuqvgBypS2
foobar logs :: "} {"{"client_ip":""}
client.ip {"","req_url":""} req.url
{"""} "}";
Generated VCL

32. log "syslog 60idOs66l4AbzuqvgBypS2 foobar logs ::
{"
{""client_ip":""} client.ip {"","}
{""req_url":""} req.url {"""}
"}";
Custom VCL

33. Big Query

34. How Not to Log

35. Only log errors
if (beresp.status == 500 || beresp.status == 503) {
log {"syslog …etc… "};
}

36. if (randombool(std.atoi(table.lookup(logging, "percentage"))),
100) {
log {"syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test :: "} var.logstr;
}
curl -X PATCH -H "Fastly-Key: <api_token>" -d "item_value=5"
"https://api.fastly.com/service/<service_id>/dictionary/<dict_id>/
item/percentage"
Adjustable percentage

37. Per URL matching
if (table.lookup(panic_mode_logging, req.url)) {
log syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test ::
}
table panic_mode_logging {
“/my_broken_url” : “1”
}

38. declare local var.logstr STRING;
set var.logstr = …;
declare local var.endpoint STRING;
set var.endpoint = randomstr(1, "1234");
if (var.endpoint == "1") {
log {"syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test1 :: "} var.logstr;
} else if (var.endpoint == "2") {
log {"syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test2 :: "} var.logstr;
} else if (var.endpoint == "3") {
log {"syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test3 :: "} var.logstr;
} else { /* only "4" left */
log {"syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test4 :: "} var.logstr;
}
Load spreading

39. Edge Applications

40. backend default {
.host = “127.0.0.1”;
.port = “8080”;
.connect_timeout = 60s;
.first_byte_timeout = 60s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
acl purge {
“127.0.0.1”;
“localhost”;
}
sub vcl_recv {
set req.grace = 2m;
#Set X-Forwarded-For header for logging in nginx
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
Sample syntax highlighting style guide
Typography - Roboto Mono
Blue - action
Red - variables
Green - string
Orange - value
Grey - comment
Colors are in keynote