Our Drupal 8 websites are true applications, often very complex ones. More and more workload is being delegated to external systems, usually microservices, that are used for many different tasks. Software architectures are becoming more distributed and fragmented. To track down problems and optimize for performance, it will become mandatory to trace the lifecycle of a single request as it originates from a client, passes through all Drupal subsystems, reaches external (micro)services and comes back. This is often time consuming and without the right tools may become very difficult. A simple, unstructured log stream isn't enough anymore; we need to find a way to observe the details of what is going on. Observability is what it’s all about. This is based on structured logs, metrics and traces. In this talk you will see how to implement these techniques in Drupal, which tools and which modules to use to trace and log all requests that reach our website and how to expose and display useful metrics. We will integrate Drupal with OpenTracing, Prometheus, Monolog, Grafana and many more.

3. Luca Lusso Developer
Java, PHP, Go, Drupal, QA, Testing, ...
Cloud architect
AWS, Docker, Terraform, Ansible, ...
Open Source
Devel, Monolog, ...
Social
@lussoluca, drupal.org/u/lussoluca

4. So ware development
Design/UI/UX
Web marketing
QA & Testing
SEO & ML
www.wellnet.it

5. Almost everyone is working with distributed systems.
There are microservices, containers, cloud, serverless,
and a lot of combinations of these technologies. All of
these increase the number of failures that systems will
have because there are too many parts interacting.
And because of the distributed system’s diversity, it’s
complex to understand present problems and predict
future ones

6. "observability is a measure of how well
internal states of a system can be
inferred from knowledge of its external
outputs"

7. We want to observe production environments and
generic metrics like CPU and memory usage are not
sufficient anymore

8. 3 PILLARS OF OBSERVABILITY3 PILLARS OF OBSERVABILITY
1. Structured logs
2. Metrics
3. (Distributed) Traces

10. STRUCTURED LOGSSTRUCTURED LOGS

11. 1. Logs are about storing specific events

12. In Drupal 8 (an in general in PHP world) we have a
battle tested library to do logging
Monolog

13. Monolog is a PSR-3 compliant library from Jordi
Boggiano.
There is a module for both Drupal 7 and Drupal 8

14. We have to download the Monolog module using
Composer, to have both the module and the library
"require": {
"drupal/monolog": "1.3",

15. Then we have to create a file in the sites/default folder
of our Drupal website, for example
monolog.services.yml

16. monolog.services.yml
parameters:
monolog.channel_handlers:
default:
handlers: ['rotating_file']
formatter: 'json'
monolog.processors: [
'message_placeholder',
'current_user',
'request_uri',
'ip',
'referer',
'filter_backtrace',
'introspection'
]

17. monolog.services.yml (cont)
services:
monolog.handler.rotating_file:
class: MonologHandlerRotatingFileHandler
arguments: [
'public://logs/debug.log',
10,
'%monolog.level.info%'
]

18. Then in settings.php we add monolog.services.yml to
the list of container yamls
settings.php
$settings['container_yamls'][] =
DRUPAL_ROOT . '/sites/default/monolog.services.yml';

19. {
"message": "Session opened for admin.",
"context": [],
"level": 250,
"level_name": "NOTICE",
"channel": "user",
"datetime": {
"date": "2019-10-06 19:50:34.641160",
"timezone_type": 3,
"timezone": "UTC"
},
"extra": {
"file": "/var/www/html/web/core/modules/user/user.module",
"line": 556,
"class": null,
"function": "user_login_finalize",
"referer": "http://web.d86.docker.localhost/user/login",
"ip": "172.20.0.2",
"request_uri": "http://web.d86.docker.localhost/user/login
"uid": "1",
"user": "admin"
}
}

20. Structured logs makes it simple to query them for any
sort of useful information
We can write custom Monolog processors to add
application specific data to our logs

21. METRICSMETRICS

22. 1. Logs are about storing specific events
2. Metrics are a measurement at a point in time for the
system

23. Examples of the sort of metrics you might have would
be:
the number of times you received HTTP requests
how much time was spent handling requests
how many requests are currently in progress
the number of errors occurred

24. To instrument our application and record real-time
metrics we will use Prometheus (prometheus.io)

25. Prometheus was the second project to join the Cloud
Native Computing Foundation a er Kubernetes

27. To gather information from our production
environment we need two things:
instrument our application
extract data from the system

28. 1. Instrument our application

29. We start writing a simple module to implement
Observability in Drupal 8:
https://www.drupal.org/sandbox/lussoluca/3054802

30. The O11y module uses the
jimdo/prometheus_client_php library to implement a
Prometheus client (so you have to install it using
Composer)
Using the APC storage will reset metrics if server is
restarted, use the Redis storage to overcome this
$registry =
new PrometheusCollectorRegistry(
new PrometheusStorageAPC()
);

31. Prometheus has three types of metric:
1. Counters (represents a single monotonically
increasing counter)
$registry
->getOrRegisterCounter(
$namespace,
$name,
$help,
$labels
);

32. 2. Gauges (represents a single numerical value that
can arbitrarily go up and down)
$registry
->getOrRegisterGauge(
$namespace,
$name,
$help,
$labels
);

33. 3. Histograms (samples observations and counts them
in buckets)
$registry
->getOrRegisterHistogram(
$namespace,
$name,
$help,
$labels
);

34. o11y.module
function o11y_entity_insert(EntityInterface $entity) {
/** @var Drupalo11yMetrics $metrics */
$metrics = Drupal::service('o11y.metrics');
$counter = $metrics
->getOrRegisterCounter(
'drupal',
'entity_insert',
'Insert a new entity',
['type', 'bundle']
);
$counter
->incBy(
1,
[
$entity->getEntityTypeId(),
$entity->bundle()
]
);
}

35. The module esposes an URL with metrics in
Prometheus format (/metrics)
# HELP drupal_entity_insert Insert a new entity
# TYPE drupal_entity_insert counter
drupal_entity_insert{type="comment",bundle="comment"} 2
drupal_entity_insert{type="node",bundle="article"} 1

36. 2. Extract data from the system

37. Node exporter
Prometheus exporter for hardware and OS metrics
exposed by *NIX kernels, written in Go with pluggable
metric collectors

38. cpu
meminfo
filesystem
diskstats
netdev
...

39. Now we need to configure and run a Prometheus
server

40. prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'drupal'
scrape_interval: 5s
static_configs:
- targets: ['apache:80']
- job_name: 'nodeexporter'
scrape_interval: 5s
static_configs:
- targets: ['nodeexporter:9100']

41. docker-compose.yml
prometheus:
image: prom/prometheus:v2.13.0
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_librari
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention=200h'
- '--web.enable-lifecycle'
expose:
- 9090
ports:
- 9090:9090

42. How to query and visualize data?

43. Grafana
Grafana allows you to query, visualize, alert on and
understand your metrics (and logs)

44. docker-compose.yml
grafana:
image: grafana/grafana:6.4.3
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
expose:
- 3000
ports:
- 3000:3000

45. Grafana uses PromQL to let the user select and
aggregate time series data in real time

46. // PHP request per second
increase(php_request[5m])
// Entity created per second
increase(drupal_entity_insert{bundle="page"}[5m])
// PHP Memory peak
avg_over_time(php_memory_peak[5m])
// CPU usage
rate(node_cpu_seconds_total{mode="user"}[5m])

48. Now that we have Grafana up and running we can use
it also for viewing logs
Loki is a new project from Grafana Labs to scrape and
aggregate logs inspired by Prometheus

50. DISTRIBUTEDDISTRIBUTED
TRACESTRACES

51. 1. Logs are about storing specific events
2. Metrics are a measurement at a point in time for the
system
3. Distributed traces deals with information that is
request-scoped

52. https://opentracing.io
Vendor-neutral APIs and instrumentation for
distributed tracing
OpenTracing will be merged with OpenCensus to a
new project call OpenTelemetry

53. Per-process logging and metric monitoring have their
place, but neither can reconstruct the elaborate
journeys that transactions take as they propagate
across a distributed system. Distributed traces are
these journeys

54. We take for example a Drupal Commerce 2 website
with product prices that comes from a remote
microservice

55. src/Resolver/PriceResolver.php
public function resolve(
PurchasableEntityInterface $entity,
$quantity,
Context $context) {
$resp = $this->client->request(
'GET',
'http://localhost:10080?sku=' . $entity->get('sku')->value
);
$data = json_decode($resp->getBody()->getContents());
$price = new Price((string) $data->price->price, 'EUR');
return $price;
}

56. We want to trace the jouney of a request that arrives to
Drupal, traverses all Drupal layers, goes to the external
service and comes back

57. https://www.jaegertracing.io/
Distributed tracing system released as open source by
Uber Technologies
Jaeger clients are language specific implementations
of the OpenTracing API

59. There are different Jaeger clients for PHP
The most complete at the moment seems
jukylin/jaeger-php (the code is a little bit messy )

60. The O11y module instrument the application to:
generate a TraceID
trace Events
trace Twig
propagate TraceID to Guzzle calls

61. src/StackMiddleware/OpentracingMiddleware.php
public function handle(
Request $request,
$type = self::MASTER_REQUEST
$catch = TRUE) {
$span = $this->tracing->startSpan(
'Request',
[
'http.method' => $request->getMethod(),
'http.url' => $request->getRequestUri()
]
);
$handle = $this->httpKernel->handle($request, $type, $catch);
$span->finish();
$this->tracing->flush();
return $handle;
}

63. src/Http/ClientFactory.php
class ClientFactory extends CoreClientFactory {
public function fromOptions(array $config = []) {
$headers = [];
$tracer = $this->tracing->getTracer();
$span = $this->tracing->startSpan('external_call');
$tracer->inject(
$span->getContext(),
FormatsTEXT_MAP,
$headers
);
return parent::fromOptions(['headers' => $headers]);
}
}

66. One last thing we need is to correlate traces with logs,
so when we found a problem with a request we can go
from the trace to the logs (and viceversa)

67. The O11y module provides a new processor for
Monolog that adds a trace_id argument to every log

68. parameters:
monolog.channel_handlers:
default:
handlers: ['rotating_file']
formatter: 'json'
monolog.processors: [
'message_placeholder',
'current_user',
'request_uri',
'ip',
'referer',
'filter_backtrace',
'introspection',
'opentracing'
]

69. {
"message": "Session opened for admin.",
"context": [],
"level": 250,
"level_name": "NOTICE",
"channel": "user",
"datetime": {
"date": "2019-10-06 19:50:34.641160",
"timezone_type": 3,
"timezone": "UTC"
},
"extra": {
"file": "/var/www/html/web/core/modules/user/user.module",
"line": 556,
"class": null,
"function": "user_login_finalize",
"referer": "http://web.d86.docker.localhost/user/login",
"ip": "172.20.0.2",
"request_uri": "http://web.d86.docker.localhost/user/login
"uid": "1",
"user": "admin",
"trace_id": "15d12733c6ed916115d12733c6eb1881"
}
}

70. Next steps
Finish the O11y module
Integrate O11y with Webprofiler
Write instructions to setup all the stack

71. References
https://peter.bourgon.org/blog/2017/02/21/metrics-
tracing-and-logging.html
https://medium.com/opentracing/towards-turnkey-
distributed-tracing-5f4297d1736
https://medium.com/jaegertracing/jaeger-and-
opentelemetry-1846f701d9f2

75. questions?