This document discusses observability tools for distributed systems like Drupal websites. It recommends using Monolog for structured logging, the Prometheus monitoring system for metrics collection, and OpenTelemetry with Tempo for distributed tracing. The Observability suite module can integrate all three for Drupal. Monolog logs are scraped by Promtail and sent to Loki. Metrics are exposed via Prometheus and scraped. OpenTelemetry instruments code and sends traces to Tempo. This provides insights across logs, metrics and traces for observability of distributed applications.

2. lussoluca
Do you know what your Drupal is doing?
Observe it!

3. Luca Lusso
Drupal / PHP / Go developer @ SparkFabrik
Drupal contributor (WebProfiler, Monolog, …) and speaker
Drupal.org: https://www.drupal.org/u/lussoluca
Twitter: https://www.twitter.com/lussoluca
LinkedIn: www.linkedin.com/in/lussoluca
Slack (drupal.slack.com): lussoluca

4. WE ARE A TECH COMPANY OF ENGINEERS,
DEVELOPERS AND DESIGNERS WHO WILL
THINK, DESIGN AND BUILD YOUR CUSTOM APPLICATIONS,
MODERNIZE YOUR LEGACY AND TAKE YOU TO THE
CLOUD NATIVE ERA.
www.sparkfabrik.com
SPARKFABRIK

5. We help italian businesses to bridge
the gap with China thanks to our
Official Partnership with
Alibaba Cloud.
SparkFabrik is Cloud Native
Computing Foundation
Silver Member.
SparkFabrik is Google Cloud Platform
Technology Partner.
SparkFabrik is AWS
Official Partner.
PROUD OF OUR PARTNERSHIPS

6. Almost everyone is working with distributed systems.
There are microservices, containers, cloud, serverless, headless, message queues
and a lot of combinations of these technologies.

7. All of these, increase the number of failures that systems may encounter
because there are too many parts interacting.
And because of the distributed system’s diversity, it’s
complex to understand present problems and predict
future ones

8. Observability is a measure of how well
internal states of a system can be
inferred from knowledge of its external
outputs

9. We need more data to correlate classic metrics like CPU
and memory spikes with application behaviours

10. 3 pillars of
observability
● Structured logs
● Metrics
● (Distributed) Traces

12. Tools
We need different tools and technologies to collect logs, traces
and metrics, and we need a way to query and visualize them.
Logs -> Monolog + Promtail + Loki
Metrics -> Prometheus
Traces -> OpenTelemetry + Tempo
Visualization -> Grafana

13. Tools
Grafana
Grafana allows to query, visualize, alert on
and understand metrics, traces and logs no
matter where they are stored.
grafana.com

14. STRUCTURED LOGS

15. 1. Logs are about storing specific events

16. Tools
Monolog
Monolog is a standard PHP library and it can
be included in a Drupal website using a
contrib module (drupal.org/project/monolog)
Monolog sends logs to files, sockets,
inboxes, databases and various web
services.
Monolog implements the PSR-3 interface
that can be type-hint against in code to keep
a maximum of interoperability.
github.com/Seldaek/monolog

17. Download the Monolog module using
Composer, to have both the Drupal module and the PHP library
composer require drupal/monolog:^2.2

18. Monolog module doesn't have a UI, it's configured using yml files,
for example
sites/default/monolog.services.yml

19. services:
monolog.handler.rotating_file:
class: MonologHandlerRotatingFileHandler
arguments: ['private://logs/debug.log', 10, 'INFO']
First of all define a new service in the service container
sites/default/monolog.services.yml

20. parameters:
monolog.channel_handlers:
default:
handlers:
- name: 'rotating_file'
formatter: 'json'
monolog.processors: [
'message_placeholder', 'current_user',
'request_uri', 'ip', 'referer',
'filter_backtrace', 'introspection'
]
Then define handlers, formatter
and processors using service
container parameters. Here we're
configuring the default channel to
catch all log messages and to save
them using the
monolog.handler.rotating_file
service, in json format and after
being processed by a set of
processors
sites/default/monolog.services.yml

21. $settings['container_yamls'][] =
DRUPAL_ROOT . '/sites/default/monolog.services.yml';
Add monolog.services.yml to
the list of container’s yamls in settings.php file
sites/default/settings.php

22. Drupal::logger('drupalcon')->notice('Data from remote microservice.');

23. {
"message": "Data from remote microservice.",
"context": {},
"level": 250,
"level_name": "NOTICE",
"channel": "drupalcon",
"datetime": "2022-09-20T10:05:53.811568+02:00",
"extra": {
"referer": "",
"ip": "172.20.0.9",
"request_uri": "https://drupal10.ddev.site/microservice",
"uid": "1",
"user": "admin",
"file": "/.../drupalcon/src/Controller/MicroserviceController.php",
"line": 26,
"class": "DrupaldrupalconControllerMicroserviceController",
"function": "view"
}
}
private://logs/debug-2022-09-20.log

24. Structured logs makes it simple to query them for any
sort of useful information
We can write custom Monolog processors to add
application’s custom data to our logs
(for example: pod name, cluster id, order id, …)

25. In a Cloud Native environment, the application runs on multiple servers (or pods). We
need a way to export all those logs generated by every instance of the application.
In this case our logs are files stored in the local filesystem of every instance.
We have to discover, scrape and send them to a log collector.
Promtail is an agent which ships the contents of local logs to a private Grafana Loki
instance or Grafana Cloud. It is usually deployed to every machine that has applications
needed to be monitored.

26. Scraping logs with Grafana Promtail
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /mnt/ddev_config/o11y/loki/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: drupal
pipeline_stages:
- json:
expressions:
level: level_name
uid: extra.uid
- labels:
level:
uid:
- static_labels:
filename: 'drupal.log'
static_configs:
- targets:
- localhost
labels:
job: drupal
__path__: /var/www/html/web/sites/default/files/private/logs/*log

27. Scraping and sending logs to Grafana
Monolog -> logs (filesystem) -> Promtail -> Loki -> Grafana

28. METRICS

29. 1. Logs are about storing specific events
2. Metrics are a measurement at a point in time for the system

30. Tools
Prometheus
Prometheus is an open-source systems
monitoring and alerting toolkit originally built
at SoundCloud. It is now a standalone open
source project and it’s maintained
independently of any company.
Prometheus collects and stores metrics as
time series data.
Prometheus was the second project to join
the Cloud Native Computing Foundation after
Kubernetes.
prometheus.io

31. Examples of metrics you might need:
● the number of times you receive an HTTP request
● how much time was spent handling requests
● how many users/nodes has been created (over time)
● the number of modules that need a security update
● The number of orders in an ecommerce site
● And many more …

32. There’s a module for that!
Observability suite
https://www.drupal.org/project/o11y
composer require drupal/o11y:1.x-dev
drush pm:enable o11y_metrics o11y_metrics_requests

33. /admin/people/permissions
/admin/config/system/o11y_metrics/plugins-settings

34. Prometheus scrapes data at the /metrics endpoint at a configured rate
PHP uses a shared-nothing architecture by default
o11y needs a way to store data between one scrape and the next
default implementation uses Database as a storage backend (but you can use Memcache
or Redis as well)

35. The main module (o11y_metrics) provides the following metrics:
PHP info
Node count: total and with bundle labels
Node revision count: total and with bundle labels
Extensions: list of modules/themes/profiles installed with name and version labels
Queue size: list of queues with number of items in them
User count: total, with status (active/blocked) and role labels
A set of submodules can be installed to provide additional metrics:
o11y_metrics_cache: cache total hits, miss with bin labels. Tag invalidations, with tag and request path labels
o11y_metrics_config: information whether the drupal config is out of sync or not
o11y_metrics_database: histograms for time spent on select queries, with database target name and route labels
o11y_metrics_requests: histograms for time spent on requests, with http method, route name and http code status labels
o11y_metrics_update: info about existing core/module/theme updates
o11y_metrics_comment: comments count, with status labels

36. The module exposes an URL with metrics in
Prometheus format (/metrics)
# HELP drupal_http_requests Timing metrics for requests.
# TYPE drupal_http_requests histogram
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.005"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.01"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.025"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.05"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.075"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.1"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.25"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.5"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="0.75"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="1"} 0
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="2.5"} 4
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="5"} 6
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="7.5"} 6
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="10"} 6
drupal_http_requests_bucket{method="GET",route="user_admin_create",status="2xx",le="+Inf"} 6
drupal_http_requests_count{method="GET",route="user_admin_create",status="2xx"} 6

37. scrape_configs:
- job_name: 'drupal'
scrape_interval: 5s
static_configs:
- targets: ['web:80']
Scraping and sending metrics to Grafana Prometheus

38. o11y_metrics -> local database -> Prometheus -> Grafana

39. DISTRIBUTED TRACES

40. 1. Logs are about storing specific events
2. Metrics are a measurement at a point in time for the system
3. Distributed traces deals with information that is request-scoped

41. Tools
OpenTelemetry
OpenTelemetry is a collection of tools, APIs,
and SDKs. It can be used to instrument,
generate, collect, and export telemetry data
(metrics, logs, and traces) to help analyze
software’s performance and behavior.
OpenTelemetry is an incubating project from
the Cloud Native Computing Foundation,
created after the merger of OpenCensus
(from Google) and OpenTracing (from Uber).
The data collected with OpenTelemetry is
vendor-agnostic and can be exported in
many formats.
https://opentelemetry.io

42. Main cloud vendor support for OpenTelemetry
● AWS Distro for OpenTelemetry: aws-otel.github.io/
● Google Cloud OpenTelemetry: google-cloud-opentelemetry.readthedocs.io
● Azure Monitor OpenTelemetry:
docs.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview

43. Tools
OpenTelemetry
Language support
PHP:
Go:
Javascript:
Tracing Metrics Logging
pre-alpha pre-alpha not yet implemented
Tracing Metrics Logging
stable alpha not yet implemented
Tracing Metrics Logging
stable RC in development

44. The OpenTelemetry Collector offers a vendor-agnostic implementation of how to receive, process and export
telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors. It allows your service to
offload data quickly and the collector can take care of additional handling like retries, batching, encryption or even
sensitive data filtering.

45. Tools
Tempo
Grafana Tempo is an open source,
easy-to-use, and high-scale distributed
tracing backend. Tempo is cost-efficient,
requiring only object storage to operate, and
is deeply integrated with Grafana,
Prometheus, and Loki. Tempo can ingest
common open source tracing protocols,
including Jaeger, Zipkin, and
OpenTelemetry.
grafana.com/oss/tempo

46. We will use the Observability suite module to instrument our
application
Internally the module uses OpenTelemetry to do the hard work
drush pm:enable o11y_traces

47. $class_loader->addPsr4('Drupaltracer', [ __DIR__ . '/../../modules/contrib/tracer/src']);
$settings['container_base_class'] = 'DrupaltracerDependencyInjectionTraceableContainer';
$settings['tracer_plugin'] = 'o11y_tracer';
sites/default/settings.php
There should be only one module that instruments the code at a time. We need to replace a lot
of services and subsystems with traceable versions.
To avoid code duplication we create a third project: Tracer
(https://www.drupal.org/project/tracer) which both WebProfiler and Observability suite
depend on.
https://blog.sparkfabrik.com/en/webprofiler-updates

48. Per-process logging and metric monitoring have their
place, but neither can reconstruct the elaborate
journeys that transactions take as they propagate
across a distributed system. Distributed traces are
these journeys

49. We take for example a Drupal 10 website
that renders a page with some data that comes from a
remote microservice

50. /microservice1

51. class MicroserviceController extends ControllerBase {
private Client $httpClient;
public static function create(ContainerInterface $container) {
return new static(
$container->get('http_client')
);
}
final public function __construct(Client $httpClient) {
$this->httpClient = $httpClient;
}
public function view() {
$response = $this->httpClient->get('http://ddev-drupal10-microservice:8080/endpoint1');
$json = json_decode($response->getBody()->getContents());
$this->loggerFactory->get(drupalcon)->notice($json->message);
return [
'#theme => 'microservice',
'#message' => $json->message,
];
}
}

52. receivers:
otlp:
protocols:
http:
processors:
batch:
exporters:
otlphttp:
endpoint: http://tempo:4318
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]
OpenTelemetry collector to send traces to a
Grafana Tempo instance

53. Observability suite automatically instrument
● Events
● Twig templates
● HTTP calls
● Database queries
● Services (optional)
but you can trace your own code too!

54. class MicroserviceController extends ControllerBase {
public function view() {
$response =
$this->httpClient->get('http://ddev-drupal10-microservice:8080/endpoint2');
$json = json_decode($response->getBody()->getContents());
$this->someComplexMethod();
return [...];
}
private function someComplexMethod() {
$tracer = Drupal::service('tracer.tracer');
$span = $tracer->start('custom','someComplexMethod',
['someAttribute' => 'someValue']
);
sleep(1);
$tracer->stop($span);
}
}

55. o11y_traces -> OpenTelemetry collector -> Tempo -> Grafana

56. /microservice2

58. One last thing we need, is to correlate traces with logs,
so when we found a problem with a request we can go
from the trace to the logs (and viceversa)

59. The o11y module provides a new processor for
Monolog that adds a trace_id argument to every log

60. parameters:
monolog.channel_handlers:
default:
handlers:
- name: 'rotating_file'
formatter: 'json'
monolog.processors: [
'message_placeholder', 'current_user',
'request_uri', 'ip', 'referer',
'filter_backtrace', 'introspection', 'tracer'
]

62. ❯ curl -I https://drupalcon-prague-2022.ddev.site/en
HTTP/2 200
server: nginx/1.20.1
date: Mon, 19 Sep 2022 10:39:26 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
cache-control: must-revalidate, no-cache, private
x-drupal-dynamic-cache: MISS
x-drupal-trace-id: 355f26d048b5f8bb5e4aa232ce35a81d
content-language: en
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
permissions-policy: interest-cohort=()
expires: Sun, 19 Nov 1978 05:00:00 GMT
x-generator: Drupal 10 (https://www.drupal.org)
x-debug-token: a79d3a
x-debug-token-link:
https://drupalcon-prague-2022.ddev.site/en/admin/reports/profiler/view/a79d3a
x-drupal-cache: HIT

63. Example stack and source code:
https://github.com/lussoluca/drupalcon_prague_2022

64. Questions?

65. Join us for
contribution opportunities
20-23 September, 2022
Room C2 + C3
Mentored
Contribution
First Time
Contributor Workshop
General
Contribution
#DrupalContributions
20 - 22 September: 9:00 - 18:00
Room C3
23 September: 9:00 - 18:00
Room C2 + C3
20 September: 17:15 - 18:00
Room D9
21 September: 10:30 - 11:15
Room D9
23 September: 09:00 - 12:30
Room C2
23 September: 09:00 - 18:00
Room C2 + C3

66. What did you think?
Please fill in this session survey directly from the Mobile App.

67. We appreciate your feedback!
Please take a moment to fill out:
the general
conference survey
Flash the QR code
OR
It will be sent by email
the Individual
session surveys
(located under each session description)
1 2