A high level introduction to logging in Kubernetes and Konfigurator for dynamic log configuration

1. Dynamic log processing
with Fluentd and
Konfigurator
By Stakater

2. Logging
● one of the pillars of observability
● recording discrete events
● provide detailed information useful to understand the flow of events

3. Fluentd
● open source data collector for unified logging
● flexible plugin architecture
● Fluentd collects the logs, filter and then forward to configured
locations.
● On Kubernetes
○ deploy as a DaemonSet
○ all Nodes run a copy of the fluentd Pod
○ any new node getting added to the cluster will automatically get a fluentd pod

4. Logging flow
1. Containerized application writes to stdout and stderr is handled and redirected to a logging
driver, which is configured in Kubernetes to write to a file in json format.
○ Symlinks to the log files are created at /var/log/containers/*.log
2. The fluentd input plugin has responsibility for generating Fluentd event from data sources.
○ The in_tail Input plugin allows Fluentd to read and parse events from the tail of text files.
○ This position of last read is recorded in a position file
3. Fluentd creates an event reading from the source, which includes:
○ tag: Where an event comes from
○ time: When an event happens
○ record: Actual log content
4. Incoming events are matched with a Tag and correspondingly use the Output plugin type
○ The out_elasticsearch Output plugin writes records into Elasticsearch

5. Log parsing
● The parsing configuration for fluentd includes a regular expression
● this regular expression will be specific to the incoming logs
● quickly becomes unwieldy when deploying multiple applications or
updating a log format in an existing deployment
● will need to update fluentd configuration and reload fluentd

6. Konfigurator
● https://github.com/stakater/Konfigurator
● open source Kubernetes operator developed by Stakater for such as
use case
● dynamically generate app configuration when kubernetes resources
change
● watches Pods, Services and Ingresses for changes to configuration,
and then render configurations to ConfigMap and Secret.

7. Konfigurator
● Use Case
○ Every application may have different log format
○ Each new application will need to add a new format, e.g. regex, to fluentd
configuration
○ New application provides its format/regex
○ Konfigurator reads this, updates and remounts fluentd configuration
● Usage Steps
○ Create a config template for the app. This is a CRD (Custom Resource Definition)
called KonfiguratorTemplate.
○ Attach app specific configuration to the app’s pods

8. Contact
Email: hello@stakater.com
Web: http://stakater.com
Opensource: https://github.com/stakater/