How fluentd fits into the modern software landscape

Blog.mp3monster.org @mp3monster ‹#›
© 2021 Phil Wilkins. All rights reserved.
Making Logs Work for you
with Fluentd
Phil Wilkins
Technology Evangelist
Oracle Ace Director
uk.linkedin.com/in/philWilkins
@PhilAtCapgemini /
@MP3Monster
Oracle-integration.cloud /
APIPlatform.cloud /
Blog.mp3monster.org

The About Me …
Me in 5:
• Husband, Father, Blogger & Author
• Technical Architect, Tech Evangelist
• Work for Capgemini UK as part of a
multi award winning team
• Work with primarily open source +
Oracle cloud & middleware
• Know more – mp3monster.org
https://www.manning.com/
books/logging-in-action

Capgemini is One of the World's Largest Consulting,
Technology, and Outsourcing Firms & a global “full
service” business transformation provider
Group Workforce: 200,000+ Globally
Asia Pacific
Latin America
Canada
United States
Mexico
Brazil
Argentina
Europe
Morocco
Australia
People’s Republic of China
India
Chile
Guatemala
Russia
Singapore
Hong Kong
North
America
UK & Ireland
Nordics
Benelux
“It is the quality of our people, and their
capacity to deliver fitting solutions, with you
and for you, that drive real business results.”
Across 40+ countries, 100 nationalities
5Businesses
Revenue
12,8
Billion EUR (2017)
Central Europe
Morocco
Net Profit
€1,18B
 Targeting Value
 Mitigating Risk
 Optimising
Capabilities
 Aligning the
Organisation
Elements to
successful
collaboration
Application Services
Infrastructure
Services
Business Process
Outsourcing
Consulting
(Capgemini Consulting)
Local Professional
4

Putting Monitoring & Log Analytics
into Perspective
Host / Infrastructure
Monitoring
Virtualization /
Container Monitoring
Application Monitoring
Business Application
Monitoring
Security
/
SIEM
Capacity
Monitoring
Metrics – Typically
Numerical (sample based
data e.g. CPU use)
Logs Textual (event based
data e.g. app logging,
SNMP traps)
Observability Pillars / Key
Traces (execution flow &
timing – transaction
based, sampled)
Cambridge English Dictionary definition of
Monitor:
to look at something for a period of time,
especially something that is changing or moving
Observe:
to watch carefully the way something happens or
the way someone does something, especially in
order to learn more about it
The idea that there are 3 Pillars of
Observability has been around for a while:

Application of Logs
• Spotting unexpected errors
• performance issues
• Who did what & when
• Blend for end to end picture
• What is happening and how does it
impact our business!!

Increasing Complexity of End to End
Solutions
Evolution
Sophistication
/
Complexity
Apache
Tomcat /
Servlets
Async BEPL in
Clustered SOA
Async BEPL in
Clustered SOA +
JMS multi-
consumer
Node.js /
Kafka
consumer
s
Purist
Containerized
Microservices
/ Function as
a Service
OSB
Sync Flows in
SOA
No
Concurrency
Concurrency
through threads –
1 thread ran the
transaction
Asynchronous
Concurrency
(thread sharing /
reactive models)
Distributed
Concurrency
Modern
Development
Techniques
SOA /
Service Bus
Techniques

Highly Pluggable Framework
Input
• TCP/UDP
• Unix Sockets
• HTTP
• Many file formats
• SNMP traps
• OS (Linux/UNIX)
• Log4J, SLF4J and other related
frameworks for .Net, JavaScript
Output
• ALM solutions e.g. Splunk, cloud native
solutions, loggly, logzio etc.
• HTTP
• Prometheus
• Grafana
• Many file formats
• DB (SQL/NoSQL)
• Event Streams e.g. Kafka, SS, Kenesis, MQTT
• Social notifications e.g. Jabber, Slack, emai,
twilio l etc
• Support mgmt tools like pagerduty
Buffer /
Cache
• Custom in memory cache
• Redis
Storage
• S3 buckets
• Db
• Redis
Formatter
• XML
• JSON
• CSV/TSV etc.
• Compressed formats
Parser
• Multline text to single event
• Event info extraction e.g. date & time
Filters
• Value based conditions
• REGEX expressions
Custom components
Amusing Ruby Gems it is possible to build
any custom components using the framework
provided

• e.g. myInput
• Logical name defined by source
definition
Tag
• e.g. 1362050500.000000000
• Defined by time into Fluentd unless
mapped from received event
TimeStamp
• e.g. {“doYou” : “believe”, “this” :
“content” }
• Can be received as anything but
treated as a JSON object
Record
Log Event Make Up

Data to Actionable Information
Information
Source Capture
• Infra structure such as
CPU, memory use
• JVM use
• App Log Files
• SNMP Traps
Structure
& Route
• Get the raw data
to the
appropriate
tooling in a
format that can
be processed
Aggregate &
Analyze
•Data from
multiple sources
•Merge in time
series
Visualize
Data
• Search for log
events
• Present trends
e.g. memory
consumption,
• Rate of storage
consumption
Notify &
Alert
•Push events
into JIRA Svc
Desk, Slack, etc
•Rules on
severity dictate
behaviours
Fluentd – Optimal
Fluentd – Leverage Other Tools

ELK / EFK Logging Stacks
Log
Aggregation
/ Unification
Analytics
& Search
Visualization
Kibana
Elastic Search
Logstash Beats
(Lightweight
variant)
ELK
Elastic Search
Kibana
Fluent Bit
(Lightweight
variant)
Fluentd
EFK

Central Node (Node 2)
(Single Instance)
Node 1
(Instance n)
Multi Node Demo
• Transsform (event 
message)
• Match + copy:
• Out file
• relabel
labelPipeline
basic-file.txt
basic-file2.txt
label-pipeline-file-output.*
forwarder
common
• filter
Slack
Stdout /
Monitoring
Warboard
Op Analytics
filters

Central Node (Node 2)
(Single Instance)
Node 1
(Instance n)
Multi Node Demo
• Transsform (event 
message)
• Match + copy:
• Out file
• relabel
labelPipeline
basic-file.txt
basic-file2.txt
label-pipeline-file-output.*
forwarder
common
• filter
Slack
stdout

Fluentd Config Example – Node 1
<system>
Log_Level info
</system>
#### begin - tail basic file
<source>
@type tail
path ./Chapter6/basic-file.txt
read_lines_limit 5
tag basicFile
pos_file ./Chapter6/basic-file-read.pos_file
read_from_head true
<parse>
@type json
</parse>
@label labelPipeline
</source>
#### end - file 1
#### begin - tail basic-file2
<source>
@type tail
path ./Chapter6/basic-file2.txt
read_lines_limit 5
tag basicFILE2
pos_file ./Chapter6/basic-file-
read2.pos_file
read_from_head true
<parse>
@type json
</parse>
@label forwarder
</source>
#### end - tail basic-file2
<label labelPipeline>
align naming
<filter *>
@type record_transformer
<record>
message ${record["event"]}
transformed yes
event -
</record>
</filter>
#### begin - file out 1
<match *>
@type copy
<store>
@type file
path ./Chapter6/label-pipeline-file-output
<buffer>
delayed_commit_timeout 10
flush_at_shutdown true
chunk_limit_records 50
flush_interval 15
flush_mode interval
</buffer>
<format>
@type out_file
delimiter comma
output_tag true
</format>
</store>
<store>
@type relabel
@label forwarder
</store>
</match>
</label>
<label forwarder>
<match *>
@type forward
buffer_type memory
flush_interval 5s
<server>
host 127.0.0.1
port 28080
</server>
</match>
</label>

Fluentd Config Example – Node 2
<system>
Log_Level info
</system>
<source>
@type forward
port 28080
bind 0.0.0.0
@label common
</source>
#### begin -
for log events with the label common
<label common>
#<filter *>
# @type stdout
#</filter>
<filter *>
@type grep
<regexp>
key message
pattern /omputer/
</regexp>
</filter>
<filter *>
@type stdout
</filter>
<match *>
@type slack
token xoxb-xyz1234567890
username me
icon_emoji :ghost: # if you don't want to use
icon_url, delete this param.
channel general
message Node2 says - %s
message_keys message
title %s
title_keys tag
flush_interval 1s
time_key time
</match>
</label>

Fluentd Config Example – Test Tool Config
SOURCE=.testDatamedium-source.txt
TARGETFILE=./Demo/basic-file.txt
#%t (+1 or timestamp format %l = log level e.
g. DEBUG %c class e.g. demo.code.com %p proc
ess or thread id %m mlog message
SOURCEFORMAT=%t %m
TARGETFORMAT={"source":"basicFile", "event":"%
m", "cycle":%i}
OUTPUTTYPE=file
DEFAULT-PROPCESS=Thread-1
DEFAULT-LOCATION=com.demo
DEFAULT-LOGLEVEL=INFO
REPEAT=5
VERBOSE=true
SOURCE=.testDatasmall-fact-source.txt
TARGETFILE=./Demo/basic-file2.txt
#%t (+1 or timestamp format %l = log level e.g.
DEBUG %c class e.g. demo.code.com %p process or
thread id %m mlog message
SOURCEFORMAT=%t %m
TARGETFORMAT={"message":"%m", "stream":"basicFILE
2", "iter":%i}
OUTPUTTYPE=file
DEFAULT-PROCESS=Thread-1
DEFAULT-LOCATION=org.demonstration
DEFAULT-LOGLEVEL=INFO
REPEAT=7
VERBOSE=false
Source 1 Config Source 2 Config

Demo Output to Slack

Realworld & Possible scaling &
deployment approaches

Addressing Real-World Challenges
In the realworld we have more significant challenges …
• Highly distributed solutions that need to have logging and monitoring
consolidated
• Tracing for reactive and solutions – context switching rather than threads in
the execution
• Often different teams want to use different tools – security want Splunk, DBAs
want OEM, infrastructure teams want Nagios – making setup of environments
more complex than need be
• Some operational events are more critical than others – need to filter those
out
• Make legacy solutions easier to operate, isolate log events and tag them with
operational code references – so process & care embedded without impacting
the app

FluentD – Scaling & Aggregation
App A
(Front End)
Server
App B
Server
Server
Shared Persistence / Analytics Platform
Ops
Alerting
Service
Server
App A
(Mid Tier)
Server
App A
(Mid Tier)
Server
App C
Server
Server Server

Kubernetes / Container
App A
(Front End)
Pod
App B
Pod
Svc
Ops
Alerting
Service
Pod
App A
(Mid Tier)
Pod
App A
(Mid Tier)
Pod
App C
Pod
Svc
Pod
Worker Node
Worker Node

Kubernetes / Container
App A
(Front End)
Pod
App B
Pod
Pod
Ops
Alerting
Service
Pod
App A
(Mid Tier)
Pod
App A
(Mid Tier)
Pod
App C
Pod
Pod Pod
Worker Node
Worker Node

Kubernetes / Container – using Side Cars
App A
(Front End)
Pod
App B
Pod
Pod
Ops
Alerting
Service
Pod
App A
(Mid Tier)
Pod
App A
(Mid Tier)
Pod
App C
Pod
Pod Pod
Worker Node
Worker Node
Side
Car
Side
Car
Side
Car

Prometheus Server
Retrieval TSDB
HTTP
Server
Storage Prometheu
s UI
Prometheus
Push Gateway
Grafana
API Clients
Exporters
Prometheus
Alert Mgr
Exporters
Exporter
Fluentd
Fluentd
eMail
Other
External
Process Push
Fluentd
Process
Service Discovery
Kubernetes File AD
Fluentd with Prometheus

Real Use Case
Cloud Native
Monitoring
/ Cloud Watch
Elasticsearch
– analytics /
search
JIRA
search process
may result in
values set to
Prometheus
Analytics
Fluentd
(Aggregate & Route)
Notification
channels like
Slack
Information Source
Capture
Structure & Route Aggregate & Analyze
Visualize
Data
Notify &
Alert
Limited to subset
of SEIM events
App
(Open
Tracing
API)
Collector

Thankyou
Phil Wilkins
Technology Evangelist
Oracle Ace Director
uk.linkedin.com/in/philWilkins
@PhilAtCapgemini /
@MP3Monster
Oracle-integration.cloud /
APIPlatform.cloud /
Blog.mp3monster.org

With more than 190,000 people, Capgemini is present in over 40 countries and
celebrates its 50th Anniversary year in 2018. A global leader in consulting, technology
and outsourcing services, the Group reported 2016 global revenues of EUR 12.5 billion.
Together with its clients, Capgemini creates and delivers business, technology and
digital solutions that fit their needs, enabling them to achieve innovation and
competitiveness. A deeply multicultural organization, Capgemini has developed its own
way of working, the Collaborative Business Experience™, and draws on Rightshore®, its
worldwide delivery model.
About Capgemini
Learn more about us at
www.capgemini.com
This message contains information that may be privileged or confidential and is
the property of the Capgemini Group.
Copyright © 2018 Capgemini. All rights reserved.
Rightshore® is a trademark belonging to Capgemini.
This message is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to
read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please
notify the sender immediately and delete all copies of this message.

How fluentd fits into the modern software landscape

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How fluentd fits into the modern software landscape

Similar to How fluentd fits into the modern software landscape (20)

More from Phil Wilkins

More from Phil Wilkins (20)

Recently uploaded

Recently uploaded (20)

How fluentd fits into the modern software landscape

Editor's Notes