SlideShare a Scribd company logo
Log aggregation
using Elasticsearch, Fluentd/Fluentbit and Kibana
(EFK)
Who is this guy?
.
© 2018 Think Stack Limited thinkstack.io 2
Who is this guy and why is he talking to me?
• I've worked in the IT industry for over 18 years
• I started with VB6 & ASP and then .NET development
• I progressed into deployment/release management and
ultimately DevOps in 2012
• I've never presented a talk before - eek
© 2018 Think Stack Limited thinkstack.io 3
What I'm covering
• Introduction to log aggregation and why you want it
• Why I'm using Fluentd/Fluentbit as opposed to Logstash
• What's so great this setup?
• Demo: Collect and parse logs from a MySQL container
© 2018 Think Stack Limited thinkstack.io 4
What's not being covered
• Elasticsearch or Kibana and their features
• Detailed technical information that can be read online
• Any of the supporting tech being used e.g.
• Feel free to reach out to me if you have questions
© 2018 Think Stack Limited thinkstack.io 5
What is log aggregation?
© 2018 Think Stack Limited thinkstack.io 6
© 2018 Think Stack Limited thinkstack.io 7
Why am I using Fluentd?
• Well supported pluggable architecture
• Easy to understand configuration
• Lightweight
• Out-of-the-box compatibility in Kubernetes via Fluentbit
• Buffering capabilities
© 2018 Think Stack Limited thinkstack.io 8
Available Fluentd Plugin Types
.
© 2018 Think Stack Limited thinkstack.io 9
Fluentd Event Structure
• tag: From where an event originated; used for message routing
• time: The epoch time at which an event occurred
• record: The event log content as a JSON object
© 2018 Think Stack Limited thinkstack.io 10
What's so great about this setup? (1/2)
• I wanted something that used Fluentd
• I wanted to easily ingest logs from a variety of sources:
• via the Docker logging driver
• via the fluentd gem installed in a Ruby environment on Centos
• via the td-agent apt package installed on Ubuntu
• via the td-agent-bit yum package installed on Centos
© 2018 Think Stack Limited thinkstack.io 11
What's so great about this setup? (2/2)
• I wanted to play with non-trivial configurations, for example:
• TLS encryption
• Parsing of multi-line Java logs
• Using the ReadonlyREST plugin for security
• The use of Fluentd configuration include directives
© 2018 Think Stack Limited thinkstack.io 12
Fluentd Plugins Used
• in_forward: capture logs securely on port 24224 and unsecurely on port 24223
• parser_multi_format: parse logs where the log stream has more than one format e.g
Redis
• filter_record_transformer: used to add a 'source' key value pair
• out_elasticsearch: forward logs to Elasticsearch targetting different indices as
appropriate
• out_copy: copies logs to more than one output source e.g. Elasticsearch AND stdout
• out_rewrite_tag_filter: used to rewrite the tags from k8s and re-emit logs to process
© 2018 Think Stack Limited thinkstack.io 13
Local dev environment architecture
.
© 2018 Think Stack Limited thinkstack.io 14
Demo Time
© 2018 Think Stack Limited thinkstack.io 15
Sorry, I was drinking , eating or
What did I miss?
That's ok, during this demo we...
• launched a new MySQL container into our Minikube node
• captured MySQL logs from Kubernetes Fluentbit daemonset
• forwarded the logs to our Fluentd aggregator
• augmented the logs with a source key-value pair
• parsed the logs using the standard MySQL format
• wrote the logs to a new index mysql-* in Elasticsearch
• viewed the logs in Kibana's UI
© 2018 Think Stack Limited thinkstack.io 16
Any Questions?
Project is available publicly on GitHub
github.com/deploymentking/efk
Please feel free to contact me via...
mail@thinkstack.io
twitter.com/thinkstackio
linkedin.com/in/leemyring

More Related Content

What's hot

Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using Prometheus
Marco Pas
 
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaPrometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
Sridhar Kumar N
 
Hashicorp Vault - OPEN Public Sector
Hashicorp Vault - OPEN Public SectorHashicorp Vault - OPEN Public Sector
Hashicorp Vault - OPEN Public Sector
Kangaroot
 
Terraform
TerraformTerraform
Terraform
Phil Wilkins
 
Monitoring with prometheus
Monitoring with prometheusMonitoring with prometheus
Monitoring with prometheus
Kasper Nissen
 
Deep dive into Kubernetes Networking
Deep dive into Kubernetes NetworkingDeep dive into Kubernetes Networking
Deep dive into Kubernetes Networking
Sreenivas Makam
 
Docker.pptx
Docker.pptxDocker.pptx
Docker.pptx
balaji257
 
Server monitoring using grafana and prometheus
Server monitoring using grafana and prometheusServer monitoring using grafana and prometheus
Server monitoring using grafana and prometheus
Celine George
 
OpenTelemetry Introduction
OpenTelemetry Introduction OpenTelemetry Introduction
OpenTelemetry Introduction
DimitrisFinas1
 
Linux container, namespaces & CGroup.
Linux container, namespaces & CGroup. Linux container, namespaces & CGroup.
Linux container, namespaces & CGroup.
Neeraj Shrimali
 
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - PreviewVictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - Preview
VictoriaMetrics
 
Terraform
TerraformTerraform
Terraform -- Infrastructure as Code
Terraform -- Infrastructure as CodeTerraform -- Infrastructure as Code
Terraform -- Infrastructure as Code
Martin Schütte
 
Introduction to GitHub Actions
Introduction to GitHub ActionsIntroduction to GitHub Actions
Introduction to GitHub Actions
Knoldus Inc.
 
Prometheus design and philosophy
Prometheus design and philosophy   Prometheus design and philosophy
Prometheus design and philosophy
Docker, Inc.
 
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Amazon Web Services
 
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIData Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Altinity Ltd
 
Gitlab, GitOps & ArgoCD
Gitlab, GitOps & ArgoCDGitlab, GitOps & ArgoCD
Gitlab, GitOps & ArgoCD
Haggai Philip Zagury
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
Lucas Jellema
 
Docker Basic to Advance
Docker Basic to AdvanceDocker Basic to Advance
Docker Basic to Advance
Paras Jain
 

What's hot (20)

Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using Prometheus
 
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaPrometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
 
Hashicorp Vault - OPEN Public Sector
Hashicorp Vault - OPEN Public SectorHashicorp Vault - OPEN Public Sector
Hashicorp Vault - OPEN Public Sector
 
Terraform
TerraformTerraform
Terraform
 
Monitoring with prometheus
Monitoring with prometheusMonitoring with prometheus
Monitoring with prometheus
 
Deep dive into Kubernetes Networking
Deep dive into Kubernetes NetworkingDeep dive into Kubernetes Networking
Deep dive into Kubernetes Networking
 
Docker.pptx
Docker.pptxDocker.pptx
Docker.pptx
 
Server monitoring using grafana and prometheus
Server monitoring using grafana and prometheusServer monitoring using grafana and prometheus
Server monitoring using grafana and prometheus
 
OpenTelemetry Introduction
OpenTelemetry Introduction OpenTelemetry Introduction
OpenTelemetry Introduction
 
Linux container, namespaces & CGroup.
Linux container, namespaces & CGroup. Linux container, namespaces & CGroup.
Linux container, namespaces & CGroup.
 
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - PreviewVictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs: Open Source Log Management System - Preview
 
Terraform
TerraformTerraform
Terraform
 
Terraform -- Infrastructure as Code
Terraform -- Infrastructure as CodeTerraform -- Infrastructure as Code
Terraform -- Infrastructure as Code
 
Introduction to GitHub Actions
Introduction to GitHub ActionsIntroduction to GitHub Actions
Introduction to GitHub Actions
 
Prometheus design and philosophy
Prometheus design and philosophy   Prometheus design and philosophy
Prometheus design and philosophy
 
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
 
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UIData Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
Data Warehouses in Kubernetes Visualized: the ClickHouse Kubernetes Operator UI
 
Gitlab, GitOps & ArgoCD
Gitlab, GitOps & ArgoCDGitlab, GitOps & ArgoCD
Gitlab, GitOps & ArgoCD
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
 
Docker Basic to Advance
Docker Basic to AdvanceDocker Basic to Advance
Docker Basic to Advance
 

Similar to Log aggregation: using Elasticsearch, Fluentd/Fluentbit and Kibana (EFK)

Fluentd and docker monitoring
Fluentd and docker monitoringFluentd and docker monitoring
Fluentd and docker monitoring
Vinay Krishna
 
Implementing Observability for Kubernetes.pdf
Implementing Observability for Kubernetes.pdfImplementing Observability for Kubernetes.pdf
Implementing Observability for Kubernetes.pdf
Jose Manuel Ortega Candel
 
P4_tutorial.pdf
P4_tutorial.pdfP4_tutorial.pdf
P4_tutorial.pdf
PramodhN3
 
Python Linters at Scale.pdf
Python Linters at Scale.pdfPython Linters at Scale.pdf
Python Linters at Scale.pdf
Jimmy Lai
 
Scaling Prometheus on Kubernetes with Thanos
Scaling Prometheus on Kubernetes with ThanosScaling Prometheus on Kubernetes with Thanos
Scaling Prometheus on Kubernetes with Thanos
Thomas Riley
 
Kubernetes and container security
Kubernetes and container securityKubernetes and container security
Kubernetes and container security
Volodymyr Shynkar
 
Distributed Logging Architecture in the Container Era
Distributed Logging Architecture in the Container EraDistributed Logging Architecture in the Container Era
Distributed Logging Architecture in the Container Era
Glenn Davis
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container Era
SATOSHI TAGOMORI
 
Netflix Open Source: Building a Distributed and Automated Open Source Program
Netflix Open Source:  Building a Distributed and Automated Open Source ProgramNetflix Open Source:  Building a Distributed and Automated Open Source Program
Netflix Open Source: Building a Distributed and Automated Open Source Program
aspyker
 
Building a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at NetflixBuilding a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at Netflix
All Things Open
 
Introducing TiDB Operator
Introducing TiDB OperatorIntroducing TiDB Operator
Introducing TiDB Operator
Kevin Xu
 
A TRUE STORY ABOUT DATABASE ORCHESTRATION
A TRUE STORY ABOUT DATABASE ORCHESTRATIONA TRUE STORY ABOUT DATABASE ORCHESTRATION
A TRUE STORY ABOUT DATABASE ORCHESTRATION
InfluxData
 
The State of Logging on Docker
The State of Logging on DockerThe State of Logging on Docker
The State of Logging on Docker
Trevor Parsons
 
How to scale MongoDB
How to scale MongoDBHow to scale MongoDB
How to scale MongoDB
Igor Donchovski
 
Integrating Existing C++ Libraries into PySpark with Esther Kundin
Integrating Existing C++ Libraries into PySpark with Esther KundinIntegrating Existing C++ Libraries into PySpark with Esther Kundin
Integrating Existing C++ Libraries into PySpark with Esther Kundin
Databricks
 
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Codemotion
 
DevOps Supercharged with Docker on Exadata
DevOps Supercharged with Docker on ExadataDevOps Supercharged with Docker on Exadata
DevOps Supercharged with Docker on Exadata
MarketingArrowECS_CZ
 
Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...
Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...
Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...
ITCamp
 
Scalable Clusters On Demand
Scalable Clusters On DemandScalable Clusters On Demand
Scalable Clusters On Demand
Bogdan Kyryliuk
 
Docker and Fluentd
Docker and FluentdDocker and Fluentd
Docker and Fluentd
N Masahiro
 

Similar to Log aggregation: using Elasticsearch, Fluentd/Fluentbit and Kibana (EFK) (20)

Fluentd and docker monitoring
Fluentd and docker monitoringFluentd and docker monitoring
Fluentd and docker monitoring
 
Implementing Observability for Kubernetes.pdf
Implementing Observability for Kubernetes.pdfImplementing Observability for Kubernetes.pdf
Implementing Observability for Kubernetes.pdf
 
P4_tutorial.pdf
P4_tutorial.pdfP4_tutorial.pdf
P4_tutorial.pdf
 
Python Linters at Scale.pdf
Python Linters at Scale.pdfPython Linters at Scale.pdf
Python Linters at Scale.pdf
 
Scaling Prometheus on Kubernetes with Thanos
Scaling Prometheus on Kubernetes with ThanosScaling Prometheus on Kubernetes with Thanos
Scaling Prometheus on Kubernetes with Thanos
 
Kubernetes and container security
Kubernetes and container securityKubernetes and container security
Kubernetes and container security
 
Distributed Logging Architecture in the Container Era
Distributed Logging Architecture in the Container EraDistributed Logging Architecture in the Container Era
Distributed Logging Architecture in the Container Era
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container Era
 
Netflix Open Source: Building a Distributed and Automated Open Source Program
Netflix Open Source:  Building a Distributed and Automated Open Source ProgramNetflix Open Source:  Building a Distributed and Automated Open Source Program
Netflix Open Source: Building a Distributed and Automated Open Source Program
 
Building a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at NetflixBuilding a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at Netflix
 
Introducing TiDB Operator
Introducing TiDB OperatorIntroducing TiDB Operator
Introducing TiDB Operator
 
A TRUE STORY ABOUT DATABASE ORCHESTRATION
A TRUE STORY ABOUT DATABASE ORCHESTRATIONA TRUE STORY ABOUT DATABASE ORCHESTRATION
A TRUE STORY ABOUT DATABASE ORCHESTRATION
 
The State of Logging on Docker
The State of Logging on DockerThe State of Logging on Docker
The State of Logging on Docker
 
How to scale MongoDB
How to scale MongoDBHow to scale MongoDB
How to scale MongoDB
 
Integrating Existing C++ Libraries into PySpark with Esther Kundin
Integrating Existing C++ Libraries into PySpark with Esther KundinIntegrating Existing C++ Libraries into PySpark with Esther Kundin
Integrating Existing C++ Libraries into PySpark with Esther Kundin
 
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
 
DevOps Supercharged with Docker on Exadata
DevOps Supercharged with Docker on ExadataDevOps Supercharged with Docker on Exadata
DevOps Supercharged with Docker on Exadata
 
Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...
Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...
Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...
 
Scalable Clusters On Demand
Scalable Clusters On DemandScalable Clusters On Demand
Scalable Clusters On Demand
 
Docker and Fluentd
Docker and FluentdDocker and Fluentd
Docker and Fluentd
 

Recently uploaded

Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Dorra BARTAGUIZ
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
DianaGray10
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 

Recently uploaded (20)

Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 

Log aggregation: using Elasticsearch, Fluentd/Fluentbit and Kibana (EFK)

  • 1. Log aggregation using Elasticsearch, Fluentd/Fluentbit and Kibana (EFK)
  • 2. Who is this guy? . © 2018 Think Stack Limited thinkstack.io 2
  • 3. Who is this guy and why is he talking to me? • I've worked in the IT industry for over 18 years • I started with VB6 & ASP and then .NET development • I progressed into deployment/release management and ultimately DevOps in 2012 • I've never presented a talk before - eek © 2018 Think Stack Limited thinkstack.io 3
  • 4. What I'm covering • Introduction to log aggregation and why you want it • Why I'm using Fluentd/Fluentbit as opposed to Logstash • What's so great this setup? • Demo: Collect and parse logs from a MySQL container © 2018 Think Stack Limited thinkstack.io 4
  • 5. What's not being covered • Elasticsearch or Kibana and their features • Detailed technical information that can be read online • Any of the supporting tech being used e.g. • Feel free to reach out to me if you have questions © 2018 Think Stack Limited thinkstack.io 5
  • 6. What is log aggregation? © 2018 Think Stack Limited thinkstack.io 6
  • 7. © 2018 Think Stack Limited thinkstack.io 7
  • 8. Why am I using Fluentd? • Well supported pluggable architecture • Easy to understand configuration • Lightweight • Out-of-the-box compatibility in Kubernetes via Fluentbit • Buffering capabilities © 2018 Think Stack Limited thinkstack.io 8
  • 9. Available Fluentd Plugin Types . © 2018 Think Stack Limited thinkstack.io 9
  • 10. Fluentd Event Structure • tag: From where an event originated; used for message routing • time: The epoch time at which an event occurred • record: The event log content as a JSON object © 2018 Think Stack Limited thinkstack.io 10
  • 11. What's so great about this setup? (1/2) • I wanted something that used Fluentd • I wanted to easily ingest logs from a variety of sources: • via the Docker logging driver • via the fluentd gem installed in a Ruby environment on Centos • via the td-agent apt package installed on Ubuntu • via the td-agent-bit yum package installed on Centos © 2018 Think Stack Limited thinkstack.io 11
  • 12. What's so great about this setup? (2/2) • I wanted to play with non-trivial configurations, for example: • TLS encryption • Parsing of multi-line Java logs • Using the ReadonlyREST plugin for security • The use of Fluentd configuration include directives © 2018 Think Stack Limited thinkstack.io 12
  • 13. Fluentd Plugins Used • in_forward: capture logs securely on port 24224 and unsecurely on port 24223 • parser_multi_format: parse logs where the log stream has more than one format e.g Redis • filter_record_transformer: used to add a 'source' key value pair • out_elasticsearch: forward logs to Elasticsearch targetting different indices as appropriate • out_copy: copies logs to more than one output source e.g. Elasticsearch AND stdout • out_rewrite_tag_filter: used to rewrite the tags from k8s and re-emit logs to process © 2018 Think Stack Limited thinkstack.io 13
  • 14. Local dev environment architecture . © 2018 Think Stack Limited thinkstack.io 14
  • 15. Demo Time © 2018 Think Stack Limited thinkstack.io 15
  • 16. Sorry, I was drinking , eating or What did I miss? That's ok, during this demo we... • launched a new MySQL container into our Minikube node • captured MySQL logs from Kubernetes Fluentbit daemonset • forwarded the logs to our Fluentd aggregator • augmented the logs with a source key-value pair • parsed the logs using the standard MySQL format • wrote the logs to a new index mysql-* in Elasticsearch • viewed the logs in Kibana's UI © 2018 Think Stack Limited thinkstack.io 16
  • 17. Any Questions? Project is available publicly on GitHub github.com/deploymentking/efk Please feel free to contact me via... mail@thinkstack.io twitter.com/thinkstackio linkedin.com/in/leemyring