SlideShare a Scribd company logo

Fluentd – Making Logging Easy & Effective in a Multi-cloud & Hybrid Environment - Deverloper Week Europe 22.pptx

Phil Wilkins
Phil Wilkins
Phil WilkinsCloud Developer Evangelist (Author & Oracle Ace Director Alumni) at Oracle

Presentation I gave to Developer Week Europe 2022 on the use of Fluentd in Hybrid and distributed use cases. This builds on previous Fluentd presentations

Fluentd – Making Logging Easy & Effective in a Multi-cloud & Hybrid Environment - Deverloper Week Europe 22.pptx

1 of 24
Download to read offline
Fluentd – Making Logging Easy & Effective
in a Multi-cloud & Hybrid Environment
Name
Phil Wilkins
OCI
April 2022
Speaker
PhilWilkins
Cloud Developer Evangelist
Copyright © 2022, Oracle and/or its affiliates
Philip.Wilkins@Oracle.com
cloud-native.info / mp3monster.org / oracle-
integration.cloud
www.linkedin.com/in/philwilkins/
@mp3monster
3 Copyright © 2022, Oracle and/or its affiliates
https://www.manning.com/books/unified
-logging-with-fluentd
https://bit.ly/FluentdBook
Philip.Wilkins@Oracle.com
cloud-native.info / mp3monster.org / oracle-
integration.cloud
www.linkedin.com/in/philwilkins/
@mp3monster
The following is intended to present ideas & application of open
technologies. It is intended for information purposes only, and may
not be incorporated into any contract. It is not a commitment to
deliver any material, code, or functionality, and should not be relied
upon in making purchasing decisions. The development, release,
timing, and pricing of any features or functionality described for
Oracle’s products may change and remains at the sole discretion of
Oracle Corporation.
This presentation contains the thoughts and ideas of
this presenter & does not necessarily represent the
views of Oracle Corporation.
4 Copyright © 2022, Oracle and/or its affiliates
Putting Monitoring & Log Analytics into Perspective
Our view of monitoring is heavily influenced by our role.
• Devs think app logs –(structured) text
• Infra think numeric – server performance/resource
consumption etc.
• Business / Low Coders use both numeric & text –
transaction counts & status
DevOps is challenging that and ideas like 3 Pillars of
Observability have matured:
Copyright © 2022, Oracle and/or its affiliates
Metrics – Typically
Numerical (sample based
data e.g. CPU use)
Logs Textual (event based
data e.g. app logging,
SNMP traps)
Observability Pillars / Key
Traces (execution flow &
timing – transaction
based, sampled)
Host / Infrastructure
Monitoring
Virtualization /
Container Monitoring
Application Monitoring
Business Application
Monitoring
Security
/
SIEM
Capacity
Monitoring
5
What is Fluentd ?
6 Copyright © 2022, Oracle and/or its affiliates
Ad

Recommended

Is 12 Factor App Right About Logging
Is 12 Factor App Right About LoggingIs 12 Factor App Right About Logging
Is 12 Factor App Right About LoggingPhil Wilkins
 
Best practices for application migration to public clouds interop presentation
Best practices for application migration to public clouds interop presentationBest practices for application migration to public clouds interop presentation
Best practices for application migration to public clouds interop presentationesebeus
 
Breaking the Monolith
Breaking the MonolithBreaking the Monolith
Breaking the MonolithVMware Tanzu
 
Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle GoldenGate Roadmap Oracle OpenWorld 2020 Oracle GoldenGate Roadmap Oracle OpenWorld 2020
Oracle GoldenGate Roadmap Oracle OpenWorld 2020 Oracle
 
(ZDM) Zero Downtime DB Migration to Oracle Cloud
(ZDM) Zero Downtime DB Migration to Oracle Cloud(ZDM) Zero Downtime DB Migration to Oracle Cloud
(ZDM) Zero Downtime DB Migration to Oracle CloudRuggero Citton
 
What’s New in Documentum 7.3
What’s New in Documentum 7.3What’s New in Documentum 7.3
What’s New in Documentum 7.3Michael Mohen
 
Building Cloud Native Applications with Oracle Autonomous Database.
Building Cloud Native Applications with Oracle Autonomous Database.Building Cloud Native Applications with Oracle Autonomous Database.
Building Cloud Native Applications with Oracle Autonomous Database.Oracle Developers
 
OOW-5185-Hybrid Cloud
OOW-5185-Hybrid CloudOOW-5185-Hybrid Cloud
OOW-5185-Hybrid CloudBen Duan
 

More Related Content

Similar to Fluentd – Making Logging Easy & Effective in a Multi-cloud & Hybrid Environment - Deverloper Week Europe 22.pptx

Automated Deployment and Management of Edge Clouds
Automated Deployment and Management of Edge CloudsAutomated Deployment and Management of Edge Clouds
Automated Deployment and Management of Edge CloudsJay Bryant
 
Design and Deploy Secure Clouds for Financial Services Use Cases
Design and Deploy Secure Clouds for Financial Services Use CasesDesign and Deploy Secure Clouds for Financial Services Use Cases
Design and Deploy Secure Clouds for Financial Services Use CasesPLUMgrid
 
FluentD for end to end monitoring
FluentD for end to end monitoringFluentD for end to end monitoring
FluentD for end to end monitoringPhil Wilkins
 
4th SDN Interest Group Seminar-Session 2-2(130313)
4th SDN Interest Group Seminar-Session 2-2(130313)4th SDN Interest Group Seminar-Session 2-2(130313)
4th SDN Interest Group Seminar-Session 2-2(130313)NAIM Networks, Inc.
 
Oracle Ravello Presentation 7Dec16 v1
Oracle Ravello Presentation 7Dec16 v1Oracle Ravello Presentation 7Dec16 v1
Oracle Ravello Presentation 7Dec16 v1Kurt Liu
 
GDG Cloud Southlake #8 Steve Cravens: Infrastructure as-Code (IaC) in 2022: ...
GDG Cloud Southlake #8  Steve Cravens: Infrastructure as-Code (IaC) in 2022: ...GDG Cloud Southlake #8  Steve Cravens: Infrastructure as-Code (IaC) in 2022: ...
GDG Cloud Southlake #8 Steve Cravens: Infrastructure as-Code (IaC) in 2022: ...James Anderson
 
.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los AngelesVMware Tanzu
 
Service Mesh and Serverless Chatbots with Linkerd, K8s and OpenFaaS
Service Mesh and Serverless Chatbots with Linkerd, K8s and OpenFaaSService Mesh and Serverless Chatbots with Linkerd, K8s and OpenFaaS
Service Mesh and Serverless Chatbots with Linkerd, K8s and OpenFaaSSoftware Guru
 
Software Innovations and Control Plane Evolution in the new SDN Transport Arc...
Software Innovations and Control Plane Evolution in the new SDN Transport Arc...Software Innovations and Control Plane Evolution in the new SDN Transport Arc...
Software Innovations and Control Plane Evolution in the new SDN Transport Arc...Cisco Canada
 
Replicate data between environments
Replicate data between environmentsReplicate data between environments
Replicate data between environmentsDLT Solutions
 
Hybrid Cloud Keynote
Hybrid Cloud Keynote Hybrid Cloud Keynote
Hybrid Cloud Keynote gcamarda
 
Oracle - Continuous Delivery NYC meetup, June 07, 2018
Oracle - Continuous Delivery NYC meetup, June 07, 2018Oracle - Continuous Delivery NYC meetup, June 07, 2018
Oracle - Continuous Delivery NYC meetup, June 07, 2018Oracle Developers
 
MOUS 2020 - Hyperion 11.2 vs. Cloud: Should I Stay or Should I Go?
MOUS 2020 - Hyperion 11.2 vs. Cloud: Should I Stay or Should I Go?MOUS 2020 - Hyperion 11.2 vs. Cloud: Should I Stay or Should I Go?
MOUS 2020 - Hyperion 11.2 vs. Cloud: Should I Stay or Should I Go?Datavail
 
PLNOG14: The benefits of "OPEN" in networking for operators - Joerg Ammon, Br...
PLNOG14: The benefits of "OPEN" in networking for operators - Joerg Ammon, Br...PLNOG14: The benefits of "OPEN" in networking for operators - Joerg Ammon, Br...
PLNOG14: The benefits of "OPEN" in networking for operators - Joerg Ammon, Br...PROIDEA
 
Hit Refresh with Oracle GoldenGate Microservices
Hit Refresh with Oracle GoldenGate MicroservicesHit Refresh with Oracle GoldenGate Microservices
Hit Refresh with Oracle GoldenGate MicroservicesBobby Curtis
 
Oracle goldegate microservice
Oracle goldegate microserviceOracle goldegate microservice
Oracle goldegate microserviceMojtaba Khandan
 
MySQL day Dublin - OCI & Application Development
MySQL day Dublin - OCI & Application DevelopmentMySQL day Dublin - OCI & Application Development
MySQL day Dublin - OCI & Application DevelopmentHenry J. Kröger
 
Databarracks & SolidFire - How to run tier 1 applications in the cloud
Databarracks & SolidFire - How to run tier 1 applications in the cloud Databarracks & SolidFire - How to run tier 1 applications in the cloud
Databarracks & SolidFire - How to run tier 1 applications in the cloud NetApp
 

Similar to Fluentd – Making Logging Easy & Effective in a Multi-cloud & Hybrid Environment - Deverloper Week Europe 22.pptx (20)

Automated Deployment and Management of Edge Clouds
Automated Deployment and Management of Edge CloudsAutomated Deployment and Management of Edge Clouds
Automated Deployment and Management of Edge Clouds
 
Design and Deploy Secure Clouds for Financial Services Use Cases
Design and Deploy Secure Clouds for Financial Services Use CasesDesign and Deploy Secure Clouds for Financial Services Use Cases
Design and Deploy Secure Clouds for Financial Services Use Cases
 
FluentD for end to end monitoring
FluentD for end to end monitoringFluentD for end to end monitoring
FluentD for end to end monitoring
 
4th SDN Interest Group Seminar-Session 2-2(130313)
4th SDN Interest Group Seminar-Session 2-2(130313)4th SDN Interest Group Seminar-Session 2-2(130313)
4th SDN Interest Group Seminar-Session 2-2(130313)
 
Oracle Ravello Presentation 7Dec16 v1
Oracle Ravello Presentation 7Dec16 v1Oracle Ravello Presentation 7Dec16 v1
Oracle Ravello Presentation 7Dec16 v1
 
GDG Cloud Southlake #8 Steve Cravens: Infrastructure as-Code (IaC) in 2022: ...
GDG Cloud Southlake #8  Steve Cravens: Infrastructure as-Code (IaC) in 2022: ...GDG Cloud Southlake #8  Steve Cravens: Infrastructure as-Code (IaC) in 2022: ...
GDG Cloud Southlake #8 Steve Cravens: Infrastructure as-Code (IaC) in 2022: ...
 
.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles
 
Service Mesh and Serverless Chatbots with Linkerd, K8s and OpenFaaS
Service Mesh and Serverless Chatbots with Linkerd, K8s and OpenFaaSService Mesh and Serverless Chatbots with Linkerd, K8s and OpenFaaS
Service Mesh and Serverless Chatbots with Linkerd, K8s and OpenFaaS
 
Software Innovations and Control Plane Evolution in the new SDN Transport Arc...
Software Innovations and Control Plane Evolution in the new SDN Transport Arc...Software Innovations and Control Plane Evolution in the new SDN Transport Arc...
Software Innovations and Control Plane Evolution in the new SDN Transport Arc...
 
Replicate data between environments
Replicate data between environmentsReplicate data between environments
Replicate data between environments
 
OpenStack Murano
OpenStack MuranoOpenStack Murano
OpenStack Murano
 
Hybrid Cloud Keynote
Hybrid Cloud Keynote Hybrid Cloud Keynote
Hybrid Cloud Keynote
 
Oracle - Continuous Delivery NYC meetup, June 07, 2018
Oracle - Continuous Delivery NYC meetup, June 07, 2018Oracle - Continuous Delivery NYC meetup, June 07, 2018
Oracle - Continuous Delivery NYC meetup, June 07, 2018
 
MOUS 2020 - Hyperion 11.2 vs. Cloud: Should I Stay or Should I Go?
MOUS 2020 - Hyperion 11.2 vs. Cloud: Should I Stay or Should I Go?MOUS 2020 - Hyperion 11.2 vs. Cloud: Should I Stay or Should I Go?
MOUS 2020 - Hyperion 11.2 vs. Cloud: Should I Stay or Should I Go?
 
PLNOG14: The benefits of "OPEN" in networking for operators - Joerg Ammon, Br...
PLNOG14: The benefits of "OPEN" in networking for operators - Joerg Ammon, Br...PLNOG14: The benefits of "OPEN" in networking for operators - Joerg Ammon, Br...
PLNOG14: The benefits of "OPEN" in networking for operators - Joerg Ammon, Br...
 
Hit Refresh with Oracle GoldenGate Microservices
Hit Refresh with Oracle GoldenGate MicroservicesHit Refresh with Oracle GoldenGate Microservices
Hit Refresh with Oracle GoldenGate Microservices
 
Oracle goldegate microservice
Oracle goldegate microserviceOracle goldegate microservice
Oracle goldegate microservice
 
MySQL day Dublin - OCI & Application Development
MySQL day Dublin - OCI & Application DevelopmentMySQL day Dublin - OCI & Application Development
MySQL day Dublin - OCI & Application Development
 
Databarracks & SolidFire - How to run tier 1 applications in the cloud
Databarracks & SolidFire - How to run tier 1 applications in the cloud Databarracks & SolidFire - How to run tier 1 applications in the cloud
Databarracks & SolidFire - How to run tier 1 applications in the cloud
 
Exadata Cloud Service Overview(v2)
Exadata Cloud Service Overview(v2) Exadata Cloud Service Overview(v2)
Exadata Cloud Service Overview(v2)
 

More from Phil Wilkins

API Design – More than just a Payload Definition
API Design – More than just a Payload DefinitionAPI Design – More than just a Payload Definition
API Design – More than just a Payload DefinitionPhil Wilkins
 
APIs, STOP Polling, lets go Streaming
APIs, STOP Polling, lets go StreamingAPIs, STOP Polling, lets go Streaming
APIs, STOP Polling, lets go StreamingPhil Wilkins
 
GitHub Actions - using Free Oracle Cloud Infrastructure (OCI)
GitHub Actions - using Free Oracle Cloud Infrastructure (OCI)GitHub Actions - using Free Oracle Cloud Infrastructure (OCI)
GitHub Actions - using Free Oracle Cloud Infrastructure (OCI)Phil Wilkins
 
Oracle OCI APIs and SDK
Oracle OCI APIs and SDKOracle OCI APIs and SDK
Oracle OCI APIs and SDKPhil Wilkins
 
Api more than payload (2021 Update)
Api more than payload (2021 Update)Api more than payload (2021 Update)
Api more than payload (2021 Update)Phil Wilkins
 
API more than payload
API more than payloadAPI more than payload
API more than payloadPhil Wilkins
 
How fluentd fits into the modern software landscape
How fluentd fits into the modern software landscapeHow fluentd fits into the modern software landscape
How fluentd fits into the modern software landscapePhil Wilkins
 
gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20
gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20
gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20Phil Wilkins
 
Meetups - The Oracle Ace Way
Meetups - The Oracle Ace WayMeetups - The Oracle Ace Way
Meetups - The Oracle Ace WayPhil Wilkins
 
Apiary - A Developers Perspective
Apiary - A Developers PerspectiveApiary - A Developers Perspective
Apiary - A Developers PerspectivePhil Wilkins
 
Secrets of Custom API Policies on the Oracle API Platform
Secrets of Custom API Policies on the Oracle API PlatformSecrets of Custom API Policies on the Oracle API Platform
Secrets of Custom API Policies on the Oracle API PlatformPhil Wilkins
 
Oracle London Developer Meetup November 2018
Oracle London Developer Meetup November 2018Oracle London Developer Meetup November 2018
Oracle London Developer Meetup November 2018Phil Wilkins
 
London Oracle Developer Meetup - June 18 - Drones with APIs
London Oracle Developer Meetup - June 18 - Drones with APIsLondon Oracle Developer Meetup - June 18 - Drones with APIs
London Oracle Developer Meetup - June 18 - Drones with APIsPhil Wilkins
 
London Oracle Developer Meetup April 18
London Oracle Developer Meetup April 18London Oracle Developer Meetup April 18
London Oracle Developer Meetup April 18Phil Wilkins
 
Oracle Developer Meetup March 2018
Oracle Developer Meetup March 2018Oracle Developer Meetup March 2018
Oracle Developer Meetup March 2018Phil Wilkins
 
OracleDeveloperMeetup - London 19-12-17
OracleDeveloperMeetup - London 19-12-17OracleDeveloperMeetup - London 19-12-17
OracleDeveloperMeetup - London 19-12-17Phil Wilkins
 
Look at Oracle Integration Cloud – its relationship to ICS. Customer use Case...
Look at Oracle Integration Cloud – its relationship to ICS. Customer use Case...Look at Oracle Integration Cloud – its relationship to ICS. Customer use Case...
Look at Oracle Integration Cloud – its relationship to ICS. Customer use Case...Phil Wilkins
 
API Platform Cloud Service best practice - OOW17
API Platform Cloud Service best practice - OOW17API Platform Cloud Service best practice - OOW17
API Platform Cloud Service best practice - OOW17Phil Wilkins
 
Oracle integration cloud service (ICS) best practices learned from the field ...
Oracle integration cloud service (ICS) best practices learned from the field ...Oracle integration cloud service (ICS) best practices learned from the field ...
Oracle integration cloud service (ICS) best practices learned from the field ...Phil Wilkins
 

More from Phil Wilkins (20)

API Design – More than just a Payload Definition
API Design – More than just a Payload DefinitionAPI Design – More than just a Payload Definition
API Design – More than just a Payload Definition
 
APIs, STOP Polling, lets go Streaming
APIs, STOP Polling, lets go StreamingAPIs, STOP Polling, lets go Streaming
APIs, STOP Polling, lets go Streaming
 
GitHub Actions - using Free Oracle Cloud Infrastructure (OCI)
GitHub Actions - using Free Oracle Cloud Infrastructure (OCI)GitHub Actions - using Free Oracle Cloud Infrastructure (OCI)
GitHub Actions - using Free Oracle Cloud Infrastructure (OCI)
 
Oracle OCI APIs and SDK
Oracle OCI APIs and SDKOracle OCI APIs and SDK
Oracle OCI APIs and SDK
 
Api more than payload (2021 Update)
Api more than payload (2021 Update)Api more than payload (2021 Update)
Api more than payload (2021 Update)
 
API more than payload
API more than payloadAPI more than payload
API more than payload
 
How fluentd fits into the modern software landscape
How fluentd fits into the modern software landscapeHow fluentd fits into the modern software landscape
How fluentd fits into the modern software landscape
 
gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20
gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20
gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20
 
Meetups - The Oracle Ace Way
Meetups - The Oracle Ace WayMeetups - The Oracle Ace Way
Meetups - The Oracle Ace Way
 
Apiary - A Developers Perspective
Apiary - A Developers PerspectiveApiary - A Developers Perspective
Apiary - A Developers Perspective
 
Secrets of Custom API Policies on the Oracle API Platform
Secrets of Custom API Policies on the Oracle API PlatformSecrets of Custom API Policies on the Oracle API Platform
Secrets of Custom API Policies on the Oracle API Platform
 
Terraform
TerraformTerraform
Terraform
 
Oracle London Developer Meetup November 2018
Oracle London Developer Meetup November 2018Oracle London Developer Meetup November 2018
Oracle London Developer Meetup November 2018
 
London Oracle Developer Meetup - June 18 - Drones with APIs
London Oracle Developer Meetup - June 18 - Drones with APIsLondon Oracle Developer Meetup - June 18 - Drones with APIs
London Oracle Developer Meetup - June 18 - Drones with APIs
 
London Oracle Developer Meetup April 18
London Oracle Developer Meetup April 18London Oracle Developer Meetup April 18
London Oracle Developer Meetup April 18
 
Oracle Developer Meetup March 2018
Oracle Developer Meetup March 2018Oracle Developer Meetup March 2018
Oracle Developer Meetup March 2018
 
OracleDeveloperMeetup - London 19-12-17
OracleDeveloperMeetup - London 19-12-17OracleDeveloperMeetup - London 19-12-17
OracleDeveloperMeetup - London 19-12-17
 
Look at Oracle Integration Cloud – its relationship to ICS. Customer use Case...
Look at Oracle Integration Cloud – its relationship to ICS. Customer use Case...Look at Oracle Integration Cloud – its relationship to ICS. Customer use Case...
Look at Oracle Integration Cloud – its relationship to ICS. Customer use Case...
 
API Platform Cloud Service best practice - OOW17
API Platform Cloud Service best practice - OOW17API Platform Cloud Service best practice - OOW17
API Platform Cloud Service best practice - OOW17
 
Oracle integration cloud service (ICS) best practices learned from the field ...
Oracle integration cloud service (ICS) best practices learned from the field ...Oracle integration cloud service (ICS) best practices learned from the field ...
Oracle integration cloud service (ICS) best practices learned from the field ...
 

Recently uploaded

maximum subarray ppt for killing camp students
maximum subarray ppt for killing camp studentsmaximum subarray ppt for killing camp students
maximum subarray ppt for killing camp studentsssuser82c38d
 
Essence of Requirements Engineering: Pragmatic Insights for 2024
Essence of Requirements Engineering: Pragmatic Insights for 2024Essence of Requirements Engineering: Pragmatic Insights for 2024
Essence of Requirements Engineering: Pragmatic Insights for 2024Asher Sterkin
 
Manual de la Mezcladora SoundCraft Notepad -12Fx
Manual de la Mezcladora SoundCraft Notepad -12FxManual de la Mezcladora SoundCraft Notepad -12Fx
Manual de la Mezcladora SoundCraft Notepad -12Fxjavierdavidvelasco17
 
unit I lecture 2 - Software Engineering Ethics - Software Process.pdf
unit I lecture 2 - Software Engineering Ethics - Software Process.pdfunit I lecture 2 - Software Engineering Ethics - Software Process.pdf
unit I lecture 2 - Software Engineering Ethics - Software Process.pdfStephenTec
 
Les02 Restricting and Sorting Data using SQL.ppt
Les02 Restricting and Sorting Data using SQL.pptLes02 Restricting and Sorting Data using SQL.ppt
Les02 Restricting and Sorting Data using SQL.pptDrZeeshanBhatti
 
unit I lecture 4 - AGILE DEVELOPMENT AND PLAN-DRIVEN.pdf
unit I lecture 4 - AGILE DEVELOPMENT AND PLAN-DRIVEN.pdfunit I lecture 4 - AGILE DEVELOPMENT AND PLAN-DRIVEN.pdf
unit I lecture 4 - AGILE DEVELOPMENT AND PLAN-DRIVEN.pdfStephenTec
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...confluent
 
Enabling Enterprise-wide OT Data access with Matrikon Data Broker.pdf
Enabling Enterprise-wide OT Data access  with Matrikon Data Broker.pdfEnabling Enterprise-wide OT Data access  with Matrikon Data Broker.pdf
Enabling Enterprise-wide OT Data access with Matrikon Data Broker.pdfJohn Archer
 
unit I lecture 5 - Software Development Life Cycle.pdf
unit I lecture 5 - Software Development Life Cycle.pdfunit I lecture 5 - Software Development Life Cycle.pdf
unit I lecture 5 - Software Development Life Cycle.pdfStephenTec
 
Getting Started with Trello for Beginners.pptx
Getting Started with Trello for Beginners.pptxGetting Started with Trello for Beginners.pptx
Getting Started with Trello for Beginners.pptxmavinoikein
 
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)GDSCNiT
 
India's_Generative_AI_Startup_Landscape_Report_2023_Inc42 (1).pdf
India's_Generative_AI_Startup_Landscape_Report_2023_Inc42 (1).pdfIndia's_Generative_AI_Startup_Landscape_Report_2023_Inc42 (1).pdf
India's_Generative_AI_Startup_Landscape_Report_2023_Inc42 (1).pdfgranitesrijan
 
AUTOKEYUNLOCKER-BRANDS-SUPPORT-STANDARD-VERSION.pdf
AUTOKEYUNLOCKER-BRANDS-SUPPORT-STANDARD-VERSION.pdfAUTOKEYUNLOCKER-BRANDS-SUPPORT-STANDARD-VERSION.pdf
AUTOKEYUNLOCKER-BRANDS-SUPPORT-STANDARD-VERSION.pdfAutokey
 
Embracing Change - The Impact of Generative AI on Strategic Portfolio Management
Embracing Change - The Impact of Generative AI on Strategic Portfolio ManagementEmbracing Change - The Impact of Generative AI on Strategic Portfolio Management
Embracing Change - The Impact of Generative AI on Strategic Portfolio ManagementOnePlan Solutions
 
App Builder - Hierarchical Data Apps.pptx
App Builder - Hierarchical Data Apps.pptxApp Builder - Hierarchical Data Apps.pptx
App Builder - Hierarchical Data Apps.pptxPoojitha B
 
Self scaling Multi cloud nomad workloads
Self scaling Multi cloud nomad workloadsSelf scaling Multi cloud nomad workloads
Self scaling Multi cloud nomad workloadsBram Vogelaar
 
Sql server types of joins with example.pptx
Sql server types of joins with example.pptxSql server types of joins with example.pptx
Sql server types of joins with example.pptxsameer gaikwad
 
unit I lecture 3 - Software Process Models.pdf
unit I lecture 3 - Software Process Models.pdfunit I lecture 3 - Software Process Models.pdf
unit I lecture 3 - Software Process Models.pdfStephenTec
 
Steps to Build a PWA with Odoo.pdf
Steps to Build a PWA with Odoo.pdfSteps to Build a PWA with Odoo.pdf
Steps to Build a PWA with Odoo.pdfayushinwizards
 

Recently uploaded (20)

maximum subarray ppt for killing camp students
maximum subarray ppt for killing camp studentsmaximum subarray ppt for killing camp students
maximum subarray ppt for killing camp students
 
Essence of Requirements Engineering: Pragmatic Insights for 2024
Essence of Requirements Engineering: Pragmatic Insights for 2024Essence of Requirements Engineering: Pragmatic Insights for 2024
Essence of Requirements Engineering: Pragmatic Insights for 2024
 
Manual de la Mezcladora SoundCraft Notepad -12Fx
Manual de la Mezcladora SoundCraft Notepad -12FxManual de la Mezcladora SoundCraft Notepad -12Fx
Manual de la Mezcladora SoundCraft Notepad -12Fx
 
unit I lecture 2 - Software Engineering Ethics - Software Process.pdf
unit I lecture 2 - Software Engineering Ethics - Software Process.pdfunit I lecture 2 - Software Engineering Ethics - Software Process.pdf
unit I lecture 2 - Software Engineering Ethics - Software Process.pdf
 
Les02 Restricting and Sorting Data using SQL.ppt
Les02 Restricting and Sorting Data using SQL.pptLes02 Restricting and Sorting Data using SQL.ppt
Les02 Restricting and Sorting Data using SQL.ppt
 
unit I lecture 4 - AGILE DEVELOPMENT AND PLAN-DRIVEN.pdf
unit I lecture 4 - AGILE DEVELOPMENT AND PLAN-DRIVEN.pdfunit I lecture 4 - AGILE DEVELOPMENT AND PLAN-DRIVEN.pdf
unit I lecture 4 - AGILE DEVELOPMENT AND PLAN-DRIVEN.pdf
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
Enabling Enterprise-wide OT Data access with Matrikon Data Broker.pdf
Enabling Enterprise-wide OT Data access  with Matrikon Data Broker.pdfEnabling Enterprise-wide OT Data access  with Matrikon Data Broker.pdf
Enabling Enterprise-wide OT Data access with Matrikon Data Broker.pdf
 
unit I lecture 5 - Software Development Life Cycle.pdf
unit I lecture 5 - Software Development Life Cycle.pdfunit I lecture 5 - Software Development Life Cycle.pdf
unit I lecture 5 - Software Development Life Cycle.pdf
 
Getting Started with Trello for Beginners.pptx
Getting Started with Trello for Beginners.pptxGetting Started with Trello for Beginners.pptx
Getting Started with Trello for Beginners.pptx
 
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
 
India's_Generative_AI_Startup_Landscape_Report_2023_Inc42 (1).pdf
India's_Generative_AI_Startup_Landscape_Report_2023_Inc42 (1).pdfIndia's_Generative_AI_Startup_Landscape_Report_2023_Inc42 (1).pdf
India's_Generative_AI_Startup_Landscape_Report_2023_Inc42 (1).pdf
 
Features of IETM Software -Code and Pixels
Features of IETM Software -Code and PixelsFeatures of IETM Software -Code and Pixels
Features of IETM Software -Code and Pixels
 
AUTOKEYUNLOCKER-BRANDS-SUPPORT-STANDARD-VERSION.pdf
AUTOKEYUNLOCKER-BRANDS-SUPPORT-STANDARD-VERSION.pdfAUTOKEYUNLOCKER-BRANDS-SUPPORT-STANDARD-VERSION.pdf
AUTOKEYUNLOCKER-BRANDS-SUPPORT-STANDARD-VERSION.pdf
 
Embracing Change - The Impact of Generative AI on Strategic Portfolio Management
Embracing Change - The Impact of Generative AI on Strategic Portfolio ManagementEmbracing Change - The Impact of Generative AI on Strategic Portfolio Management
Embracing Change - The Impact of Generative AI on Strategic Portfolio Management
 
App Builder - Hierarchical Data Apps.pptx
App Builder - Hierarchical Data Apps.pptxApp Builder - Hierarchical Data Apps.pptx
App Builder - Hierarchical Data Apps.pptx
 
Self scaling Multi cloud nomad workloads
Self scaling Multi cloud nomad workloadsSelf scaling Multi cloud nomad workloads
Self scaling Multi cloud nomad workloads
 
Sql server types of joins with example.pptx
Sql server types of joins with example.pptxSql server types of joins with example.pptx
Sql server types of joins with example.pptx
 
unit I lecture 3 - Software Process Models.pdf
unit I lecture 3 - Software Process Models.pdfunit I lecture 3 - Software Process Models.pdf
unit I lecture 3 - Software Process Models.pdf
 
Steps to Build a PWA with Odoo.pdf
Steps to Build a PWA with Odoo.pdfSteps to Build a PWA with Odoo.pdf
Steps to Build a PWA with Odoo.pdf
 

Fluentd – Making Logging Easy & Effective in a Multi-cloud & Hybrid Environment - Deverloper Week Europe 22.pptx

  • 1. Fluentd – Making Logging Easy & Effective in a Multi-cloud & Hybrid Environment Name Phil Wilkins OCI April 2022
  • 2. Speaker PhilWilkins Cloud Developer Evangelist Copyright © 2022, Oracle and/or its affiliates Philip.Wilkins@Oracle.com cloud-native.info / mp3monster.org / oracle- integration.cloud www.linkedin.com/in/philwilkins/ @mp3monster
  • 3. 3 Copyright © 2022, Oracle and/or its affiliates https://www.manning.com/books/unified -logging-with-fluentd https://bit.ly/FluentdBook Philip.Wilkins@Oracle.com cloud-native.info / mp3monster.org / oracle- integration.cloud www.linkedin.com/in/philwilkins/ @mp3monster
  • 4. The following is intended to present ideas & application of open technologies. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. This presentation contains the thoughts and ideas of this presenter & does not necessarily represent the views of Oracle Corporation. 4 Copyright © 2022, Oracle and/or its affiliates
  • 5. Putting Monitoring & Log Analytics into Perspective Our view of monitoring is heavily influenced by our role. • Devs think app logs –(structured) text • Infra think numeric – server performance/resource consumption etc. • Business / Low Coders use both numeric & text – transaction counts & status DevOps is challenging that and ideas like 3 Pillars of Observability have matured: Copyright © 2022, Oracle and/or its affiliates Metrics – Typically Numerical (sample based data e.g. CPU use) Logs Textual (event based data e.g. app logging, SNMP traps) Observability Pillars / Key Traces (execution flow & timing – transaction based, sampled) Host / Infrastructure Monitoring Virtualization / Container Monitoring Application Monitoring Business Application Monitoring Security / SIEM Capacity Monitoring 5
  • 6. What is Fluentd ? 6 Copyright © 2022, Oracle and/or its affiliates
  • 7. Highly Pluggable Framework Input • TCP/UDP • Unix Sockets • HTTP • Many file formats • SNMP traps • OS (Linux/UNIX) • Log4J, SLF4J and other related frameworks for .Net, JavaScript Output • ALM solutions e.g. Splunk, cloud native solutions, loggly, logzio etc. • HTTP • Prometheus • Grafana • Many file formats • DB (SQL/NoSQL) • Event Streams e.g. Kafka, Kenesis, MQTT • Social notifications e.g. Jabber, Slack, emai, twilio l etc • Support mgmt tools like Pager Duty Buffer / Cache • Custom in memory cache • Redis & Coherence Storage • S3 buckets • DB (No) SQL • File Formatter • XML • JSON • CSV/TSV etc. • Compressed formats Parser • Multline text to single event • Event info extraction e.g. date & time Filters • Value based conditions • REGEX expressions Custom components Amusing Ruby Gems it is possible to build any custom components using the framework provided
  • 8. Fluentd Provided Language Libraries Copyright © 2022, Oracle and/or its affiliates 8
  • 9. Language Logging Frameworks with Fluentd direct support Copyright © 2022, Oracle and/or its affiliates 9 Java, .Net Rust - logrs Perl NodeJS - Pino Multiple Open Source Appenders/Adaptors Multiple Open Source Appenders/Adaptors Serilog – VB.net R – lgr
  • 10. Data to Actionable Information 10 Copyright © 2022, Oracle and/or its affiliates Information Source Capture • Infra structure such as CPU, memory use • JVM use • App Log Files • SNMP Traps Structure & Route • Get the raw data to the appropriate tooling in a format that can be processed Aggregate & Analyze •Data from multiple sources •Merge in time series Visualize Data • Search for log events • Present trends e.g. memory consumption, • Rate of storage consumption Notify & Alert •Push events into JIRA Svc Desk, Slack, etc •Rules on severity dictate behaviours Fluentd – Optimal Fluentd – Leverage Other Tools
  • 11. How Does this relate to Oracle and other hyperscalers? 11 Copyright © 2022, Oracle and/or its affiliates • For Oracle Cloud Log management… • leverages Fluentd – offers Fluentd compliant endpoints. • can share logging content back to Fluentd endpoints • Other Hyperscalers (AWS, GCP) also leverage Fluentd to varying degrees (GCP lead the way) • Fluentd has plugins for Cloud Vendor monitoring integration • Verrazano (open-source pure CNCF platform) as a pure multi-cloud solution uses Fluentd • Other Container platforms like OpenShift also leverage Fluentd
  • 12. Data Egress costs $€£¥ 12 Copyright © 2022, Oracle and/or its affiliates OCI $0.02 - $0.16 per GB $0.08 per GB $0.09 per GB $0.01 - $0.02 per GB $0.0085 - $0.05 per GB $0.01 - $0.15 per GB $0.085 per GB $0.043 per GB Traffic costs vary by Geographic region – South America & Far East most expensive Europe & North America cheapest
  • 13. Balancing $€£¥ with visibility 13 Copyright © 2022, Oracle and/or its affiliates OCI - Consider adapting a centre of mass for critical E2E visibility - Centre of mass = where most log traffic is generated - Do need to trade off technology need, capacity & bandwidth cost - But have regional richer logs in the event of needing more detail
  • 14. Addressing Real-world Challenges In the realworld, we have more significant challenges … • Highly distributed solutions that need to have logging and monitoring consolidated • Tracing for reactive and solutions – context switching rather than threads in the execution • Often different teams want to use different tools – security want Splunk, DBAs want OEM, infrastructure teams want Nagios – making the setup of environments more complex than needs be • Some operational events are more critical than others – need to filter those out • Make legacy solutions easier to operate, isolate log events and tag them with operational code references – so process & care are embedded without impacting the app 14 Copyright © 2022, Oracle and/or its affiliates
  • 15. Possible scaling & deployment approaches 15 Copyright © 2022, Oracle and/or its affiliates Single instance - Often how people tend to think of log unification tools (multiples when app bundles solution - Presents lots of issues for multi- cloud / hybrid Resilient Pairing - Monolith approach of hot standby approach (Presents lots of issues for multi-cloud / hybrid) - OR 1 Node per cloud/DC Distributed Instances with Central Focus - More resilient, - Ability to control data flow, local control of connection failure - Supports highly distributed Fluentd has a very small footprint - Lots of deployments – becomes an issue of patching/config change - Even smaller footprint possible with Fluent Bit - From deployment on IoT to containers, servers and Mainframes
  • 16. Possible scaling & deployment approaches 16 Copyright © 2022, Oracle and/or its affiliates Single instance - Often how people tend to think of log unification tools (multiples when app bundles solution - Presents lots of issues for multi- cloud / hybrid Resilient Pairing - Monolith approach of hot standby approach - Presents lots of issues for multi- cloud / hybrid Distributed Instances with Central Focus - More resilient, - Ability to control data flow, local control of connection failure - Supports highly distributed Fluentd has a very small footprint - Lots of deployments – becomes an issue of patching/config change - Even smaller footprint possible with Fluent Bit - From deployment on IoT to containers, servers and Mainframes
  • 17. Fluentd – Scaling & Aggregation 17 Copyright © 2022, Oracle and/or its affiliates App A (Front End) Svr App B VM Svc Shared Persistence / Analytics Platform Ops Alerting Service Pod App A (Mid Tier) Svr App A (Mid Tier) VM App C VM Svc Pod
  • 18. Fluentd – Scaling & Aggregation – Cloud / DC 18 Copyright © 2022, Oracle and/or its affiliates App A (Front End) Svr App B VM Svc Shared Persistence / Analytics Platform Ops Alerting Service Pod App A (Mid Tier) Svr App A (Mid Tier) VM App C VM Svc Pod Cloud Region Own DC
  • 19. Fluentd – Scaling & Aggregation - Kubernetes / Container 19 Copyright © 2022, Oracle and/or its affiliates App A (Front End) Pod App B Pod Svc Shared Persistence / Analytics Platform Ops Alerting Service Pod App A (Mid Tier) Pod App A (Mid Tier) Pod App C Pod Svc Pod Worker Node Worker Node
  • 20. Fluentd – Scaling & Aggregation - Kubernetes / Container – using Side Cars 20 Copyright © 2021, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted App A (Front End) Pod App B Pod Pod Shared Persistence / Analytics Platform Ops Alerting Service Pod App A (Mid Tier) Pod App A (Mid Tier) Pod App C Pod Pod Pod Worker Node Worker Node Side Car Side Car Side Car
  • 21. Demo… Copyright © 2022, Oracle and/or its affiliates
  • 22. 22 Copyright © 2022, Oracle and/or its affiliates Central Node (Node 2) (Single Instance) Node 1 (Instance n) common • filter Slack Stdout / Monitoring Warboard Op Analytics & AI Ops filters • Transsform (event  message) • Match + copy: • Out file • relabel labelPipeline basic-file.txt basic-file2.txt label-pipeline-file-output.* forwarder We need to control this flow to avoid a notification storm!
  • 23. Questions / Thank you Copyright © 2022, Oracle and/or its affiliates Phil Wilkins Cloud Developer Evangelist Philip.Wilkins@Oracle.com cloud-native.info / mp3monster.org / oracle- integration.cloud www.linkedin.com/in/philwilkins/ @mp3monster

Editor's Notes

  1. https://threadreaderapp.com/thread/1020188389721530368.html Twitter had an observability team ten years ago Hungarian-American engineer Rudolf E. Kálmán formalized the idea of observability in a paper describing characteristics of linear dynamic systems 1961 https://en.wikipedia.org/wiki/Rudolf_E._K%C3%A1lm%C3%A1n His work was important Nasa Google’s 4 golden signals Latency Traffic Errors saturation
  2. Django – Python Crates.io – RUST package manager https://github.com/technologize/fluency-log4j-appender https://github.com/komamitsu/fluency https://github.com/vmware/kube-fluentd-operator https://www.npmjs.com/package/pino-fluentd https://githubhot.com/repo/MCKanpolat/Log4net.Appenders.Fluentd https://metacpan.org/dist/Log-Log4perl-Appender-Fluent https://crates.io/crates/log4rs-fluentd/0.2.0/dependencies https://google.github.io/flogger/ https://rdrr.io/cran/lgr/ https://serilog.net/ https://github.com/Seldaek/monolog - php monolog
  3. https://www.hostdime.com/blog/data-egress-fees-cloud/ https://azure.microsoft.com/en-gb/pricing/details/bandwidth/ https://aws.amazon.com/ec2/pricing/on-demand https://www.oracle.com/uk/cloud/networking/pricing/#fastconnect/ https://cloud.google.com/vpc/network-pricing#all-networking-pricing https://www.oracle.com/uk/cloud/networking/pricing/ 100 Mb Leased line from BT Wholesale = 343 343 / (30*24) = 0.4763 (per hour) 100Mb can peak at 28GB per hour 0.4763 / 28 = 0.017 (@ 50% utilization 0.034) GBP  USD = 1.28 $0.022 (100% use) $0.043 (50%)