SlideShare a Scribd company logo
ELK Logging
Optimizations & Updates for 2021
Bao Huynh
Site Reliability Engineer
29-Dec-2020 1
Private & Confidential 2
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes
Private & Confidential 3
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes
4
1. Logging – the crucial part !
5
1. Logging – the crucial part !
6
1. Logging – the crucial part !
Clearly states that:
- Applications should not attempt to write to / manage logsfiles
- Instead, each running process writes its event stream  STDOUT
- During “local development”: observe on the terminal
- During “staging/production”:
Capture & collected by agents.
Then, routed to one final destination for viewing & long-term archival.
7
1. Logging – the crucial part !
Private & Confidential 8
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes
9
2. Motivation for changes
- Could not ship log for external helm-chart services (ingress-nginx, redis, es,..)
- Developers have to add “additional fields” (svc + env) in their log lines
- Once search, it goes through all indices (logstash-*) to find matching =>
longer result return
- Hard to scale with multi-cluster architecture
Private & Confidential 10
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes
11
3. How changes benefit developers
3.1 Faster lookup/search log pattern
- Logs from *different env/ns* is allocated into *different indices*
12
3. How changes benefit developers
3.2 Dev do what to get log shipping ?
- Kibana.vndev.shopee.vn
- Everything is ready for you
13
3. How changes benefit developers
3.3 Fresher, cleaner & better log format
- Remove unneccessary fields
- Compact fields:
+ services_.* (metadata related to your application)
+ ingress_nginx.* (metadata related to client requests)
14
3. How changes benefit developers
3.4 Your log is the “real” log
- Include more filters
- No more trash logs
15
3. How changes benefit developers
3.5 Enable Cross function/team log correlation checking
- BE can check logs from FE/SRE if suspects issues from their side
- Remove the dependency between teams for error checking
16
3. How changes benefit developers
3.5 Enable Logs2Metrics capability
Private & Confidential 17
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes
18
4. Multi-cluster logging architecture
4.1 Logging basic
19
4. Multi-cluster logging architecture
4.1 Logging basic
Filebeat:
- Select messages for shipping (/var/logs/container/*.log)
- Select fields for shipping along with msg (kubernetes.namespace/imageVersion,…)
- Enrich msg with additional metadata (env, service_name, rename fields)
- Message processing AMAP to reduce workload for Logstash
Logstash:
- Drop more fields that can not removed at Filebeat level
- Using built-in module to parse specific log type (NGINX log)
- Log aggregator & sent to Elasticsearch for storage
ElasticSearch:
- Messages allocated to indices
- Index retention by ILM (Index Lifecyle Management)
Kibana:
- Visualization & Analysis
20
4. Multi-cluster logging architecture
4.2 Logging at Scale (multi-cluster architecture)
Private & Confidential 21
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes
22
5. Usages & Notes
- Deploy fresh service
- Observe how log shipping
- Enjoy reading logs in Kibana
Private & Confidential 23
THANKS FOR YOUR ATTENTION

More Related Content

What's hot

Why observability matters - now and in the future (w/guest Grafana)
Why observability matters - now and in the future (w/guest Grafana)Why observability matters - now and in the future (w/guest Grafana)
Why observability matters - now and in the future (w/guest Grafana)
Weaveworks
 
"Using Automation Tools To Deploy And Operate Applications In Real World Scen...
"Using Automation Tools To Deploy And Operate Applications In Real World Scen..."Using Automation Tools To Deploy And Operate Applications In Real World Scen...
"Using Automation Tools To Deploy And Operate Applications In Real World Scen...
ConSol Consulting & Solutions Software GmbH
 
Git ops: Git based application deployment patterns for Kubernetes
Git ops: Git based application deployment patterns for KubernetesGit ops: Git based application deployment patterns for Kubernetes
Git ops: Git based application deployment patterns for Kubernetes
Shahidh K Muhammed
 
DevOps Fest 2020. Дмитрий Кудрявцев. Реализация GitOps на Kubernetes. ArgoCD
DevOps Fest 2020. Дмитрий Кудрявцев. Реализация GitOps на Kubernetes. ArgoCDDevOps Fest 2020. Дмитрий Кудрявцев. Реализация GitOps на Kubernetes. ArgoCD
DevOps Fest 2020. Дмитрий Кудрявцев. Реализация GitOps на Kubernetes. ArgoCD
DevOps_Fest
 
GitOps for Helm Users by Scott Rigby
GitOps for Helm Users by Scott RigbyGitOps for Helm Users by Scott Rigby
GitOps for Helm Users by Scott Rigby
Weaveworks
 
Quarkus: From developer joy to Kubernetes nirvana! | DevNation Tech Talk
Quarkus: From developer joy to Kubernetes nirvana! | DevNation Tech TalkQuarkus: From developer joy to Kubernetes nirvana! | DevNation Tech Talk
Quarkus: From developer joy to Kubernetes nirvana! | DevNation Tech Talk
Red Hat Developers
 
Monitoring Cockpit for OpenShift Clusters
Monitoring Cockpit for OpenShift ClustersMonitoring Cockpit for OpenShift Clusters
Monitoring Cockpit for OpenShift Clusters
ConSol Consulting & Solutions Software GmbH
 
Git overview
Git overviewGit overview
Git overview
Gowarthini
 
Operator development made easy with helm
Operator development made easy with helmOperator development made easy with helm
Operator development made easy with helm
ConSol Consulting & Solutions Software GmbH
 
Meetup 23 - 03 - Application Delivery on K8S with GitOps
Meetup 23 - 03 - Application Delivery on K8S with GitOpsMeetup 23 - 03 - Application Delivery on K8S with GitOps
Meetup 23 - 03 - Application Delivery on K8S with GitOps
Vietnam Open Infrastructure User Group
 
The Power of GitOps with Flux & GitOps Toolkit
The Power of GitOps with Flux & GitOps ToolkitThe Power of GitOps with Flux & GitOps Toolkit
The Power of GitOps with Flux & GitOps Toolkit
Weaveworks
 
Best Practices for Microservice CI/CD: Lessons from Expedia and Codefresh
Best Practices for Microservice CI/CD: Lessons from Expedia and CodefreshBest Practices for Microservice CI/CD: Lessons from Expedia and Codefresh
Best Practices for Microservice CI/CD: Lessons from Expedia and Codefresh
Codefresh
 
Kubernetes based Cloud-region support in ONAP to bring up VM and container ba...
Kubernetes based Cloud-region support in ONAP to bring up VM and container ba...Kubernetes based Cloud-region support in ONAP to bring up VM and container ba...
Kubernetes based Cloud-region support in ONAP to bring up VM and container ba...
Victor Morales
 
Kubernetes GitOps featuring GitHub, Kustomize and ArgoCD
Kubernetes GitOps featuring GitHub, Kustomize and ArgoCDKubernetes GitOps featuring GitHub, Kustomize and ArgoCD
Kubernetes GitOps featuring GitHub, Kustomize and ArgoCD
Sunnyvale
 
Kubernetes Logging
Kubernetes LoggingKubernetes Logging
Kubernetes Logging
Denys Havrysh
 
Remote debugging of Application in Kubernetes
Remote debugging of Application in KubernetesRemote debugging of Application in Kubernetes
Remote debugging of Application in Kubernetes
ConSol Consulting & Solutions Software GmbH
 
Continious integration pipeline
Continious integration pipelineContinious integration pipeline
Continious integration pipeline
GomathiNayagam S
 
GitOps w/argocd
GitOps w/argocdGitOps w/argocd
GitOps w/argocd
Jean-Philippe Bélanger
 
Android and Hard Real Time
Android and Hard Real TimeAndroid and Hard Real Time
Android and Hard Real TimeAkshar Desai
 
GitOps - Modern best practices for high velocity app dev using cloud native t...
GitOps - Modern best practices for high velocity app dev using cloud native t...GitOps - Modern best practices for high velocity app dev using cloud native t...
GitOps - Modern best practices for high velocity app dev using cloud native t...
Weaveworks
 

What's hot (20)

Why observability matters - now and in the future (w/guest Grafana)
Why observability matters - now and in the future (w/guest Grafana)Why observability matters - now and in the future (w/guest Grafana)
Why observability matters - now and in the future (w/guest Grafana)
 
"Using Automation Tools To Deploy And Operate Applications In Real World Scen...
"Using Automation Tools To Deploy And Operate Applications In Real World Scen..."Using Automation Tools To Deploy And Operate Applications In Real World Scen...
"Using Automation Tools To Deploy And Operate Applications In Real World Scen...
 
Git ops: Git based application deployment patterns for Kubernetes
Git ops: Git based application deployment patterns for KubernetesGit ops: Git based application deployment patterns for Kubernetes
Git ops: Git based application deployment patterns for Kubernetes
 
DevOps Fest 2020. Дмитрий Кудрявцев. Реализация GitOps на Kubernetes. ArgoCD
DevOps Fest 2020. Дмитрий Кудрявцев. Реализация GitOps на Kubernetes. ArgoCDDevOps Fest 2020. Дмитрий Кудрявцев. Реализация GitOps на Kubernetes. ArgoCD
DevOps Fest 2020. Дмитрий Кудрявцев. Реализация GitOps на Kubernetes. ArgoCD
 
GitOps for Helm Users by Scott Rigby
GitOps for Helm Users by Scott RigbyGitOps for Helm Users by Scott Rigby
GitOps for Helm Users by Scott Rigby
 
Quarkus: From developer joy to Kubernetes nirvana! | DevNation Tech Talk
Quarkus: From developer joy to Kubernetes nirvana! | DevNation Tech TalkQuarkus: From developer joy to Kubernetes nirvana! | DevNation Tech Talk
Quarkus: From developer joy to Kubernetes nirvana! | DevNation Tech Talk
 
Monitoring Cockpit for OpenShift Clusters
Monitoring Cockpit for OpenShift ClustersMonitoring Cockpit for OpenShift Clusters
Monitoring Cockpit for OpenShift Clusters
 
Git overview
Git overviewGit overview
Git overview
 
Operator development made easy with helm
Operator development made easy with helmOperator development made easy with helm
Operator development made easy with helm
 
Meetup 23 - 03 - Application Delivery on K8S with GitOps
Meetup 23 - 03 - Application Delivery on K8S with GitOpsMeetup 23 - 03 - Application Delivery on K8S with GitOps
Meetup 23 - 03 - Application Delivery on K8S with GitOps
 
The Power of GitOps with Flux & GitOps Toolkit
The Power of GitOps with Flux & GitOps ToolkitThe Power of GitOps with Flux & GitOps Toolkit
The Power of GitOps with Flux & GitOps Toolkit
 
Best Practices for Microservice CI/CD: Lessons from Expedia and Codefresh
Best Practices for Microservice CI/CD: Lessons from Expedia and CodefreshBest Practices for Microservice CI/CD: Lessons from Expedia and Codefresh
Best Practices for Microservice CI/CD: Lessons from Expedia and Codefresh
 
Kubernetes based Cloud-region support in ONAP to bring up VM and container ba...
Kubernetes based Cloud-region support in ONAP to bring up VM and container ba...Kubernetes based Cloud-region support in ONAP to bring up VM and container ba...
Kubernetes based Cloud-region support in ONAP to bring up VM and container ba...
 
Kubernetes GitOps featuring GitHub, Kustomize and ArgoCD
Kubernetes GitOps featuring GitHub, Kustomize and ArgoCDKubernetes GitOps featuring GitHub, Kustomize and ArgoCD
Kubernetes GitOps featuring GitHub, Kustomize and ArgoCD
 
Kubernetes Logging
Kubernetes LoggingKubernetes Logging
Kubernetes Logging
 
Remote debugging of Application in Kubernetes
Remote debugging of Application in KubernetesRemote debugging of Application in Kubernetes
Remote debugging of Application in Kubernetes
 
Continious integration pipeline
Continious integration pipelineContinious integration pipeline
Continious integration pipeline
 
GitOps w/argocd
GitOps w/argocdGitOps w/argocd
GitOps w/argocd
 
Android and Hard Real Time
Android and Hard Real TimeAndroid and Hard Real Time
Android and Hard Real Time
 
GitOps - Modern best practices for high velocity app dev using cloud native t...
GitOps - Modern best practices for high velocity app dev using cloud native t...GitOps - Modern best practices for high velocity app dev using cloud native t...
GitOps - Modern best practices for high velocity app dev using cloud native t...
 

Similar to ELK - Optimizations & Updates

Meetup milano #4 log management and anypoint advanced monitoring
Meetup milano #4   log management and anypoint advanced monitoringMeetup milano #4   log management and anypoint advanced monitoring
Meetup milano #4 log management and anypoint advanced monitoring
Gonzalo Marcos Ansoain
 
IBM Cognos 10.2.2 New Features for Professional Authors
IBM Cognos 10.2.2 New Features for Professional AuthorsIBM Cognos 10.2.2 New Features for Professional Authors
IBM Cognos 10.2.2 New Features for Professional Authors
Senturus
 
Openmiconwhatsnewindomino9socialedition 130411102852-phpapp01
Openmiconwhatsnewindomino9socialedition 130411102852-phpapp01Openmiconwhatsnewindomino9socialedition 130411102852-phpapp01
Openmiconwhatsnewindomino9socialedition 130411102852-phpapp01
Ranjit Rai
 
Kafka/SMM Crash Course
Kafka/SMM Crash CourseKafka/SMM Crash Course
Kafka/SMM Crash Course
DataWorks Summit
 
Sitecore user group mumbai sitecore commerce extension
Sitecore user group mumbai  sitecore commerce extensionSitecore user group mumbai  sitecore commerce extension
Sitecore user group mumbai sitecore commerce extension
Jitendra Soni
 
What's New for Report Authors in Cognos 10.2
What's New for Report Authors in Cognos 10.2What's New for Report Authors in Cognos 10.2
What's New for Report Authors in Cognos 10.2
Senturus
 
Pivotal Cloud Foundry 2.4: A First Look
Pivotal Cloud Foundry 2.4: A First LookPivotal Cloud Foundry 2.4: A First Look
Pivotal Cloud Foundry 2.4: A First Look
VMware Tanzu
 
CollabSphere 2020 Live - HCL Notes 11.0.1 FP1 - Performance Boost Re-Reloaded
CollabSphere 2020 Live - HCL Notes 11.0.1 FP1 - Performance Boost Re-ReloadedCollabSphere 2020 Live - HCL Notes 11.0.1 FP1 - Performance Boost Re-Reloaded
CollabSphere 2020 Live - HCL Notes 11.0.1 FP1 - Performance Boost Re-Reloaded
Christoph Adler
 
CollabSphere 2020 - INF105 - HCL Notes 11.0.1 FP1 - Performance Boost Re-Relo...
CollabSphere 2020 - INF105 - HCL Notes 11.0.1 FP1 - Performance Boost Re-Relo...CollabSphere 2020 - INF105 - HCL Notes 11.0.1 FP1 - Performance Boost Re-Relo...
CollabSphere 2020 - INF105 - HCL Notes 11.0.1 FP1 - Performance Boost Re-Relo...
panagenda
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
confluent
 
IBM Connect 2014 BP103: Ready, Aim, Fire: Mastering the Latest in the Adminis...
IBM Connect 2014 BP103: Ready, Aim, Fire: Mastering the Latest in the Adminis...IBM Connect 2014 BP103: Ready, Aim, Fire: Mastering the Latest in the Adminis...
IBM Connect 2014 BP103: Ready, Aim, Fire: Mastering the Latest in the Adminis...
Benedek Menesi
 
Your notes DNA
Your notes DNAYour notes DNA
Your notes DNA
Jon Pyke FBCS CITP
 
State of Development - Icinga Meetup Linz August 2019
State of Development - Icinga Meetup Linz August 2019State of Development - Icinga Meetup Linz August 2019
State of Development - Icinga Meetup Linz August 2019
Icinga
 
SCOM Tips and Tricks
SCOM Tips and TricksSCOM Tips and Tricks
SCOM Tips and Tricks
Christian Heitkamp
 
Application depolyment
Application depolymentApplication depolyment
Application depolymentshriikantL
 
Using The Mysql Binary Log As A Change Stream
Using The Mysql Binary Log As A Change StreamUsing The Mysql Binary Log As A Change Stream
Using The Mysql Binary Log As A Change Stream
Luís Soares
 
CIMtrek migrator to share point user guide
CIMtrek migrator to share point user guideCIMtrek migrator to share point user guide
CIMtrek migrator to share point user guide
Jon Pyke FBCS CITP
 
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Piyush Kumar
 
IBM Cloud Integration Platform High Availability - Integration Tech Conference
IBM Cloud Integration Platform High Availability - Integration Tech ConferenceIBM Cloud Integration Platform High Availability - Integration Tech Conference
IBM Cloud Integration Platform High Availability - Integration Tech Conference
Robert Nicholson
 
Sitecore user group chennai
Sitecore user group chennaiSitecore user group chennai
Sitecore user group chennai
Jitendra Soni
 

Similar to ELK - Optimizations & Updates (20)

Meetup milano #4 log management and anypoint advanced monitoring
Meetup milano #4   log management and anypoint advanced monitoringMeetup milano #4   log management and anypoint advanced monitoring
Meetup milano #4 log management and anypoint advanced monitoring
 
IBM Cognos 10.2.2 New Features for Professional Authors
IBM Cognos 10.2.2 New Features for Professional AuthorsIBM Cognos 10.2.2 New Features for Professional Authors
IBM Cognos 10.2.2 New Features for Professional Authors
 
Openmiconwhatsnewindomino9socialedition 130411102852-phpapp01
Openmiconwhatsnewindomino9socialedition 130411102852-phpapp01Openmiconwhatsnewindomino9socialedition 130411102852-phpapp01
Openmiconwhatsnewindomino9socialedition 130411102852-phpapp01
 
Kafka/SMM Crash Course
Kafka/SMM Crash CourseKafka/SMM Crash Course
Kafka/SMM Crash Course
 
Sitecore user group mumbai sitecore commerce extension
Sitecore user group mumbai  sitecore commerce extensionSitecore user group mumbai  sitecore commerce extension
Sitecore user group mumbai sitecore commerce extension
 
What's New for Report Authors in Cognos 10.2
What's New for Report Authors in Cognos 10.2What's New for Report Authors in Cognos 10.2
What's New for Report Authors in Cognos 10.2
 
Pivotal Cloud Foundry 2.4: A First Look
Pivotal Cloud Foundry 2.4: A First LookPivotal Cloud Foundry 2.4: A First Look
Pivotal Cloud Foundry 2.4: A First Look
 
CollabSphere 2020 Live - HCL Notes 11.0.1 FP1 - Performance Boost Re-Reloaded
CollabSphere 2020 Live - HCL Notes 11.0.1 FP1 - Performance Boost Re-ReloadedCollabSphere 2020 Live - HCL Notes 11.0.1 FP1 - Performance Boost Re-Reloaded
CollabSphere 2020 Live - HCL Notes 11.0.1 FP1 - Performance Boost Re-Reloaded
 
CollabSphere 2020 - INF105 - HCL Notes 11.0.1 FP1 - Performance Boost Re-Relo...
CollabSphere 2020 - INF105 - HCL Notes 11.0.1 FP1 - Performance Boost Re-Relo...CollabSphere 2020 - INF105 - HCL Notes 11.0.1 FP1 - Performance Boost Re-Relo...
CollabSphere 2020 - INF105 - HCL Notes 11.0.1 FP1 - Performance Boost Re-Relo...
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
 
IBM Connect 2014 BP103: Ready, Aim, Fire: Mastering the Latest in the Adminis...
IBM Connect 2014 BP103: Ready, Aim, Fire: Mastering the Latest in the Adminis...IBM Connect 2014 BP103: Ready, Aim, Fire: Mastering the Latest in the Adminis...
IBM Connect 2014 BP103: Ready, Aim, Fire: Mastering the Latest in the Adminis...
 
Your notes DNA
Your notes DNAYour notes DNA
Your notes DNA
 
State of Development - Icinga Meetup Linz August 2019
State of Development - Icinga Meetup Linz August 2019State of Development - Icinga Meetup Linz August 2019
State of Development - Icinga Meetup Linz August 2019
 
SCOM Tips and Tricks
SCOM Tips and TricksSCOM Tips and Tricks
SCOM Tips and Tricks
 
Application depolyment
Application depolymentApplication depolyment
Application depolyment
 
Using The Mysql Binary Log As A Change Stream
Using The Mysql Binary Log As A Change StreamUsing The Mysql Binary Log As A Change Stream
Using The Mysql Binary Log As A Change Stream
 
CIMtrek migrator to share point user guide
CIMtrek migrator to share point user guideCIMtrek migrator to share point user guide
CIMtrek migrator to share point user guide
 
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
 
IBM Cloud Integration Platform High Availability - Integration Tech Conference
IBM Cloud Integration Platform High Availability - Integration Tech ConferenceIBM Cloud Integration Platform High Availability - Integration Tech Conference
IBM Cloud Integration Platform High Availability - Integration Tech Conference
 
Sitecore user group chennai
Sitecore user group chennaiSitecore user group chennai
Sitecore user group chennai
 

More from Huynh Thai Bao

Service Mesh 101 - Digging into your service
Service Mesh 101 - Digging into your service Service Mesh 101 - Digging into your service
Service Mesh 101 - Digging into your service
Huynh Thai Bao
 
K8s Webhook Admission
K8s Webhook AdmissionK8s Webhook Admission
K8s Webhook Admission
Huynh Thai Bao
 
Cassandra - decentralized structured database
Cassandra - decentralized structured databaseCassandra - decentralized structured database
Cassandra - decentralized structured database
Huynh Thai Bao
 
Skaffold - faster development on K8S
Skaffold - faster development on K8SSkaffold - faster development on K8S
Skaffold - faster development on K8S
Huynh Thai Bao
 
Kubernetes - A Rising Hero
Kubernetes - A Rising HeroKubernetes - A Rising Hero
Kubernetes - A Rising Hero
Huynh Thai Bao
 
Vault - Enhancement for K8S secret security
Vault - Enhancement for K8S secret securityVault - Enhancement for K8S secret security
Vault - Enhancement for K8S secret security
Huynh Thai Bao
 
GCP Best Practices for SRE Team
GCP Best Practices for SRE TeamGCP Best Practices for SRE Team
GCP Best Practices for SRE Team
Huynh Thai Bao
 

More from Huynh Thai Bao (7)

Service Mesh 101 - Digging into your service
Service Mesh 101 - Digging into your service Service Mesh 101 - Digging into your service
Service Mesh 101 - Digging into your service
 
K8s Webhook Admission
K8s Webhook AdmissionK8s Webhook Admission
K8s Webhook Admission
 
Cassandra - decentralized structured database
Cassandra - decentralized structured databaseCassandra - decentralized structured database
Cassandra - decentralized structured database
 
Skaffold - faster development on K8S
Skaffold - faster development on K8SSkaffold - faster development on K8S
Skaffold - faster development on K8S
 
Kubernetes - A Rising Hero
Kubernetes - A Rising HeroKubernetes - A Rising Hero
Kubernetes - A Rising Hero
 
Vault - Enhancement for K8S secret security
Vault - Enhancement for K8S secret securityVault - Enhancement for K8S secret security
Vault - Enhancement for K8S secret security
 
GCP Best Practices for SRE Team
GCP Best Practices for SRE TeamGCP Best Practices for SRE Team
GCP Best Practices for SRE Team
 

Recently uploaded

AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdfAKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
SamSarthak3
 
road safety engineering r s e unit 3.pdf
road safety engineering  r s e unit 3.pdfroad safety engineering  r s e unit 3.pdf
road safety engineering r s e unit 3.pdf
VENKATESHvenky89705
 
Final project report on grocery store management system..pdf
Final project report on grocery store management system..pdfFinal project report on grocery store management system..pdf
Final project report on grocery store management system..pdf
Kamal Acharya
 
Student information management system project report ii.pdf
Student information management system project report ii.pdfStudent information management system project report ii.pdf
Student information management system project report ii.pdf
Kamal Acharya
 
weather web application report.pdf
weather web application report.pdfweather web application report.pdf
weather web application report.pdf
Pratik Pawar
 
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
AJAYKUMARPUND1
 
ethical hacking-mobile hacking methods.ppt
ethical hacking-mobile hacking methods.pptethical hacking-mobile hacking methods.ppt
ethical hacking-mobile hacking methods.ppt
Jayaprasanna4
 
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
MdTanvirMahtab2
 
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdfGoverning Equations for Fundamental Aerodynamics_Anderson2010.pdf
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf
WENKENLI1
 
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
bakpo1
 
The role of big data in decision making.
The role of big data in decision making.The role of big data in decision making.
The role of big data in decision making.
ankuprajapati0525
 
Hierarchical Digital Twin of a Naval Power System
Hierarchical Digital Twin of a Naval Power SystemHierarchical Digital Twin of a Naval Power System
Hierarchical Digital Twin of a Naval Power System
Kerry Sado
 
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...
Amil Baba Dawood bangali
 
J.Yang, ICLR 2024, MLILAB, KAIST AI.pdf
J.Yang,  ICLR 2024, MLILAB, KAIST AI.pdfJ.Yang,  ICLR 2024, MLILAB, KAIST AI.pdf
J.Yang, ICLR 2024, MLILAB, KAIST AI.pdf
MLILAB
 
space technology lecture notes on satellite
space technology lecture notes on satellitespace technology lecture notes on satellite
space technology lecture notes on satellite
ongomchris
 
Fundamentals of Electric Drives and its applications.pptx
Fundamentals of Electric Drives and its applications.pptxFundamentals of Electric Drives and its applications.pptx
Fundamentals of Electric Drives and its applications.pptx
manasideore6
 
Water Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation and Control Monthly - May 2024.pdfWater Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation & Control
 
Cosmetic shop management system project report.pdf
Cosmetic shop management system project report.pdfCosmetic shop management system project report.pdf
Cosmetic shop management system project report.pdf
Kamal Acharya
 
WATER CRISIS and its solutions-pptx 1234
WATER CRISIS and its solutions-pptx 1234WATER CRISIS and its solutions-pptx 1234
WATER CRISIS and its solutions-pptx 1234
AafreenAbuthahir2
 
Investor-Presentation-Q1FY2024 investor presentation document.pptx
Investor-Presentation-Q1FY2024 investor presentation document.pptxInvestor-Presentation-Q1FY2024 investor presentation document.pptx
Investor-Presentation-Q1FY2024 investor presentation document.pptx
AmarGB2
 

Recently uploaded (20)

AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdfAKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf
 
road safety engineering r s e unit 3.pdf
road safety engineering  r s e unit 3.pdfroad safety engineering  r s e unit 3.pdf
road safety engineering r s e unit 3.pdf
 
Final project report on grocery store management system..pdf
Final project report on grocery store management system..pdfFinal project report on grocery store management system..pdf
Final project report on grocery store management system..pdf
 
Student information management system project report ii.pdf
Student information management system project report ii.pdfStudent information management system project report ii.pdf
Student information management system project report ii.pdf
 
weather web application report.pdf
weather web application report.pdfweather web application report.pdf
weather web application report.pdf
 
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
 
ethical hacking-mobile hacking methods.ppt
ethical hacking-mobile hacking methods.pptethical hacking-mobile hacking methods.ppt
ethical hacking-mobile hacking methods.ppt
 
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
 
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdfGoverning Equations for Fundamental Aerodynamics_Anderson2010.pdf
Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf
 
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
 
The role of big data in decision making.
The role of big data in decision making.The role of big data in decision making.
The role of big data in decision making.
 
Hierarchical Digital Twin of a Naval Power System
Hierarchical Digital Twin of a Naval Power SystemHierarchical Digital Twin of a Naval Power System
Hierarchical Digital Twin of a Naval Power System
 
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...
 
J.Yang, ICLR 2024, MLILAB, KAIST AI.pdf
J.Yang,  ICLR 2024, MLILAB, KAIST AI.pdfJ.Yang,  ICLR 2024, MLILAB, KAIST AI.pdf
J.Yang, ICLR 2024, MLILAB, KAIST AI.pdf
 
space technology lecture notes on satellite
space technology lecture notes on satellitespace technology lecture notes on satellite
space technology lecture notes on satellite
 
Fundamentals of Electric Drives and its applications.pptx
Fundamentals of Electric Drives and its applications.pptxFundamentals of Electric Drives and its applications.pptx
Fundamentals of Electric Drives and its applications.pptx
 
Water Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation and Control Monthly - May 2024.pdfWater Industry Process Automation and Control Monthly - May 2024.pdf
Water Industry Process Automation and Control Monthly - May 2024.pdf
 
Cosmetic shop management system project report.pdf
Cosmetic shop management system project report.pdfCosmetic shop management system project report.pdf
Cosmetic shop management system project report.pdf
 
WATER CRISIS and its solutions-pptx 1234
WATER CRISIS and its solutions-pptx 1234WATER CRISIS and its solutions-pptx 1234
WATER CRISIS and its solutions-pptx 1234
 
Investor-Presentation-Q1FY2024 investor presentation document.pptx
Investor-Presentation-Q1FY2024 investor presentation document.pptxInvestor-Presentation-Q1FY2024 investor presentation document.pptx
Investor-Presentation-Q1FY2024 investor presentation document.pptx
 

ELK - Optimizations & Updates

  • 1. ELK Logging Optimizations & Updates for 2021 Bao Huynh Site Reliability Engineer 29-Dec-2020 1
  • 2. Private & Confidential 2 AGENDA 1. Logging - the crucial part ! 2. Motivation for changes 3. How changes benefit developers 4. Multi-cluster logging architecture 5. Usages & Notes
  • 3. Private & Confidential 3 AGENDA 1. Logging - the crucial part ! 2. Motivation for changes 3. How changes benefit developers 4. Multi-cluster logging architecture 5. Usages & Notes
  • 4. 4 1. Logging – the crucial part !
  • 5. 5 1. Logging – the crucial part !
  • 6. 6 1. Logging – the crucial part ! Clearly states that: - Applications should not attempt to write to / manage logsfiles - Instead, each running process writes its event stream  STDOUT - During “local development”: observe on the terminal - During “staging/production”: Capture & collected by agents. Then, routed to one final destination for viewing & long-term archival.
  • 7. 7 1. Logging – the crucial part !
  • 8. Private & Confidential 8 AGENDA 1. Logging - the crucial part ! 2. Motivation for changes 3. How changes benefit developers 4. Multi-cluster logging architecture 5. Usages & Notes
  • 9. 9 2. Motivation for changes - Could not ship log for external helm-chart services (ingress-nginx, redis, es,..) - Developers have to add “additional fields” (svc + env) in their log lines - Once search, it goes through all indices (logstash-*) to find matching => longer result return - Hard to scale with multi-cluster architecture
  • 10. Private & Confidential 10 AGENDA 1. Logging - the crucial part ! 2. Motivation for changes 3. How changes benefit developers 4. Multi-cluster logging architecture 5. Usages & Notes
  • 11. 11 3. How changes benefit developers 3.1 Faster lookup/search log pattern - Logs from *different env/ns* is allocated into *different indices*
  • 12. 12 3. How changes benefit developers 3.2 Dev do what to get log shipping ? - Kibana.vndev.shopee.vn - Everything is ready for you
  • 13. 13 3. How changes benefit developers 3.3 Fresher, cleaner & better log format - Remove unneccessary fields - Compact fields: + services_.* (metadata related to your application) + ingress_nginx.* (metadata related to client requests)
  • 14. 14 3. How changes benefit developers 3.4 Your log is the “real” log - Include more filters - No more trash logs
  • 15. 15 3. How changes benefit developers 3.5 Enable Cross function/team log correlation checking - BE can check logs from FE/SRE if suspects issues from their side - Remove the dependency between teams for error checking
  • 16. 16 3. How changes benefit developers 3.5 Enable Logs2Metrics capability
  • 17. Private & Confidential 17 AGENDA 1. Logging - the crucial part ! 2. Motivation for changes 3. How changes benefit developers 4. Multi-cluster logging architecture 5. Usages & Notes
  • 18. 18 4. Multi-cluster logging architecture 4.1 Logging basic
  • 19. 19 4. Multi-cluster logging architecture 4.1 Logging basic Filebeat: - Select messages for shipping (/var/logs/container/*.log) - Select fields for shipping along with msg (kubernetes.namespace/imageVersion,…) - Enrich msg with additional metadata (env, service_name, rename fields) - Message processing AMAP to reduce workload for Logstash Logstash: - Drop more fields that can not removed at Filebeat level - Using built-in module to parse specific log type (NGINX log) - Log aggregator & sent to Elasticsearch for storage ElasticSearch: - Messages allocated to indices - Index retention by ILM (Index Lifecyle Management) Kibana: - Visualization & Analysis
  • 20. 20 4. Multi-cluster logging architecture 4.2 Logging at Scale (multi-cluster architecture)
  • 21. Private & Confidential 21 AGENDA 1. Logging - the crucial part ! 2. Motivation for changes 3. How changes benefit developers 4. Multi-cluster logging architecture 5. Usages & Notes
  • 22. 22 5. Usages & Notes - Deploy fresh service - Observe how log shipping - Enjoy reading logs in Kibana
  • 23. Private & Confidential 23 THANKS FOR YOUR ATTENTION