This document discusses optimizations and updates to ELK logging in 2021. It covers the importance of logging, motivations for changes to improve logging, benefits to developers, and an overview of the multi-cluster logging architecture. Key points include faster log searches by indexing logs from different environments separately, automatic log shipping setup, cleaner log formats, and enabling log correlation and conversion to metrics. The logging flow involves Filebeat selecting logs and metadata, Logstash parsing and aggregating logs, Elasticsearch storing logs in indices, and Kibana for visualization and analysis.

1. ELK Logging
Optimizations & Updates for 2021
Bao Huynh
Site Reliability Engineer
29-Dec-2020 1

2. Private & Confidential 2
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes

3. Private & Confidential 3
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes

4. 4
1. Logging – the crucial part !

5. 5
1. Logging – the crucial part !

6. 6
1. Logging – the crucial part !
Clearly states that:
- Applications should not attempt to write to / manage logsfiles
- Instead, each running process writes its event stream  STDOUT
- During “local development”: observe on the terminal
- During “staging/production”:
Capture & collected by agents.
Then, routed to one final destination for viewing & long-term archival.

7. 7
1. Logging – the crucial part !

8. Private & Confidential 8
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes

9. 9
2. Motivation for changes
- Could not ship log for external helm-chart services (ingress-nginx, redis, es,..)
- Developers have to add “additional fields” (svc + env) in their log lines
- Once search, it goes through all indices (logstash-*) to find matching =>
longer result return
- Hard to scale with multi-cluster architecture

10. Private & Confidential 10
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes

11. 11
3. How changes benefit developers
3.1 Faster lookup/search log pattern
- Logs from *different env/ns* is allocated into *different indices*

12. 12
3. How changes benefit developers
3.2 Dev do what to get log shipping ?
- Kibana.vndev.shopee.vn
- Everything is ready for you

13. 13
3. How changes benefit developers
3.3 Fresher, cleaner & better log format
- Remove unneccessary fields
- Compact fields:
+ services_.* (metadata related to your application)
+ ingress_nginx.* (metadata related to client requests)

14. 14
3. How changes benefit developers
3.4 Your log is the “real” log
- Include more filters
- No more trash logs

15. 15
3. How changes benefit developers
3.5 Enable Cross function/team log correlation checking
- BE can check logs from FE/SRE if suspects issues from their side
- Remove the dependency between teams for error checking

16. 16
3. How changes benefit developers
3.5 Enable Logs2Metrics capability

17. Private & Confidential 17
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes

18. 18
4. Multi-cluster logging architecture
4.1 Logging basic

19. 19
4. Multi-cluster logging architecture
4.1 Logging basic
Filebeat:
- Select messages for shipping (/var/logs/container/*.log)
- Select fields for shipping along with msg (kubernetes.namespace/imageVersion,…)
- Enrich msg with additional metadata (env, service_name, rename fields)
- Message processing AMAP to reduce workload for Logstash
Logstash:
- Drop more fields that can not removed at Filebeat level
- Using built-in module to parse specific log type (NGINX log)
- Log aggregator & sent to Elasticsearch for storage
ElasticSearch:
- Messages allocated to indices
- Index retention by ILM (Index Lifecyle Management)
Kibana:
- Visualization & Analysis

20. 20
4. Multi-cluster logging architecture
4.2 Logging at Scale (multi-cluster architecture)

21. Private & Confidential 21
AGENDA
1. Logging - the crucial part !
2. Motivation for changes
3. How changes benefit developers
4. Multi-cluster logging architecture
5. Usages & Notes

22. 22
5. Usages & Notes
- Deploy fresh service
- Observe how log shipping
- Enjoy reading logs in Kibana

23. Private & Confidential 23
THANKS FOR YOUR ATTENTION