How to design Log Management System for microservices architect. Focus on Log classification, with two groups: Business Log and Operation Log.

1. Log Management
System
for Microservices
Võ Duy Tuấn
Founder
Teamcrop.com

2. Vo Duy Tuan
 PHP, Javascript, Flutter
 Web Scalability & Optimization
 Data Mining
 ERP System
 Docker & Microservices
Interests
Founder @ Teamcrop.com

3. Why log management?
• System troubleshooting
• Application debugging
• Performance optimization
• Security
• Business decision making

4. Type of Logs
• Business
– Auditing
– Feature Usage
– Critical Function
• Operation
– Operating System
– General
– API Request
– SQL Query
– Distributed Tracing

5. Abstract Logging Design
Storage
Collecto
r
Dashboard
Syslog
PHP Log
Nginx Log
Access Log
Error Log
Activity Log
…
PHP
React JS
Grafana
File
Syslog
RabbitMQ
MySQL
Clickhouse

6. Anatomy of a Service @Teamcrop
Host (Linux)
Service Container (Docker)
Web Server (Nginx) + PHP
Application Request (eg: order)
SQL Query 1
SQL Query 2
Call another services request (Eg: customer, cashflow…)
OS log
Syslog
Access, Error log
API Request, Audit…
SQL Query log
Distributed tracing

7. | Business1

8. 1.1. Auditing Log
• Metrics:
– Employee
– Service
– Section (controller)
– Action (add, update, delete, import..)
– Severity
– Change log
– IP Address

9. 1.1. Store Auditing Log
Audit Log
Message
Queue
(RabbitMQ)
Queue Worker
(PHP)
Teamcrop API
(PHP)
Teamcrop Log
(MySQL)

10. 1.1. View Auditing Log
Teamcrop
Audit Log API
(PHP)
Teamcrop Log
(MySQL)
Dashboard
(React JS)
Request Query

11. 1.1. Auditing Log Stack
• Collector
– PHP
• Storage
– MySQL
• Dashboard
– React JS

12. 1.1. Auditing Log Example

13. 1.1. Auditing Log Example (cont.)

14. 1.2. Feature Usage Log
• Metrics:
– Visitor
– Page View
– Time on Page
– Technology
– …
• Stack:
– Google Analytics
– Self-hosted application

15. 1.2. Feature Usage Log Example

16. 1.3. Critical Function Log
• Metrics:
– Raw POST data
• Stack:
– Collector: PHP
– Storage: File System
– Dashboard: React JS

17. 1.3. Critical Function Log Example

18. | Operation2

19. 2.1. Operating System Log
• Metrics:
– Memory (RAM)
– CPU
– Disk
– Network
– …
• Stack:
– NodeQuery.com

20. 2.1. Operating System Log Example

21. 2.1. Operating System Log Example
(cont.)

22. 2.1. Operating System Log Example
(cont.)

23. 2.2. General Log
• Metrics:
– PHP: error, warning…
– Nginx: access, error…
– Custom Log Message
– Cronjob Log
– Queue Log
– Others

24. 2.2. Store General Log
General Log
(Access, Error Log)
Syslog
(syslog-ng)
UDP
UDP Log Server
(PHP)
Message
Queue
(RabbitMQ)
UDP Log Worker
(PHP)
Clickhouse
(Syslog Table)

25. 2.2. View General Log
Teamcrop
Analytics API
(PHP)
Clickhouse
(Syslog Table)
Dashboard
(React JS)
Request Query

26. 2.2. General Log Stack
• Collector:
– Syslog-ng
• Storage:
– RabbitMQ (Queue)
– Clickhouse
• Dashboard:
– PHP
– React JS

27. 2.2. General Log Example

28. 2.2. General Log Example (cont.)

29. 2.3. API Request Log
• Metrics:
– HTTP Status
– HTTP Method
– Tenant
– Service name
– Function (controller, action)
– Execution time
– Memory
– Source
– …

30. 2.3. Store API Request Log
API Request Log Syslog
(syslog-ng)
UDP
UDP Log Server
(PHP)
Message
Queue
(RabbitMQ)
UDP Log Worker
(PHP)Clickhouse

31. 2.3. View API Request Log
Teamcrop
Analytics API
(PHP) Clickhouse
(API Request
Log Table)
Dashboard
(React JS)
Request Query
Grafana UI Grafana API
Request
Query

32. 2.3. API Request Log Stack
• Collector:
– Syslog-ng
– Monolog (PHP)
• Storage:
– RabbitMQ (Queue)
– Clickhouse
• Dashboard:
– PHP
– React JS
– Grafana

33. 2.3. API Request Log Example

34. 2.3. API Request Log Example
(cont.)

35. 2.3. API Request Log Example
(cont.)

36. 2.3. API Request Log Example
(cont.)

37. 2.4. SQL Query Log
• Metrics:
– Replication
– Table
– Query type (SELECT, INSERT…)
– Execution
– Tenant
– …

38. 2.4. Store SQL Query Log
SQL Query Log Syslog
(syslog-ng)
UDP
UDP Log Server
(PHP)
Message
Queue
(RabbitMQ)
UDP Log Worker
(PHP)Clickhouse

39. 2.4. View SQL Query Log
Teamcrop
Analytics API
(PHP) Clickhouse
(SQL Query
Log Table)
Dashboard
(React JS)
Request Query
Grafana UI Grafana API
Request
Query

40. 2.4. SQL Query Log Stack
• Collector:
– Syslog-ng
– Monolog (PHP)
• Storage:
– RabbitMQ (Queue)
– Clickhouse
• Dashboard:
– PHP
– React JS
– Grafana

41. 2.4. SQL Query Log Example

42. 4. SQL Query Log Example (cont.)

43. 4. SQL Query Log Example (cont.)

44. 2.5. Distributed Tracing Log
• Metrics:
– Chain of requests
– Duration
– Execution
– Tenant
– Service
– …

45. 2.5. Store Distributed Tracing Log
OpenTracing Log
(ZipKin format)
Syslog
(syslog-ng)
UDP
UDP Log Server
(PHP)
Message
Queue
(RabbitMQ)
UDP Log Worker
(PHP)
Jaeger
API
Jaeger
DB

46. 2.5. Distributed Tracing Log Stack
• Collector:
– Syslog-ng
– Monolog (PHP)
– Zipkin Client Library (PHP)
• Storage:
– RabbitMQ (Queue)
– Jaeger
• Dashboard:
– Jaeger UI

47. 2.5. Distributed Tracing Model
• OpenTracing Specification

48. 2.5. Distributed Tracing Log Example

49. DEMO
RabbitMQ
Jeager UI
Clickhouse GUI Client
Customized Log Dashboard
NodeQuery
Grafana
Teamcrop Log Audit

50. Thanks!
CONTACT:
voduytuan@teamcrop.com
bloghoctap.com