Lessons learned observing in microservices

1. Lessons learned observing
in Microservices
Matt Ho

2. SoftLeader RD
Matt Ho
https://github.com/shihyuho

3. Microservices!

4. Move from Spring Cloud
to Kubernetes native!

5. 好多 App, 平常沒事, But...

6. Infrastructure0

7. Tracing
App
App
App
container platform

8. Monitoring
App
App
App
container platform
cpu/memory/VM heap …
cpu/memory/VM heap …

9. Loggin
g
App
App
App
container platform

10. Kubernetes on-premises

11. 原⽣的, ⼀裝好什麼都沒有!

12. 不能客製呦!

13. 不是在做比較

14. Tracing1

15. Using OpenTracing#

16. github.com/opentracing-contrib/
java-spring-cloud

17. Put tracing in response#

18. Appfilter
response
request
api
add trace info

19. Add tracing in logger MDC#

20. App A
log
MDC

26. Create your Spans#

27. time
Span A
Span B Span C
Span D
外部呼叫
trace

29. Propagating messages#

31. App A App B
App C
App D
Queue
Baggage

32. App A App B
App C
App D
REST
gRPC
Queue
Enqueue
Dequeue
Baggage

34. Monitoring2

35. Prometheus & exporters#

36. - node-exporter - Hardware/OS metric
s
- cAdvisor - Container metric
s
- kube-state-metrics - Kubernetes Object metrics
環境類 exporters

37. github.com/prometheus-operator/
kube-prometheus

41. - blackbox - Probing over HTTP, HTTPS, DNS
…
- micrometer.io - Java app metrics
服務類 exporters

44. https://grafana.com/grafana/
dashboards

45. Logging3

46. Define logging format#

47. JSON logging format

48. github.com/logstash/
logstash-logback-encoder

49. Append app information#

50. app_name, app_version…

53. ES index strategy#

54. By namespace and date

56. Forwarding logs from { … }#

57. Fluentd
Forward
Log format
OCP
DockerSwarm
Forward
Other Cluster

58. App
App
App
App
App
App

59. 好多 App, 平常沒事, But...

60. Thanks