Session 42 Speaker: http://www.devfestdc.org/speakers/ray-tsang/ Session: http://www.devfestdc.org/sessions/ray-kubernetes/

1. @saturnism @googlecloud
Debugging & Troubleshooting Microservices
w/ Kubernetes
It runs in production, but it's not enough!

2. @saturnism @googlecloud
Ray Tsang
Developer Advocate
Google Cloud Platform
@saturnism | +RayTsang

3. @saturnism @googlecloud
Ray Tsang
Developer
Architect
Traveler
Photographer
flickr.com/saturnism

4. @saturnism @googlecloud

5. @saturnism @googlecloud #kubernetes #devoxx

6. @saturnism @googlecloud

7. @saturnism @googlecloud
Hello World
Service - Greet
Guestbook
Service - Create
Guestbook Service -
Retrieve

8. @saturnism @googlecloud
Guestbook
UI
Hello World
Service
Redis
session replication
greeting
MySQL
Guestbook
Service
CRUD

9. @saturnism @googlecloud
Visibility & Insight is Key

10. @saturnism @googlecloud
Let's see it!

11. @saturnism @googlecloud
Production vs Non-Production :(

12. @saturnism @googlecloud
Distributed Systems are hard!

13. @saturnism @googlecloud
We've been there, done that

14. @saturnism @googlecloud
Information - Logs!
kubectl logs -f pod_id

15. @saturnism @googlecloud
Inspecting the process / container
kubectl exec -ti pod_id /bin/bash

16. @saturnism @googlecloud
Port forward to container
kubectl port-forward pod_id local_port:remote_port

17. @saturnism @googlecloud
Isolate problematic instance w/ Labels
kubectl label pod pod_id --overewrite serving=false

18. @saturnism @googlecloud
Pod
frontend
Pod
serving = true
version = 1.0
Pod
serving = true
version = 1.0
Service
Label selectors:
version = 1.0
serving = true
Pod
serving = true
version = 1.0

19. @saturnism @googlecloud
Pod
frontend
Pod
serving = true
version = 1.0
Pod
serving = true
version = 1.0
Service
Label selectors:
version = 1.0
serving = true
Pod
serving = false
version = 1.0

20. @saturnism @googlecloud
Log aggregation, at scale

21. @saturnism @googlecloud
Stackdriver Logging
https://cloud.google.com/logging/

22. @saturnism @googlecloud

23. @saturnism @googlecloud
Metrics from Logs → Alerting
Near-realtime insight

24. @saturnism @googlecloud
Search and Query Your Logs

25. @saturnism @googlecloud
Export to BigQuery, Storage, Pub/Sub
Near-realtime insight

26. @saturnism @googlecloud

27. @saturnism @googlecloud

28. @saturnism @googlecloud
Tracing at Google - Dapper
https://research.google.com/pubs/pub36356.html

29. @saturnism @googlecloud
Zipkin

30. @saturnism @googlecloud
Stackdriver Trace
https://cloud.google.com/trace/

31. @saturnism @googlecloud
Zipkin → Stackdriver Trace
https://cloud.google.com/trace/docs/zipkin

32. @saturnism @googlecloud

33. @saturnism @googlecloud

34. @saturnism @googlecloud

35. @saturnism @googlecloud

36. @saturnism @googlecloud
Automatic reports
Detect performance regression

37. @saturnism @googlecloud
Oops… I forgot a log message. Darn!

38. @saturnism @googlecloud
Don't you wish you can use a debugger?

39. @saturnism @googlecloud
Stackdriver Debugger - Production Debugger
https://cloud.google.com/debugger/

40. @saturnism @googlecloud
Add debugger agent
java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE

41. @saturnism @googlecloud
Logpoint - dynamically add log messages!
No redeployment, Magic.
gcloud debug logpoints create HelloworldService.java:35
"Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT

42. @saturnism @googlecloud
Snapshots
Inspect call stack and variables

43. @saturnism @googlecloud

44. @saturnism @googlecloud

45. @saturnism @googlecloud
Trace
Log
Debug

46. @saturnism @googlecloud
Trace → Stackdriver Trace
Logs → Stackdriver Logging
Debug → Stackdriver Debugger
https://cloud.google.com/stackdriver/

47. @saturnism @googlecloud
Thanks!
Source: https://github.com/saturnism/spring-boot-docker
Stackdriver: https://cloud.google.com/stackdriver/
Ray Tsang
@saturnism
Google Cloud Platform