This is the Docker Logging & Monitoring workshop completed during DockerCon 2018 Europe. We cover how to build native tools in Docker, deploy an ELK stack, and Prometheus with cAdvisor, node-exporter, Prometheus, and Grafana stack

1. Transparency into your workloads
Docker Monitoring & Logging

2. Agenda
- [ ] Introduction
- [ ] Operations Overview
- [ ] Logging Workshop
- [ ] Monitoring Workshop
- [ ] Best Practices & Recap

3. Docker Captain
Site Reliability Engineer / Co-Founder
56K.Cloud
Brian Christner

4. Site Reliability Engineer / Co-Founder
56K.Cloud
Darragh Grealish
Docker Captain / Solutions Architect
BoxBoat
Brandon Mitchell

6. Agenda
- [ X ] Introduction
- [ ] Operations Overview
- [ ] Logging Workshop
- [ ] Monitoring Workshop
- [ ] Best Practices & Recap

7. Zugerberg, Zug, Switzerland

10. Ops Paradise
• Everything is Automated
• Reduce Costs
• No support calls

11. • Total Downtime: Just under 4
minutes
• 502 error messages total: 12 000
• People affected by the 502 error
who did not get their bargain: 400
Website Down?

12. Ops Firefighting

13. Operational Models

14. Users Care About 3 Things
● Availability - Is my system online yes/no
● Latency - Does it take a long time to access application x,y,z
● Reliability - Can the user rely on using the application

15. Brain Based Tools
• We can track 8 objects on average
• 4 Moving Objects
• Build Dashboards & Tools
accordingly

16. SRE

17. SRE is treats Operations as if it
were a Software Problem
“Hope is not a
strategy.”
Traditional SRE saying
SRE (Site Reliability Engineering)
www.google.com/sre

18. 4 Golden Signals
www.google.com/sre
Latency
Traffic
Errors
Saturation

19. R.E.D (Microservice level)
(Request) Rate: the number of requests, per second, you
services are serving.
(Request) Errors: the number of failed requests per second.
Utilization: the average time that the resource was busy servicing
work
(Request) Duration: distributions of the amount of time each
request takes.

20. U.S.E (Low Level / Infrastructure)
For every resource, check Utilization, Saturation, and Errors
Resource: all physical server functional components (CPUs,
disks, busses, ...)
Utilization: the average time that the resource was busy
servicing work
Saturation: the degree to which the resource has extra work
which it can't service, often queued
Errors: the count of error events

21. Black box vs. White box
Monitoring
Black Box Monitoring White Box Monitoring
App metrics, requests,
responses, process times
HTTP, Ping, etc
External App Metrics Internal App Metrics

22. Healthchecks

23. Readiness / Liveness Probes

24. What to Monitor

25. Operating
Systems
Understanding Failure Modes
Config Mgt Monitoring LoggingCI/CD ..more..Images Networking Volumes
PhysicalVirtualizationPublic Cloud
Developer
Services
Registry
Services
Access
Policies
App Lifecycle
Management
Automation &
Extensibility
Networking Orchestration Storage
Container Engine
ENTERPRISE PLATFORM
Platform
Security

26. Operating
Systems
Host / Hardware
Config Mgt Monitoring LoggingCI/CD ..more..Images Networking Volumes
PhysicalVirtualizationPublic Cloud
Developer
Services
Registry
Services
Access
Policies
App Lifecycle
Management
Automation &
Extensibility
Networking Orchestration Storage
Container Engine
ENTERPRISE PLATFORM
Platform
Security
CPU
Memory
Liveness
File Descriptors
Storage Capacity

27. Operating
Systems
Networking
Config Mgt Monitoring LoggingCI/CD ..more..Images Networking Volumes
PhysicalVirtualizationPublic Cloud
Developer
Services
Registry
Services
Access
Policies
App Lifecycle
Management
Automation &
Extensibility
Networking Orchestration Storage
Container Engine
ENTERPRISE PLATFORM
Platform
Security
Reachability
Link Utilization
File Descriptors
Storage Capacity

28. Operating
Systems
Orchestration
Config Mgt Monitoring LoggingCI/CD ..more..Images Networking Volumes
PhysicalVirtualizationPublic Cloud
Developer
Services
Registry
Services
Access
Policies
App Lifecycle
Management
Automation &
Extensibility
Networking Orchestration Storage
Container Engine
ENTERPRISE PLATFORM
Platform
Security
State
Deployment Rates
Capacity
Scheduling Events

29. Operating
Systems
Applications
Config Mgt Monitoring LoggingCI/CD ..more..Images Networking Volumes
PhysicalVirtualizationPublic Cloud
Developer
Services
Registry
Services
Access
Policies
App Lifecycle
Management
Automation &
Extensibility
Networking Orchestration Storage
Container Engine
ENTERPRISE PLATFORM
Platform
Security
Response Times
Error Rates
App Specific Metrics
Availability

30. Improve incrementally

32. Agenda
- [ X ] Introduction
- [ X ] Operations Overview
- [ ] Logging Workshop
- [ ] Monitoring Workshop
- [ ] Best Practices & Recap

33. Logging Tools Overview

34. github.com/56kcloud/Training

35. https://ee-labs.play-with-docker.com

36. ● What should we log?
● Where and how long should we store logs
● Analysis
Logging Challenges

37. ● docker logs
● docker-compose logs
● docker service logs
● log drivers
Logging Tools

38. Logging Workshop

39. github.com/56kcloud/Training

40. Agenda
- [ X ] Introduction
- [ X ] Operations Overview
- [ X ] Logging Workshop
- [ ] Monitoring Workshop
- [ ] Best Practices & Recap

41. ● What's Broken?
● Why is it Broken?
● How Long has it been broken?
The Basis of Monitoring

42. ● Docker Stats
● Docker Top
● Docker df
● cAdvisor
● Prometheus
Monitoring Tools

43. • Live container resources
• All containers or single
• Very basic but useful info
Docker Stats

44. Docker Top
● Display running process in a container

45. • Developed by Google
• Real-time Data
• Clean UI
• Exposes Metrics
• Integrates well

47. PrometheusStack

48. Federated Configuration

49. github.com/56kcloud/Training

50. Agenda
- [ X ] Introduction
- [ X ] Operations Overview
- [ X ] Logging Workshop
- [ X ] Monitoring Workshop
- [ ] Best Practices & Recap

51. • Start small & increment
• Don’t Overlert yourself
• Set Resource Limits
• Aim for actionable Information
• Run separate from Workload
• Test for Failures
• Know your Failure Models
Best Practices

52. - 56K.Cloud - https://56K.Cloud
- Prometheus - https://github.com/vegasbrianc/prometheus
- ELK - https://github.com/deviantony/docker-elk
- Labs – github.com/56kcloud/Training/tree/master/DockerCon
- Docker Resource Link - https://awesome-docker.netlify.com
Resources

53. Agenda
- [ X ] Introduction
- [ X ] Operations Overview
- [ X ] Monitoring Workshop
- [ X ] Logging Workshop
- [ X ] Best Practices & Recap

54. Thank You
Brian Christner
brian@56K.cloud
@idomyowntricks

55. Transparency into your workloads
Docker Monitoring & Logging