This document summarizes Christian Beedgen's presentation on logging and Docker at the Seattle Docker Meetup on October 13, 2015. It discusses the evolution of logging in Docker from simply collecting stdout/stderr pre-Docker 1.6, to the introduction of log drivers in 1.6 like syslog and null. It also covers enhancements in newer Docker versions like additional log drivers, options for the json-file driver, and upcoming log tags in 1.9 to identify logs by container metadata. The document also notes Sumo Logic's work on containerizing their log collectors and vision for comprehensive Docker monitoring.

1. Sumo Logic Confidential
Logging & Docker
Christian Beedgen, CTO & Co-Founder, Sumo Logic
Seattle Docker Meetup, October 13, 2015

2. Sumo Logic Confidential
$ whoami
• Co-Founder & CTO, Sumo Logic
Cloud-based Machine Data Analytics Service
Applications, Operations, Security
• Chief Architect, ArcSight
Major SIEM player in the enterprise space
Log Management for security and compliance

3. Sumo Logic Confidential
December 2014, New York City

4. Sumo Logic Confidential
December 2014, New York City

5. Sumo Logic Confidential
December 2014, New York City

6. Sumo Logic Confidential
December 2014, New York City

7. Sumo Logic Confidential
December 2014, New York City

8. Sumo Logic Confidential
December 2014, New York City

9. Sumo Logic Confidential
December 2014, New York City

10. Sumo Logic Confidential
December 2014, New York City

11. Sumo Logic Confidential
December 2014, New York City

12. Sumo Logic Confidential
December 2014, New York City

13. Sumo Logic Confidential
December 2014, New York City

14. Sumo Logic Confidential
December 2014, New York City

15. Sumo Logic Confidential
December 2014, New York City

16. Sumo Logic Confidential
December 2014, New York City

17. Sumo Logic Confidential
December 2014, New York Cityhttp://www.slideshare.net/raychaser/
6-million-ways-to-log-in-docker-nyc-
docker-meetup-12172014

18. Sumo Logic Confidential
Season 2
Where Are We In Late 2015?

19. Sumo Logic Confidential
Basics
• Logging in Docker as per 12factor.net

20. Sumo Logic Confidential
Basics
• Logging in Docker as per 12factor.net

21. Sumo Logic Confidential
Basics
• Logging in Docker as per 12factor.net
• Also, one process per container, plz!

22. Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container
• Wrapped in a bit of JSON and stored on disk

23. Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container
• Wrapped in a bit of JSON and stored on disk

24. Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container
• Wrapped in a bit of JSON and stored on disk

25. Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container
• Wrapped in a bit of JSON and stored on disk

26. Sumo Logic Confidential
Pre-Docker 1.6
• Early hardcore crowd would just collect /var/lib/docker/containers/**
• And then of course there’s the UX: docker logs
• docker logs is using a daemon API for getting the logs
• This leads to logspout – attach to API, forward to Syslog
• https://github.com/gliderlabs/logspout

27. Sumo Logic Confidential
Docker 1.6 Introduced Log Drivers
• Hallelujah
• Initially supports json-file, syslog, null
• json-file – default, this is the old mechanism
– Continues to this day to be required for API access and docker logs
• docker run -–log-driver syslog …
– Sends to local Syslog, no more writing to disk
• docker run –-log-driver null
– STFU, basically

28. Sumo Logic Confidential
Docker 1.7 Introduces --log-opt
• Now we can pass parameters to the log drivers!
• docker run
--log-driver syslog
--log-opt syslog-address=(udp|tcp)://…
--log-opt syslog-facility=(kern|daemon|user|local0|…)
--log-opt syslog-tag=“myapp”
• Forward directly to local Syslog aggregator, or to a cloud-based logging service
• Docker 1.7 also added support to log to journald

29. Sumo Logic Confidential
Docker 1.8, 1.9 - Even More Log Drivers
• Fluentd
• GELF
• AWS

30. Sumo Logic Confidential
Also in Docker 1.8 – Options For json-file
• json-file still the default, still required for docker logs and /logs API
• Long standing problem – will eventually fill up your disk
• Folks have been using logrotate hacks…
• Now, json-file log driver can be configured:
• Basically, keep up to max-file files, roll current at max-size

31. Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags
• Many containers can share a single aggregator downstream the log driver
• All this muxing creates a problem – which log from which container?
• Basically, there is a loss of meta data
• Log Tags enable to use of container meta data as part of each message
• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.

32. Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags
• Many containers can share a single aggregator downstream the log driver
• All this muxing creates a problem – which log from which container?
• Basically, there is a loss of meta data
• Log Tags enable to use of container meta data as part of each message
• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.

33. Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags
• Many containers can share a single aggregator downstream the log driver
• All this muxing creates a problem – which log from which container?
• Basically, there is a loss of meta data
• Log Tags enable to use of container meta data as part of each message
• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.

34. Sumo Logic Confidential
Coming In Docker 1.9 – Log Tags
• Many containers can share a single aggregator downstream the log driver
• All this muxing creates a problem – which log from which container?
• Basically, there is a loss of meta data
• Log Tags enable to use of container meta data as part of each message
• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.

35. Sumo Logic Confidential
What Is Sumo Working On?
• We have containerized our collectors
– https://github.com/SumoLogic/sumologic-collector-docker
– docker run -d -p 514:514 -p 514:514/udp
--name="sumo-logic-collector"
sumologic/collector:latest-syslog
[Access ID] [Access key]
– https://www.sumologic.com/2015/09/09/update-on-logging-with-docker/

36. Sumo Logic Confidential
What Is Sumo Working On?
• We are working towards our vision of Comprehensive Monitoring
– https://www.sumologic.com/2015/06/16/comprehensive-monitoring-for-docker-more-than-just-logs/
• We have released an initial App for Docker at DockerCon 2015