Comprehensive Monitoring for Docker

Comprehensive Monitoring for Docker

• 1.
  Sumo Logic Confidential ComprehensiveMonitoring for Docker Christian Beedgen, CTO & Co-Founder, Sumo Logic San Francisco Docker Meetup, February 9, 2016
• 2.
  Sumo Logic Confidential $whoami • Co-Founder & CTO, Sumo Logic Cloud-based Machine Data Analytics Service Applications, Operations, Security • Chief Architect, ArcSight Major SIEM player in the enterprise space Log Management for security and compliance
• 3.
  Sumo Logic Confidential Agenda •The Logging Story So Far • Comprehensive Monitoring • If I Had A Wish…
• 4.
  Sumo Logic Confidential December2014, New York City
• 5.
  Sumo Logic Confidential December2014, New York City
• 6.
  Sumo Logic Confidential December2014, New York City
• 7.
  Sumo Logic Confidential December2014, New York City
• 8.
  Sumo Logic Confidential December2014, New York City
• 9.
  Sumo Logic Confidential December2014, New York City
• 10.
  Sumo Logic Confidential December2014, New York City
• 11.
  Sumo Logic Confidential December2014, New York City
• 12.
  Sumo Logic Confidential December2014, New York City
• 13.
  Sumo Logic Confidential December2014, New York City
• 14.
  Sumo Logic Confidential December2014, New York City
• 15.
  Sumo Logic Confidential December2014, New York City
• 16.
  Sumo Logic Confidential December2014, New York City
• 17.
  Sumo Logic Confidential December2014, New York City
• 18.
  Sumo Logic Confidential December2014, New York Cityhttp://www.slideshare.net/raychaser/ 6-million-ways-to-log-in-docker-nyc- docker-meetup-12172014
• 19.
  Sumo Logic Confidential Season2 Where Are We In Early 2016?
• 20.
  Sumo Logic Confidential Basics •Logging in Docker as per 12factor.net
• 21.
  Sumo Logic Confidential Basics •Logging in Docker as per 12factor.net
• 22.
  Sumo Logic Confidential Basics •Logging in Docker as per 12factor.net • Also, one process per container, plz!
• 23.
  Sumo Logic Confidential Pre-Docker1.6 • Docker simply collects stdout and stderr from a container • Wrapped in a bit of JSON and stored on disk
• 24.
  Sumo Logic Confidential Pre-Docker1.6 • Docker simply collects stdout and stderr from a container • Wrapped in a bit of JSON and stored on disk
• 25.
  Sumo Logic Confidential Pre-Docker1.6 • Docker simply collects stdout and stderr from a container • Wrapped in a bit of JSON and stored on disk
• 26.
  Sumo Logic Confidential Pre-Docker1.6 • Docker simply collects stdout and stderr from a container • Wrapped in a bit of JSON and stored on disk
• 27.
  Sumo Logic Confidential Pre-Docker1.6 • Early hardcore crowd would just collect /var/lib/docker/containers/** • And then of course there’s the UX: docker logs • docker logs is using a daemon API for getting the logs • This leads to logspout – attach to API, forward to Syslog • https://github.com/gliderlabs/logspout
• 28.
  Sumo Logic Confidential Docker1.6 Introduced Log Drivers • Hallelujah • Initially supports json-file, syslog, null • json-file – default, this is the old mechanism – Continues to this day to be required for API access and docker logs – Exception: As of 1.8, the journald driver (introduced in 1.7) supports docker logs • docker run -–log-driver syslog – Sends to local Syslog, no more writing to disk • docker run –-log-driver null – STFU, basically
• 29.
  Sumo Logic Confidential Docker1.7 Introduces --log-opt • Now we can pass parameters to the log drivers! • docker run --log-driver syslog --log-opt syslog-address=(udp|tcp)://… --log-opt syslog-facility=(kern|daemon|user|local0|…) --log-opt syslog-tag=“myapp” • Forward directly to local Syslog aggregator, or to a cloud-based logging service • Docker 1.7 also added support to log to journald
• 30.
  Sumo Logic Confidential Alsoin Docker 1.8 – Options For json-file • json-file still the default, still required for docker logs and /logs API • Long standing problem – will eventually fill up your disk • Folks have been using logrotate hacks… • Now, json-file log driver can be configured: • Basically, keep up to max-file files, roll current at max-size
• 31.
  Sumo Logic Confidential Docker1.9 Introduces Log Tags For The Syslog Driver • Many containers can share a single aggregator downstream the log driver • All this muxing creates a problem – which log from which container? • Basically, there is a loss of meta data • Log Tags enable to use of container meta data as part of each message • --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" • Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]: Hello from Docker.
• 32.
  Sumo Logic Confidential Docker1.9 Introduces Log Tags For The Syslog Driver • Many containers can share a single aggregator downstream the log driver • All this muxing creates a problem – which log from which container? • Basically, there is a loss of meta data • Log Tags enable to use of container meta data as part of each message • --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" • Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]: Hello from Docker.
• 33.
  Sumo Logic Confidential Docker1.9 Introduces Log Tags For The Syslog Driver • Many containers can share a single aggregator downstream the log driver • All this muxing creates a problem – which log from which container? • Basically, there is a loss of meta data • Log Tags enable to use of container meta data as part of each message • --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" • Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]: Hello from Docker.
• 34.
  Sumo Logic Confidential Docker1.9 Introduces Log Tags For The Syslog Driver • Many containers can share a single aggregator downstream the log driver • All this muxing creates a problem – which log from which container? • Basically, there is a loss of meta data • Log Tags enable to use of container meta data as part of each message • --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" • Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]: Hello from Docker.
• 35.
  Sumo Logic Confidential Docker1.10 Adds TCP+TLS Support for Syslog • This is really nice, because now you can send to the cloud securely • 1.10 actually has a bug that prevents this from actually working • But fear not, it will be fixed in 1.10.1 - Thanks @cavalera!
• 36.
  Sumo Logic Confidential Docker1.10 Adds TCP+TLS Support for Syslog • This is really nice, because now you can send to the cloud securely • 1.10 actually has a bug that prevents this from actually working • But fear not, it will be fixed in 1.10.1 - Thanks @cavalera!
• 37.
  Sumo Logic Confidential Forgivethe plug… We are about to release a Cloud Syslog endpoint!
• 38.
  Sumo Logic Confidential Docker1.8, 1.9, 1.10 - Even More Log Drivers • Fluentd • GELF • AWS • Splunk
• 39.
  Sumo Logic Confidential
• 40.
  Sumo Logic Confidential WhatElse? • Proposal for Google Cloud Cloud Logging driver – https://github.com/docker/docker/issues/18726 • Proposal for a Redis driver (rejected today!) – https://github.com/docker/docker/pull/20146 • Proposal for a TCP, UDP, Unix Domain Socket driver – https://github.com/docker/docker/pull/18001
• 41.
  Sumo Logic Confidential WhereWill It End? • Logging drivers have been a very large step forward in the last 12 months • As part of the engine, need to go through the engine commit protocol • This good, because there’s a lot of review  stability • But it is also suboptimal because it is not really modular • And it adds more and more dependencies on third party libraries • Can this be decoupled?
• 42.
  Sumo Logic Confidential
• 43.
  Sumo Logic Confidential Let’sget back to this a little later…
• 44.
  Sumo Logic Confidential https://www.sumologic.com/2015/06/16/comprehensive-monitoring-for-docker-more-than-just-logs
• 45.
  Sumo Logic Confidential TheWorld… Does Not Live On Logs A Alone
• 46.
  Sumo Logic Confidential ForComprehensive Monitoring & Troubleshooting 1.Events 2.Configurations 3.Logs 4.Statistics 5.Host and daemon logs
• 47.
  Sumo Logic Confidential Events •Enumerate all running containers, then – Start listening to the event stream, then… • For each running container, and each start event... – Start collecting (more on that in a minute) • Also, send each event JSON as a log – Now we can analyze what the daemon is doing! – What containers are getting started, stopped etc. etc. etc.
• 48.
  Sumo Logic Confidential
• 49.
  Sumo Logic Confidential Configurations •For each running container, and each start event – Call inspect API • Send result JSON as a log • Now we have all the configurations in the logs! – And we can quickly search for them when we troubleshoot
• 50.
  Sumo Logic Confidential Logs •For each running container, and each start event – Call logs API to open a stream • Send each log… as a log
• 51.
  Sumo Logic Confidential Statistics •For each running container, and each start event – Call stats API to open a stream • Send each received JSON as a log • Now we have monitoring!
• 52.
  Sumo Logic Confidential Host& Daemon Logs • Include collector into host images • Or run collector as a container – … and -v /var/log etc.
• 53.
  Sumo Logic Confidential ThanksTo The API, We Can Really Do This
• 54.
  Sumo Logic Confidential ThanksTo The API, We Can Really Do This
• 55.
  Sumo Logic Confidential ThanksTo The API, We Can Really Do This
• 56.
  Sumo Logic Confidential IfI Had A Wish…
• 57.
  Sumo Logic Confidential IReally, Really <3 The API • Of course, there’s always limitations • What we found – logs API requires jsonfile driver (or journald) which means it touches the disk – Race condition between start event and have logs or stats stream established – Scaling connections on both daemon and client side • 100 containers on host (or Swarm!) • 100 * (1 event stream connection + (1 logs + 1 stats stream per container) • 201 connections! – Different mechanisms between events, logs, stats
• 58.
  Sumo Logic Confidential SoWhat Is That Dream? • One combined stream for events, logs, stats • Either as an API call to pull, perhaps with a filter – Name, label, type, start time • Or as a registration API call to indicate where to send to – Daemon firehoses to the endpoint, with strict guarantees for the endpoint • I personally could deal with either approach • Can we expand #18604 ”Logging driver plugins” to allow for this?
• 59.
  Sumo Logic Confidential Thankyou! @raychaser