Maxime Petazzoni gave a presentation on Docker at SignalFx. He discussed how SignalFx uses Docker for infrastructure separation, development lifecycles, application packaging and delivery, and orchestration. He explained how SignalFx monitors Docker containers using CollectD and their Docker plugin to collect host and container metrics. SignalFx provides curated dashboards and anomaly detection for correlated system, container and application metrics.

1. MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
DOCKER {AT,WITH} SIGNALFX

2. MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Me?
github.com/mpetazzoni
@mpetazzoni

3. MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
What is SignalFx?

4. The pitch
• Real-time monitoring system for modern applications
• User-defined, streaming and interactive analytics
• Alerting and anomaly detection
• Easy to deploy, to use and to integrate
• Scales; from small deployments to large, complex infrastructures

5. MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Docker at SignalFx

6. Docker at SignalFx
• Started using Docker in November 2013
• Running Docker in prod ever since we’ve had a “prod”
• Back when Docker’s README said “DO NOT RUN IN PRODUCTION”

7. Objectives
• Separate infrastructure concerns
• Stronger independence from hosting provider
• Development lifecycle
• Application packaging and delivery
• Higher-level orchestration

8. Infrastructure vs application
• Separation of concerns
• Infrastructure provides appropriately spec-ed hosts and network
• Abstracts from cloud provider
• Currently on AWS
• Amazon Linux AMIs + Salt to setup a few things, up to Docker

9. Infrastructure vs application (cont.)
• “Application” stack takes it up from there
• Everything is containerized, including 3rd party components
• No dependence on Amazon specifics
• We can spin up the stack anywhere (laptop, GCE, DO, …)

10. SDLC
• Run locally, in tests and in prod environments
• Same bits, same tools, same arguments, etc.
• Crucial for software validation
• Maven, Jenkins, Quay.io, MaestroNG
• maestro start

11. SDLC (cont.)
docker-maven-plugin

12. SDLC (cont.)
• Continuous build of the release branch
• Builds and pushes :latest tagged Docker images
• Promotion track through :lab, :prod, :rc and :mon
• One-click promote + upgrade, orchestrated by MaestroNG
• Automated integration tests environment

13. Application packaging and delivery
• All components of the stack are packaged as Docker images
• 3rd-party component packaging was figured out once (Dockerfile)
• SignalFx components all have their Dockerfile too
• Built by docker-maven-plugin
• All components have a run.py init script

14. Discovery
• ZooKeeper / Curator service discovery
• We built announcing wrappers for Cassandra and ElasticSearch
• Only need to know “realm” name and zkConnectString
• Everything else is discovered from ZooKeeper
• Need to make sure to announce host IP address/port
• Maestro gives us that via environment

15. Orchestration
• We built MaestroNG
• Orchestrator of multi-host Docker environments
• github.com/signalfuse/maestro-ng
• Jinja2/Yaml environment file
• Defines what runs where, and how
• Knows and respects service dependencies

16. MAXIME PETAZZONI — PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015
Monitoring containers

17. The basics
• Docker’s /stats API provides the bare minimum
• CPU
• Memory
• Network I/O
• Disk I/O
• Pitfalls; GET /stats is blocking, up to a second
• Use CollectD and docker-collectd-plugin
• github.com/signalfx/docker-collectd-plugin

18. The basics (cont.)
• Data returned by /stats API needs “math” to yield usable numbers
• See Docker client code
• Better: use SignalFx!
• Docker Dashboard
• Collect host metrics too if you can (with CollectD)
• System/container-level metrics are very limited
• We need to go deeper

19. Application monitoring
• Usually needs more specific visibility
• Instrument first, ask questions later
• Metrics are cheap, both in CPU and $
• For 3rd-party apps, use CollectD and its plugins
• We’re doing a lot of work to improve/provide CollectD plugins for common FOSS
• And provide curated dashboards for each of them

20. Application monitoring (cont.)
• For 1st-party software, depends on language platform
• For VM-based services (Java, Scala, Go, …), get key metrics
• Heap
• # threads
• Time spent in GC and # collections
• Use metric libraries and instrument everything
• Bring it all together in dashboards and anomaly detectors

21. Docker with SignalFx
• Objective: time-to-data as short as possible
• Easiest route: use our packaged CollectD
• Good default set of host metrics
• Install/enable plugins as needed; docker-collectd-plugin
• SignalFx automatically imports curated dashboards
• That’s it!

22. More monitoring with SignalFx
• Use our client libraries or API to send your monitoring data
• Or tee off your existing Graphite, StatsD, … pipeline towards SignalFx
• Leverage the power of the real-time, streaming analytics
• Correlate system, container and application metrics
• Build better dashboards and alerts from higher-level signals

23. Thanks!
github.com/signalfuse/maestro-ng
github.com/signalfx/docker-collectd-plugin
signalfx.com/free-trial.html
@signalfx — signalfx.com/careers.html
@mpetazzoni — github.com/mpetazzoni