Download as PDF, PPTX
Uploaded bySignalFx
Docker at and with SignalFx
Maxime Petazzoni gave a presentation on Docker at SignalFx. He discussed how SignalFx uses Docker for infrastructure separation, development lifecycles, application packaging and delivery, and orchestration. He explained how SignalFx monitors Docker containers using CollectD and their Docker plugin to collect host and container metrics. SignalFx provides curated dashboards and anomaly detection for correlated system, container and application metrics.
More Related Content
Similar to Docker at and with SignalFx
Docker at and with SignalFx
- 1. MAXIME PETAZZONI —PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015 DOCKER {AT,WITH} SIGNALFX
- 2. MAXIME PETAZZONI —PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015 Me? github.com/mpetazzoni @mpetazzoni
- 3. MAXIME PETAZZONI —PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015 What is SignalFx?
- 4. The pitch • Real-timemonitoring system for modern applications • User-defined, streaming and interactive analytics • Alerting and anomaly detection • Easy to deploy, to use and to integrate • Scales; from small deployments to large, complex infrastructures
- 5. MAXIME PETAZZONI —PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015 Docker at SignalFx
- 6. Docker at SignalFx •Started using Docker in November 2013 • Running Docker in prod ever since we’ve had a “prod” • Back when Docker’s README said “DO NOT RUN IN PRODUCTION”
- 7. Objectives • Separate infrastructureconcerns • Stronger independence from hosting provider • Development lifecycle • Application packaging and delivery • Higher-level orchestration
- 8. Infrastructure vs application •Separation of concerns • Infrastructure provides appropriately spec-ed hosts and network • Abstracts from cloud provider • Currently on AWS • Amazon Linux AMIs + Salt to setup a few things, up to Docker
- 9. Infrastructure vs application(cont.) • “Application” stack takes it up from there • Everything is containerized, including 3rd party components • No dependence on Amazon specifics • We can spin up the stack anywhere (laptop, GCE, DO, …)
- 10. SDLC • Run locally,in tests and in prod environments • Same bits, same tools, same arguments, etc. • Crucial for software validation • Maven, Jenkins, Quay.io, MaestroNG • maestro start
- 11.
- 12. SDLC (cont.) • Continuousbuild of the release branch • Builds and pushes :latest tagged Docker images • Promotion track through :lab, :prod, :rc and :mon • One-click promote + upgrade, orchestrated by MaestroNG • Automated integration tests environment
- 13. Application packaging anddelivery • All components of the stack are packaged as Docker images • 3rd-party component packaging was figured out once (Dockerfile) • SignalFx components all have their Dockerfile too • Built by docker-maven-plugin • All components have a run.py init script
- 14. Discovery • ZooKeeper /Curator service discovery • We built announcing wrappers for Cassandra and ElasticSearch • Only need to know “realm” name and zkConnectString • Everything else is discovered from ZooKeeper • Need to make sure to announce host IP address/port • Maestro gives us that via environment
- 15. Orchestration • We builtMaestroNG • Orchestrator of multi-host Docker environments • github.com/signalfuse/maestro-ng • Jinja2/Yaml environment file • Defines what runs where, and how • Knows and respects service dependencies
- 16. MAXIME PETAZZONI —PALO ALTO DOCKER MEETUP — SEPT. 1ST, 2015 Monitoring containers
- 17. The basics • Docker’s/stats API provides the bare minimum • CPU • Memory • Network I/O • Disk I/O • Pitfalls; GET /stats is blocking, up to a second • Use CollectD and docker-collectd-plugin • github.com/signalfx/docker-collectd-plugin
- 18. The basics (cont.) •Data returned by /stats API needs “math” to yield usable numbers • See Docker client code • Better: use SignalFx! • Docker Dashboard • Collect host metrics too if you can (with CollectD) • System/container-level metrics are very limited • We need to go deeper
- 19. Application monitoring • Usuallyneeds more specific visibility • Instrument first, ask questions later • Metrics are cheap, both in CPU and $ • For 3rd-party apps, use CollectD and its plugins • We’re doing a lot of work to improve/provide CollectD plugins for common FOSS • And provide curated dashboards for each of them
- 20. Application monitoring (cont.) •For 1st-party software, depends on language platform • For VM-based services (Java, Scala, Go, …), get key metrics • Heap • # threads • Time spent in GC and # collections • Use metric libraries and instrument everything • Bring it all together in dashboards and anomaly detectors
- 21. Docker with SignalFx •Objective: time-to-data as short as possible • Easiest route: use our packaged CollectD • Good default set of host metrics • Install/enable plugins as needed; docker-collectd-plugin • SignalFx automatically imports curated dashboards • That’s it!
- 22. More monitoring withSignalFx • Use our client libraries or API to send your monitoring data • Or tee off your existing Graphite, StatsD, … pipeline towards SignalFx • Leverage the power of the real-time, streaming analytics • Correlate system, container and application metrics • Build better dashboards and alerts from higher-level signals
- 23.