Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Docker and Maestro for fun, development and profit

Presentation on MaestroNG, an orchestration and management tool for multi-host container deployments with Docker.

#lspe meetup, February 20th, 2014 at Yahoo!'s URL café.

Docker and Maestro for fun, development and profit

  1. 1. Docker and Maestro For fun, development and profit
  2. 2. Maxime Petazzoni Software Engineer at SignalFuse, Inc. (also, Jérôme’s cousin) !
  3. 3. Real-time monitoring, instrumentation, observability and analytics Still in “stealth” mode Get updates at
  4. 4. “Docker is awesome!” –You, some time in the last hour (hopefully).
  5. 5. A versatile foundation Service or application containment, security, software delivery, host and environment isolation, …and so much more.
  6. 6. Power at your fingertips Complete control through the remote API Available programmatic clients like docker-py
  7. 7. docker:$ docker -d -H tcp:// ! client:$ cat << EOF | python import docker from pprint import pprint as pp pp(docker.client.Client(‘tcp://docker:4243') .images('')) EOF ! ! [{u’Created': 1391202535, u’Id': u’37de13d273eb9a02cd64…’, u’Repository': u'', u'Size': 155663843, u'Tag': u'0.1.6', u'VirtualSize': 774767942}]
  8. 8. Docker’s Achilles: orchestration Single-host is alright with links, but multi-host just isn’t there.
  9. 9. How do I orchestrate the deployment and control of a full, multi-host, Docker-based environment?
  10. 10. (And more importantly:) How do I make this process one and the same for development, testing and production environments?
  11. 11. Enter: Maestro The totally not scalable, pet project that solved my use case. (and maybe yours)
  12. 12. Maestro is actually MaestroNG, a re-invention of Kimbro Staken’s Maestro (formerly, dockermix)
  13. 13. Takes in a definition of services, their dependencies , configuration and target host… ! …and automates the deployment (and control) of their corresponding containers on these hosts.
  14. 14. Classic use case: a pool of “dumb” workers on your favorite cloud/hosting provider that just run Docker. ! No need to (ma)ssh into anything, no need to pre-configure anything. ! Everything is remote controlled.
  15. 15. Other typical use case: running all the components of your stack in a single, local virtual machine. ! Useful for development, integration testing, etc.
  16. 16. Philosophy: lightweight application/service containers. ! Represent and control your software stack and its dependencies. ! Docker images are the output of your CI process (automation!). ! Start fast, fail faster. Not for heavyweight, complex container “VMs”.
  17. 17. Each service instance (container) defines where it runs and which ports it exposes, among other things. ! Like Docker links, Maestro works by injecting this information in the container’s environment about each container’s service’s dependencies.
  20. 20. Using this information, you can configure your application at container start time. ! If you like Python, Maestro helps you by providing a set of guest helper functions in maestro.guestutils to easily extract and use this data.
  21. 21. #!/usr/bin/env python ! # This is my cool container’s “init script” ! import os from maestro.guestutils import * ! os.execl(‘java’, ‘java’, ‘-jar’, ‘my-app.jar’, ‘-DlistenPort={}’.format(get_port(‘service’)), ‘-DzkServers={}’.format( get_node_list(‘zookeeper’, ports=[‘peer’])))
  22. 22. Dependency order is respected on start; inverse order on stop. ! Can be overridden to stop individual services or containers.
  23. 23. MyApp Start order: 1. ZooKeeper 2. Kafka 3. MyApp Kafka ZK Stop order: 1. MyApp 2. Kafka 3. ZooKeeper Works on subsets of services too.
  24. 24. So how do you wield this power? A bit clunkily, with YAML (and a bit of Jinja2). ! ! ! (sorry)
  25. 25. # Yay, YAML! name: lspe ! registries: # Define custom image registries for # private registries, with credentials. ! ships: # Declare each target host. # (Docker daemon locations) ! services: # Declare each service, their # instances, dependencies and # configuration
  26. 26. registries: # with Maestro robot account registry: email: username: signalfuse+maestro password: {{ env.SUPER_SECRET }} When starting a container, Maestro will automatically login and pull the image from the right place if the image name matches a configured registry.
  27. 27. ships: # Local virtual machine vm: ip: docker_port: 4243 timeout: 10 # Slow VM is slow # A shorter form… vm2: {ip:, timeout: 5} Ships carry containers and are referred to by name in the configuration.
  28. 28. services: # ZooKeeper zookeeper: image: ! # Our zoo isn’t too wild, # only one keeper is enough. zk-node-1: ship: vm ports: client: 2181 peer: 2888/tcp leader_election: “3888/tcp:3888/tcp” # Keep persistent data on the host. volumes: /var/lib/zookeeper: /data/zookeeper # Environment can be passed-in too. env: JVM_FLAGS: “-Xmx1g”
  29. 29. # Kafka kafka: image: requires: [ zookeeper ] env: ZOOKEEPER_BASE: /lspe/kafka RETENTION_HOURS: 48 broker-1: ship: vm ports: {broker: 9092, jmx: “7199:17199”} # Keep persistent data on the host. volumes: /var/lib/kafka: /data/kafka env: BROKER_ID: 0 More flexibility in port mappings, volume bindings, and environment variables definition not shown here.
  30. 30. See for full syntax details and features
  31. 31. Demo time! Be prepared for it to fail, because demos always do.
  32. 32. What’s next? More flexible service status detection (not only port pinging) Soft and hard service dependencies Parallel startup of independent services and instances of a service
  33. 33. That’s it! Thanks for listening! :)
  34. 34. SignalFuse is hiring world class engineers!