Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Getting Deep on Orchestration: APIs, Actors, and Abstractions in a Distributed System by Docker Captain Jeff Nickoloff


Published on

Orchestration platforms let us work with higher level ideas like services and jobs; but there is more to a platform than scheduling and service discovery. A platform is a collection of actors and APIs that work together and provide those higher level abstractions on a distributed system. In this session we'll go deep on the architecture of open source orchestration platforms, consider scaling pains, reveal extension points, and reflect on an orchestration platform at Amazon. We'll finish with a demo of a homemade abstraction deployed on a live, multi-cloud Swarm cluster.

Published in: Technology

Getting Deep on Orchestration: APIs, Actors, and Abstractions in a Distributed System by Docker Captain Jeff Nickoloff

  1. 1. Getting Deep on Orchestration Jeff Nickoloff Engineer, Author, and Consultant
  2. 2. What is orchestration? Abstractions Examples Agenda Components and patterns APIs System of Record Agents Demo: Entropy About failure Architecture Break stuff
  3. 3. High quality abstractions are force multipliers for communication and reasoning about more complex (or potentially variable) ideas and systems. Using a computer or programming is an act of communication. <tba> (Software that provides such an abstraction gives a user amazing power.) Software like Docker. About Abstractions
  4. 4. Orchestration What is it anyway?
  5. 5. “Orchestrate (v): to organize or plan (something that is complicated).” — Merriam-Webster
  6. 6. “Orchestrating (tv): to arrange or combine so as to achieve a desired or maximum effect.” — Merriam-Webster
  7. 7. (for orchestral music uses see Podium) A system that provides organization and planning abstractions for other abstractions. In the OSS world we have: • Swarm • Kubernetes • Mesosphere Orchestration Platform (Software)
  8. 8. (for orchestral music uses see Podium) A system that provides organization and planning abstractions for other abstractions. Orchestration Platform (Software) In the OSS world we have: • Docker + Swarm + Compose + etcd / consul / zk • Docker + Flannel + Systemd + etcd + kubelet + kube (api, scheduler, controller-manager, proxy, DNS) • Mesos + Marathon + Calico + zk
  9. 9. Force multiplying ideas: a few lower level examples • Engine (runs containers) • Cluster (nodes as an engine) • Network (routing, overlay, etc) • and network name resolution • Volume (named sharable storage) Abstractions
  10. 10. Force multiplying ideas: higher level examples • Service (long running event handler) • Job (process with a linear lifecycle) • Feed (Job with an input document) • Report (Job with an output document) • Request (process a single request) • Cron Job (run a periodic job) Abstractions
  11. 11. Architecture Platform components and patterns
  12. 12. … probably many! Users and other software compositions need some means to interact with the orchestration platform. • One or a collection of APIs • Command Line Program • Web or Native GUI An Interface
  13. 13. Managed state of the platform Accounting for entities (container, pod, service, volume, etc) and their state. Commonly provide: • KV semantics • Record observation (watches) • Update / Delete semantics with fencing tokens • Distributed locks • HA with strong consistency (Paxos / Raft) System of Record
  14. 14. API’s provide interfaces between the platform and users or other tools. Control loops are the platform’s automata. • Passively react to state changes • Manage system state based on active monitoring • Might require leadership election for HA Agents / Control Loops
  15. 15. Event driven agents that coordinate changes in the system • Container (re)scheduler • Cluster node registrar • Service node registrar • Low entropy network registrar • Local supervisor / init • Service controller • Job controller • Report workflow controller • Feed workflow controller • Distributed cron controller Agents / Control Loops
  16. 16. Event driven agents that coordinate changes in the system • State Observation • Feedback for control loops • Entity Lifecycle Graph • Registration and Discovery • Route to an IP • Engine in a cluster • Replica of a service • Endpoint of a service Patterns
  17. 17. Getting dirty <insert illustration of a Swarm cluster> <insert illustration of a Kubernetes cluster> <insert illustration of a Mesosphere cluster> Deep Dive: Cluster
  18. 18. Even more examples <tba> Deep Dive: Service
  19. 19. An Example Demo a new abstraction: Entropy
  20. 20. A powerful and complex idea Build confidence in complex distributed systems by injecting realistic failures and comparing operations against a steady state. • • Failure mode and effects analysis • Netflix / SimianArmy Failure Injection
  21. 21. … and container based platforms All container platforms (clustered or not): • enable high-entropy systems • add new components and failure modes • provide new mechanisms for handling failure __ extension point, failure injection interface (the kernel), cross cutting concern Failure Injection
  22. 22. An orchestration abstraction for failure injection Features: • Probabilistic failure injection policies • Failure modes • Latency, partition, GC pause, etc • Applied to existing containers filtered by label • An event stream • Notifications • Integrates with the Docker API Project: Entropy
  23. 23. An orchestration abstraction for failure injection Components: • Microservice API • CLI • Policy manager (control loop) • Failure injection agents (pluggable) On GitHub: __ insert URL Project: Entropy
  24. 24. Demo Slide $ # Container platform is running. Start a service. $ docker-compose up -d -p election -f election.yml $ # Start the entropy platform $ docker-compose up -d -p entropy -f entropy.yml $ # Define a policy $ entropy -H $DOCKER_HOST run -f 1s -p .1 -t servicename=voter recv_drop
  25. 25. Thank you! Checkout: