Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pull, don’t push: Architectures for monitoring and configuration in a microservices era

59 views

Published on

Applications today are increasingly being designed using a share-nothing, microservices architecture that is resilient to the failure of individual components, even when built atop cloud infrastructure that can suffer infrequent-but-massive outages. Yet we still see many supporting tools for application monitoring, observability, configuration management and release management using a centralized “orchestration” approach that depends on pushing changes to unreliable distributed systems.

In this Sensu Summit 2018 talk, Chef's Julian Dunn & Fletcher Nichol give you a primer about promise theory and the autonomous actor model that underlies the design of products like Sensu and Habitat, why it leads to not only higher overall system reliability but human comprehension for easier operations. They argue that you should consider designing all of your applications and supporting systems in this way. They may even show a demo or two to illustrate how inverting the design radically changes the notion of “application release orchestration”, so that you can retain orchestration-type semantics even with an eventually-consistent system design.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Pull, don’t push: Architectures for monitoring and configuration in a microservices era

  1. 1. Pull, don’t push! Architectures for monitoring and configuration in a microservices era Julian Dunn, Director of Product Marketing, Chef @julian_dunn Fletcher Nichol, Senior Software Development Engineer, Chef @fnichol
  2. 2. • Modular, self-contained, pre-fabricated components • Neighbors share components • Complex shares services as a whole
  3. 3. Orchestration
  4. 4. An ordered set of operations Across a set of independent machines Connected to an orchestrator only via a network.
  5. 5. Humans acting on Microsoft Visio acting on machines Humans acting on code acting on machines
  6. 6. An ordered set of operations Defined in code Across a set of independent machines Connected to an orchestrator only via a network.
  7. 7. mylaptop:~$ ./disable-load-balancer.sh mylaptop:~$ ssh db01 do-database-migration.sh mylaptop:~$ for i in app01 app02; do > ssh $i do-deployment.sh > done mylaptop:~$ ./enable-load-balancer.sh
  8. 8. Problems with Orchestration Resilience Scalability Deployment Technical Operational Cognitive
  9. 9. Deployment Resilience for i in app01 app02 app03; do do-deploy.sh –server $i done
  10. 10. Deployment Resilience for i in app01 app02 app03; do do-deploy.sh –server $i if $? != 0; then failed=$i break end done # what goes down here? # roll back $failed? # roll back all others? # ignore it?
  11. 11. Operational Resilience
  12. 12. Operational Resilience Orchestration Backplane – must be up at all times! Application Plane – delegated resilience to the backplane
  13. 13. Operational Resilience Orchestration Backplane Application Plane Orchestration Backplane
  14. 14. Cognitive Scalability
  15. 15. Cognitive Scalability
  16. 16. Technical Scalability
  17. 17. Mainframes Time Sharing Client/Server Web 1.0 Web 2.0 Cloud Internet of Things Edge Time Distributed Centralized The Future Is Distributed
  18. 18. Distributed Devices Need Distributed Management • Adaptive Learning • Configuration Updates • Software Updates
  19. 19. Distributed, Autonomous Systems Make progress towards promised desired state Expose interfaces to allow others to verify promises Can promise to take certain behaviors in the face of failure of others
  20. 20. The Design of Sensu and The Design of Habitat
  21. 21. The Design of Sensu vs. Traditional “Monitoring” Nagios master Agent 1 Agent 2 1. Poll (orchestrate) 2. Run checks 1. Run checks Agent 1 Agent 2 Sensu Backend 2. Post data
  22. 22. Habitat supervisor in a nutshell •Network-connected supervision system •Like systemd+consul/etcd (process supervision with lifecycle hooks + shared state for reactive realtime change management) •Eventually-consistent global state using SWIM masterless (peer-to-peer) membership protocol
  23. 23. sensu- backend hab-sup sensu- backend hab-sup sensu- backend hab-sup backend.default sensu- agent hab-sup agent.default --bind sensu:backend.default Resolve symbol “sensu” in configs to properties of service group backend.default
  24. 24. Let’s See it in Action! Demo: Sensu running under Habitat
  25. 25. • Modern architectures demand a choreographed rather than an orchestrated approach • At scale, fleet management and cognitive complexity is the biggest problem • Habitat and Sensu are both examples of edge-centric, autonomous actor systems, and they work well together 😺

×