Public and private cloud infrastructures promise to make fully dynamic infrastructure a reality – compute instances can be provisioned and terminated at a moments notice, all in response to customer demand. Though “auto-scaling” was once held as the pinnacle of infrastructure automation, it is now considered table stakes. And while this has relieved certain operational burdens (developers can now have access to “on-demand” compute!), it has also created new challenges.

1. Monitoring
Challenges in a
World of
Automation
Monitoring is hard enough on its own.
Automation makes it harder.

2. Anthony Goddard
VP Operations
Sensu, Inc.
@anthonygoddard // @sensu

3. ● Open core monitoring framework, released in 2011
● Enterprise offering launched in 2015
● Sensu Inc formed in January 2017
● 20 employees & growing!
About Sensu

4. What is Sensu?
● An open source, cloud native monitoring framework
● The monitoring router
● Infrastructure, service, and application monitoring
● Designed for automation
● Cross platform (linux, Windows, BSD, AIX, Solaris, MacOS, etc)
● Learn more: https://sensuapp.org

5. Mission Statement
Obviate the need to (re)build custom monitoring solutions.

6. This isn't a talk about
Sensu.

7. Purpose of this talk
● Discuss challenges of monitoring ephemeral systems
● Review basic cloud native monitoring requirements
○ Automated discovery
○ Automated monitoring
○ Automated decommissioning
● Talk about cloud native monitoring anti-patterns
● Live demo! (what could possibly go wrong?)

8. Let's do this

9. Cloud computing has
changed the world.

10. Which came first? Cloud computing or DevOps?

11. Problem Statement
● Cloud platforms and automation systems cause changes in
infrastructure that increase the complexity of monitoring
● New systems/endpoints must be discovered and monitored
automatically
● Monitoring must now distinguish the subtle differences between
"down" and "decommissioned"

12. Expectations
Our infrastructure is becoming increasingly more automated and ephemeral.
Shouldn't we expect similar capabilities from our monitoring?

13. Cloud Native Monitoring Requirements
Overview
1. Automated discovery
2. Automated monitoring
3. Automated decommissioning

14. 1. Automated
Discovery

15. New systems should be
automatically discovered.
Cloud Native Monitoring Requirements

16. Cloud concepts
● Provisioning events create and replace instances
● Cloud providers automate replication of instances (e.g.
auto-scaling groups, etc)
● APIs allow external systems to invoke provisioning events
Automated Discovery

17. Automated Discovery
Cloud monitoring anti-patterns
● Polling-based discovery (regardless of protocol)
● Discovery that precludes complex network topologies
● Punching holes in firewalls (ingress traffic)

18. Polling is not a reliable discovery solution.

19. Automated Discovery
Cloud-native monitoring requirements
● New systems must be discovered in realtime
● Provide push-based or event-based discovery + discovery APIs

20. 2. Automated
Monitoring

21. New systems should be
monitored automatically.
Cloud Native Monitoring Requirements

22. Automated Monitoring
Cloud concepts
● Almost all infrastructures are distributed systems
● Disparate systems fulfill unique roles (e.g. db, web service)
● Simple architectures = one or more roles per system
● Complex architectures = one role per system

23. Automated Monitoring
Cloud monitoring anti-patterns
● Monitoring configuration mapped to individual systems
● Monitoring via remote access (e.g. SSH, WinRM, NRPE)

24. Nope.

25. Automated Monitoring
Cloud-native monitoring requirements
● Monitoring configuration should be mapped to roles
● Monitoring should begin the moment systems come online

26. Automated monitoring should "just work"

27. 3. Automated
Decommissioning

28. Terminated systems should be
automatically removed
from monitoring.
Cloud Native Monitoring Requirements

29. Automated Decommissioning
Cloud Concepts
● Utility computing incentivizes cost savings
● Decommission systems when not in use, or during reduced load
● Intentional actions look very similar to failure scenarios

30. Automated Decommissioning
Cloud monitoring anti-patterns
● Making assumptions about the lack of monitoring data
● Making assumptions about the loss of network connectivity
● Using a monitoring system as a source of absolute truth

31. Cloud-native monitoring requirements
● Should be invoked by the terminated system (i.e. stop signal)
● May be triggered by the provisioning system (i.e. via APIs)
● Optionally verified via external source(s) of truth (as needed)
● Must be the most reliable function of the monitoring system
Automated Decommissioning

32. When you can no longer trust your monitoring alerts.

33. Demo!
But first, some questions…

34. Public/Private Cloud (IaaS)
Who knows what "the cloud" is?
Who understands basic cloud computing
concepts like ASGs and ELBs?
Who is currently using a IaaS provider like
AWS, GCP, Azure, or OpenStack?
Kubernetes
Who knows what Kubernetes is?
Who has Kubernetes on their roadmap?
Who is currently using Kubernetes?
Audience participation time!

35. (DEMO)

38. QUESTIONS?

39. Conclusion
● Cloud computing introduces challenges that demand
cloud-native monitoring solutions.
● Monitoring solutions must automatically discover new systems.
● Monitoring configuration should be applied automatically.
● Monitoring should comprehend "down" vs "decommissioned".

40. Thank You