The rules have changed. The shift from static to dynamic infrastructure requires a change in approach to monitoring, from host-based to functional role-based. Connectivity moves from remote polling to publish-subscribe and push APIs. The control plane moves from point-and-click interfaces to infrastructure-as-code workflows and self-service, developer-friendly APIs. Now — more than ever before — organizations are faced with a deluge of data in various formats. As operators and as natural system integrators, our response to these challenges is likely to roll our own solution, construct a scalable data collection and processing pipeline and combine a number of best-of-breed tools. This is an idea and journey that I am all too familiar with. In 2010, as an operator, working for an early Cloud adopter, I was faced with these challenges, and fought for better visibility. I hacked for the greater good. In this talk, I will give an overview of what I consider to be attributes of an effective monitoring pipeline. I will recount my experience in creating Sensu, the open source monitoring pipeline, and the pitfalls the project has encountered. I will then live demo Sensu monitoring pipelines in action and make my case for not rolling your own solution from scratch.

1. 1
Why you need one, and why you should
stop rolling your own.
By Sean Porter (@PorterTech)
Monitoring Event
Pipelines

2. ● Sean Porter
● Creator of Sensu
● Co-founder & CTO
● @PorterTech
Who am I?
2

3. Overview
3
● Our shared reality
● My experience in building Sensu
● What is a monitoring pipeline?
● Attributes of an effective pipeline
● Demo time
● The future of pipelines

4. 4
OUR SHARED
REALITY

5. 5
COMPLEXITY
TIME

6. 6
#OFTHINGS
TIME
Containers
Servers
VMs
Functions

7. Ephemeral compute!
● Host-based -> role-based monitoring
● Polling -> publish-subscribe & push APIs
● Point-and-click -> Infrastructure as Code
The paradigm shift
7

8. “We gotta find a way to make [this] fit into
the hole for (this), using nothing but {that}.”
8

9. 9
AMOUNTOFDATA
TIME

10. 10

11. 11
THE TOOLS YOU KNOW

12. 12
INTEGRATE
NEED TO BUILD
SCALE

13. 13

14. 14
NEED TO MAINTAIN

15. 15
HOLD THAT THOUGHT

16. 16
BUILDING
SENSU

17. Sensu origin story
17
I joined Sonian in 2010 as an “Automation Engineer”
● Early adopters
● High rate of change
○ Growing team
○ Evolving software stack

18. 18

19. Sensu origin story
19
I started a project with some goals (July 2011)
● Handle ephemeral compute
● Leverage existing and familiar technologies
● Easy to drive with config management
● Easy to scale horizontally
● APIs!!!

20. Sensu origin story
20
● Agent based system with auto-discovery
● Message bus for communication
● Simple key-value data store for state
● Central service check scheduler (pub-sub)
● JSON configuration
● REST APIs

21. 21

22. 22

23. Sensu origin story
23
● Designed for the cloud
○ Proved to handle ephemeral compute
○ Operated securely on public net (AWS)
● Focused on composability & extensibility
○ Reusable components / building blocks

24. Sensu origin story
24

25. Sensu origin story
25
● Named it Sensu
● Sonian sponsored development!
○ Deployed to production after 2 months
○ Replaced a number of tools!
● Open sourced (MIT) on November 1st, 2011

26. 26

27. 27
FRAMEWORK
THE MONITORING ROUTER
BUS

28. 28
THE MONITORING PIPELINE

29. 29
THE MONITORING PIPELINE
OBSERVABILITY

30. 30
WHAT IS A
MONITORING PIPELINE?

31. Unified data collection and processing for all
types of monitoring events:
● Service checks
● Metrics
● Traces
● Logs
● Inventory
What is a monitoring pipeline?
31

32. There are two critical layers:
● Data plane
● Control plane
What is a monitoring pipeline?
32

33. ● Data input
● Data transportation & routing
● Load balancing & failover
● The layer developers interact with (APIs)
The data plane
33

34. ● Central management unit
○ Orchestrator
○ Configuration
○ Security (auth)
● APIs, agents, data processors, etc.
● The layer operators interact with
The control plane
34

35. 35
https://www.youtube.com/watch?v=CM2Y6B1yuDg

36. 36

37. 37
https://bravenewgeek.com/the-observability-pipeline/

38. 38
WAIT… BUT WHY?

39. 39
CHANGE YOUR MIND

40. 40
CHANGE DATASTORES

41. 41
CHANGE FORMATS

42. 42
CHANGE VISUALIZATION

43. 43
CHANGE SAMPLING

44. 44
CHANGE PLATFORMS

45. 45
MAKE CHANGE INEXPENSIVE

46. 46
MAKE IT FUTURE PROOF

47. ● Fewer agents
○ Less “edge” service to support
○ Resource utilization (i.e. fewer sidecars)
● Cost savings*
Wait… But why?
47

48. 48
ATTRIBUTES OF AN
EFFECTIVE PIPELINE

49. ● Unified data format(s)
● Unique IDs
● Capture context at collection time
● Support additional metadata
● Support efficient debugging
Event payload(s)
49

50. Event payload(s)
50

51. {
metadata: {
annotations: { … }
},
entity: {
name: “osmc.de”
},
timestamp: now()
}
Event payload(s)
51

52. {
metadata: {
annotations: { … }
},
check: {
output: “the system is down”
},
timestamp: now()
}
Event payload(s)
52

53. {
metadata: {
annotations: { … }
},
metrics: {
points: [ … ]
},
timestamp: now()
}
Event payload(s)
53

54. ● Lightweight (think sidecars)
● Multi-platform support
● Initiates connections to backends
● Bi-directional communication
Collection Agent
54

55. ● Auto-registration with the backend
● Auto-discovery (context)
○ Platform info
○ System details
○ Roles / responsibilities
Collection Agent
55

56. ● Keepalive / heartbeat mechanism
● Service check execution support
○ Commonly overlooked / underappreciated
○ Leverage over a decade of investment
● Durable outbound data queue
Collection Agent
56

57. ● Several data inputs (APIs)
○ Industry standards*
■ Metrics (StatsD, OpenTelemetry, Graphite)
■ Trace (OpenTracing)
■ Structured log (JSON)
Collection Agent
57

58. ● Standard cryptography (TLS v1.2)
○ mTLS verification & authentication
● Standard protocol (HTTP)
● Agent initiated connections
○ Traverse complex networks
Data Transport
58

59. ● Scale horizontally
● Little coordination with peers
● Concurrency & parallelism
○ “Concurrency is about dealing with lots
of things at once. Parallelism is about
doing lots of things at once.” — Rob Pike
Data Processor
59

60. ● Easy to extend and integrate!
○ Simple APIs and clear specs
● Multi-tenancy to enable self-service
○ Namespaces
○ RBAC
Data Processor
60

61. ● The “secret sauce”
● Granular routing
○ Is it an incident?
○ Is it a resolution?
○ Metric data?
○ Production? Office hours?
Filtering
61

62. Transformation
62

63. Actions
63
● Alert notifications
● Incident management
● Metric & event storage
● Inventory
● Auto-remediation

64. 64
Monitoring Pipeline
Filter
Only incidents
Filter
Only logs
Filter
Only production
Transform
Transform
-
-
Action
Action
Action
Action
Action
Action
Events
Service Checks
Metrics
Trace
Log
Transform
Redact Sensitive
Transform
Nagios => InfluxDB
Filter
Only once every 30 min
Filter
Only metrics
Filter
Only incidents
Transform
Annotate
Transform
-

65. 65
https://bravenewgeek.com/the-observability-pipeline/

66. 66
DEMO TIME

67. 67
THE FUTURE OF
PIPELINES

68. ● Sean Porter
● @PorterTech
● https://sensu.io
Thank you!
68