As a small startup team of developers, release engineering and quality assurance was inherently problematic. To optimize these processes, Sensu implemented a full automated test infrastructure for staging and end-to-end testing, which later became known as the QA Crucible. To increase observability on the overall ecosystem, Nikki started to research the intersection of monitoring and testing in a CI/CD pipeline, and sought to implement existing tooling to further optimize the workflow. By instrumenting JSON test results within a monitoring solution, she discovered a major upside to how tests are run, visualized, responded to, and remediated. Not only can tests be run continuously, but results can also be routed through an event pipeline for data manipulation, visibility and alerting. This pattern allows operators to treat test failures as incidents, and persists test results as metrics in a time-series database for analysis. This technology stack uses Rspec for automated tests, Sensu as an event pipeline, InfluxDB for metric storage, and Grafana for visual dashboards.

1. Time Series Los Angeles Meetup 10/29/19
Nikki Attea
Software Engineer
Sensu Inc.
Testing and monitoring and
broken things

2. Hi, I’m Nikki.
- 🏐🏖
- 🍷🍺
- 🐶🐶
nikki.dev @nikkixdev

3. How does
work?
Sensu is the original monitoring event pipeline.
Events
Sensu Event Pipeline
Filter Transform Handler
nikki.dev @nikkixdev

4. nikki.dev @nikkixdev

5. Building software is easy.
nikki.dev @nikkixdev

6. nikki.dev @nikkixdev

8. Building software is easy.
nikki.dev @nikkixdev

9. Building software is easy.
Shipping software is hard.
nikki.dev @nikkixdev

10. Releases should be simple,
reliable, and reproducible.
nikki.dev @nikkixdev

12. Releases were complex,
chaotic, and inconsistent.
nikki.dev @nikkixdev

13. #1: Eliminate bias by adding
instruction.
#2: Define acceptance criteria.
#3: Automate, but don’t
over-automate.
#4: Implement test case
management.
#5: Hire employees with
diverse experience.
nikki.dev @nikkixdev

14. nikki.dev @nikkixdev

16. TESTING ,
MONITORING,
-NIKKI ATTEA

17. nikki.dev @nikkixdev
Tests fail when an assertion in
our code breaks.

18. Tests fail when an assertion in
our code breaks.
Monitoring detects and alerts
us when our system breaks.
nikki.dev @nikkixdev

19. nikki.dev @nikkixdev
monitoring
testing breakage

20. nikki.dev @nikkixdev
monitoring
testing breakage
systems/infrastructure
code/product

21. nikki.dev @nikkixdev
monitoring
testing breakage
systems/infrastructure
code/product

22. nikki.dev @nikkixdev
monitoring
testing breakage
systems/infrastructure
code/product
state

23. nikki.dev @nikkixdev

24. bonsai.sensu.io/assets/nikkixdev/sensu-serverspec-events
github.com/nikkixdev/sensu-serverspec-events
nikki.dev @nikkixdev

25. nikki.dev @nikkixdev
Stop! Demo time!

26. nikki.dev @nikkixdev

27. nikki.dev @nikkixdev

28. nikki.dev @nikkixdev

29. nikki.dev @nikkixdev

30. nikki.dev @nikkixdev
What’s next?

31. nikki.dev @nikkixdev
What does this mean for your org?

32. nikki.dev @nikkixdev
What does this mean for your
workflow?

33. nikki.dev @nikkixdev
What does this mean for my org?

34. nikki.dev @nikkixdev
What does this mean for CD?

35. nikki.dev @nikkixdev

36. nikki.dev @nikkixdev

37. datageddon

38. nikki.dev @nikkixdev

39. nikki.dev @nikkixdev

40. nikki.dev @nikkixdev

41. nikki.dev @nikkixdev

42. Time-series database
InfluxDB
Record
Data visualization
Grafana
Visualize
Monitoring event pipeline
Sensu
Collect
nikki.dev @nikkixdev

43. Application
Performance
Monitoring
Metric Aggregation Service Checks
Network
& Infrastructure
Monitoring
nikki.dev @nikkixdev

44. kube-apiserver-node-01 cpu=16.4 1550625843
InfluxDB
influxdb_line
CPU ok - CPU = 16.4 | kube-apiserver-node-01.cpu=16.4
Nagios
nagios_perfdata
kube-apiserver-node-01.cpu 16.4 1550625843
Graphite
graphite_plaintext
kube-apiserver-node-01.cpu 1550625843 16.4
OpenTSDB
opentsdb_line
nikki.dev @nikkixdev

45. ---
type: CheckConfig
api_version: core/v2
metadata:
name: check-cpu-influxdb
namespace: default
spec:
command: check-cpu.sh
interval: 10
subscriptions:
- metrics
output_metric_format: influxdb_line
output_metric_handlers:
- influxdb
Sensu Check Configuration and Event Data
---
type: Event
api_version: core/v2
spec:
check:
metadata:
name: check-cpu-influxdb
output: kube-apiserver-node-01 cpu=16.4 1550625843
metrics:
handlers:
- influxdb
points:
- name: kube-apiserver-node-01.cpu
value: 16.4
timestamp: 1550625843
docs.sensu.ioecho $(hostname) cpu=$(ps -A -o %cpu | awk '{s+=$1} END {print s}') $(date +%s)
check-cpu.sh

46. ---
type: Handler
api_version: core/v2
metadata:
namespace: default
name: influxdb
spec:
type: pipe
command: sensu-influxdb-handler -d sensu
env_vars:
- INFLUXDB_ADDR=http://influxdb.default.local:8086
- INFLUXDB_USER=sensu
- INFLUXDB_PASS=password
filters:
- has_metrics
Insert time series-database here, seriously
github.com/sensu/sensu-influxdb-handler
---
type: Event
api_version: core/v2
spec:
check:
metadata:
name: check-cpu-influxdb
output: kube-apiserver-node-01 cpu=16.4 1550625843
metrics:
handlers:
- influxdb
points:
- name: kube-apiserver-node-01.cpu
value: 16.4
timestamp: 1550625843

47. The monitoring event pipeline
The Sensu Go agent is a powerful cross-platform event producer.
Sensu Event Pipeline
Filter HandlerTransform
Sensu Client
Events
Agent
(event
producer)
Nagios Plugin
(service check)
StatsD Metrics
(instrumentation
library)
Prometheus
(/metrics endpoint)

48. Sensu
Checks
Sensu
StatsD
Server

50. Questions?
nikki.dev @nikkixdev