The document discusses shifting monitoring left from production to earlier stages. It describes monitoring a monolithic application and the challenges of microservices. Implementing monitoring in development and testing environments provides early feedback on metrics, logs, tests and detects issues like data inconsistencies and deployment errors before production. Benefits include enhanced observability and fixing problems earlier. Real examples detected include crashes, security leaks and infrastructure issues.

1. Shift monitoring left

2. About me
 Karol Szewczak
 @bvccaneer
 https://szekar1.wordpress.com/
 Test Architect @Lufthansa Systems Poland
 3rd A&A Days
 1st time conference speaker
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
2

3. Agenda
 About me
 Setting the scene
 Monolith
 Microservices
 Monitoring
 Shift left
 Brace Yourself, Change is Coming
 Benefits
 Findings
 Q&A
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
3

4. Why to monitor at all?
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
4

5. Our monolith
 15+ years old
 35,000+ flights per day
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
5

6. Working with monolith
15+
dev
test
ops
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
6

7. Our monolith
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
7

8. Microservices
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
8

9. Production-Ready Microservices
“Something I discovered in a terrifying way,
is with microservices architecture, is that
the state of the system is never the same
from one second to another.
If you’re not aware of the state of the system,
you won’t know when the system fails,
and it will fail”
Susan J. Fowler
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
9

10. And why it will fail?
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
10

11. Monitoring
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
11

12. But where is the shift?
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
12

13. Environments “pipeline”
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
13

14. Monitoring the monolith
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
14

15. Monitoring shifted left
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
15

16. Monitoring as part of test strategy
“Testing is great for known knowns
and ok for known unknowns”
– Leon Fayer
but
we still have unknown unknowns …
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
16

17. Implementation
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
17

18. Implementation
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
18
pushgateway
prometheus
targets
prometheus
alertmanager
grafana
short-lived jobs

19. Implementation
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
19

20. Implementation
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
20

21. Implementation
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
21

22. Benefits of early monitoring
 early indication about ‘new’ problems
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
22

23. Benefits of early monitoring
 enhancing observability of the application
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
23
“a failed check
is invitation
to explore”

24. Benefits of early monitoring
 enhancing observability of the application
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
24
“a failed check
is invitation
to explore”
Alarm raised by monitoring

25. Benefits of early monitoring
 feedback about:
 usefulness of implemented metrics
 readability of prepared dashboards
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
25

26. Benefits of early monitoring
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
26
 feedback about:
 levels of logging and it’s details

27. Benefits of early monitoring
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
27
 feedback about:
 test coverage on earlier stages

28. Benefits of early monitoring
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
28

29. Real life examples
 data inconsistency between services
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
29

30. Real life examples
 split brain scenarios
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
30

31. Real life examples
 deployment correctness
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
31

32. Real life examples
 program crashes, memleaks
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
32

33. Real life examples
 security data leakage
 database connection management
 aftermath of:
 ‘no-space-left’
 host reboot
 infrastructure maintenance
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
33

34. October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
34

35. Thank you!
October 15-16,
2018
Agile & Automation Days - Karol Szewczak - @bvccaneer
35