The document discusses the importance of both testing and monitoring in production environments, emphasizing that testing alone is insufficient due to the complexities of systems and the unpredictability of user behavior. It outlines various types of testing and monitoring, the need to adapt to changes, and how monitoring can directly impact business performance. The author advocates for a comprehensive approach that integrates ongoing monitoring with testing to ensure reliability and continuous improvement.

1. @papa_fire
Troubleshooting with monitoring
Testing in production
DevOps monitoring
[something] testing [something]
monitoring [something] in
production
Leon Fayer

2. ❖ @papa_fire
❖ leon@omniti.com
❖ fayerplay.com
❖ slideshare.net/LeonFayer1
THAT’S ME
WHO AM I?
๏ engineer for 20+ years
๏ professional cynic
๏ @ OmniTI
๏ build and operate big systems
๏ we are hiring!
๏ omniti.com/is/hiring

3. @papa_fire
I HATE TESTING

4. @papa_fire
testing is required

5. @papa_fire
testing is not enough

6. @papa_fire
> unit testing
> functional testing
> resilience testing
> performance testing
> …

7. @papa_fire
testing can give a false
sense of security

8. @papa_fire
testing is deterministic

9. @papa_fire
data problem

10. @papa_fire
> quantity of data
> frequency of data
> quality of data

11. @papa_fire
example
Wolfe+585

12. @papa_fire
example
Hubert Blaine Wolfeschlegelsteinhausenbergerdorffwelchevoralternwaren-
gewissenhaftschaferswessenschafewarenwohlgepflegeundsorgfaltigkeitbe
schutzenvorangreifendurchihrraubgierigfeindewelchevoralternzwolfhundert
tausendjahresvorandieerscheinenvonderersteerdemenschderraumschiff
genachtmittungsteinundsiebeniridiumelektrischmotorsgebrauchlichtalsseinur
sprungvonkraftgestartseinlangefahrthinzwischensternartigraumaufdersuchen
nachbarschaftdersternwelchegehabtbewohnbarplanetenkreisedrehensichundwo
hinderneuerassevonverstandigmenschlichkeitkonntefortpflanzenundsicher
freuenanlebenslanglichfreudeundruhemitnichteinfurchtvorangreifenvor
andererintelligentgeschopfsvonhinzwischensternartigraum, Sr.

13. @papa_fire
user problem

14. @papa_fire
“Users (n) - distributed fault injection
test suite for production

15. @papa_fire
example
Corrupted Blood bug

16. @papa_fire
example

17. @papa_fire
other factors

18. @papa_fire
> lack of foresight
(Y2K bug)
> too many use-cases
(female Tauren bug)
> change to assumptions

19. @papa_fire
testing is great for
“known knowns”

20. @papa_fire
testing is ok for
“known unknowns”

21. @papa_fire
testing is bad for
“unknown unknowns”

22. @papa_fire
enter monitoring

23. @papa_fire
why monitor?

24. @papa_fire
because testing isn’t
enough

25. @papa_fire
> software is never perfect
> systems are complex
> external dependency worry
> proactive is better than reactive
> …

26. @papa_fire
because things change

27. @papa_fire
because things change
in production

28. @papa_fire
what to monitor?

29. @papa_fire
in God we trust all others
we monitor
“

30. @papa_fire
> systems
> databases
> applications
> integration points
> performance
> user behavior
> …

31. @papa_fire
is it enough?

32. @papa_fire
is it too much?

33. @papa_fire
what is important?

34. @papa_fire
what is important?
(i.e. what to alert on)

35. @papa_fire
example
> servers up and running
> HTTP checks return 200
> tweets are lost

36. @papa_fire
s/system checks/unit tests/

37. @papa_fire
I don’t give a **** if the
datacenter is on fire as
long as I am still making
money
“ — CEO

38. @papa_fire
we monitor because things
change

39. @papa_fire
changes effect business

40. @papa_fire
top-down approach
> understand business
> define baseline
> correlate data

41. @papa_fire
example
๏ online marketing company
๏ major e-commerce component
๏ ~100 million users
๏ 1 billion emails/month
๏ 300,000 lines of code
๏5600+ metrics collected

42. @papa_fire
it all starts with a call …

43. @papa_fire
revenue

44. @papa_fire
revenue + traffic

45. @papa_fire
revenue + traffic + load time

46. @papa_fire
revenue + traffic + load time + db

47. @papa_fire
revenue + traffic + load time + db + email

48. @papa_fire
… email wasn’t monitored?
what if …

49. @papa_fire
… email wasn’t monitored?
(it would be after this)
what if …

50. @papa_fire
instrumentation
is never done

51. @papa_fire
example
> same symptoms
> higher decline rates
> all metrics are within norm

52. @papa_fire
example
> same symptoms
> higher decline rates
> all metrics are within norm
AmEx blocked

53. @papa_fire
tl;dr

54. @papa_fire
testing and monitoring
not
testing or monitoring

55. @papa_fire
understand the business

56. @papa_fire
continuous improvement

57. @papa_fire
{also bad at conclusions}

58. @papa_fire
THANK YOU
questions?