This document discusses the trade-offs between testing and monitoring in a DevOps lifecycle. It argues that while testing is important and can find many issues, it is not enough on its own as systems are complex with many unknown unknowns. Monitoring is needed to catch issues that occur in production. The key aspects of monitoring discussed are focusing on monitoring metrics that affect business outcomes rather than just technical metrics, taking a top-down approach to understand the business and what to monitor, and increasing overall observability into systems. Both testing and monitoring are important, with their combined use providing better confidence in systems than either approach alone.

1. @robtreat2
Managing Chaos In Production:
Testing vs Monitoring
Robert Treat

2. @robtreat2
monitoring,
it’s like testing,
but in production,
so actually important
Robert Treat

3. @robtreat2
a discussion on trade-offs between testing
and monitoring and how to apply them in a
devops lifecycle
Robert Treat

4. ❖ @robtreat2
❖ robert@omniti.com
❖ xzilla.net
❖ slideshare.net/xzilla
WHO AM I?
๏ Former dev, ops, dba, & more
๏ CEO @ OmniTI
๏ Build & operate systems at scale
๏ IT Services / Consulting

5. @robtreat2
TESTING

6. @robtreat2
testing is overrated

7. @robtreat2
testing is required

8. @robtreat2
testing is not enough

9. @robtreat2
> unit testing
> functional testing
> resilience testing
> performance testing
> …

10. @robtreat2
testing can give a false
sense of security

11. @robtreat2
testing is deterministic

12. @robtreat2
data problem

13. @robtreat2
> quantity of data
> frequency of data
> quality of data

14. @robtreat2
example
Wolfe+585

15. @robtreat2
example
Hubert Blaine Wolfeschlegelsteinhausenbergerdorffwelchevoralternwaren-
gewissenhaftschaferswessenschafewarenwohlgepflegeundsorgfaltigkeitbe-
schutzenvorangreifendurchihrraubgierigfeindewelchevoralternzwolfhundert-
tausendjahresvorandieerscheinenvonderersteerdemenschderraumschiff-
genachtmittungsteinundsiebeniridiumelektrischmotorsgebrauchlichtalsseinur-
sprungvonkraftgestartseinlangefahrthinzwischensternartigraumaufdersuchen-
nachbarschaftdersternwelchegehabtbewohnbarplanetenkreisedrehensichund-
wohinderneuerassevonverstandigmenschlichkeitkonntefortpflanzenundsicher-
freuenanlebenslanglichfreudeundruhemitnichteinfurchtvorangreifenvor-
andererintelligentgeschopfsvonhinzwischensternartigraum, Sr.

16. @robtreat2
user problem

17. @robtreat2
“Users (n) - distributed fault injection
test suite for production

18. @robtreat2
example
the
Corrupted Blood
incident

19. @robtreat2
example

20. @robtreat2
other factors

21. @robtreat2
> lack of foresight
> too many use-cases
> change to assumptions

22. @robtreat2
you can never add
enough tests

23. @robtreat2
100% code coverage
is not the goal

24. @robtreat2
the goal of testing is to win
the confidence game

25. @robtreat2
we want to be reasonably
confident that the code we
are going to push will not
break production

26. @robtreat2
reasonably confident

27. @robtreat2
testing is good for
“known knowns”

28. @robtreat2
testing is not so good for
“unknown unknowns”

29. @robtreat2
enter monitoring

30. @robtreat2
why monitor?

31. @robtreat2
> software is never perfect
> systems are complex
> external dependency worry
> proactive is better than reactive
> …

32. @robtreat2
because things change

33. @robtreat2
because things change
a lot in production

34. @robtreat2
what to monitor?

35. @robtreat2
in God we trust all others
we monitor
“

36. @robtreat2
> systems
> databases
> applications
> integration points
> performance
> user behavior
> …

37. @robtreat2
is it enough?

38. @robtreat2
is it too much?

39. @robtreat2
what is important?

40. @robtreat2
what is important?
(i.e. what to alert on)

41. @robtreat2
example
> servers up and running
> HTTP checks return 200
> tweets are lost

42. @robtreat2
servers working
!=
business working

43. @robtreat2
I don’t give a **** if the
datacenter is on fire as
long as I am still making
money
“ — CEO

44. @robtreat2
most people use
monitoring to focus on the
wrong things

45. @robtreat2
my fault

46. @robtreat2
monitoring vendors fault

47. @robtreat2
we need to be smarter
about how we talk about
monitoring

48. @robtreat2
often when people say
monitoring
they actually mean
visibility

49. @robtreat2
metrics collection for
“all the things”
monitor
what affects business

50. @robtreat2
metrics collections for
“all the things”
monitor
what affects business

51. @robtreat2
top-down approach
> understand business
> define baseline
> correlate data

52. @robtreat2
example
๏ online marketing company
๏ major e-commerce component
๏ ~100 million users
๏ 1 billion emails/month
๏ 300,000 lines of code
๏5600+ metrics collected

53. @robtreat2
it all starts with a call …

54. @robtreat2
revenue

55. @robtreat2
revenue
+ traffic

56. @robtreat2
revenue + traffic
+ load time

57. @robtreat2
revenue + traffic + load time
+ db

58. @robtreat2
revenue + traffic + load time + db
+ email bounces

59. @robtreat2
… email wasn’t monitored?
what if …

60. @robtreat2
… email wasn’t monitored?
(it would be after this)
what if …

61. @robtreat2
instrumentation
is never done

62. @robtreat2
example
> same symptoms
> all metrics are within norm
> higher decline rates

63. @robtreat2
example
> same symptoms
> all metrics are within norm
> higher decline rates
AmEx blocked

64. @robtreat2
tl;dr

65. @robtreat2
testing and monitoring
not
testing or monitoring

66. @robtreat2
understand the business

67. @robtreat2
increase observability
(metrics, tracing, etc..)

68. @robtreat2
monitor things that are
impactful

69. @robtreat2
alert only on actionable
emergencies

70. @robtreat2
THANK YOU
questions?