This document discusses the relationship between testing and monitoring in software development. It argues that while testing is important and gives confidence, it is not enough on its own since systems are complex and change frequently in production. Monitoring is needed to observe systems in production and catch unknown issues. The key aspects of monitoring discussed are instrumenting all parts of the system to have observability, monitoring metrics that affect the business, and only alerting on issues that require immediate action. Both testing and monitoring are needed, not one or the other.

1. @robtreat2
Production Testing
Through
Monitoring
Robert Treat

2. @robtreat2
monitoring,
it’s like testing,
but in production,
so actually important
Robert Treat

3. @robtreat2
a discussion on trade-offs between
testing and monitoring and how to
apply them in a devops lifecycle
Robert Treat

4. ❖ @robtreat2
❖ robert@omniti.com
❖ xzilla.net
❖ slideshare.net/xzilla
WHO AM I?
๏ Former dev, ops, dba, & more
๏ CEO @ OmniTI
๏ Build & operate systems at
scale
๏ IT Services / Consulting

5. @robtreat2
TESTING

6. @robtreat2
testing is overrated

7. @robtreat2
testing is required

8. @robtreat2
testing is not enough

9. @robtreat2
> unit testing
> functional testing
> resilience testing
> performance testing
> …

10. @robtreat2
testing can give a false
sense of security

11. @robtreat2
testing is deterministic

12. @robtreat2
data problem

13. @robtreat2
> quantity of data
> frequency of data
> quality of data

14. @robtreat2
example
Wolfe+585

15. @robtreat2
example
Hubert Blaine Wolfeschlegelsteinhausenbergerdorffwelchevoralternwarengewissenhaft-
schaferswessenschafewarenwohlgepflegeundsorgfaltigkeitbeschutzenvorangreifen-
durchihrraubgierigfeindewelchevoralternzwolfhunderttausendjahresvorandieer-
scheinenvonderersteerdemenschderraumschiffgenachtmittungsteinundsiebeniridium-
elektrischmotorsgebrauchlichtalsseinursprungvonkraftgestartseinlangefahrthin-
zwischensternartigraumaufdersuchennachbarschaftdersternwelchegehabtbewohnbar-
planetenkreisedrehensichundwohinderneuerassevonverstandigmenschlichkeitkonnte-
fortpflanzenundsicherfreuenanlebenslanglichfreudeundruhemitnichteinfurchtvoran-
greifenvorandererintelligentgeschopfsvonhinzwischensternartigraum, Sr.

16. @robtreat2
user problem

17. @robtreat2
Users (n) - distributed fault injection
test suite for production

18. @robtreat2
example
the
Corrupted Blood
incident

19. @robtreat2
example

20. @robtreat2
other factors

21. @robtreat2
> lack of foresight
> too many use-cases
> change to assumptions

22. @robtreat2
you can never add
enough tests

23. @robtreat2
100% code coverage
is not the goal

24. @robtreat2
the goal of testing is to win
the confidence game

25. @robtreat2
we want to be reasonably
confident that the code we
are going to push will not
break production

26. @robtreat2
reasonably confident

27. @robtreat2
testing is good for
“known knowns”

28. @robtreat2
testing is not so good for
“unknown unknowns”

29. @robtreat2
enter monitoring

30. @robtreat2
why monitor?

31. @robtreat2
> software is never perfect
> systems are complex
> external dependency worry
> proactive is better than reactive
> …

32. @robtreat2
because things change

33. @robtreat2
because things change
a lot in production

34. @robtreat2
what to monitor?

35. @robtreat2
in God we trust all others we
monitor

36. @robtreat2
> systems
> databases
> applications
> integration points
> performance
> user behavior
> …

37. @robtreat2
is it enough?

38. @robtreat2
is it too much?

39. @robtreat2
what is important?

40. @robtreat2
what is important?
(i.e. what to alert on)

41. @robtreat2
example
> servers up and running
> HTTP checks return 200
> tweets are lost

42. @robtreat2
servers working
!=
business working

43. @robtreat2
I don’t give a **** if the
datacenter is on fire as long
as I am still making money
— CEO

44. @robtreat2
most people use monitoring
to focus on the wrong things

45. @robtreat2
my fault

46. @robtreat2
monitoring vendors fault

47. @robtreat2
we need to be smarter about
how we talk about
monitoring

48. @robtreat2
often when people say
monitoring
they actually mean
observability

49. @robtreat2
add observability
to all the things
monitor
what affects business

50. @robtreat2
add observability
to all the things
monitor
what affects business

51. @robtreat2
top-down approach
> understand business
> define baseline
> correlate data

52. @robtreat2
example
๏ online marketing company
๏ major e-commerce component
๏ ~100 million users
๏ 1 billion emails/month
๏ 300,000 lines of code
๏5600+ metrics collected

53. @robtreat2
it all starts with a call …

54. @robtreat2
revenue

55. @robtreat2
revenue
+ traffic

56. @robtreat2
revenue + traffic
+ load time

57. @robtreat2
revenue + traffic + load time
+ db

58. @robtreat2
revenue + traffic + load time + db
+ email bounces

59. @robtreat2
… email wasn’t monitored?
what if …

60. @robtreat2
… email wasn’t monitored?
(it would be after this)
what if …

61. @robtreat2
instrumentation
is never done

62. @robtreat2
example
> same symptoms
> all metrics are within norm
> higher decline rates

63. @robtreat2
example
> same symptoms
> all metrics are within norm
> higher decline rates
AmEx blocked

64. @robtreat2
tl;dr

65. @robtreat2
testing and monitoring
not
testing or monitoring

66. @robtreat2
understand the business

67. @robtreat2
add observability to
everything

68. @robtreat2
monitor things that are
impactful

69. @robtreat2
alert only on actionable
emergencies

70. @robtreat2
THANK YOU
questions?