Imagine being responsible for monitoring 100 servers. Now imagine 1000. Each server has 100 different things to keep track of. What do you pay attention to and what do you ignore? What is important? In this talk Brian will show how Erlang can be used to capture more information without compromising clarity — i.e. to keep track of the forest without loosing site of the trees!

1. Monitoring
Complex
Systems

2. I do things to/with
computers.

3. I build real-time
systems.

4. I build fault-tolerant
systems.

5. I build critical
systems.

6. AdRoll

7. Less this.

8. More this.

9. Engineering + Mathematics = ads

10. Engineering + Mathematics = ads
(you’re welcome)

11. REAL- T I M E
BIDDING

12. The Problem Domain
• Low latency ( < 100 ms per transaction)

13. The Problem Domain
• Low latency ( < 100 ms per transaction)
• Firm real-time system

14. The Problem Domain
• Low latency ( < 100 ms per transaction)
• Firm real-time system
• Highly concurrent (90 billion transactions
per day)

15. The Problem Domain
• Low latency ( < 100 ms per transaction)
• Firm real-time system
• Highly concurrent (90 billion transactions
per day)
• Global, 24/7 operation

16. I build
Complex Systems

17. Complex Systems
• Non-linear feedback
• Tightly coupled to external systems
• Difficult to model, understand

18. Bad things happen when
Complex Systems fail.

19. Humans are bad at predicting the
performance of complex
systems(…). Our ability to create
large and complex systems fools us
into believing that we’re also
entitled to understand them.
CARLOS BUENO
“MATURE OPTIMIZATION HANDBOOK”

20. Complex Systems often
create worse problems
than those they solve.

21. The key challenge to
sustaining a complex
system is maintaining our
understanding of it.

22. What can be done?

23. Ahead of time verification is
not sufficient.
!
(don’t scrimp on it, though)

24. Compile-time guarantees
are not sufficient.
!
(don’t scrimp on them, either)

25. We need insight into the
running system.

26. What are we looking for?
• VM killers

27. What are we looking for?
• VM killers
• Application performance regressions

28. What are we looking for?
• VM killers
• Application performance regressions
• Abnormal application behavior

29. What are we looking for?
• VM killers
• Application performance regressions
• Abnormal application behavior
• Surprises

30. INSTRUMENTATION

31. The BEAM is
ready to ride.

32. erlang:memory/1

33. erlang:statistics/1

34. erlang:system_info/1

35. What about
our own work?

36. Exometer

37. Important Terms
metric a measurement
entry a receiver and aggregator of metrics
reporter that which samples entries periodically
and ships them to another system
subscription the definition of the regular interval on
which reporters sample entries

38. These are all loosely
coupled at runtime.

39. Configuration is static, but
you can adapt it on the fly.

40. Why exometer over
the alternatives?

41. It is extensively
documented.

42. It’s vigorously
maintained.

43. It’s vigorously
maintained.
(Ulf Wiger fan club day.)

44. It is
silly
fast.

45. Okay, great. We have
instrumentation.

46. Now what?

47. MONITORING

48. This is the
hard part.

49. Visualization

50. Alerting

51. Analysis

52. Visualization tells you
how things look but
not why.

53. A good
day.

54. Uh oh.

55. Growth is good,
but steady is better.

56. Bids
are
stable.

57. Budgets
are
stable.

58. What happened?

59. We
forgot
about
Labor
Day.

61. Alerting tells you that
something happened,
but not why.

62. A
normal
day.

63. Wat?

64. That’s
some
cliff.

65. Timeouts look good.

66. Errors
prior
are
okay.

67. What happened?

68. “Uh, hey guys, you know
Facebook is down, right?”

69. Analysis gives you why
but only if you know
how to ask for what.

70. The
memory
use of a
bidder.

71. ಠ_ಠ

72. It’s all
binaries.

73. Not in processes.

74. Not in ETS.

75. Come on now.

76. What happened?

77. A jiffy bug.

78. A jiffy bug.
(we think)

79. Shout out to Miriam Pena
for spending two weeks
tracking this down.

80. Okay, great. We
have monitoring and
instrumentation.

81. Now all our problems
are solved, right?

82. Not
quite.

83. Instruments
make up for
our lack of
insight.

84. Monitoring
makes up
for our
frailty.

85. Every
solution
brings its
own
problems.

86. Instruments may
be misleading.

87. Instruments may be
overwhelming.

88. Instruments
may be
inaccurate.

89. Instruments may
be ignored.

90. What can
be done?

91. A little
paranoia never hurt
anyone.

92. Use glass displays.

93. Train.

94. Keep sight of
the main goal.

95. Have resources
you’re willing
to sacrifice.

96. AdRoll is Hiring! :D

97. Thanks, folks!
<3
@bltroutwine