1 PM EDT The Solarwinds Monitoring Cloud companies are experts in how to do monitoring at scale in very high throughput environments. During our time running on the cloud, we’ve used many DevOps concepts, including automation, to build a fully-instrumented stack from infrastructure all the way through the application that allows us to have observability into all areas of the stack and from multiple angles.

1. Monitoring First: Instrumenting Your
Entire Stack for the Ultimate in
Observability
Dave Mangot
Head of Site Reliability Engineering,
SolarWinds Cloud Companies
@davemangot

2. © 2016 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
ABOUT ME
•
•
•
•
5/9/16 1

3. About Librato
•
•
•
•
•
2

4. 3
but we wouldn’t be upset if you did :)
You do not need to purchase these
solutions from SolarWinds!
Disclaimer:

5. Data Driven
4
If we have data, let’s look at data. If all we
have are opinions, let’s go with mine.”
- Jim Barksdale

6. MDD
Metrics Driven
Development
5

7. MDD
• Instrumentation-as-Code
6

8. MDD
• Instrumentation-as-Code
• Single Source of Truth
7
By Minesweeper (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons

9. MDD
• Instrumentation-as-Code
• Single Source of Truth
• Developers Curate Visualizations and Alerts
8

10. MDD
• Instrumentation-as-Code
• Single Source of Truth
• Developers Curate Visualizations and Alerts
• Alert on What You See
9

11. MDD
• Instrumentation-as-Code
• Single Source of Truth
• Developers Curate Visualizations and Alerts
• Alert on What You See
• Don’t Measure Everything
10

12. "Just because you can measure
everything doesn't mean that you should."
—W. Edwards Deming
11

13. MDD
• Instrumentation-as-Code
• Single Source of Truth
• Developers Curate Visualizations and Alerts
• Alert on What You See
• Don’t Measure Everything
• Show me the Graph
12

14. ChatOps
13

15. Nagios, etc.
Is a port listening?
14

16. Nagios, etc.
Is a port listening?
Is that what I really care about?
15

17. Nagios, etc.
Is a port listening?
Is that what I really care about?
Are the number of requests over a period of time below a threshold?
What are my average # of requests?
What are the min/max/std dev of latency?
16

18. DevOps??
17

19. 2nd Way of DevOps
The Second Way is about creating the right to left feedback loops. The goal of almost any process
improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually
made.
The outcomes of the Second Way include understanding and responding to all customers, internal and
external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
18http://itrevolution.com/the-three-ways-principles-underpinning-devops/

20. 19
You can’t improve what you don’t
measure

21. Follow the Packet
20

22. Uptime Monitoring
21

23. Uptime Monitoring
22

24. Monitoring Cake
• Librato Agent
• Cloudwatch
• Statsite
• Syslog-ng
• Tracelytics
23
By Przykuta (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

25. Infrastructure Metrics
24

26. Librato Agent/Collectd
25

27. Cloudwatch
26

28. Custom Metrics
27

29. Application Metrics
28

30. Dropwizard Metrics
29

31. Kafka
30

32. Cassandra
31

33. Make the right way, the easy way
32

34. Application Instrumentation
33
# Timing a potentially latent operation with custom instrumentation
statsd.time("api.measures.source_match_build.time") do
metric_source_match_build(active, explicit, wildcards, negates)
end

35. Custom
34

36. Dogfood
35

37. Logging
36

38. Logging
37
•Application Development
•Troubleshooting
•Deployments
•Search Alerts

39. Application Development/Troubleshooting
38

40. Deployment
39

41. Search Alerts
40

42. Papertrail
41

43. Distributed Tracing
42

44. Distributed Tracing
43
Web
App
Database

45. Distributed Tracing
44

46. Distributed Tracing
45

47. © 2016 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Total Observability
5/9/16 46
Distributed
Tracing
Logging
Metrics
Uptime
MDD

48. Try for yourself
47
www.solarwinds.com/cloud-monitoring

49. Thanks!
48
@davemangot