1. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Observability in DevOps with Modern Distributed Applications
Dave Mangot
Head of Site Reliability Engineering,
SolarWinds Cloud
@davemangot

2. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
About Me
• 20+ years in systems engineering, have led DevOps and operational maturity
transformations at multiple companies
• Big companies: Cable & Wireless, Salesforce, SolarWinds
• Small Companies: Terracotta, Tagged, Librato
• International DevOps contributor
@davemangot
5/9/16 2

3. © 2017 SolarWinds Worldwide, LLC. All rights reserved.
SolarWinds company overview: Quick facts
• Founded in 1999, over 18 years in the industry
• Over 250,000 customers in 170 countries, including 425 of the Fortune
500®
• Global headquarters in Austin, Texas, with 2,300 employees worldwide
• SolarWinds was taken private by private equity investment firms
Thoma Bravo and Silver Lake Partners in February 2016
3

4. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
4
but we wouldn’t be upset if you did :)
You do not need to purchase these
solutions from SolarWinds!
Disclaimer:

5. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
5

6. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
6

7. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
7

8. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
8

9. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
9

10. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
10

11. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
11

12. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
12

13. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
13

14. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
DevOps
14

15. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
DevOps
15
DevOps is a collaboration between
Development, Operations and other teams
with the recognition that we are tasked with
achieving common business goals.

16. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 3 Ways of DevOps
16http://itrevolution.com/the-three-ways-principles-underpinning-devops/

17. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 3 Ways of DevOps
17

18. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Feedback
18

19. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
2nd Way of DevOps
The Second Way is about creating the right to left feedback loops. The goal of almost any process
improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually
made.
The outcomes of the Second Way include understanding and responding to all customers, internal and
external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
19http://itrevolution.com/the-three-ways-principles-underpinning-devops/

20. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
20

21. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
21
If it’s not monitored, it doesn’t exist

22. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Uptime Monitoring
22

23. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Uptime Monitoring
23

24. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
24

25. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
25

26. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
26

27. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
27

28. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
https://blog.gigaspaces.com/amazon-found-every-100ms-of-latency-cost-them-1-in-sales/
28
Latency matters. Amazon found every
100ms of latency cost them 1% in sales.
Google found an extra .5 seconds in
search page generation time dropped
traffic by 20%.

29. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
29
You can’t improve what you don’t measure

30. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Observability
30
https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html

31. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
31

32. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Logging
(events)
32

33. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Logging
33
tail -f /var/log/messages

34. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
34

35. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
35

36. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Application Development/Troubleshooting
36

37. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Setup
37

38. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Logging
38
• Application Development
• Troubleshooting
• Deployments
• Search Alerts

39. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Shipping
39

40. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Search Alerts
40

41. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Papertrail
41

42. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
42

43. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
43

44. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Metrics
(aggregatable)
44

45. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Infrastructure Metrics
45

46. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Host Metrics
46

47. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
47
root 23706 1 0 Nov18 ? 00:00:00 /bin/sh -c start-stop-daemon --start --quiet --pidfile /var/run/appoptics/appoptics-snapteld.pid —chuid
appoptics --exec /opt/appoptics/sbin/snapteld --oknodo --config /opt/appoptics/etc/config.yaml --pidfile /var/run/appoptics/appoptics-snapteld.pid
Host Metrics

48. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Host Metrics
48

49. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Application Metrics
49

50. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Kafka
50

51. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Custom Metrics
51

52. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Metric Submission
52
api = librato.connect('email', 'token')
q = api.new_queue()
q.add('temperature', 22.1, tags={'location': 'downstairs'})
q.add('temperature', 23.1, tags={'location': 'upstairs'})
q.submit()

53. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Custom Metrics
53

54. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
54
● Latency
● Traffic
● Errors
● Saturation

55. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
55
Latency
The time it takes to service a request.

56. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
56
Traffic
How much demand is being placed
on your system.

57. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
57
Errors
The rate of requests that fail either
explicitly, implicitly, or by policy.

58. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
58
Saturation
How “full” your service is.

59. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Tracing
(request scoped)
59

60. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
60

61. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
61
Web
App
Database

62. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
62
Web
App
Database
X-Trace: 123654
X-Trace: 123654
X-Trace: 123654

63. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 63
WHAT IS DISTRIBUTED TRACING?
HELP!No, help
ME!

64. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
export APPOPTICS_SERVICE_KEY="795fXXXXXXXe5f5b17c7:serviceA"
-javaagent:/usr/local/lib/appoptics-agent.jar
64

65. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
65

66. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
66

67. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
67

68. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
68

69. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
69

70. © 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Thanks!
70
@davemangot