Not having the ability to know exactly how your production system behaves can be costly - both in terms of missed opportunities as well as recovery time when disaster strikes. However, the overall complexity of today's systems and their interactions makes it ever more difficult to have visibility into their health and performance. This talk will explore the various facets of monitoring and how combining them can lead from operational blindness to effective observability.

1. The road to monitoring Nirvana
June 2017
Pedro Araújo

2. Who am I?
Studied Computer Engineering
Did web development for a couple of years
Moved to systems administration for a couple of years
Had a run at build and automation engineering
Landed in SRE, <3 it

3. 135 million daily transactions
4.7 billion daily API calls most weeks (55k/s, 100k/s peak)
2.5 terabytes of daily log data output
250,000-per-second time series data points
14k nagios checks

4. What this talk is not going to be about

5. What is monitoring?
Different things to different people

6. Everyone starts with (nagios-style) checks
Also, black-box monitoring

7. (from: https://www.novell.com/coolsolutions/feature/16723.html)

9. Logs give you initial insights
at the cost of brittle and heavy scripting

10. RSS_URI="/rss"
LOG_FILE="/var/log/httpd/access_log"
LOG_DATE_FORMAT="%d/%b/%Y"
DATE="-1 day"
LOG_FDATE=`date -d "$DATE" "+${LOG_DATE_FORMAT}"`
# Unique IPs requesting RSS, except those reporting "subscribers":
IPSUBS=$(
fgrep "$LOG_FDATE" "$LOG_FILE"
| fgrep " $RSS_URI"
| egrep -v '[0-9]+ subscribers'
| cut -d' ' -f 1
| sort
| uniq
| wc -l
)
# Other user-agents reporting "subscribers", for which we'll use the entire
# user-agent string for uniqueness:
OTHERSUBS=$(
fgrep "$LOG_FDATE" "$LOG_FILE"
| fgrep " $RSS_URI"
| fgrep -v 'subscribers; feed-id='
| egrep '[0-9]+ subscribers'
| egrep -o '"[^"]+"$'
| sort -t( -k2 -sr
| awk '!x[$1]++'
| egrep -o '[0-9]+ subscribers'
| awk '{s+=$1} END {print s}'
)
(from: https://gist.github.com/marcoarment/3783146 (abbr.))

11. Client-side analytics
Online
Crash reporting
Synthetic monitoring
RUM
Offline
Detailed logging
Round-Robin Databases

12. (from: https://megalytic.com/blog/tips-for-segmenting-stats-by-geography)

13. (from: https://piwik.org/docs/piwik-tour/#toc-dashboard-widgets)

14. Time series are like check results saved
continuously
But we can do so much more

16. Named value at some time.
Metric identity
name
dimensions
Metric value
Timestamp
os.filesystem.size 1486469296 961130496 mount=/ type=Used
os.filesystem.size 1486469296 8903143424 mount=/ type=Free
os.filesystem.size 1486469296 1143103488 mount=/var type=Used
os.filesystem.size 1486469296 249068044288 mount=/var type=Free
os.filesystem.size 1486469296 0 mount=/dev/shm type=Used
os.filesystem.size 1486469296 50682404864 mount=/dev/shm type=Free
os.filesystem.size-inodes 1486469296 18862 mount=/ type=Used
os.filesystem.size-inodes 1486469296 2602578 mount=/ type=Free
os.filesystem.size-inodes 1486469296 14518 mount=/var type=Used
os.filesystem.size-inodes 1486469296 66504522 mount=/var type=Free
os.filesystem.size-inodes 1486469296 1 mount=/dev/shm type=Used
os.filesystem.size-inodes 1486469296 12373633 mount=/dev/shm type=Free

18. Common datapoint types
Counters

19. Counter example

20. Counter rated example

21. Common datapoint types
Gauges

22. Gauge example

23. Advanced datapoint types
Histogram

25. Example histogram sample
# HELP The number of chunks persisted per series.
# TYPE prometheus_local_storage_series_chunks_persisted histogram
prometheus_local_storage_series_chunks_persisted_bucket{le="1"} 3.205911e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="2"} 3.652375e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="4"} 4.405614e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="8"} 5.66866e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="16"} 8.226382e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="32"} 8.73615e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="64"} 8.770525e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="128"} 8.770525e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="+Inf"} 8.770525e+06
prometheus_local_storage_series_chunks_persisted_sum 5.5495433e+07
prometheus_local_storage_series_chunks_persisted_count 8.770525e+06

26. Histogram example

27. White-box monitoring
Detailed insight via native instrumentation

28. You can roll your own instrumentation.
@contextmanager
def op(what):
start = time.time()
yield
increment('hitcount.total_s',
value=(time.time() - start),
tags=["op:" + what])
while True:
with op('receive'):
req = queue.pop()
with op('compute_route'):
route = compute_route(req)
with op('update'):
db.execute('''
UPDATE hitcount WHERE route = ? SET hits=hits + 1
''', (route, ))
with op('finish'):
req.finish()
(from: https://honeycomb.io/blog/2017/01/instrumentation-measuring-capacity-through-utilization/)

29. Aspects help instrumentation.
@Controller
public class MyController {
@RequestMapping("/")
@TimeMethod(name = "app_duration_seconds", help = "Some helpful info here")
public Object handleMain() {
// Do something
}
}
c = Counter('request_failure_total', 'Description of counter')
h = Histogram('request_latency_seconds', 'Description of histogram')
@c.count_exceptions()
@h.time()
def businessFunction():
# Do something
pass

30. Collectors
or, how to get all that interesting data

31. Event-based monitoring

32. grep'ing logs across hosts is hard
I CAN HAZ LOG AGGREGATION

33. (from: https://dzone.com/articles/getting-started-splunk)

34. (from: http://blog.takipi.com/splunk-vs-elk-the-log-management-tools-decision-making-guide/)

35. Micro-services - perf debugging is hard
Distributed tracing to the rescue

36. (from: http://opentracing.io/documentation/)

37. (from: http://opentracing.io/documentation/)

38. (from: http://jaeger.readthedocs.io/en/latest/#trace-detail-view)

40. Advanced visualisations

41. (from: https://www.circonus.com/2012/09/understanding-data-with-histograms/)

42. (from: http://www.brendangregg.com/frequencytrails.html)

43. Anomaly detection

44. (from: https://eng.uber.com/argos/)

45. Alerting
not covered here

46. Conclusion

48. Thank you
Pedro Araújo
https://keybase.io/phcrva