According to Google, SRE is what you get when you treat operations as if it’s a software problem. In this video, I briefly explain distributed monitoring concepts. Youtube channel here: https://youtu.be/EgpCw15fIK8

1. SRE Demystified
Distributed Monitoring
ganesh@ganeshniyer.com
ganesh.vigneswara@gmail.com,
http://ganeshniyer.com
Dr Ganesh Neelakanta Iyer

2. SRE
•
2https://image.slidesharecdn.com/devopssreatgooglescale-190121123035/95/devops-sre-at-google-scale-30-638.jpg?cb=1548074257

3. Why Monitor?
3
Analyzing long-term trends
Comparing time/iterations
Alerting
Building dashboards
How big is my database and how fast is it
growing? How quickly is my daily-active user
count growing?
Are queries faster with Acme Bucket of Bytes
2.72 versus Ajax DB 3.14? How much better
is my memcache hit rate with an extra node?
Is my site slower than it was last week?
Something is broken,
and somebody needs to
fix it right now! Or,
something might break
soon, so attention
needed soon.
Dashboards should answer basic questions
about your service, and normally include
some form of the four golden signals
Conducting ad hoc retrospective analysis
Our latency just shot up; what else happened around the same time?
https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/

4. Setting Reasonable Expectations for Monitoring
• Simpler and faster monitoring systems, with better tools for post
hoc analysis
• Other uses of monitoring data such as capacity planning and
traffic prediction can tolerate more fragility, and thus, more
complexity
• To keep noise low and signal high, the elements of your
monitoring system that direct to a pager need to be very simple
and robust
• Rules that generate alerts for humans should be simple to
understand and represent a clear failure
4https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/

5. Example symptoms and causes
5
Symptom Cause
I’m serving HTTP 500s or 404s Database servers are refusing connections
My responses are slow
CPUs are overloaded by a bogosort, or an Ethernet
cable is crimped under a rack, visible as partial
packet loss
Users in Antarctica aren’t receiving animated cat
GIFs
Your Content Distribution Network hates scientists
and felines, and thus blacklisted some client IPs
Private content is world-readable
A new software push caused ACLs to be forgotten
and allowed all requests
https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/

6. Black-box monitoring White-box monitoring
• Restricted to external
service behaviour
• e.g. ping, http
6
• Internal metrics and logs to
surface system issues
• e.g. request_errors_total,
request_processing_time,
DevOps School

7. 7https://www.devopsschool.com/blog/white-box-monitoring-and-black-box-monitoring-explained/

8. Four Golden Signals
8

9. Choosing an Appropriate Resolution for
Measurements
• Observing CPU load over the time span of a minute won’t reveal even
quite long-lived spikes that drive high tail latencies
• For a web service targeting no more than 9 hours aggregate downtime
per year (99.9% annual uptime), probing for a 200 (success) status
more than once or twice a minute is probably unnecessarily frequent.
• Similarly, checking hard drive fullness for a service targeting 99.9%
availability more than once every 1–2 minutes is probably unnecessary
9

10. References
10

11. Dr Ganesh Neelakanta Iyer
ganesh@ganeshniyer.com
ganesh.vigneswara@gmail.com