Altitude NY 2018: Observability workshop
•
1 like•521 views
This workshop is a hands-on deep-dive into the major pillars of observability. Peter Bourgon will explore the defining characteristics of metrics, tracing, and logging, and use that knowledge to instrument an example service, including interfacing with the Fastly stats and logging pipelines. Attendees will leave with specific, actionable intelligence on how to make their own systems more observable and easier to operate at scale.
Recommended
Recommended
More Related Content
Similar to Altitude NY 2018: Observability workshop
Similar to Altitude NY 2018: Observability workshop (20)
More from Fastly
More from Fastly (20)
Recently uploaded
Recently uploaded (20)
Altitude NY 2018: Observability workshop
- 1. Fastly presents Observability workshop April 17, 2018 · Peter Bourgon
- 2. NYC presents Context-setting: The 3 Pillars of Observability Instrumentation Methodology Observability Scenarios Observability workshop
- 3. NYC presents Context-setting: The 3 Pillars of Observability Instrumentation Methodology Observability Scenarios Observability workshop
- 4. NYC presents Technique Classification Definition System Classification Context-setting: The 3 Pillars of Observability Observability workshop
- 5. NYC presents Technique Classification Definition System Classification Context-setting: The 3 Pillars of Observability Observability workshop
- 6. NYC presents ☂ Observability is an umbrella term
- 7. NYC presents ☂ Monitoring and Observability, Cindy Sridharan
- 8. NYC presents Observability: a Manifesto, Honeycomb ☂ Manifesto
- 9. NYC presents Monitoring in the time of Cloud-Native, Cindy Sridharan ☂
- 10. NYC presents Monitoring in the time of Cloud-Native, Cindy Sridharan ☂
- 11. NYC presents Monitoring in the time of Cloud-Native, Cindy Sridharan ☂
- 12. NYC presents Monitoring in the time of Cloud-Native, Cindy Sridharan ☂
- 13. “Observability is a system attribute.” Monitoring isn’t Observability, Baron Schwartz, VividCortex
- 14. Corollary: There are different techniques to achieve observability in a system.
- 15. NYC presents Technique Classification Definition System Classification Context-setting: The 3 Pillars of Observability Observability workshop
- 16. NYC presents Technique Classification Definition System Classification Context-setting: The 3 Pillars of Observability Observability workshop
- 19. NYC presents Observability Technique Classification CPU utilization Goroutine count Memory utilization HTTP 503s Last GC pause duration
- 20. NYC presents Observability Technique Classification CPU utilization Goroutine count Memory utilization Last GC pause duration Measurements, or Observations HTTP 503s
- 22. NYC presents Observability Technique Classification Metrics Counter Gauge Histogram Add Set Observe Labels K/V pairs
- 23. NYC presents Observability Technique Classification Logging Structured {"db_err_code": 14002, ...} Unstructured "[WARN] DB error: 14002 ..."
- 24. NYC presents Observability Technique Classification Tracing Trace T1 Span S1 < T1 Span S3 < T1 Span S2 < S1
- 29. NYC presents Observability Technique Classification TracingLogging Metrics Investigation Reaction OpEx CapEx Low Medium High May Contain Opinions
- 30. NYC presents Observability Technique Classification TracingLogging Metrics Investigation Reaction OpEx CapEx Low Medium High Low Medium High May Contain Opinions
- 31. NYC presents Observability Technique Classification TracingLogging Metrics Investigation Reaction OpEx CapEx Low Medium High Low Low Medium Medium High High May Contain Opinions
- 32. NYC presents Observability Technique Classification TracingLogging Metrics Investigation Reaction OpEx CapEx Low Medium High Low Low Low Medium Medium Medium High High High May Contain Opinions
- 33. All parts of a Complete Breakfast
- 34. NYC presents Technique Classification Definition System Classification Context-setting: The 3 Pillars of Observability Observability workshop
- 35. NYC presents Technique Classification Definition System Classification Context-setting: The 3 Pillars of Observability Observability workshop
- 41. NYC presents Context-setting: The 3 Pillars of Observability Instrumentation Methodology Observability Scenarios Observability workshop
- 42. NYC presents Context-setting: The 3 Pillars of Observability Instrumentation Methodology Observability Scenarios Observability workshop
- 46. NYC presents ● https://docs.fastly.com/api/analytics ● Requires API token ● Requires Service ID Observability workshop Fastly’s Real-time Analytics
- 47. NYC presents ● https://docs.fastly.com/api/logging ● Configured entirely in web UI Observability workshop Fastly’s Remote Logging
- 50. NYC presents Observability workshop Service architecture Preprocessor DB Postprocessor API HTTP Request HTTP Response
- 51. NYC presents Logging Observability workshop Logging Preprocessor DB Postprocessor API HTTP Request HTTP Response
- 52. NYC presents ● Middleware or Decorator pattern keeps code clean ● Structured logging whenever possible ● Minimize log events, maximize contextual information ○ More detail is generally better Logging methodology Observability workshop
- 53. NYC presents Metrics Observability workshop Metrics Preprocessor DB Postprocessor API HTTP Request HTTP Response
- 54. NYC presents ● RED or USE methods ● Be cognizant of label cardinality ● Be thoughtful about what you instrument ○ Fewer, more considered metrics are generally better Metrics methodology Observability workshop
- 55. NYC presents Tracing Observability workshop Tracing Preprocessor DB Postprocessor API HTTP Request HTTP Response
- 56. NYC presents ● 100% tracing “coverage” is essential ● Start small, add K/V as necessary ● Be aware of operational costs ○ In practice, traces are always highly sampled Tracing methodology Observability workshop
- 57. NYC presents Observability workshop Fully instrumented Preprocessor DB Postprocessor API HTTP Request HTTP Response
- 58. NYC presents Context-setting: The 3 Pillars of Observability Instrumentation Methodology Observability Scenarios Observability workshop
- 59. NYC presents Context-setting: The 3 Pillars of Observability Instrumentation Methodology Observability Scenarios Observability workshop
- 60. Scenario 1: Elevated rates of 401s The R & E in RED
- 61. Scenario 2: Curious latency spikes The D in RED
- 62. Scenario 3: Lorenzo is angry
- 66. NYC presents Observability Summary Further reading... ● Monitoring and Observability, Cindy Sridharan ● Monitoring isn't Observability, Baron Schwartz ● Metrics, tracing, and logging, Peter Bourgon ● USE and RED methods, Baron Schwartz ● Ch. 6: Monitoring Distributed Systems, Google SRE book ● Instrumentation Best Practices, Prometheus
- 67. Fastly presents Observability workshop April 17, 2018 · Peter Bourgon