Observability: Beyond the Three Pillars with Spring

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Jonatan Ivanov
2022-06-29
Observability
Copyright © 2022 VMware, Inc. or its aﬃliates.
Beyond the three pillars with Spring

About Me
- @jonatan_ivanov
- develotters.com
- Seattle Java User Group
- Spring Team @ VMware
- Micrometer
- Spring Cloud ...

Disclaimer
This presentation may contain product features or functionality that are currently under
development.
This over...

Cover w/ Image
Agenda
- What is Observability?
- Why do we need it?
- “The Three Pillars” (with examples)
- Logging
- Metr...

What is Observability?
Why do we need it?

What is Observability?
“In control theory, observability is a measure of how well
internal states of a system can be infer...

What is Observability?
How well we can understand the
internals of a system based on its
outputs
(Providing meaningful inf...

What is Observability?
Being able to ask arbitrary questions
without knowing ahead what you want to ask
Turning data point...

Why do we need Observability?
Today's systems are insanely complex (cloud)
(Death Star Architecture, Big Ball of Mud)

Why do we need Observability?
Complexity (cloud): LAMP stack vs. Cloud Environments
We need to face unknown unknowns
We mi...

Why do we need Observability?
Chaos
Environments can be chaotic
You turn a knob here a little and services are going down ...

Why do we need Observability?
Continuous Improvement
If you want to improve something, you need to be able to measure it ﬁ...

Why do we need Observability?
Opens the door for advanced capabilities
Chaos Engineering
Anomaly Detection
Feature ﬂags
A/...

“The Three Pillars”
(The most popular approach)

Logging - Metrics - Distributed Tracing
Metrics
What is the context?
Measure-and-Combine data
Aggregatable
Can identify tr...

Example: Latency
Metrics
“99.999% of the requests
were faster than 140ms.”
“The max was 150ms.”
So it’s quite bad.
But why...

Example: Error
Metrics
“The error rate is 0.001/sec.”
“We had 2 errors recently.”
So it’s not that bad.
But why did this h...

Logging

Application logs: classic DEBUG/INFO/WARN/ERROR events (+stacktraces)
Payload logs: Raw request and response pairs
GC logs...

SLF4J with Logback comes pre-conﬁgured but you can replace Logback
SLF4J
- Simple Logging Façade for Java
- Simple API for...

Logging with Spring: Payload, Access, GC
Payload logs: Logbook
+ logbook-spring-boot-starter (auto-conﬁgured)
Access logs:...

Metrics

Metrics 101
Time series data: data that changes over time
Trends, context, anomaly detection, visualization, alerting
Vari...

Metrics with Spring: Micrometer
Popular Metrics library on the JVM
Like SLF4J, but for metrics
Simple API
Supports the mos...

Micrometer - Like SLF4J, but for metrics
Graphite
Humio
InﬂuxDB
JMX
KairosDB
New Relic
OpenTSDB
OTLP
Prometheus
SignalFx
S...

Distributed Tracing

Distributed Tracing 101

Distributed Tracing 101 - Correlation
TraceId: 123
123
123

Distributed Tracing 101 - Span and Trace
E
F
C
D
B
A
TraceId: 123

Span (basic unit of work)
SpanId, ParentSpanId, TraceId
Timestamps (start/stop)
Events (annotations) with timestamps
Tags ...

Distributed Tracing with Spring: Spring Cloud Sleuth
Distributed Tracing Support for Spring
Provides an abstraction layer ...

All-In-One: Observation API (Micrometer.next)
Observation observation = Observation.start("test", registry);
try { // TODO...

“Non-conventional”
Observability

“Non-conventional” Observability
Is there anything else beyond Logging + Metrics + Tracing?
We are looking for:
- outputs ...

Spring Boot Actuator
auditevents
beans
caches
conditions
conﬁgprops
env
ﬂyway
health (k8s probes)
heap/thread dump
httptra...

{
"status": "UP",
"components": {
"db": {
"status": "UP",
"details": {
"database": "H2",
"validationQuery": "isValid()"
}
...

{
"status": "UP",
"components": {
[...]
"diskSpace": {
"status": "UP",
"details": {
"total": 1000240963584,
"free": 764043...

{
"status": "UP",
"components": {
[...]
"tealeafService": {
"status": "UP",
"details": {
"components": { ... }
}
},
"water...

"git": {
"branch": "main",
"commit": {
"id": "96c9ebe",
"time": "2022-04-07T19:19:19Z"
}
},
"build": {
"artifact": "tea-se...

"java": {
"vendor": "Eclipse Adoptium",
"version": "18",
"runtime": {
"name": "OpenJDK Runtime Environment",
"version": "1...

"memory": {
"total": 268435456,
"max": 268435456,
"free": 149509024
},
"cpu": {
"availableProcessors": 16
},
"gcs": [
{
"n...

"user": {
"timezone": "UTC",
"country": "US",
"language": "en",
"dir": "~/GitHub/teahouse/tea-service"
},
"os": {
"arch": ...

Info Endpoint
How to contact the dev team, where is the repo of the project?
Cloud
instanceId and type
image version
regio...

Service Registry/Discoverability
How many service instances do we have (by environment)?
What versions are deployed (by en...

Eureka

Tanzu Application Live View

Tanzu Application Live View

API Discoverability
How can I call this service?
Spring REST Docs
Generates docs from tests and hand-written docs
Spring C...

${ "id": "6b55663a", [...] "_links": { "self": { "href": "/tealeaves/6b55663a" }, "search": { "href": "/tealeaves/search/fi...$

${ "_embedded": { "tealeaves": [...] }, "_links": { "first": { "href": "/tealeaves?page=0&size=5" }, "prev": { "href": "/te...$

Questions/Feedback?
Twitter: @jonatan_ivanov
Blog: develotters.com
Try it: github.com/jonatan-ivanov/teahouse
© 2022 Sprin...