Overview and tips & tricks related to observability for Spring applications, delivered during virtual meetup for Trifork's GOTO Night in November 2020

1. I Can See Clearly Now
Observing and understanding
your Spring Applications at runtime

2. Observability
“To acquire knowledge, one must study;
but to acquire wisdom, one must observe.”
― Marilyn vos Savant

3. Observability
 Term in use since ca. 2013, increasingly popular
 Fancy term for monitoring?

4. Observability
 More focus on gaining insight into actual workings
 Not just known failure modes
 And without the need to deploy
new software
 Debugging the
“unknown unknowns”

5. The Three Pillars Of Observability

6. Logging
“it's log, log!
it's better than bad, it's good!
Everyone wants a log!
You're gonna love it, log!
Come on and get your log!
Everyone needs a log!”
― The Log Song, Ren & Stimpy

7. Logging In Spring Boot
 Logback by default, support for Log4J 2 etc.
 Some features:
 Colored Console logging
 Reusable log config fragments
 Reference Spring properties in logback-spring.xml
 Configure aspects using Spring properties,
e.g. log levels

8. Log Aggregation
 Days of grepping and tailing log files are over
 Too many services, not enough history, unknown
servers, correlate several logs, etc,
 Ship logs to log aggregation server
 Custom appender
 Beware of latency and buffering issues!
 Log shipper outside app

9. Patterns vs Structured Logging
 Logging output often follows pattern
 Problematic
 Lossy, hard to parse, multiline messages

10. Patterns vs Structured Logging
 Structured logs don’t have these problems
 Typically use JSON
 Supported by many aggregators as-is
 Easy to add additional fields
{"@timestamp":"2020-11-16T09:34:25.118+01:00",
"message":"Starting service [Tomcat]",
"logger.name":"org.apache.catalina.core.StandardService",
"logger.thread_name":"main","level":"INFO",
"service":"player-experience","region":"eu-west-1"}

11. logback-spring.xml
<configuration>
<property name="environment" value="${ENVIRONMENT:-local}"/>
<include resource="org/sfw/boot/logging/logback/defaults.xml" />
<include resource="org/sfw/boot/logging/logback/console-appender.xml" />
<include resource="console-json-appender.xml" />
…
<root level="INFO">
<if condition='property("environment").equals("local")'>
<then><appender-ref ref="CONSOLE"/></then>
<else><appender-ref ref="CONSOLE_JSON"/></else>
</if>
</root>
</configuration>

12. console-json-appender.xml
<included>
<springProperty name="service" source="spring.application.name/>
<define name="awsRegion"
class="nlo.gateway.logging.AwsRegionPropertyDefiner"/>
<appender name="CONSOLE_JSON"
class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>{"service":"${service}",
"region":"${awsRegion}"}</customFields>
</encoder>
</appender>
…
</included>

13. Logback PropertyProvider
import ch.qos.logback.core.PropertyDefinerBase;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
public class AwsRegionPropertyDefiner extends PropertyDefinerBase {
@Override
public String getPropertyValue() {
Region currentRegion = Regions.getCurrentRegion();
return currentRegion != null ? currentRegion.getName() : "-";
}
}

14. What To Log?
 Events that show what your code is doing
 More that just errors
 With sufficient contextual information!
 Parameters
 Currently executing request URL, batch job, …
 Logged in user
 Consider what goes in message vs. Mapped
Diagnostic Context (MDC)

15. MDC
 Key-value pairs associated with current thread
 Made available for logging
 Standard feature in most logging frameworks
 Many use cases
 Current user, tenant, request URL, etc.
public class MdcPopulatingFilter extends OncePerRequestFilter {
protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
FilterChain chain) throws ServletException, IOException {
String requestURI = req.getRequestURI();
try (var u = MDC.putCloseable("uri", requestURI)) {
chain.doFilter(request, response);
}
}

16. Log Levels
 Pro-tip: use Spring config for easy per-
environment log levels
 Nice to combine with centralized configuration
 Also: Spring Boot Actuator allows on-the-fly
log level adjustments
 Nice to combine with Spring Boot Admin
logging.level.nl.trifork.client.ws.MessageTracing=debug
application-local.properties

17. Metrics
“What’s measured improves”
― Peter Drucker

18. Metrics
 Pre-aggregated numeric info
 Counters, timers, gauge, distribution-summary, …
 Like mix, max, avg response times for MVC controllers
 Very different from individual log events
 Enriched with tags or dimensions
 Like HTTP method, path, status code, …
 Sent to time-series DB for storing
 For dashboarding and alerting
 Slicing & dicing based on tags

19. Metrics
 Boot 2.x ships with Micrometer integration
 Metrics façade: SLF4J for metrics
 Instruments many Spring components
 Easy to add your own metrics
 Many backends supported
 Adapts automatically
 Push / pull
 Dimensional / hierarchic
 Naming conventions / limitations

20. Metrics Benefits
Compared to logging:
 Scales much better: bigger numbers, not more
 Faster to work with for big periods of time
 Cheaper to store
 Provides lots of insight into both infra and busines
without doing any work

21. Common Tags
@Bean
MeterRegistryCustomizer<MeterRegistry> commonTags(Environment env) {
Region awsRegion = Regions.getCurrentRegion();
return registry -> registry.config().commonTags(
"service", env.getProperty("spring.application.name),
"region", awsRegion != null ? awsRegion.getName() : "-"
);
}
 Tags sent with every metric

22. Tagging Monitoring Requests
@Bean
MeterFilter monitoringRequestTaggingMeterFilter() {
return new MeterFilter() {
@Override
public Meter.Id map(Meter.Id id) {
boolean monitoringRequest = false;
ServletRequestAttributes reqAttrs = (ServletRequestAttributes)
RequestContextHolder.getRequestAttributes();
if (reqAttrs != null) {
monitoringRequest = "true".equalsIgnoreCase(
reqAttrs.getRequest().getHeader(“X-Monitoring"));
}
return id.withTag(
Tag.of("monitoring_request", Boolean.toString(monitoringRequest)));
}
};
}
* does require RequestContextListener bean registered

23. Tracing
“A lost road will remember your footsteps because
someday you may want to return, tracing the way.”
― Munia Khan

24. Traces and Slices
 Trace ID generated once per incoming request,
propagated with downstream calls
 Every hop start a new slice within the trace
 Actual network hop, or logical within a service
 Can be exported to tracing DB
 Zipkin, Jaeger, custom APM solutions, …
 Often based on sampling

25. Distributed Tracing
 Spring Cloud Sleuth allows distributed tracing
 Propagating correlation ID per logical request
 Uses OpenZipkin’s Brave
 Instruments many Spring components

26. Brave vs. AMZN Trace IDs
 AWS LBs can add X-AMZN-Trace-ID header
 Format incompatible with Brave’s Trace ID
 Self=1-67891234-12456789abcdef012345678;Root=1-
67891233-abcdef012345678912345678
 However, easy to forward through Sleuth:
spring.sleuth.baggage.remote-fields=
X-Amzn-Trace-Id,X-Monitoring

27. SLF4J MDC Integration

28. SLF4J MDC Integration

29. Expose Trace IDs in Responses
 Allows clients to reference trace ID when reporting errors
 No more browsing through logs “around 12:05”
public class TraceIdResponseHeaderFilter implements Filter {
@Autowired Tracer tracer;
public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
throws IOException, ServletException {
Span currentSpan = this.tracer.currentSpan();
if (currentSpan != null) {
((HttpServletResponse) res).addHeader(
"X-TraceId", currentSpan.context().traceIdString());
}
chain.doFilter(req, res);
}
…

30. We Want More!
 Can only explain so much in 45 minutes
 Check out Boot Loot presentation from Spring IO ‘19

31. We Want More!
 Check out code snippets on GitHub
https://gist.github.com/jkuipers
 Esp. LoggingClientHttpRequestInterceptor
 Logging outgoing HTTP requests & responses
 Including auto-configuration via RestTemplateBuilder