The document outlines 11 practical tips for developers to enhance observability in real-world applications, focusing on version tracking, alerting, logging, and metrics management. Key recommendations include implementing human-readable build versions, utilizing webhooks for error notifications, and employing structured logs for better searchability. It emphasizes the importance of centralized logging and metrics for long-term scalability and efficiency in application monitoring.

1. REAL-WORLD APPLICATION OBSERVABILITY
11 PRACTICAL DEVELOPER FOCUSED TIPS
Victor Szoltysek

2. SHORT TERM

3. VERSION TRACKING
TIP #1 - MAKE IT EASY TO IDENTIFY CURRENTLY DEPLOYED VERSIONS
▸ Human Readable Build Versions (1.3.101)
▸ Major / Minor (hot
fi
x) / Build Number
▸ JENKINS_BUILD_NUMBER /
GITHUB_BUILD_NUMBER
▸ Date / Time of Build
▸ GIT Hash
▸ GIT Branch
▸ Filename

4. plugins {
id 'org.springframework.boot' version '2.3.1.RELEASE'
id 'io.spring.dependency-management' version '1.0.9.RELEASE'
id "com.gorylenko.gradle-git-properties" version "2.2.2"
id 'java'
}
version = "1.0.${System.env.BUILD_NUMBER ?: System.env.GITHUB_RUN_NUMBER ?: '0-SNAPSHOT'}"
springBoot {
buildInfo()
}
GRADLE CODE

5. LIGHT-WEIGHT ALERTING
TIP #2 - ADD INCOMING WEBHOOKS
▸ Slack / Teams
▸ Developers Need Access
▸ CI Deployments and Failure Noti
fi
cations
▸ Separate “Bot” Channel

6. MEAN TIME TO DETECTION (MTTD)
TIP #3 - IMMEDIATELY ALERT ON SERIOUS ERRORS
▸ Logback.xml
▸ Filter on Errors
▸ Use a Custom Slack / Teams / Email Appender
▸ Fix Errors or change them to Warnings

7. <springProperty scope="context" name="APP_NAME" source="vcap.application.name"
defaultValue="local_app"/>
<springProperty scope="context" name="APP_SPACE" source="vcap.application.space_name"
defaultValue="${HOSTNAME}"/>
<appender name="SLACK" class="com.github.maricn.logback.SlackAppender">
<webhookUri>${SLACK_INCOMING_WEB_HOOK}</webhookUri>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%-4relative [%thread] %-5level %class - %msg%n</pattern>
</layout>
<username>${APP_NAME}@${APP_SPACE}</username>
<iconEmoji>:rage:</iconEmoji>
</appender>
<appender name="ASYNC_SLACK" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="SLACK" />
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>ERROR</level>
</filter>
</appender>
LOGBACK.XML CODE

8. CLIENT SIDE JAVASCRIPT ISSUES
TIP #4 - LOG JAVASCRIPT UNCAUGHT EXCEPTIONS
▸ Especially important for SPA’s (ReactJS)
▸ Send to Backend
▸ Include UserAgent

9. <script>
window.onerror = function(message, url, lineno, colno, error) {
$.post({
url: 'logging/client-error',
contentType: 'text/plain',
data: 'Source:' + url + ':' + lineno + ' Error:' +
message + ' UserAgent:’ + navigator.userAgent
});
}
</script>
JAVASCRIPT CODE

10. 3RD PARTY FAILURES
TIP #5 - NOTIFY RELEVANT TEAMS OF 3RD PARTY DEPENDENCY FAILURES
▸ i.e. Communication Failures, Database Failures,
etc
▸ Filter on Speci
fi
c Error and create a Custom
Appender
▸ Automatically Email the Team Manager

11. MEDIUM TERM

12. SCALING OBSERVABILITY
TIP #6 - ADD CENTRALIZED LOGGING
▸ Prefer SaaS Solutions
▸ Warning on DIY (ELK)
▸ Give Team Access / Ownership
▸ Consider Appenders instead of “Agents”

13. BETTER LOG SEARCHABILITY
TIP #7 - PREFER STRUCTURED LOGS
▸ Indexed Key-Value Pairs
▸ Examples:
▸ env=prod
▸ error_code=COMM_FAILURE
▸ user_id=123
▸ trace_id=234234

14. DISTRIBUTED LOGGING
TIP #8 - LINK RELATED DISTRIBUTED CALLS WITH TRACEIDS
▸ Also know as Corelation-Ids
▸ Spring Cloud Sleuth

15. BETTER LOG SEARCHABILITY
TIP #9 - IMPROVE “SIGNAL-TO-NOISE” RATIO
▸ Logging can get slow and expensive quickly!
▸ Prune unused logging
▸ “Entering method / exiting” method
▸ Focus on Business Value / Exceptions
▸ Consider alternatives to logging for high
volume, aggregatable, and numerical data …

16. LONG TERM

17. SCALING OBSERVABILITY FURTHER
TIP #10 - USE CENTRALIZED METRICS
▸ Use Micrometer (like logback but metrics)
▸ Out of the Box Metrics
▸ Prefer SaaS Solutions (Datadog, Humio, etc)
▸ Warning on DIY (Prometheus)
▸ humio.io has a “free” version
▸ Give Developers Access / Ownership
▸ Understand the difference between Logging and
Metrics
▸ Not Meant for “High Cardinality”

18. Metrics.counter("sample.counter").increment();
//Example event - { "avg": 168.837394, "max": 213.738641, "name": "sample_timer",
//“count”: 5, "sum": 844.186968 } (after 5 calls)
Metrics.summary("purchase","product_name",
getRandomPurchaseName()).record(getRandomPurchaseAmount());
//Example event - { "avg": 40, "max": 61, "name": "purchase", "count": 2,
//“sum”: 80, "product_name": "House" } (after 2 random calls)
METRICS JAVA CODE

19. END