This document discusses metrics and measurements for monitoring systems. It describes the purpose of measurements as gathering quantifiable data to check system health and identify areas for improvement. It outlines the typical flow of collecting measures from applications and services, storing them, and reporting them to dashboards. It provides examples of common metric types like counts, timers, gauges, and histograms. Finally, it explains how to implement measurements directly in code or using annotations.

1. Measure Metrics
CatalogTribe in Shanghai
debop@coupang.com 2018.01.18

2. Agenda
• Purpose of Measurements
• Measurement flow
• Dashboard examples
• Type of measures
• Count,Time, Meter, Gauge, Histogram
• How to measures
• Use metrics directly
• Use Annotations

3. Purpose of Measurements
• Gathering Objective, Quantifiable data
• System Health Check withTimely measured data
• Back data to select points for System Improvements
• Analysis measured data for Maintenance and Improvement

4. Measurements flow
Dashboards (Grafana)
Store Measures
Application
Storage
• Count
• Timers
• Meters (P95, P99)
• Gauge (JMX,Ratio,Cached)
• Histogram
Reporter

5. Measurements flow
Dashboards (Grafana)
Store Measures
Application
Storage
• Count
• Timers
• Meters (P95, P99)
• Gauge (JMX,Ratio,Cached)
• Histogram
Reporter

6. Measurements flow
Dashboards (Grafana)
Store Measures
Application
Storage
• Count
• Timers
• Meters (P95, P99)
• Gauge (JMX,Ratio,Cached)
• Histogram
Reporter

7. Measurements flow
Dashboards (Grafana)
Report periodical Store Measures
Application
Storage
• Count
• Timers
• Meters (P95, P99)
• Gauge (JMX,Ratio,Cached)
• Histogram
Reporter

8. Measurements flow
Dashboards (Grafana)
Report periodical Store Measures
Application
Storage
• Count
• Timers
• Meters (P95, P99)
• Gauge (JMX,Ratio,Cached)
• Histogram
Reporter

9. Dashboard examples

10. Dashboard

11. Deathwing Redis cache

12. Latency by API

13. Type of measurements
• Count (ex: Call count)
• Time (ex:API Latency)
• Meter (ex:API Latency in P50, P95, P99 …)
• Gauge
• JVM, Ratio, Cache
• Histogram

14. How to measure

15. How to measure - Directly
private final Counter pendingJobs = metrics.counter(name(QueueManager.class, "pending-jobs"));
public void addJob(Job job) {
pendingJobs.inc();
queue.offer(job);
}
private final Timer responses = metrics.timer(name(RequestHandler.class, "responses"));
public String handleRequest(Request request, Response response) {
final Timer.Context context = responses.time();
try {
// etc;
return "OK";
} finally {
context.stop();
}
}
Count
Elapsed Time
(Latency)

16. How to measure - Directly
public class DatabaseHealthCheck extends HealthCheck {
private final Database database;
public DatabaseHealthCheck(Database database) {
this.database = database;
}
@Override
public HealthCheck.Result check() throws Exception {
if (database.isConnected()) {
return HealthCheck.Result.healthy();
} else {
return HealthCheck.Result.unhealthy("Cannot connect to " + database.getUrl());
}
}
}
final HealthCheckRegistry healthChecks = new HealthCheckRegistry();
Health Check

17. How to measure - Annotations
• Use coupang-metrics
• Support for Guice and Spring both
• @Count, @Timed, @Meter, @Gauge …
• Support Async Call by @AsyncCount, @AsyncTimed

18. How to measure - Annotation
open class InstrumentedWithCounterParent {
@Counted(name = "counterParent", absolute = true, monotonic = true)
open fun counterParent(): String = "counterParent"
}

19. How to measure - Annotation
open class InstrumentedWithTimed {
@Timed(name = "things")
open fun doAThing(): String {
Thread.sleep(10L)
return "poop"
}
@Timed
open fun doAThingWithDefaultScope(): String {
Thread.sleep(10L)
return "defaultResult"
}
@Timed(name = "absoluteName", absolute = true)
open fun doAThingWithAbsoluteName(): String = "defaultProtected"
}

20. How to measure - Annotation
open class InstrumentedWithMetered {
@Metered(name = "things")
open fun doAThing(): String {
Thread.sleep(10L)
return "poop"
}
@Metered
open fun doAThingWithDefaultScope(): String = "defaultResult"
@Metered
open protected fun doAThingWithProtectedScope(): String = "defaultProtected"
@Metered(name = "n")
open fun doAThingWithName(): String = "withName"
@Metered(name = "absoluteName", absolute = true)
open fun doAThingWithAbsoluteName(): String = "defaultProtected"
}

21. How to measure - Annotation
@Configuration
class HealthCheckConfiguration {
@Bean
fun customHealthCheck(): CustomHealthCheck = CustomHealthCheck()
@Bean
fun healthCheckRegistry(): HealthCheckRegistry = HealthCheckRegistry().apply {
register("customHealthCheck", customHealthCheck())
}
}
@com.codahale.metrics.health.annotation.Async(period = 50, unit = TimeUnit.MILLISECONDS)
open class CustomHealthCheck() : HealthCheck() {
var shouldFail: Boolean = false
override fun check(): Result = when {
shouldFail -> Result.unhealthy(RuntimeException("Fail whale"))
else -> Result.healthy()
}
}

22. TO DO
• Add measures to exists API and sub apis
• JVM Measures (provided by dropwizard library)
• Add your business measures
• Add HealthCheck to Servers (DB, Cache …)
• Implements Dashboard with Grafana

23. Thank you!