!1
Spring Boot Actuator 2.0 &
Micrometer
2018-05-26
Toshiaki Maki (@making / tmaki@pivotal.io)
#jjug_ccc #ccc_a1
Who am I ?
2
Toshiaki Maki (@making) https://blog.ik.am
Sr. Solutions Architect @Pivotal Japan
Spring / Cloud Foundry / Concourse / Kubernetes
" Spring Boot 2": https://note.ik.am (not free!)
Trouble in containerized applications / micro services!!
!3
💥 Out of memory error!
💥 Performance issue!
🤔
GC log
Heap Dump
Thread
Dump
Method
Profiling
💥 Out of memory error!
💥 Performance issue!
What you will get from this session
!4
💥 Out of memory error!
💥 Performance issue!
What you will get from this session
!4
💥 Out of memory error!
💥 Performance issue!
What you will get from this session
!4
💥 Out of memory error!
💥 Performance issue!
What you will get from this session
!4
😎
Agenda
!5
• Spring Boot Actuator 2
• Micrometer
• Micrometer
• Histograms and percentiles
• Micrometer API
• Monitoring Spring Boot apps 

on Cloud Foundry and 

Kubernetes
Spring Boot Actuator 2
Spring Boot Actuator
!7
Additional features to help you monitor and manage
your application in production.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Supported Endpoints
!8
• auditevents
• beans
• conditions
• configprops
• env
• flyway
• health
• httptrace
https://demo-micrometer.cfapps.io/actuator
https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html
• info
• loggers
• liquibase
• metrics
• mappings
• scheduledtasks
• sessions
• shutdown
• threaddump
• heapdump
• jolokia
• logfile
• prometheus
/actuator/*
•renamed
•added
Technology Support
!9
Spring Boot Actuator 1.x Spring Boot Actuator 2.x
•Spring MVC •Spring MVC
•Spring WebFlux
•Jersey
How to enable endpoints
!10
Default exposed endpoints are /info and /health.
management.endpoints.web.exposure.include=*
management.endpoints.web.exposure.include=info,health,env
All endpoints are not secured by default.
Secure endpoints (Spring MVC)
!11
http.authorizeRequests()
.mvcMatchers("/actuator/health", "/actuator/info")
.permitAll()
.mvcMatchers("/actuator/**")

.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()...
Secure endpoints (Spring MVC)
!11
http.authorizeRequests()
.mvcMatchers("/actuator/health", "/actuator/info")
.permitAll()
.mvcMatchers("/actuator/**")

.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()...
spring.security.user.name=hoge
spring.security.user.password=foo
spring.security.user.roles=ACTUATOR
Secure endpoints (Spring MVC)
!12
http.authorizeRequests()
.requestMatchers(EndpointRequest.to("health", "info"))
.permitAll()
.requestMatchers(EndpointRequest.toAnyEndpoint())

.hasRole("ACTUATOR")
.anyRequest()
.permitAll()
.and()
.httpBasic()
.and()...
Secure endpoints (Spring WebFlux)
!13
http.authorizeExchange()
.matchers(EndpointRequest.to("health", "info"))
.permitAll()
.matchers(EndpointRequest.toAnyEndpoint())

.hasRole("ACTUATOR")
.anyExchange()
.permitAll()
.and()
.httpBasic()
.and()...
Health Check
!14
management.endpoint.health.show-details=when_authorized
when authorized
Cloud Foundry Support
!15 https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-cloudfoundry.html
https://run.pivotal.io/
Micrometer
Micrometer
!17
Move to Micrometer
Think SLF4J, but for Metrics
Instrument without vendor lock-in:
• Prometheus
• Netflix Atlas
• Datadog
• InfluxDB
• ...
Multi-dementional metrics
https://micrometer.io/
Multi-dimensional VS Hierarchical
!18
{
"counter.200.foo": 100,
"counter.400.foo": 3,
"counter.200.bar": 25,
"counter.200.baz": 50
}
Status URI
Hierarchical (Spring Boot 1.x)
Multi-dimensional VS Hierarchical
!18
{
"counter.200.foo": 100,
"counter.400.foo": 3,
"counter.200.bar": 25,
"counter.200.baz": 50
}
Status URI Method...? 😨
Hierarchical (Spring Boot 1.x)
Multi-dimensional VS Hierarchical
!19
{
"names": "counter",
"tags": [
{"tag": "uri",
"values":
["foo", "bar", "baz"]},
{"tag": "status",
"values": [200, 400]},
{"tag": "method",
"values": ["GET"]}
]
}
Multi-dimensional (Spring Boot 2.0)
Multi-dimensional VS Hierarchical
!20
{
"names": "counter",
"tags": [
{"tag": "uri",
"values":
["foo", "bar", "baz"]},
{"tag": "status",
"values": [200, 400]},
{"tag": "method",
"values": ["GET"]}
]
}
Multi-dimensional (Spring Boot 2.0)
counter?tag=uri:foo&tag=status:200
{
"names": "counter",
"measurements":[
{
"statistic": "COUNT",
"value": 100
}
}
}
Multi-dimensional VS Hierarchical
!21
{
"names": "counter",
"tags": [
{"tag": "uri",
"values":
["foo", "bar", "baz"]},
{"tag": "status",
"values": [200, 400]},
{"tag": "method",
"values": ["GET", "POST"]}
]
}
Multi-dimensional (Spring Boot 2.0)
{
"names": "counter",
"measurements":[
{
"statistic": "COUNT",
"value": 100
},
"availableTags":[
{"tag":"method",
"values":["GET", "POST"]}
]]}
counter?tag=uri:foo&tag=status:200
Supported monitoring systems
!22
Server polls Client pushes
Prometheus
Atlas, Datadog, Datadog StatsD,
Influx, SignalFx, Telegraf StatsD,
Wavefront, New Relic
Dimensional
Graphite, Ganglia, JMX, Etsy StatsD,
PCF Metrics 1.4
Hierarchical
push
poll
Prometheus
!23
monitoring system and time series database.
https://prometheus.io/
Prometheus
!24
Prometheus ServerExporter
pull metrics
scrape_configs:
- job_name: foo
static_configs:
- targets:
- 192.168.1.2:8080
metrics_path: /metrics
Prometheus
!25
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
/actuator/prometheus
!26
system_cpu_count 4.0
system_load_average_1m 1.64
jvm_memory_max_bytes{area="heap",id="PS Eden Space",}
1.05906176E8
jvm_memory_max_bytes{area="heap",id="PS Survivor Space",}
2.1495808E7
jvm_memory_max_bytes{area="heap",id="PS Old Gen",} 3.00941312E8
http_server_requests_seconds_count{exception="None",method="GET"
,status="200",uri="/hello",} 4469.0
http_server_requests_seconds_sum{exception="None",method="GET",s
tatus="200",uri="/hello",} 297.942920787
http_server_requests_seconds_max{exception="None",method="GET",s
tatus="200",uri="/hello",} 0.401581731
https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-metrics.html#production-ready-
metrics-meter
Prometheus
!27
Prometheus ServerExporter
pull metrics
scrape_configs:
- job_name: my-boot-app
static_configs:
- targets:
- localhost:8080
metrics_path: /actuator/prometheus
Supported Metrics
!28
• JVM Metrics
• Memory, Buffer pools
• GC
• Thread
• Classes
• CPU
• File Descriptors
• Logback
• Uptime
• Tomcat
• Spring MVC
• Spring WebFlux
• RestTemplate
• Cache
• DataSource
• RabbitMQ
Prometheus
!29
Grafana
!30
Datadog
!31
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-datadog</artifactId>
</dependency>
management.metrics.export.datadog.api-key=YOUR-API-KEY
Datadog
!32
PCF Metrics
!33
Bind Metrics Forwarder Service
https://github.com/cloudfoundry/java-buildpack-metric-writer Use Metric Writer 2.4.0+ which comes with JBP 4.10+
Also work with Spring Boot 1
!34
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-spring-legacy</artifactId>
<version>${micrometer.version}</version>
</dependency>
Histograms and percentiles
Histogram
!36
management.metrics.distribution.percentiles-
histogram.http.server.requests=true
Histogram in Prometheus
!37
http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",} 15646.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0
The number of requests that took less than 0.447392426 sec.
Query percentiles in Prometheus
!38
histogram_quantile(0.9,
sum(rate(http_server_requests_seconds_bucket{status="200"
}[5m])) by (app, uri, le))
https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile()
SLA (Service Level Agreement)
!39
management.metrics.distribution.sla.http.server.requests=
100ms, 200ms, 400ms
SLA in Prometheus
!40
http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",} 15646.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.1",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.2",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.4",} 15687.0
http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0
Client-side Percentile
!41
management.metrics.distribution.percentiles.http.server.r
equests=0.5, 0.9, 0.95, 0.99, 0.999
http_server_requests_seconds{...,uri="/hello",quantile="0.5",} 0.050855935
http_server_requests_seconds{...,uri="/hello",quantile="0.9",} 0.051380223
http_server_requests_seconds{...,uri="/hello",quantile="0.95",} 0.40265318
http_server_requests_seconds{...,uri="/hello",quantile="0.99",} 0.40265318
http_server_requests_seconds{...,uri="/hello",quantile="0.999",} 0.40265318
Micrometer API
Meter (Timer, Counter, Gauge, ...)
!43
Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte
r, FunctionTimer, and TimeGauge
@Component
public class FooService {
final Counter counter;
public FooService(MeterRegistry registry) {
this.counter = Counter.builder("received.messages")
.register(registry);
}
public void handleMessage() {
this.counter.increment();
}
}
Meter (Timer, Counter, Gauge, ...)
!44
@Component
public class FooService {
final Timer timer;
public FooService(MeterRegistry registry) {
this.timer = Timer.builder("foo").register(registry);
}
public String foo() {
return this.timer.record(() -> {
// ...
});
}
}
Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte
r, FunctionTimer, and TimeGauge
@Timed
!45
@Component
public class FooService {
@Timed("foo")
public String foo() {
// ...
}
}
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(meterRegistry);
} // Not auto-configured in Spring Boot 2.0
https://github.com/micrometer-metrics/micrometer/issues/361
Tag
!46
@Component
public class TweetCounter {
final MeterRegistry registry;
public FooService(MeterRegistry registry) {
this.counter = registry;
}
public void countTweet(String name, String hashTag) {
this.registry.counter("tweet",
"screen_name", name,
"hash_tag", hashTag)
.increment();
}
}
MeterFilter
!47
@Bean
public MeterFilter meterFilter() {
return MeterFilter.deny(id -> {
String uri = id.getTag("uri");
return id != null && uri.startsWith("/actuator");
}
);
}
Exclude metrics against Actuator endpoints
Add MeterBinders
!48
@Bean
public HystrixMetricsBinder hystrixMetricsBinder() {
return new HystrixMetricsBinder();
}
@Bean
public HibernateMetrics hibernateMetrics() {
return new HibernateMetrics(...);
}
Check io.micrometer.core.instrument.binder package
Some binders are auto-configured https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-
metrics.html#production-ready-metrics-meter
Monitoring Spring Boot Apps
on Cloud Foundry and
Kubernetes
On Cloud Foundry
!50
App (instance 0)
App (instance 1)
GoRouter
Load Balanced... 🤔
http://example.com/actuator/prometheus
scrape_configs:
- job_name: example
static_configs:
- targets:
- example.com:80
metrics_path: /actuator/prometheus
prometheus.yml
scrape
On Cloud Foundry
!51
@Bean
public MeterRegistryCustomizer meterRegistryCustomizer(
@Value("${vcap.application.instance_id:}") String id) {
return registry -> registry.config()
.commonTags("cf_app_instance_id", id);
}
On Cloud Foundry
!52
system_cpu_count{cf_app_instance_id="31388366-...",}
} 4.0
system_load_average_1m{cf_app_instance_id="31388366-...",} 1.64
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea
p",id="PS Eden Space",} 1.05906176E8
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea
p",id="PS Survivor Space",} 2.1495808E7
jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea
p",id="PS Old Gen",} 3.00941312E8
On Cloud Foundry
!53
App (instance 0)
App (instance 1)
GoRouter
Prometheus
scrape_configs:
- job_name: example
dns_configs:
- names:
- example.apps.internal
type: A
port: 8080
metrics_path: /actuator/prometheus
http://demo.apps.inetrnal:8080/actuator/prometheus
https://github.com/making/demo-prom-grafana-on-cf
Another solution - Promregater
!54
https://github.com/promregator/promregator
On Kubernetes
!55
Pod (instance 0)
Pod (instance 1)
Prometheus
http://xxx.yyy.zzz:8080/actuator/prometheus
!56
scrape_configs:
- job_name: k8s-pods
kubernetes_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __address__
regex: ([^:]+)(?::d+)?;(d+)
replacement: $1:$2
On Kubernetes
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
On Kubernetes
!57
apiVersion: apps/v1
kind: Deployment
spec:
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: /actuator/prometheus

# ...
Related Links
!58
Thank you for your attention!
• https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-
ready.html
• https://micrometer.io/docs
• https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-
metrics.html
• https://blog.ik.am/entries/448
• https://github.com/making/demo-micrometer
• https://github.com/making/demo-prom-grafana-on-cf (CF env)
• https://github.com/categolj/blog-prometheus/tree/master/local (Local &k8s env)
Related Talks
!59
• More Spring Boot 2? => 13:30-14:15 @Room E+F

Pivotal Reactive
Spring 5 & Spring Boot 2
• More Observability? => 14:30-15:15 @Room I

How to Properly Blame Things for Causing Latency:
An Introduction to Distributed Tracing and Zipkin

Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1

  • 1.
    !1 Spring Boot Actuator2.0 & Micrometer 2018-05-26 Toshiaki Maki (@making / tmaki@pivotal.io) #jjug_ccc #ccc_a1
  • 2.
    Who am I? 2 Toshiaki Maki (@making) https://blog.ik.am Sr. Solutions Architect @Pivotal Japan Spring / Cloud Foundry / Concourse / Kubernetes " Spring Boot 2": https://note.ik.am (not free!)
  • 3.
    Trouble in containerizedapplications / micro services!! !3 💥 Out of memory error! 💥 Performance issue! 🤔 GC log Heap Dump Thread Dump Method Profiling
  • 4.
    💥 Out ofmemory error! 💥 Performance issue! What you will get from this session !4
  • 5.
    💥 Out ofmemory error! 💥 Performance issue! What you will get from this session !4
  • 6.
    💥 Out ofmemory error! 💥 Performance issue! What you will get from this session !4
  • 7.
    💥 Out ofmemory error! 💥 Performance issue! What you will get from this session !4 😎
  • 8.
    Agenda !5 • Spring BootActuator 2 • Micrometer • Micrometer • Histograms and percentiles • Micrometer API • Monitoring Spring Boot apps 
 on Cloud Foundry and 
 Kubernetes
  • 9.
  • 10.
    Spring Boot Actuator !7 Additionalfeatures to help you monitor and manage your application in production. <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency>
  • 11.
    Supported Endpoints !8 • auditevents •beans • conditions • configprops • env • flyway • health • httptrace https://demo-micrometer.cfapps.io/actuator https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html • info • loggers • liquibase • metrics • mappings • scheduledtasks • sessions • shutdown • threaddump • heapdump • jolokia • logfile • prometheus /actuator/* •renamed •added
  • 12.
    Technology Support !9 Spring BootActuator 1.x Spring Boot Actuator 2.x •Spring MVC •Spring MVC •Spring WebFlux •Jersey
  • 13.
    How to enableendpoints !10 Default exposed endpoints are /info and /health. management.endpoints.web.exposure.include=* management.endpoints.web.exposure.include=info,health,env All endpoints are not secured by default.
  • 14.
    Secure endpoints (SpringMVC) !11 http.authorizeRequests() .mvcMatchers("/actuator/health", "/actuator/info") .permitAll() .mvcMatchers("/actuator/**")
 .hasRole("ACTUATOR") .anyRequest() .permitAll() .and() .httpBasic() .and()...
  • 15.
    Secure endpoints (SpringMVC) !11 http.authorizeRequests() .mvcMatchers("/actuator/health", "/actuator/info") .permitAll() .mvcMatchers("/actuator/**")
 .hasRole("ACTUATOR") .anyRequest() .permitAll() .and() .httpBasic() .and()... spring.security.user.name=hoge spring.security.user.password=foo spring.security.user.roles=ACTUATOR
  • 16.
    Secure endpoints (SpringMVC) !12 http.authorizeRequests() .requestMatchers(EndpointRequest.to("health", "info")) .permitAll() .requestMatchers(EndpointRequest.toAnyEndpoint())
 .hasRole("ACTUATOR") .anyRequest() .permitAll() .and() .httpBasic() .and()...
  • 17.
    Secure endpoints (SpringWebFlux) !13 http.authorizeExchange() .matchers(EndpointRequest.to("health", "info")) .permitAll() .matchers(EndpointRequest.toAnyEndpoint())
 .hasRole("ACTUATOR") .anyExchange() .permitAll() .and() .httpBasic() .and()...
  • 18.
  • 19.
    Cloud Foundry Support !15https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-cloudfoundry.html https://run.pivotal.io/
  • 20.
  • 21.
    Micrometer !17 Move to Micrometer ThinkSLF4J, but for Metrics Instrument without vendor lock-in: • Prometheus • Netflix Atlas • Datadog • InfluxDB • ... Multi-dementional metrics https://micrometer.io/
  • 22.
    Multi-dimensional VS Hierarchical !18 { "counter.200.foo":100, "counter.400.foo": 3, "counter.200.bar": 25, "counter.200.baz": 50 } Status URI Hierarchical (Spring Boot 1.x)
  • 23.
    Multi-dimensional VS Hierarchical !18 { "counter.200.foo":100, "counter.400.foo": 3, "counter.200.bar": 25, "counter.200.baz": 50 } Status URI Method...? 😨 Hierarchical (Spring Boot 1.x)
  • 24.
    Multi-dimensional VS Hierarchical !19 { "names":"counter", "tags": [ {"tag": "uri", "values": ["foo", "bar", "baz"]}, {"tag": "status", "values": [200, 400]}, {"tag": "method", "values": ["GET"]} ] } Multi-dimensional (Spring Boot 2.0)
  • 25.
    Multi-dimensional VS Hierarchical !20 { "names":"counter", "tags": [ {"tag": "uri", "values": ["foo", "bar", "baz"]}, {"tag": "status", "values": [200, 400]}, {"tag": "method", "values": ["GET"]} ] } Multi-dimensional (Spring Boot 2.0) counter?tag=uri:foo&tag=status:200 { "names": "counter", "measurements":[ { "statistic": "COUNT", "value": 100 } } }
  • 26.
    Multi-dimensional VS Hierarchical !21 { "names":"counter", "tags": [ {"tag": "uri", "values": ["foo", "bar", "baz"]}, {"tag": "status", "values": [200, 400]}, {"tag": "method", "values": ["GET", "POST"]} ] } Multi-dimensional (Spring Boot 2.0) { "names": "counter", "measurements":[ { "statistic": "COUNT", "value": 100 }, "availableTags":[ {"tag":"method", "values":["GET", "POST"]} ]]} counter?tag=uri:foo&tag=status:200
  • 27.
    Supported monitoring systems !22 Serverpolls Client pushes Prometheus Atlas, Datadog, Datadog StatsD, Influx, SignalFx, Telegraf StatsD, Wavefront, New Relic Dimensional Graphite, Ganglia, JMX, Etsy StatsD, PCF Metrics 1.4 Hierarchical push poll
  • 28.
    Prometheus !23 monitoring system andtime series database. https://prometheus.io/
  • 29.
    Prometheus !24 Prometheus ServerExporter pull metrics scrape_configs: -job_name: foo static_configs: - targets: - 192.168.1.2:8080 metrics_path: /metrics
  • 30.
  • 31.
    /actuator/prometheus !26 system_cpu_count 4.0 system_load_average_1m 1.64 jvm_memory_max_bytes{area="heap",id="PSEden Space",} 1.05906176E8 jvm_memory_max_bytes{area="heap",id="PS Survivor Space",} 2.1495808E7 jvm_memory_max_bytes{area="heap",id="PS Old Gen",} 3.00941312E8 http_server_requests_seconds_count{exception="None",method="GET" ,status="200",uri="/hello",} 4469.0 http_server_requests_seconds_sum{exception="None",method="GET",s tatus="200",uri="/hello",} 297.942920787 http_server_requests_seconds_max{exception="None",method="GET",s tatus="200",uri="/hello",} 0.401581731 https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-metrics.html#production-ready- metrics-meter
  • 32.
    Prometheus !27 Prometheus ServerExporter pull metrics scrape_configs: -job_name: my-boot-app static_configs: - targets: - localhost:8080 metrics_path: /actuator/prometheus
  • 33.
    Supported Metrics !28 • JVMMetrics • Memory, Buffer pools • GC • Thread • Classes • CPU • File Descriptors • Logback • Uptime • Tomcat • Spring MVC • Spring WebFlux • RestTemplate • Cache • DataSource • RabbitMQ
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
    PCF Metrics !33 Bind MetricsForwarder Service https://github.com/cloudfoundry/java-buildpack-metric-writer Use Metric Writer 2.4.0+ which comes with JBP 4.10+
  • 39.
    Also work withSpring Boot 1 !34 <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-spring-legacy</artifactId> <version>${micrometer.version}</version> </dependency>
  • 40.
  • 41.
  • 42.
    Histogram in Prometheus !37 http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",}15646.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0 The number of requests that took less than 0.447392426 sec.
  • 43.
    Query percentiles inPrometheus !38 histogram_quantile(0.9, sum(rate(http_server_requests_seconds_bucket{status="200" }[5m])) by (app, uri, le)) https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile()
  • 44.
    SLA (Service LevelAgreement) !39 management.metrics.distribution.sla.http.server.requests= 100ms, 200ms, 400ms
  • 45.
    SLA in Prometheus !40 http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",}15646.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.1",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.2",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.4",} 15687.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0
  • 46.
    Client-side Percentile !41 management.metrics.distribution.percentiles.http.server.r equests=0.5, 0.9,0.95, 0.99, 0.999 http_server_requests_seconds{...,uri="/hello",quantile="0.5",} 0.050855935 http_server_requests_seconds{...,uri="/hello",quantile="0.9",} 0.051380223 http_server_requests_seconds{...,uri="/hello",quantile="0.95",} 0.40265318 http_server_requests_seconds{...,uri="/hello",quantile="0.99",} 0.40265318 http_server_requests_seconds{...,uri="/hello",quantile="0.999",} 0.40265318
  • 47.
  • 48.
    Meter (Timer, Counter,Gauge, ...) !43 Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte r, FunctionTimer, and TimeGauge @Component public class FooService { final Counter counter; public FooService(MeterRegistry registry) { this.counter = Counter.builder("received.messages") .register(registry); } public void handleMessage() { this.counter.increment(); } }
  • 49.
    Meter (Timer, Counter,Gauge, ...) !44 @Component public class FooService { final Timer timer; public FooService(MeterRegistry registry) { this.timer = Timer.builder("foo").register(registry); } public String foo() { return this.timer.record(() -> { // ... }); } } Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte r, FunctionTimer, and TimeGauge
  • 50.
    @Timed !45 @Component public class FooService{ @Timed("foo") public String foo() { // ... } } @Bean public TimedAspect timedAspect(MeterRegistry registry) { return new TimedAspect(meterRegistry); } // Not auto-configured in Spring Boot 2.0 https://github.com/micrometer-metrics/micrometer/issues/361
  • 51.
    Tag !46 @Component public class TweetCounter{ final MeterRegistry registry; public FooService(MeterRegistry registry) { this.counter = registry; } public void countTweet(String name, String hashTag) { this.registry.counter("tweet", "screen_name", name, "hash_tag", hashTag) .increment(); } }
  • 52.
    MeterFilter !47 @Bean public MeterFilter meterFilter(){ return MeterFilter.deny(id -> { String uri = id.getTag("uri"); return id != null && uri.startsWith("/actuator"); } ); } Exclude metrics against Actuator endpoints
  • 53.
    Add MeterBinders !48 @Bean public HystrixMetricsBinderhystrixMetricsBinder() { return new HystrixMetricsBinder(); } @Bean public HibernateMetrics hibernateMetrics() { return new HibernateMetrics(...); } Check io.micrometer.core.instrument.binder package Some binders are auto-configured https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready- metrics.html#production-ready-metrics-meter
  • 54.
    Monitoring Spring BootApps on Cloud Foundry and Kubernetes
  • 55.
    On Cloud Foundry !50 App(instance 0) App (instance 1) GoRouter Load Balanced... 🤔 http://example.com/actuator/prometheus scrape_configs: - job_name: example static_configs: - targets: - example.com:80 metrics_path: /actuator/prometheus prometheus.yml scrape
  • 56.
    On Cloud Foundry !51 @Bean publicMeterRegistryCustomizer meterRegistryCustomizer( @Value("${vcap.application.instance_id:}") String id) { return registry -> registry.config() .commonTags("cf_app_instance_id", id); }
  • 57.
    On Cloud Foundry !52 system_cpu_count{cf_app_instance_id="31388366-...",} }4.0 system_load_average_1m{cf_app_instance_id="31388366-...",} 1.64 jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea p",id="PS Eden Space",} 1.05906176E8 jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea p",id="PS Survivor Space",} 2.1495808E7 jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea p",id="PS Old Gen",} 3.00941312E8
  • 58.
    On Cloud Foundry !53 App(instance 0) App (instance 1) GoRouter Prometheus scrape_configs: - job_name: example dns_configs: - names: - example.apps.internal type: A port: 8080 metrics_path: /actuator/prometheus http://demo.apps.inetrnal:8080/actuator/prometheus https://github.com/making/demo-prom-grafana-on-cf
  • 59.
    Another solution -Promregater !54 https://github.com/promregator/promregator
  • 60.
    On Kubernetes !55 Pod (instance0) Pod (instance 1) Prometheus http://xxx.yyy.zzz:8080/actuator/prometheus
  • 61.
    !56 scrape_configs: - job_name: k8s-pods kubernetes_configs: -role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __address__ regex: ([^:]+)(?::d+)?;(d+) replacement: $1:$2 On Kubernetes https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
  • 62.
    On Kubernetes !57 apiVersion: apps/v1 kind:Deployment spec: template: metadata: annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" prometheus.io/path: /actuator/prometheus
 # ...
  • 63.
    Related Links !58 Thank youfor your attention! • https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production- ready.html • https://micrometer.io/docs • https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready- metrics.html • https://blog.ik.am/entries/448 • https://github.com/making/demo-micrometer • https://github.com/making/demo-prom-grafana-on-cf (CF env) • https://github.com/categolj/blog-prometheus/tree/master/local (Local &k8s env)
  • 64.
    Related Talks !59 • MoreSpring Boot 2? => 13:30-14:15 @Room E+F
 Pivotal Reactive Spring 5 & Spring Boot 2 • More Observability? => 14:30-15:15 @Room I
 How to Properly Blame Things for Causing Latency: An Introduction to Distributed Tracing and Zipkin