Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1

7,183 views

Published on

Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
@JJUG CCC 2018 Spring / 2018-05-26

Published in: Technology
  • Be the first to comment

Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1

  1. 1. !1 Spring Boot Actuator 2.0 & Micrometer 2018-05-26 Toshiaki Maki (@making / tmaki@pivotal.io) #jjug_ccc #ccc_a1
  2. 2. Who am I ? 2 Toshiaki Maki (@making) https://blog.ik.am Sr. Solutions Architect @Pivotal Japan Spring / Cloud Foundry / Concourse / Kubernetes " Spring Boot 2": https://note.ik.am (not free!)
  3. 3. Trouble in containerized applications / micro services!! !3 💥 Out of memory error! 💥 Performance issue! 🤔 GC log Heap Dump Thread Dump Method Profiling
  4. 4. 💥 Out of memory error! 💥 Performance issue! What you will get from this session !4
  5. 5. 💥 Out of memory error! 💥 Performance issue! What you will get from this session !4
  6. 6. 💥 Out of memory error! 💥 Performance issue! What you will get from this session !4
  7. 7. 💥 Out of memory error! 💥 Performance issue! What you will get from this session !4 😎
  8. 8. Agenda !5 • Spring Boot Actuator 2 • Micrometer • Micrometer • Histograms and percentiles • Micrometer API • Monitoring Spring Boot apps 
 on Cloud Foundry and 
 Kubernetes
  9. 9. Spring Boot Actuator 2
  10. 10. Spring Boot Actuator !7 Additional features to help you monitor and manage your application in production. <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency>
  11. 11. Supported Endpoints !8 • auditevents • beans • conditions • configprops • env • flyway • health • httptrace https://demo-micrometer.cfapps.io/actuator https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html • info • loggers • liquibase • metrics • mappings • scheduledtasks • sessions • shutdown • threaddump • heapdump • jolokia • logfile • prometheus /actuator/* •renamed •added
  12. 12. Technology Support !9 Spring Boot Actuator 1.x Spring Boot Actuator 2.x •Spring MVC •Spring MVC •Spring WebFlux •Jersey
  13. 13. How to enable endpoints !10 Default exposed endpoints are /info and /health. management.endpoints.web.exposure.include=* management.endpoints.web.exposure.include=info,health,env All endpoints are not secured by default.
  14. 14. Secure endpoints (Spring MVC) !11 http.authorizeRequests() .mvcMatchers("/actuator/health", "/actuator/info") .permitAll() .mvcMatchers("/actuator/**")
 .hasRole("ACTUATOR") .anyRequest() .permitAll() .and() .httpBasic() .and()...
  15. 15. Secure endpoints (Spring MVC) !11 http.authorizeRequests() .mvcMatchers("/actuator/health", "/actuator/info") .permitAll() .mvcMatchers("/actuator/**")
 .hasRole("ACTUATOR") .anyRequest() .permitAll() .and() .httpBasic() .and()... spring.security.user.name=hoge spring.security.user.password=foo spring.security.user.roles=ACTUATOR
  16. 16. Secure endpoints (Spring MVC) !12 http.authorizeRequests() .requestMatchers(EndpointRequest.to("health", "info")) .permitAll() .requestMatchers(EndpointRequest.toAnyEndpoint())
 .hasRole("ACTUATOR") .anyRequest() .permitAll() .and() .httpBasic() .and()...
  17. 17. Secure endpoints (Spring WebFlux) !13 http.authorizeExchange() .matchers(EndpointRequest.to("health", "info")) .permitAll() .matchers(EndpointRequest.toAnyEndpoint())
 .hasRole("ACTUATOR") .anyExchange() .permitAll() .and() .httpBasic() .and()...
  18. 18. Health Check !14 management.endpoint.health.show-details=when_authorized when authorized
  19. 19. Cloud Foundry Support !15 https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-cloudfoundry.html https://run.pivotal.io/
  20. 20. Micrometer
  21. 21. Micrometer !17 Move to Micrometer Think SLF4J, but for Metrics Instrument without vendor lock-in: • Prometheus • Netflix Atlas • Datadog • InfluxDB • ... Multi-dementional metrics https://micrometer.io/
  22. 22. Multi-dimensional VS Hierarchical !18 { "counter.200.foo": 100, "counter.400.foo": 3, "counter.200.bar": 25, "counter.200.baz": 50 } Status URI Hierarchical (Spring Boot 1.x)
  23. 23. Multi-dimensional VS Hierarchical !18 { "counter.200.foo": 100, "counter.400.foo": 3, "counter.200.bar": 25, "counter.200.baz": 50 } Status URI Method...? 😨 Hierarchical (Spring Boot 1.x)
  24. 24. Multi-dimensional VS Hierarchical !19 { "names": "counter", "tags": [ {"tag": "uri", "values": ["foo", "bar", "baz"]}, {"tag": "status", "values": [200, 400]}, {"tag": "method", "values": ["GET"]} ] } Multi-dimensional (Spring Boot 2.0)
  25. 25. Multi-dimensional VS Hierarchical !20 { "names": "counter", "tags": [ {"tag": "uri", "values": ["foo", "bar", "baz"]}, {"tag": "status", "values": [200, 400]}, {"tag": "method", "values": ["GET"]} ] } Multi-dimensional (Spring Boot 2.0) counter?tag=uri:foo&tag=status:200 { "names": "counter", "measurements":[ { "statistic": "COUNT", "value": 100 } } }
  26. 26. Multi-dimensional VS Hierarchical !21 { "names": "counter", "tags": [ {"tag": "uri", "values": ["foo", "bar", "baz"]}, {"tag": "status", "values": [200, 400]}, {"tag": "method", "values": ["GET", "POST"]} ] } Multi-dimensional (Spring Boot 2.0) { "names": "counter", "measurements":[ { "statistic": "COUNT", "value": 100 }, "availableTags":[ {"tag":"method", "values":["GET", "POST"]} ]]} counter?tag=uri:foo&tag=status:200
  27. 27. Supported monitoring systems !22 Server polls Client pushes Prometheus Atlas, Datadog, Datadog StatsD, Influx, SignalFx, Telegraf StatsD, Wavefront, New Relic Dimensional Graphite, Ganglia, JMX, Etsy StatsD, PCF Metrics 1.4 Hierarchical push poll
  28. 28. Prometheus !23 monitoring system and time series database. https://prometheus.io/
  29. 29. Prometheus !24 Prometheus ServerExporter pull metrics scrape_configs: - job_name: foo static_configs: - targets: - 192.168.1.2:8080 metrics_path: /metrics
  30. 30. Prometheus !25 <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency>
  31. 31. /actuator/prometheus !26 system_cpu_count 4.0 system_load_average_1m 1.64 jvm_memory_max_bytes{area="heap",id="PS Eden Space",} 1.05906176E8 jvm_memory_max_bytes{area="heap",id="PS Survivor Space",} 2.1495808E7 jvm_memory_max_bytes{area="heap",id="PS Old Gen",} 3.00941312E8 http_server_requests_seconds_count{exception="None",method="GET" ,status="200",uri="/hello",} 4469.0 http_server_requests_seconds_sum{exception="None",method="GET",s tatus="200",uri="/hello",} 297.942920787 http_server_requests_seconds_max{exception="None",method="GET",s tatus="200",uri="/hello",} 0.401581731 https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-metrics.html#production-ready- metrics-meter
  32. 32. Prometheus !27 Prometheus ServerExporter pull metrics scrape_configs: - job_name: my-boot-app static_configs: - targets: - localhost:8080 metrics_path: /actuator/prometheus
  33. 33. Supported Metrics !28 • JVM Metrics • Memory, Buffer pools • GC • Thread • Classes • CPU • File Descriptors • Logback • Uptime • Tomcat • Spring MVC • Spring WebFlux • RestTemplate • Cache • DataSource • RabbitMQ
  34. 34. Prometheus !29
  35. 35. Grafana !30
  36. 36. Datadog !31 <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-datadog</artifactId> </dependency> management.metrics.export.datadog.api-key=YOUR-API-KEY
  37. 37. Datadog !32
  38. 38. PCF Metrics !33 Bind Metrics Forwarder Service https://github.com/cloudfoundry/java-buildpack-metric-writer Use Metric Writer 2.4.0+ which comes with JBP 4.10+
  39. 39. Also work with Spring Boot 1 !34 <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-spring-legacy</artifactId> <version>${micrometer.version}</version> </dependency>
  40. 40. Histograms and percentiles
  41. 41. Histogram !36 management.metrics.distribution.percentiles- histogram.http.server.requests=true
  42. 42. Histogram in Prometheus !37 http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",} 15646.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0 The number of requests that took less than 0.447392426 sec.
  43. 43. Query percentiles in Prometheus !38 histogram_quantile(0.9, sum(rate(http_server_requests_seconds_bucket{status="200" }[5m])) by (app, uri, le)) https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile()
  44. 44. SLA (Service Level Agreement) !39 management.metrics.distribution.sla.http.server.requests= 100ms, 200ms, 400ms
  45. 45. SLA in Prometheus !40 http_server_requests_seconds_bucket{...,uri="/hello",le="0.061516456",} 15646.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.067108864",} 15657.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.089478485",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.1",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.111848106",} 15679.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.134217727",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.156587348",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.178956969",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.2",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.20132659",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.223696211",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.246065832",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.268435456",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.357913941",} 15680.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.4",} 15687.0 http_server_requests_seconds_bucket{...,uri="/hello",le="0.447392426",} 16475.0
  46. 46. Client-side Percentile !41 management.metrics.distribution.percentiles.http.server.r equests=0.5, 0.9, 0.95, 0.99, 0.999 http_server_requests_seconds{...,uri="/hello",quantile="0.5",} 0.050855935 http_server_requests_seconds{...,uri="/hello",quantile="0.9",} 0.051380223 http_server_requests_seconds{...,uri="/hello",quantile="0.95",} 0.40265318 http_server_requests_seconds{...,uri="/hello",quantile="0.99",} 0.40265318 http_server_requests_seconds{...,uri="/hello",quantile="0.999",} 0.40265318
  47. 47. Micrometer API
  48. 48. Meter (Timer, Counter, Gauge, ...) !43 Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte r, FunctionTimer, and TimeGauge @Component public class FooService { final Counter counter; public FooService(MeterRegistry registry) { this.counter = Counter.builder("received.messages") .register(registry); } public void handleMessage() { this.counter.increment(); } }
  49. 49. Meter (Timer, Counter, Gauge, ...) !44 @Component public class FooService { final Timer timer; public FooService(MeterRegistry registry) { this.timer = Timer.builder("foo").register(registry); } public String foo() { return this.timer.record(() -> { // ... }); } } Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounte r, FunctionTimer, and TimeGauge
  50. 50. @Timed !45 @Component public class FooService { @Timed("foo") public String foo() { // ... } } @Bean public TimedAspect timedAspect(MeterRegistry registry) { return new TimedAspect(meterRegistry); } // Not auto-configured in Spring Boot 2.0 https://github.com/micrometer-metrics/micrometer/issues/361
  51. 51. Tag !46 @Component public class TweetCounter { final MeterRegistry registry; public FooService(MeterRegistry registry) { this.counter = registry; } public void countTweet(String name, String hashTag) { this.registry.counter("tweet", "screen_name", name, "hash_tag", hashTag) .increment(); } }
  52. 52. MeterFilter !47 @Bean public MeterFilter meterFilter() { return MeterFilter.deny(id -> { String uri = id.getTag("uri"); return id != null && uri.startsWith("/actuator"); } ); } Exclude metrics against Actuator endpoints
  53. 53. Add MeterBinders !48 @Bean public HystrixMetricsBinder hystrixMetricsBinder() { return new HystrixMetricsBinder(); } @Bean public HibernateMetrics hibernateMetrics() { return new HibernateMetrics(...); } Check io.micrometer.core.instrument.binder package Some binders are auto-configured https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready- metrics.html#production-ready-metrics-meter
  54. 54. Monitoring Spring Boot Apps on Cloud Foundry and Kubernetes
  55. 55. On Cloud Foundry !50 App (instance 0) App (instance 1) GoRouter Load Balanced... 🤔 http://example.com/actuator/prometheus scrape_configs: - job_name: example static_configs: - targets: - example.com:80 metrics_path: /actuator/prometheus prometheus.yml scrape
  56. 56. On Cloud Foundry !51 @Bean public MeterRegistryCustomizer meterRegistryCustomizer( @Value("${vcap.application.instance_id:}") String id) { return registry -> registry.config() .commonTags("cf_app_instance_id", id); }
  57. 57. On Cloud Foundry !52 system_cpu_count{cf_app_instance_id="31388366-...",} } 4.0 system_load_average_1m{cf_app_instance_id="31388366-...",} 1.64 jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea p",id="PS Eden Space",} 1.05906176E8 jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea p",id="PS Survivor Space",} 2.1495808E7 jvm_memory_max_bytes{cf_app_instance_id="31388366-...",area="hea p",id="PS Old Gen",} 3.00941312E8
  58. 58. On Cloud Foundry !53 App (instance 0) App (instance 1) GoRouter Prometheus scrape_configs: - job_name: example dns_configs: - names: - example.apps.internal type: A port: 8080 metrics_path: /actuator/prometheus http://demo.apps.inetrnal:8080/actuator/prometheus https://github.com/making/demo-prom-grafana-on-cf
  59. 59. Another solution - Promregater !54 https://github.com/promregator/promregator
  60. 60. On Kubernetes !55 Pod (instance 0) Pod (instance 1) Prometheus http://xxx.yyy.zzz:8080/actuator/prometheus
  61. 61. !56 scrape_configs: - job_name: k8s-pods kubernetes_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __address__ regex: ([^:]+)(?::d+)?;(d+) replacement: $1:$2 On Kubernetes https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
  62. 62. On Kubernetes !57 apiVersion: apps/v1 kind: Deployment spec: template: metadata: annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" prometheus.io/path: /actuator/prometheus
 # ...
  63. 63. Related Links !58 Thank you for your attention! • https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production- ready.html • https://micrometer.io/docs • https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready- metrics.html • https://blog.ik.am/entries/448 • https://github.com/making/demo-micrometer • https://github.com/making/demo-prom-grafana-on-cf (CF env) • https://github.com/categolj/blog-prometheus/tree/master/local (Local &k8s env)
  64. 64. Related Talks !59 • More Spring Boot 2? => 13:30-14:15 @Room E+F
 Pivotal Reactive Spring 5 & Spring Boot 2 • More Observability? => 14:30-15:15 @Room I
 How to Properly Blame Things for Causing Latency: An Introduction to Distributed Tracing and Zipkin

×