Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Monitoring Alfresco SOLR with Prometheus and Grafana

274 views

Published on

https://github.com/aborroy/alfresco-solr-monitoring
https://github.com/aborroy/alfresco-solr-monitoring/tree/add-metric-heap-usage

Published in: Technology
  • Be the first to comment

Monitoring Alfresco SOLR with Prometheus and Grafana

  1. 1. Angel Borroy Search Team February 13, 2020 Monitoring Alfresco SOLR with Prometheus and Grafana
  2. 2. 22 Monitoring Alfresco SOLR with Prometheus and Grafana • Building Components • Docker Compose demo • Creating new Metrics and Dashboards • Shard Support • Deployment considerations Contents
  3. 3. 33 Building Components
  4. 4. 4 Map of Components Alfresco Search Service 1.0 to 1.4+ SOLR 6.6.x Prometheus Exporter SOLR 7.3.0+ SOLR Metrics Prometheus Server 2.x Grafana UI 6.x 8083 9854 9090 3000 Exporter Config XML JOB YAML Dashboard JSON
  5. 5. 5 SOLR Metrics Classes of meters • counters - simply count events. Single long value. • meters - additionally compute rates of events. Provide a count and mean, 1-, 5-, and 15-minute rates. • histograms - calculate approximate distribution of events according to their values. Provide the following approximate statistics, with a similar exponential decay as above: mean (arithmetic average), median, maximum, minimum, standard deviation, and 75th, 95th, 98th, 99th and 999th percentiles. • timers - measure the number and duration of events. Count and histogram of timings. • gauges - offer instantaneous reading of a current value. http://localhost:8083/solr/admin/metrics Metric Registries
  6. 6. 6 Prometheus Exporter https://github.com/aborroy/alfresco-solr-monitoring/blob/master/solr-exporter/solr6-exporter-config.xml <str> .metrics["solr.jvm"] | to_entries | .[] | select(.key | startswith("gc.")) | select(.key | endswith(".time")) as $object | $object.key | split(".")[1] as $item | ($object.value.value / 1000) as $value | { name : "solr_metrics_jvm_gc_seconds_total", type : "COUNTER", help : "Help", label_names : ["item"], label_values : [$item], value : $value } </str> http://localhost:9854/metrics Exporter Config XML
  7. 7. 7 Prometheus Server https://github.com/aborroy/alfresco-solr-monitoring/blob/master/prometheus/prometheus.yml scrape_configs: - job_name: 'solr' static_configs: - targets: ['solr-exporter:9854'] http://localhost:9090/targets JOB YAML
  8. 8. 8 Grafana UI https://github.com/aborroy/alfresco-solr-monitoring/blob/master/grafana/grafana-solr6-dashboard.json "targets": [ { "expr": "increase(solr_metrics_jvm_gc_total{base_url=~"$base_url"}[1m])", "format": "time_series", "intervalFactor": 2, "legendFormat": "{{base_url}} {{item}}", "refId": "A" } ] Dashboard JSON http://localhost::3000/d/GqfWBzbWz/solr-dashboard
  9. 9. 99 Docker Compose Demo
  10. 10. 1010 Demo settings • Single SOLR server • Every service running in the same Docker Compose • Default templates for exporter and dashboard
  11. 11. 11 Docker Compose solr-exporter: image: solr:7.3.0 depends_on: - solr6 ports: - "9854:9854" entrypoint: - "/opt/solr/contrib/prometheus-exporter/bin/solr-exporter" - "-p" - "9854" - "-b" - "http://solr6:8983/solr" - "-f" - "/opt/solr/contrib/prometheus-exporter/conf/solr-exporter-config.xml" - "-n" - "8" volumes: - ./solr-exporter/solr6-exporter-config.xml:/opt/solr/contrib/prometheus-exporter/conf/solr-exporter-config.xml
  12. 12. 12 Docker Compose prometheus-server: image: prom/prometheus ports: - 9090:9090 volumes: - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml grafana-ui: image: grafana/grafana ports: - 3000:3000 environment: - GF_SECURITY_ADMIN_PASSWORD=secret links: - prometheus-server:prometheus volumes: - ./grafana/datasources/prometheus.yml:/etc/grafana/provisioning/datasources/prometheus.yml - ./grafana/dashboards/solr.yml:/etc/grafana/provisioning/dashboards/solr.yml - ./grafana/grafana-solr6-dashboard.json:/var/lib/grafana/dashboards/grafana-solr6-dashboard.json
  13. 13. 1313 Demo Time
  14. 14. 1414 Creating new Metrics and Dashboards
  15. 15. 1515 Creating new metrics http://127.0.0.1:8083/solr/admin/metrics http://127.0.0.1:9854/metrics http://127.0.0.1:3000 Prometheus https://github.com/aborroy/alfresco-solr-monitoring/tree/add-metric-heap-usage
  16. 16. 1616 Exposing new metrics Prometheus Exporter <str> .metrics["solr.jvm"] | to_entries | .[] | select(.key | startswith("memory.heap.usage")) as $object | $object.key | split(".")[1] as $item | $object.value.value as $value | { name : "solr_metrics_memory_heap_usage", type : "GAUGE", help : "See following URL: https://lucene.apache.org/...", label_names : ["item"], label_values : [$item], value : $value } </str>
  17. 17. 1717 Adding new panels Grafana { "datasource": "Prometheus", "id": 61, "options": { "fieldOptions": { "calcs": [ "last" ], "defaults": { "max": 1.0, "min": 0.0, "thresholds": [ { "color": "green", "value": null }, { "color": "red", "value": 0.75 } ], "unit": "percentunit" }, "override": {}, "values": false }, "orientation": "auto", }, "targets": [ { "expr": "solr_metrics_memory_heap_usage{base_url=~"$ base_url"}", "format": "time_series", "intervalFactor": 2, "legendFormat": "{{base_url}} {{item}}", "refId": "A" } ], "timeFrom": null, "timeShift": null, "title": "Memory Heap Usage", "type": "gauge" },
  18. 18. 1818 Final Considerations
  19. 19. 1919 Shard is not supported • Prometheus Exporter only supports ZooKeeper (SolrCloud) • Alternatively several Prometheus Exporter endpoints can be configured, one for each SOLR ShardX
  20. 20. 2020 Alfresco Deployment SOLR Server
  21. 21. Thank you! Monitoring Alfresco SOLR with Prometheus and Grafana
  22. 22. 22 References • Monitoring Alfresco SOLR with Prometheus and Grafana blog post https://hub.alfresco.com/t5/alfresco-content-services-blog/monitoring-alfresco-solr-with-prometheus-and-grafana/ba- p/294157 • Docker Compose sample template https://github.com/aborroy/alfresco-solr-monitoring https://github.com/aborroy/alfresco-solr-monitoring/tree/add-metric-heap-usage • SOLR Metrics Reporting https://lucene.apache.org/solr/guide/6_6/metrics-reporting.html • SOLR Prometheus Exporter https://github.com/apache/lucene-solr/tree/master/solr/contrib/prometheus-exporter • Grafana Dashboard JSON Reference https://grafana.com/docs/grafana/latest/reference/dashboard/

×