Solr Performance Monitoring with SPM

9,821 views

Published on

Originally delivered as Lightning Talk at Lucene Eurocon 2011 in Barcelona, this quick presentation shows how to use Sematext's SPM service to monitor Solr, OS, JVM, and more.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,821
On SlideShare
0
From Embeds
0
Number of Embeds
6,677
Actions
Shares
0
Downloads
41
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Solr Performance Monitoring with SPM

  1. 1. Solr Performance Monitoring with Scalable Performance Monitoring SaaS Otis Gospodneti ć – Sematext International @otisg ◦ @sematext ◦ sematext.com sematext.com/spm
  2. 2. SPM Overview <ul><li>SPM = Scalable Performance Monitoring </li></ul><ul><li>SaaS </li></ul><ul><ul><li>Small client-side agent attached to the JVM </li></ul></ul><ul><ul><li>Collectd for OS, FS, and other metrics </li></ul></ul><ul><ul><li>Near Real Time </li></ul></ul><ul><ul><li>Distributed backend: Hadoop, HBase, Flume, etc. </li></ul></ul><ul><li>Support: Solr , HBase , etc. </li></ul>
  3. 3. SPM Functionality <ul><li>Look at one or multiple Solr instances , Request Handlers , or Solr Cores at once </li></ul><ul><li>See performance of any time period with no loss of resolution </li></ul><ul><li>Show/hide extended time periods </li></ul><ul><li>Compare two time periods </li></ul><ul><li>Monitor multiple Solr clusters </li></ul><ul><li>Get Alerts (soon) </li></ul>
  4. 4. Solr Metrics <ul><li>All Solr metrics and more: </li></ul><ul><ul><li>All cache metrics (hit %,evictions, etc.) </li></ul></ul><ul><ul><li>Query rate and latency </li></ul></ul><ul><ul><li>Warmup time </li></ul></ul><ul><ul><li>Latency breakdown by SearchComponent (soon) </li></ul></ul><ul><ul><li>Latency breakdown by classes & methods (soon) </li></ul></ul><ul><ul><li>Index size on disk, # files, # segments, # docs, # deletions </li></ul></ul>
  5. 5. OS Metrics <ul><li>OS metrics </li></ul><ul><ul><li>CPU </li></ul></ul><ul><ul><li>Memory </li></ul></ul><ul><ul><li>Load </li></ul></ul><ul><ul><li>Disk IO by device </li></ul></ul><ul><ul><li>Network IO by device </li></ul></ul><ul><ul><li>Swap (soon) </li></ul></ul><ul><ul><li>Disk usage (soon) </li></ul></ul>
  6. 6. JVM Metrics <ul><li>Garbage Collection </li></ul><ul><ul><li>Major & Minor </li></ul></ul><ul><ul><li>Counts </li></ul></ul><ul><ul><li>Time spent in GC </li></ul></ul><ul><li>Thread count </li></ul><ul><li>Memory </li></ul><ul><ul><li>Heap memory </li></ul></ul><ul><ul><li>Non-heap memory </li></ul></ul><ul><ul><li>Memory pools (soon) </li></ul></ul>
  7. 7. Lucene / Solr Index segment merge # deleted docs
  8. 8. Solr Caches see the increase! did you know you had these dips?
  9. 9. Search Rate & Latency Select 1 or more Solr instances Select 1 or more Request Handlers Select any time periods Compare 2 time periods Select data granularity
  10. 10. Warmup Warmup time broken by warmup type – caches and searcher Select 1 or more Solr Cores
  11. 11. OS & FS
  12. 12. JVM
  13. 13. Open Sourcing <ul><li>2 open-source projects: </li></ul><ul><li>github.com/sematext/HBaseWD </li></ul><ul><li>github.com/sematext/HBaseHUT </li></ul><ul><li>See sematext.com/open-source/index.html </li></ul><ul><li>Patches for Flume and HBase blog.sematext.com/tag/flume/ </li></ul>
  14. 14. About Sematext <ul><li>Products & Services </li></ul><ul><li>Consulting, Development, Tech Support: </li></ul><ul><li>Search (Lucene, Solr, ElasticSearch...) </li></ul><ul><li>Big Data (Hadoop, HBase, Voldemort...) </li></ul><ul><li>Web Crawling (Nutch, Droids) </li></ul><ul><li>Machine Learning (Mahout) </li></ul>
  15. 15. Sematext Metrics <ul><li>100% organic : no GMO, no VC </li></ul><ul><li>4 years old </li></ul><ul><li>< 10 people </li></ul><ul><li>7 countries </li></ul><ul><li>3 timezones </li></ul><ul><li>2 continents </li></ul><ul><li>> 100 customers </li></ul>
  16. 16. We're Hiring <ul><li>Dig Search ? </li></ul><ul><li>Dig Analytics ? </li></ul><ul><li>Dig Big Data ? </li></ul><ul><li>Dig Performance ? </li></ul><ul><li>Dig working with and in open-source ? </li></ul><ul><li>We're hiring world-wide ! </li></ul><ul><li>sematext.com/about/jobs.html </li></ul>
  17. 17. <ul><li>sematext.com </li></ul><ul><li>blog.sematext.com </li></ul><ul><li>@sematext </li></ul><ul><li>@otisg </li></ul><ul><li>[email_address] </li></ul><ul><li>sematext.com/spm </li></ul>Contact

×