Large Scale Performance Monitoring for ElasticSearch, HBase, Solr, SenseiDB, etc.

SPM: Large Scale Performance Monitoring for ElasticSearch HBase Solr & Friends #spmbuzz #bbuzz Otis Gospodnetić – Sematext International @otisg ◦ @sematext ◦ sematext.com

About SematextProducts & ServicesConsulting, Development, Tech Support:Search; Lucene, Solr, ElasticSearch...Big Data Analytics; Hadoop, HBase, Voldemort...Web Crawling; Nutch, DroidsMachine Learning; Mahout 4 Copyright 2012 Sematext Intl. All rights reserved.

Scalable Performance MonitoringMulti-tenant APM SaaSWith: HBase, Hadoop, Flume, Jetty, Esper...For: ElasticSearch, HBase, Solr, Sensei...Amazon Cloud apps.sematext.com ganz frei 6 Copyright 2012 Sematext Intl. All rights reserved.

SPM FunctionalityPerformance graphs for any time periodNo loss of metric precision/granularitySlice & dice metrics with filtersCompare two time periodsIntegrated alertsIntegrated email reportsMonitor multiple (types of) systemsMetric arithmetic (ganz soon!) 7 Copyright 2012 Sematext Intl. All rights reserved.

SPM: Client SideIn-JVM Agent – app stats collectorCollectd – system stats collectorOut-of-JVM Sender – collector & senderJMX, bytecode instrumentation, Flume, HttpClient, local buffering 9 Copyright 2012 Sematext Intl. All rights reserved.

SPM: Client Sideapt-get install collectdrun SPM client installer-Dcom.sun.management.jmxremote -javaagent:/spm/spm-monitor/lib/spm- monitor-es-1.5.0-withdeps.jar=/spm/spm- monitor/conf/spm-monitor-config-token-here- default.xml/etc/init.d/spm-sender start 10 Copyright 2012 Sematext Intl. All rights reserved.

SPM: Receiver & FlumeApache + Jetty + KeepAliveFlume tail + HBase sinkIn-memory metric pre-aggregation - HBaseHUTFlume tail + Esper sink = Performance Alerts 11 Copyright 2012 Sematext Intl. All rights reserved.

SPM: Front EndJavaScript UI, jQuery, FlotFlot → HighCharts → d3Filters: Time, Host, JVM Core, Request Handler (Solr) Node, Index, and Shard (ElasticSearch) Network interface Disk ... 14 Copyright 2012 Sematext Intl. All rights reserved.

Lessons: nofile> ulimit -n Default – often way too low1024 > sudo vim /etc/security/limits.conf Bump it up!spmsender - nofile 32000> sudo vim /etc/pam.d/susession required pam_limits.so 20 Copyright 2012 Sematext Intl. All rights reserved.

Lessons: HTable vs. asynchbaseHTable is blocking and serialSolution: https://github.com/stumbleupon/asynchbaseSee: https://issues.apache.org/jira/browse/HBASE-5539http://tsunanet.net/~tsuna/asynchbase/benchmark/viz.html 24 Copyright 2012 Sematext Intl. All rights reserved.

Were Hiring Dig Search? Dig Analytics? Dig Big Data? Dig Performance? Dig working with and in open-source? Were hiring world-wide! http://sematext.com/about/jobs.html 28 Copyright 2012 Sematext Intl. All rights reserved.

http://blog.sematext.com	5884
http://feeds.feedburner.com	13
http://www.tuicool.com	3
http://webcache.googleusercontent.com	2
http://www.newsblur.com	1

Large Scale Performance Monitoring for ElasticSearch, HBase, Solr, SenseiDB, etc.

by Sematext Group, Inc. on Jun 09, 2012

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 5,903

Statistics

Large Scale Performance Monitoring for ElasticSearch, HBase, Solr, SenseiDB, etc. Presentation Transcript