Your SlideShare is downloading. ×
0
SPM: Large Scale Performance Monitoring                  for  ElasticSearch HBase Solr & Friends         #spmbuzz         ...
AgendaIntroductionsSPM ArchitectureSPM Client SideSPM Receiver & FlumeSPM & HBaseSPM Front EndLessonsInteresting Metrics  ...
About Otis GospodnetićASF Member: Lucene, Solr, Nutch, MahoutAuthor: Lucene in Action 1 & 2Entrepreneur: Sematext, Simpy  ...
About SematextProducts & ServicesConsulting, Development, Tech Support:Search; Lucene, Solr, ElasticSearch...Big Data Anal...
Sematext Metrics100% organic: keine GMO / VC5 years old< 10 people7 countries3 timezones3 continents> 100 customers       ...
Scalable Performance MonitoringMulti-tenant APM SaaSWith: HBase, Hadoop, Flume, Jetty, Esper...For: ElasticSearch, HBase, ...
SPM FunctionalityPerformance graphs for any time periodNo loss of metric precision/granularitySlice & dice metrics with fi...
Birds Eye View                                                                8          Copyright 2012 Sematext Intl. All...
SPM: Client SideIn-JVM Agent – app stats collectorCollectd – system stats collectorOut-of-JVM Sender – collector & senderJ...
SPM: Client Sideapt-get install collectdrun SPM client installer-Dcom.sun.management.jmxremote  -javaagent:/spm/spm-monito...
SPM: Receiver & FlumeApache + Jetty + KeepAliveFlume tail + HBase sinkIn-memory metric pre-aggregation - HBaseHUTFlume tai...
SPM: HBaseRaw metrics tableAggregated metrics tableAggregations via MapReduceAggregations on the fly - HBaseHUT           ...
SPM: HBase AggregationsGeneric aggregations frameworkLarge dataset report building (HDFS/HBase)                           ...
SPM: Front EndJavaScript UI, jQuery, FlotFlot → HighCharts → d3Filters:  Time, Host, JVM  Core, Request Handler (Solr)  No...
ElasticSearch Indexing Perf.                                                                 15           Copyright 2012 S...
SPM Numbers>1000 data points/second6 HBase nodesBB raw rows & MM aggregate rows                                           ...
LessonsEC2LinuxKeepAlivesHBase                                                                   17             Copyright ...
Lessons: EC2Agreement?Benchmarking impossibleSpot instance prices spikeAre you feeling lucky?                             ...
Lessons: swapiness                                                               19         Copyright 2012 Sematext Intl. ...
Lessons: nofile> ulimit -n                     Default – often way too low1024  > sudo vim /etc/security/limits.conf      ...
Lessons: KeepAlivesKeepAlives matter at high request rates                                                                ...
Lessons: HBase KeysBad keys can hotspot RegionServersSolution: HBaseWD https://github.com/sematext/HBaseWD                ...
Lessons: Append-only UpdatesRead-Modify-Write can be slowSolution: HBaseHUT http://github.com/sematext/HBaseHUT           ...
Lessons: HTable vs. asynchbaseHTable is blocking and serialSolution: https://github.com/stumbleupon/asynchbaseSee:  https:...
Poll: Search Monitoring                                                                25          Copyright 2012 Sematext...
Interesting MetricsSolrElasticSearchHBase                                                                      26         ...
BuzzworthinessBig Data                               Large ScaleSearch                                 Multi-tenantAnalyti...
Were Hiring   Dig Search?   Dig Analytics?   Dig Big Data?   Dig Performance?   Dig working with and in open-source?   Wer...
Contact      sematext.com      blog.sematext.com      github.com/sematext      @sematext      @otisg      otis@sematext.co...
Upcoming SlideShare
Loading in...5
×

Large Scale Performance Monitoring for ElasticSearch, HBase, Solr, SenseiDB, etc.

11,582

Published on

Description of how Sematext SPM Performance Monitoring service is built and how it works. Originally presented at Berlin Buzzwords 2012.

Published in: Technology, Business
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
11,582
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
75
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Transcript of "Large Scale Performance Monitoring for ElasticSearch, HBase, Solr, SenseiDB, etc."

  1. 1. SPM: Large Scale Performance Monitoring for ElasticSearch HBase Solr & Friends #spmbuzz #bbuzz Otis Gospodnetić – Sematext International @otisg ◦ @sematext ◦ sematext.com
  2. 2. AgendaIntroductionsSPM ArchitectureSPM Client SideSPM Receiver & FlumeSPM & HBaseSPM Front EndLessonsInteresting Metrics 2 Copyright 2012 Sematext Intl. All rights reserved.
  3. 3. About Otis GospodnetićASF Member: Lucene, Solr, Nutch, MahoutAuthor: Lucene in Action 1 & 2Entrepreneur: Sematext, Simpy 3 Copyright 2012 Sematext Intl. All rights reserved.
  4. 4. About SematextProducts & ServicesConsulting, Development, Tech Support:Search; Lucene, Solr, ElasticSearch...Big Data Analytics; Hadoop, HBase, Voldemort...Web Crawling; Nutch, DroidsMachine Learning; Mahout 4 Copyright 2012 Sematext Intl. All rights reserved.
  5. 5. Sematext Metrics100% organic: keine GMO / VC5 years old< 10 people7 countries3 timezones3 continents> 100 customers 5 Copyright 2012 Sematext Intl. All rights reserved.
  6. 6. Scalable Performance MonitoringMulti-tenant APM SaaSWith: HBase, Hadoop, Flume, Jetty, Esper...For: ElasticSearch, HBase, Solr, Sensei...Amazon Cloud apps.sematext.com ganz frei 6 Copyright 2012 Sematext Intl. All rights reserved.
  7. 7. SPM FunctionalityPerformance graphs for any time periodNo loss of metric precision/granularitySlice & dice metrics with filtersCompare two time periodsIntegrated alertsIntegrated email reportsMonitor multiple (types of) systemsMetric arithmetic (ganz soon!) 7 Copyright 2012 Sematext Intl. All rights reserved.
  8. 8. Birds Eye View 8 Copyright 2012 Sematext Intl. All rights reserved.
  9. 9. SPM: Client SideIn-JVM Agent – app stats collectorCollectd – system stats collectorOut-of-JVM Sender – collector & senderJMX, bytecode instrumentation, Flume, HttpClient, local buffering 9 Copyright 2012 Sematext Intl. All rights reserved.
  10. 10. SPM: Client Sideapt-get install collectdrun SPM client installer-Dcom.sun.management.jmxremote -javaagent:/spm/spm-monitor/lib/spm- monitor-es-1.5.0-withdeps.jar=/spm/spm- monitor/conf/spm-monitor-config-token-here- default.xml/etc/init.d/spm-sender start 10 Copyright 2012 Sematext Intl. All rights reserved.
  11. 11. SPM: Receiver & FlumeApache + Jetty + KeepAliveFlume tail + HBase sinkIn-memory metric pre-aggregation - HBaseHUTFlume tail + Esper sink = Performance Alerts 11 Copyright 2012 Sematext Intl. All rights reserved.
  12. 12. SPM: HBaseRaw metrics tableAggregated metrics tableAggregations via MapReduceAggregations on the fly - HBaseHUT 12 Copyright 2012 Sematext Intl. All rights reserved.
  13. 13. SPM: HBase AggregationsGeneric aggregations frameworkLarge dataset report building (HDFS/HBase) 13 Copyright 2012 Sematext Intl. All rights reserved.
  14. 14. SPM: Front EndJavaScript UI, jQuery, FlotFlot → HighCharts → d3Filters: Time, Host, JVM Core, Request Handler (Solr) Node, Index, and Shard (ElasticSearch) Network interface Disk ... 14 Copyright 2012 Sematext Intl. All rights reserved.
  15. 15. ElasticSearch Indexing Perf. 15 Copyright 2012 Sematext Intl. All rights reserved.
  16. 16. SPM Numbers>1000 data points/second6 HBase nodesBB raw rows & MM aggregate rows 16 Copyright 2012 Sematext Intl. All rights reserved.
  17. 17. LessonsEC2LinuxKeepAlivesHBase 17 Copyright 2012 Sematext Intl. All rights reserved.
  18. 18. Lessons: EC2Agreement?Benchmarking impossibleSpot instance prices spikeAre you feeling lucky? 18 Copyright 2012 Sematext Intl. All rights reserved.
  19. 19. Lessons: swapiness 19 Copyright 2012 Sematext Intl. All rights reserved.
  20. 20. Lessons: nofile> ulimit -n Default – often way too low1024 > sudo vim /etc/security/limits.conf Bump it up!spmsender - nofile 32000> sudo vim /etc/pam.d/susession required pam_limits.so 20 Copyright 2012 Sematext Intl. All rights reserved.
  21. 21. Lessons: KeepAlivesKeepAlives matter at high request rates 21 Copyright 2012 Sematext Intl. All rights reserved.
  22. 22. Lessons: HBase KeysBad keys can hotspot RegionServersSolution: HBaseWD https://github.com/sematext/HBaseWD 22 Copyright 2012 Sematext Intl. All rights reserved.
  23. 23. Lessons: Append-only UpdatesRead-Modify-Write can be slowSolution: HBaseHUT http://github.com/sematext/HBaseHUT 23 Copyright 2012 Sematext Intl. All rights reserved.
  24. 24. Lessons: HTable vs. asynchbaseHTable is blocking and serialSolution: https://github.com/stumbleupon/asynchbaseSee: https://issues.apache.org/jira/browse/HBASE-5539http://tsunanet.net/~tsuna/asynchbase/benchmark/viz.html 24 Copyright 2012 Sematext Intl. All rights reserved.
  25. 25. Poll: Search Monitoring 25 Copyright 2012 Sematext Intl. All rights reserved.
  26. 26. Interesting MetricsSolrElasticSearchHBase 26 Copyright 2012 Sematext Intl. All rights reserved.
  27. 27. BuzzworthinessBig Data Large ScaleSearch Multi-tenantAnalytics SaaSReal-time Cloud 27 Copyright 2012 Sematext Intl. All rights reserved.
  28. 28. Were Hiring Dig Search? Dig Analytics? Dig Big Data? Dig Performance? Dig working with and in open-source? Were hiring world-wide! http://sematext.com/about/jobs.html 28 Copyright 2012 Sematext Intl. All rights reserved.
  29. 29. Contact sematext.com blog.sematext.com github.com/sematext @sematext @otisg otis@sematext.com Stop by our booth 29 Copyright 2012 Sematext Intl. All rights reserved.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×