Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HBaseCon 2013: Using Metrics to Monitor and Debug Apache HBase

4,266 views

Published on

Presented by: Elliott Clark, Cloudera

Published in: Technology, Sports
  • Be the first to comment

HBaseCon 2013: Using Metrics to Monitor and Debug Apache HBase

  1. 1. 1 Using Metrics How to monitor and debug Apache HBase with metrics
  2. 2. 2 Hi! I’m Elliott eclark@apache.org @elliottnclark Engineer @ Cloudera
  3. 3. Rules of HBase Metrics 1. Collect Metrics 2. Collect Metrics 3. Know your steady state 4. Explore 5. Correlate 3
  4. 4. Examples • User Error • Region Skew • GC Pressure • Suboptimal HBase • Balancer not Balancing 4
  5. 5. Region Skew 5 Photo From: http://www.flickr.com/photos/64618542@N00/
  6. 6. Region Skew 6
  7. 7. Region Skew Symptoms • Long tail latencies • Unbalanced WALs • Crashing region servers 7
  8. 8. Region Skew Fixes • Hash Row Key • Add random salt to start of Row Key • Re-Arrange The Row Key • Add Caching 8
  9. 9. GC Pressure 9 Photo by: http://www.flickr.com/photos/wwarby/
  10. 10. GC Pressure 10
  11. 11. GC Pressure Symptoms • Crashing region servers • ZK Connection Issues • Spiky latency 11
  12. 12. GC Pressure Fixes • Switch map reduce to Bulk Load • Pre-Aggregate Puts • Reduce data returned per scan#next 12
  13. 13. Bad Balancer 13 Photo By: http://www.flickr.com/photos/jimmiehomeschoolmom/
  14. 14. Bad Balancer 14
  15. 15. Bad Balancer Symptoms • Bad latency after balancer • Bad cache hit on some servers 15
  16. 16. Coming Soon • Better balancer • More per region metrics • Renamed Metrics • Metrics2 based metrics 16
  17. 17. Metrics Options Cloudera Manger OpenTSDB 17 Ganglia Ganglia Image From:http://www.flickr.com/photos/hongiiv/
  18. 18. 18

×