1
Using Metrics
How to monitor and debug Apache HBase with
metrics
2
Hi! I’m Elliott
eclark@apache.org @elliottnclark Engineer @ Cloudera
Rules of HBase Metrics
1. Collect Metrics
2. Collect Metrics
3. Know your steady state
4. Explore
5. Correlate
3
Examples
• User Error
• Region Skew
• GC Pressure
• Suboptimal HBase
• Balancer not Balancing
4
Region Skew
5 Photo From: http://www.flickr.com/photos/64618542@N00/
Region Skew
6
Region Skew Symptoms
• Long tail latencies
• Unbalanced WALs
• Crashing region servers
7
Region Skew Fixes
• Hash Row Key
• Add random salt to start of Row Key
• Re-Arrange The Row Key
• Add Caching
8
GC Pressure
9 Photo by: http://www.flickr.com/photos/wwarby/
GC Pressure
10
GC Pressure Symptoms
• Crashing region servers
• ZK Connection Issues
• Spiky latency
11
GC Pressure Fixes
• Switch map reduce to Bulk Load
• Pre-Aggregate Puts
• Reduce data returned per scan#next
12
Bad Balancer
13 Photo By: http://www.flickr.com/photos/jimmiehomeschoolmom/
Bad Balancer
14
Bad Balancer Symptoms
• Bad latency after balancer
• Bad cache hit on some servers
15
Coming Soon
• Better balancer
• More per region metrics
• Renamed Metrics
• Metrics2 based metrics
16
Metrics Options
Cloudera Manger OpenTSDB
17
Ganglia
Ganglia Image From:http://www.flickr.com/photos/hongiiv/
18
Upcoming SlideShare
Loading in...5
×

HBaseCon 2013: Using Metrics to Monitor and Debug Apache HBase

1,701

Published on

Presented by: Elliott Clark, Cloudera

Published in: Technology, Sports
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,701
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • Slow DOWN!NO THUMBS!
  • Talk about the two different catagories.Suboptimal usage of hbaseThings hbase should handle for you.
  • Explain that region skew is an example.
  • Don’t talk about the symtoms.
  • Lots of hot regions on the same region serverRegions fighting over space in the Block Cache.
  • Go forth and collect metrics.
  • HBaseCon 2013: Using Metrics to Monitor and Debug Apache HBase

    1. 1. 1 Using Metrics How to monitor and debug Apache HBase with metrics
    2. 2. 2 Hi! I’m Elliott eclark@apache.org @elliottnclark Engineer @ Cloudera
    3. 3. Rules of HBase Metrics 1. Collect Metrics 2. Collect Metrics 3. Know your steady state 4. Explore 5. Correlate 3
    4. 4. Examples • User Error • Region Skew • GC Pressure • Suboptimal HBase • Balancer not Balancing 4
    5. 5. Region Skew 5 Photo From: http://www.flickr.com/photos/64618542@N00/
    6. 6. Region Skew 6
    7. 7. Region Skew Symptoms • Long tail latencies • Unbalanced WALs • Crashing region servers 7
    8. 8. Region Skew Fixes • Hash Row Key • Add random salt to start of Row Key • Re-Arrange The Row Key • Add Caching 8
    9. 9. GC Pressure 9 Photo by: http://www.flickr.com/photos/wwarby/
    10. 10. GC Pressure 10
    11. 11. GC Pressure Symptoms • Crashing region servers • ZK Connection Issues • Spiky latency 11
    12. 12. GC Pressure Fixes • Switch map reduce to Bulk Load • Pre-Aggregate Puts • Reduce data returned per scan#next 12
    13. 13. Bad Balancer 13 Photo By: http://www.flickr.com/photos/jimmiehomeschoolmom/
    14. 14. Bad Balancer 14
    15. 15. Bad Balancer Symptoms • Bad latency after balancer • Bad cache hit on some servers 15
    16. 16. Coming Soon • Better balancer • More per region metrics • Renamed Metrics • Metrics2 based metrics 16
    17. 17. Metrics Options Cloudera Manger OpenTSDB 17 Ganglia Ganglia Image From:http://www.flickr.com/photos/hongiiv/
    18. 18. 18

    ×