• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
HBaseCon 2013: Using Metrics to Monitor and Debug Apache HBase
 

HBaseCon 2013: Using Metrics to Monitor and Debug Apache HBase

on

  • 1,551 views

Presented by: Elliott Clark, Cloudera

Presented by: Elliott Clark, Cloudera

Statistics

Views

Total Views
1,551
Views on SlideShare
1,551
Embed Views
0

Actions

Likes
3
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Slow DOWN!NO THUMBS!
  • Talk about the two different catagories.Suboptimal usage of hbaseThings hbase should handle for you.
  • Explain that region skew is an example.
  • Don’t talk about the symtoms.
  • Lots of hot regions on the same region serverRegions fighting over space in the Block Cache.
  • Go forth and collect metrics.

HBaseCon 2013: Using Metrics to Monitor and Debug Apache HBase HBaseCon 2013: Using Metrics to Monitor and Debug Apache HBase Presentation Transcript

  • 1 Using Metrics How to monitor and debug Apache HBase with metrics
  • 2 Hi! I’m Elliott eclark@apache.org @elliottnclark Engineer @ Cloudera
  • Rules of HBase Metrics 1. Collect Metrics 2. Collect Metrics 3. Know your steady state 4. Explore 5. Correlate 3
  • Examples • User Error • Region Skew • GC Pressure • Suboptimal HBase • Balancer not Balancing 4
  • Region Skew 5 Photo From: http://www.flickr.com/photos/64618542@N00/
  • Region Skew 6
  • Region Skew Symptoms • Long tail latencies • Unbalanced WALs • Crashing region servers 7
  • Region Skew Fixes • Hash Row Key • Add random salt to start of Row Key • Re-Arrange The Row Key • Add Caching 8
  • GC Pressure 9 Photo by: http://www.flickr.com/photos/wwarby/
  • GC Pressure 10
  • GC Pressure Symptoms • Crashing region servers • ZK Connection Issues • Spiky latency 11
  • GC Pressure Fixes • Switch map reduce to Bulk Load • Pre-Aggregate Puts • Reduce data returned per scan#next 12
  • Bad Balancer 13 Photo By: http://www.flickr.com/photos/jimmiehomeschoolmom/
  • Bad Balancer 14
  • Bad Balancer Symptoms • Bad latency after balancer • Bad cache hit on some servers 15
  • Coming Soon • Better balancer • More per region metrics • Renamed Metrics • Metrics2 based metrics 16
  • Metrics Options Cloudera Manger OpenTSDB 17 Ganglia Ganglia Image From:http://www.flickr.com/photos/hongiiv/
  • 18