• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
20121022 tm hbasecanarytool
 

20121022 tm hbasecanarytool

on

  • 707 views

 

Statistics

Views

Total Views
707
Views on SlideShare
363
Embed Views
344

Actions

Likes
0
Downloads
5
Comments
0

6 Embeds 344

http://dumbointaiwan.blogspot.tw 215
http://dumbointaiwan.blogspot.com 122
http://trender.trendmicro.com 4
http://webcache.googleusercontent.com 1
http://dumbointaiwan.blogspot.sg 1
http://dumbointaiwan.blogspot.jp 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    20121022 tm hbasecanarytool 20121022 tm hbasecanarytool Presentation Transcript

    • HBase Canary Tool2012/10/22 Scott Miao
    • Another way to monitor HBase processes • org.apache.hadoop.hbase.tool.Canary – Be used to do "canary monitoring" of a running HBase cluster. – For each region tries to get one row per column family and outputs some information about failure or latencyUsage: bin/hbase org.apache.hadoop.hbase.tool.Canary [opts] [table 1 [table 2...]]where [opts] are: -help Show this help and exit. -daemon Continuous check at defined intervals. # 6sec -interval <N> Interval between checks (sec) # specify how many secs you want https://issues.apache.org/jira/browse/HBASE-4393
    • Canary Tool studyprivate void sniff() throws Exception { for (HTableDescriptor table : admin.listTables()) { sniff(table); } } private void sniff(HTableDescriptor tableDesc) throws Exception { HTable table = null; try { table = new HTable(admin.getConfiguration(), tableDesc.getName()); } catch (TableNotFoundException e) { return; } for (HRegionInfo region : admin.getTableRegions(tableDesc.getName())) { try { sniffRegion(region, table); } catch (Exception e) { sink.publishReadFailure(region); } } }
    • Canary Tool studyprivate void sniffRegion(HRegionInfo region, HTable table) throws Exception { HTableDescriptor tableDesc = table.getTableDescriptor(); for (HColumnDescriptor column : tableDesc.getColumnFamilies()) { Get get = new Get(region.getStartKey()); get.addFamily(column.getName()); try { long startTime = System.currentTimeMillis(); table.get(get); long time = System.currentTimeMillis() - startTime; sink.publishReadTiming(region, column, time); } catch (Exception e) { sink.publishReadFailure(region, column); } } }
    • Canary Tool studypublic interface Sink { public void publishReadFailure(HRegionInfo region); public void publishReadFailure(HRegionInfo region, HColumnDescriptor column); public void publishReadTiming(HRegionInfo region, HColumnDescriptorcolumn, long msTime); } public static class StdOutSink implements Sink { public void publishReadFailure(HRegionInfo region) { LOG.error(String.format("read from region %s failed", region.getRegionNameAsString())); } public void publishReadFailure(HRegionInfo region, HColumnDescriptor column) { LOG.error(String.format("read from region %s column family %s failed", region.getRegionNameAsString(), column.getNameAsString())); } public void publishReadTiming(HRegionInfo region, HColumnDescriptor column, long msTime) { LOG.info(String.format("read from region %s column family %s in %dms", region.getRegionNameAsString(), column.getNameAsString(), msTime)); } }
    • Canary Tool study //constructors public Canary() { this(new StdOutSink()); } public Canary(Sink sink) { this.sink = sink; }
    • Canary Tool in Circus Send mail if any Fix problem abnormal Canary-toolStart here Nagios Server Tm-puppet operation server hbase- Write to canary.log Read from /var/log/hbase/
    • Canary Tool in Circuscom.trendmicro.spn.ops.hbase.RunCanaryToolprivate static class CustomSink implements Canary.Sink { public void publishReadFailure(HRegionInfo regionInfo) { //... LOG.error(String.format("Read from table:%s, region:%s failed", tableName, regionName)); } public void publishReadFailure(HRegionInfo regionInfo, HColumnDescriptor colDescriptor) { //... LOG.error(String.format("Read from table:%s, region:%s, columnFamily:%sfailed", tableName, regionName, colFamilyName)); } public void publishReadTiming(HRegionInfo regionInfo, HColumnDescriptor colDescriptor, long msTime) { //... LOG.info(String.format("Read from table:%s, region:%s, columnFamily:%s in%dms", tableName, regionName, colFamilyName, msTime)); }}
    • Canary Tool in Circuscom.trendmicro.spn.ops.hbase.RunCanaryToolpublic static void main(String[] args) throws Exception { Canary canary = new Canary(new CustomSink()); int exitCode = ToolRunner.run(canary, args); System.exit(exitCode);} hbase-canary-monitor.sh su - hbase <<EOF kinit -kt /etc/hbase/conf/hbase.keytab hbase/$(hostname -f) java -cp $CLASSPATH com.trendmicro.spn.ops.hbase.RunCanaryTool $@ EOF