• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
20121022 tm hbasecanarytool

20121022 tm hbasecanarytool






Total Views
Views on SlideShare
Embed Views



6 Embeds 344

http://dumbointaiwan.blogspot.tw 215
http://dumbointaiwan.blogspot.com 122
http://trender.trendmicro.com 4
http://webcache.googleusercontent.com 1
http://dumbointaiwan.blogspot.sg 1
http://dumbointaiwan.blogspot.jp 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    20121022 tm hbasecanarytool 20121022 tm hbasecanarytool Presentation Transcript

    • HBase Canary Tool2012/10/22 Scott Miao
    • Another way to monitor HBase processes • org.apache.hadoop.hbase.tool.Canary – Be used to do "canary monitoring" of a running HBase cluster. – For each region tries to get one row per column family and outputs some information about failure or latencyUsage: bin/hbase org.apache.hadoop.hbase.tool.Canary [opts] [table 1 [table 2...]]where [opts] are: -help Show this help and exit. -daemon Continuous check at defined intervals. # 6sec -interval <N> Interval between checks (sec) # specify how many secs you want https://issues.apache.org/jira/browse/HBASE-4393
    • Canary Tool studyprivate void sniff() throws Exception { for (HTableDescriptor table : admin.listTables()) { sniff(table); } } private void sniff(HTableDescriptor tableDesc) throws Exception { HTable table = null; try { table = new HTable(admin.getConfiguration(), tableDesc.getName()); } catch (TableNotFoundException e) { return; } for (HRegionInfo region : admin.getTableRegions(tableDesc.getName())) { try { sniffRegion(region, table); } catch (Exception e) { sink.publishReadFailure(region); } } }
    • Canary Tool studyprivate void sniffRegion(HRegionInfo region, HTable table) throws Exception { HTableDescriptor tableDesc = table.getTableDescriptor(); for (HColumnDescriptor column : tableDesc.getColumnFamilies()) { Get get = new Get(region.getStartKey()); get.addFamily(column.getName()); try { long startTime = System.currentTimeMillis(); table.get(get); long time = System.currentTimeMillis() - startTime; sink.publishReadTiming(region, column, time); } catch (Exception e) { sink.publishReadFailure(region, column); } } }
    • Canary Tool studypublic interface Sink { public void publishReadFailure(HRegionInfo region); public void publishReadFailure(HRegionInfo region, HColumnDescriptor column); public void publishReadTiming(HRegionInfo region, HColumnDescriptorcolumn, long msTime); } public static class StdOutSink implements Sink { public void publishReadFailure(HRegionInfo region) { LOG.error(String.format("read from region %s failed", region.getRegionNameAsString())); } public void publishReadFailure(HRegionInfo region, HColumnDescriptor column) { LOG.error(String.format("read from region %s column family %s failed", region.getRegionNameAsString(), column.getNameAsString())); } public void publishReadTiming(HRegionInfo region, HColumnDescriptor column, long msTime) { LOG.info(String.format("read from region %s column family %s in %dms", region.getRegionNameAsString(), column.getNameAsString(), msTime)); } }
    • Canary Tool study //constructors public Canary() { this(new StdOutSink()); } public Canary(Sink sink) { this.sink = sink; }
    • Canary Tool in Circus Send mail if any Fix problem abnormal Canary-toolStart here Nagios Server Tm-puppet operation server hbase- Write to canary.log Read from /var/log/hbase/
    • Canary Tool in Circuscom.trendmicro.spn.ops.hbase.RunCanaryToolprivate static class CustomSink implements Canary.Sink { public void publishReadFailure(HRegionInfo regionInfo) { //... LOG.error(String.format("Read from table:%s, region:%s failed", tableName, regionName)); } public void publishReadFailure(HRegionInfo regionInfo, HColumnDescriptor colDescriptor) { //... LOG.error(String.format("Read from table:%s, region:%s, columnFamily:%sfailed", tableName, regionName, colFamilyName)); } public void publishReadTiming(HRegionInfo regionInfo, HColumnDescriptor colDescriptor, long msTime) { //... LOG.info(String.format("Read from table:%s, region:%s, columnFamily:%s in%dms", tableName, regionName, colFamilyName, msTime)); }}
    • Canary Tool in Circuscom.trendmicro.spn.ops.hbase.RunCanaryToolpublic static void main(String[] args) throws Exception { Canary canary = new Canary(new CustomSink()); int exitCode = ToolRunner.run(canary, args); System.exit(exitCode);} hbase-canary-monitor.sh su - hbase <<EOF kinit -kt /etc/hbase/conf/hbase.keytab hbase/$(hostname -f) java -cp $CLASSPATH com.trendmicro.spn.ops.hbase.RunCanaryTool $@ EOF