HBase Canary Tool2012/10/22 Scott Miao
Another way to monitor HBase processes  • org.apache.hadoop.hbase.tool.Canary     – Be used to do "canary monitoring" of a...
Canary Tool studyprivate void sniff() throws Exception {   for (HTableDescriptor table : admin.listTables()) {     sniff(t...
Canary Tool studyprivate void sniffRegion(HRegionInfo region, HTable table) throws Exception {   HTableDescriptor tableDes...
Canary Tool studypublic interface Sink {   public void publishReadFailure(HRegionInfo region);   public void publishReadFa...
Canary Tool study    //constructors    public Canary() {        this(new StdOutSink());      }     public Canary(Sink sink...
Canary Tool in Circus                                                   Send mail if any                        Fix proble...
Canary Tool in Circuscom.trendmicro.spn.ops.hbase.RunCanaryToolprivate static class CustomSink implements Canary.Sink { pu...
Canary Tool in Circuscom.trendmicro.spn.ops.hbase.RunCanaryToolpublic static void main(String[] args) throws Exception {  ...
Upcoming SlideShare
Loading in...5
×

20121022 tm hbasecanarytool

625

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
625
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

20121022 tm hbasecanarytool

  1. 1. HBase Canary Tool2012/10/22 Scott Miao
  2. 2. Another way to monitor HBase processes • org.apache.hadoop.hbase.tool.Canary – Be used to do "canary monitoring" of a running HBase cluster. – For each region tries to get one row per column family and outputs some information about failure or latencyUsage: bin/hbase org.apache.hadoop.hbase.tool.Canary [opts] [table 1 [table 2...]]where [opts] are: -help Show this help and exit. -daemon Continuous check at defined intervals. # 6sec -interval <N> Interval between checks (sec) # specify how many secs you want https://issues.apache.org/jira/browse/HBASE-4393
  3. 3. Canary Tool studyprivate void sniff() throws Exception { for (HTableDescriptor table : admin.listTables()) { sniff(table); } } private void sniff(HTableDescriptor tableDesc) throws Exception { HTable table = null; try { table = new HTable(admin.getConfiguration(), tableDesc.getName()); } catch (TableNotFoundException e) { return; } for (HRegionInfo region : admin.getTableRegions(tableDesc.getName())) { try { sniffRegion(region, table); } catch (Exception e) { sink.publishReadFailure(region); } } }
  4. 4. Canary Tool studyprivate void sniffRegion(HRegionInfo region, HTable table) throws Exception { HTableDescriptor tableDesc = table.getTableDescriptor(); for (HColumnDescriptor column : tableDesc.getColumnFamilies()) { Get get = new Get(region.getStartKey()); get.addFamily(column.getName()); try { long startTime = System.currentTimeMillis(); table.get(get); long time = System.currentTimeMillis() - startTime; sink.publishReadTiming(region, column, time); } catch (Exception e) { sink.publishReadFailure(region, column); } } }
  5. 5. Canary Tool studypublic interface Sink { public void publishReadFailure(HRegionInfo region); public void publishReadFailure(HRegionInfo region, HColumnDescriptor column); public void publishReadTiming(HRegionInfo region, HColumnDescriptorcolumn, long msTime); } public static class StdOutSink implements Sink { public void publishReadFailure(HRegionInfo region) { LOG.error(String.format("read from region %s failed", region.getRegionNameAsString())); } public void publishReadFailure(HRegionInfo region, HColumnDescriptor column) { LOG.error(String.format("read from region %s column family %s failed", region.getRegionNameAsString(), column.getNameAsString())); } public void publishReadTiming(HRegionInfo region, HColumnDescriptor column, long msTime) { LOG.info(String.format("read from region %s column family %s in %dms", region.getRegionNameAsString(), column.getNameAsString(), msTime)); } }
  6. 6. Canary Tool study //constructors public Canary() { this(new StdOutSink()); } public Canary(Sink sink) { this.sink = sink; }
  7. 7. Canary Tool in Circus Send mail if any Fix problem abnormal Canary-toolStart here Nagios Server Tm-puppet operation server hbase- Write to canary.log Read from /var/log/hbase/
  8. 8. Canary Tool in Circuscom.trendmicro.spn.ops.hbase.RunCanaryToolprivate static class CustomSink implements Canary.Sink { public void publishReadFailure(HRegionInfo regionInfo) { //... LOG.error(String.format("Read from table:%s, region:%s failed", tableName, regionName)); } public void publishReadFailure(HRegionInfo regionInfo, HColumnDescriptor colDescriptor) { //... LOG.error(String.format("Read from table:%s, region:%s, columnFamily:%sfailed", tableName, regionName, colFamilyName)); } public void publishReadTiming(HRegionInfo regionInfo, HColumnDescriptor colDescriptor, long msTime) { //... LOG.info(String.format("Read from table:%s, region:%s, columnFamily:%s in%dms", tableName, regionName, colFamilyName, msTime)); }}
  9. 9. Canary Tool in Circuscom.trendmicro.spn.ops.hbase.RunCanaryToolpublic static void main(String[] args) throws Exception { Canary canary = new Canary(new CustomSink()); int exitCode = ToolRunner.run(canary, args); System.exit(exitCode);} hbase-canary-monitor.sh su - hbase <<EOF kinit -kt /etc/hbase/conf/hbase.keytab hbase/$(hostname -f) java -cp $CLASSPATH com.trendmicro.spn.ops.hbase.RunCanaryTool $@ EOF
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×