Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Presentation slide for HBase Meetup on the Night before Strata/HW @ Google in Chelsea 
Copyright © 2014 NTT DATA Corporati...
What is HTrace? 
https://github.com/cloudera/htrace 
 
Tracing tool for parallel distributed system 
like Google's Dapper ...
Tracing spans 
Span represents traced processing unit and time 
Spans have parent-‐‑‒child relationship 
Passing tracing i...
Starting root span 
org.htrace.Trace#startSpan create span 
new trace ID is assigned when root span starts 
FsShell shell ...
Starting passive span 
Starting child span only when there is parent 
For server-‐‑‒side custom tracing span 
 
if (Trace....
Passing tracing info along with RPC 
RPC header has optional field for tracing 
RPC with tracing info start span on server...
Span receivers 
Each process loads receiver module 
Receivers receive spans from in-‐‑‒process queue  
Receivers send span...
Passing tracing info between threads 
Ongoing tracing span is stored in ThreadLocal 
You need to pass tracing info between...
JIRAs 
Already available in 
HBase (HBASE-‐‑‒6449) 
HDFS (HDFS-‐‑‒5274) 
 
Working on 
YARN (YARN-‐‑‒1418) 
Copyright © 20...
Configurations 
Setting receiver class impl turns on tracing 
Each receiver impl has its own additional confs 
! 
property...
Tracing from HBase shell 
trace command start/stop tracing span 
# You need configuration on client node 
! 
$ hbase shell...
Example: Creating table in HBase 
Copyright © 2014 NTT DATA Corporation 12
Example: Tracing of putting 200MB of a file to HDFS 
Copyright © 2014 NTT DATA Corporation 13
Example: Tracing of getting 200MB of a file from HDFS 
Copyright © 2014 NTT DATA Corporation 14
Example: Zipkin UI 
Copyright © 2014 NTT DATA Corporation 15
Todo 
Adding granular tracing spans 
Sampling and filtering spans 
Dynamic reconfiguration (HDFS-‐‑‒6956) 
sink and viewer...
Copyright © 2011 NTT DATA Corporation 
Copyright © 2014 NTT DATA Corporation
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
distributed tracing in 5 minutes
Next
Download to read offline and view in fullscreen.

15

Share

Download to read offline

HTrace: Tracing in HBase and HDFS (HBase Meetup)

Download to read offline

HTrace: Tracing in HBase and HDFS
HBase Meetup on the Night before Strata/HW @ Google in Chelsea
2014/10/15

NTT DATA
Masatake Iwasaki (OSS Professional Services)

Related Books

Free with a 30 day trial from Scribd

See all

HTrace: Tracing in HBase and HDFS (HBase Meetup)

  1. 1. Presentation slide for HBase Meetup on the Night before Strata/HW @ Google in Chelsea Copyright © 2014 NTT DATA Corporation HTrace: Tracing in HBase and HDFS 10/15/2014 Masatake Iwasaki
  2. 2. What is HTrace? https://github.com/cloudera/htrace Tracing tool for parallel distributed system like Google's Dapper Effective for finding bottleneck Effective for code analysis Low Overhead Copyright © 2014 NTT DATA Corporation 2
  3. 3. Tracing spans Span represents traced processing unit and time Spans have parent-‐‑‒child relationship Passing tracing info along with RPC Time Span Atrace id: 12345 node 1parent: root Span Btrace id: 12345 RPC node 2parent: A RPCRPC Span Ctrace id: 12345Span D node 3parent: B parent: B trace id: 12345 Copyright © 2014 NTT DATA Corporation 3
  4. 4. Starting root span org.htrace.Trace#startSpan create span new trace ID is assigned when root span starts FsShell shell = new FsShell();! conf.setQuietMode(false);! shell.setConf(conf);! int res = 0;! SpanReceiverHost.getInstance(new HdfsConfiguration());! TraceScope ts = null;! try {! ts = Trace.startSpan(FsShell, Sampler.ALWAYS);! res = ToolRunner.run(shell, argv);! } finally {! shell.close();! if (ts != null) ts.close();! } Copyright © 2014 NTT DATA Corporation 4
  5. 5. Starting passive span Starting child span only when there is parent For server-‐‑‒side custom tracing span if (Trace.isTracing()) {! traceScope = Trace.startSpan(method.getName());! } Copyright © 2014 NTT DATA Corporation 5
  6. 6. Passing tracing info along with RPC RPC header has optional field for tracing RPC with tracing info start span on server-‐‑‒side message RequestHeader {! optional uint32 call_id = 1;! optional RPCTInfo trace_info = 2;! optional string method_name = 3;! ...! }! ! message RPCTInfo {! optional int64 trace_id = 1;! optional int64 parent_id = 2;! }! Copyright © 2014 NTT DATA Corporation 6
  7. 7. Span receivers Each process loads receiver module Receivers receive spans from in-‐‑‒process queue Receivers send spans to collector asynchronously SpanReceiver Server SpanReceiver RPC SpanReceiver Client Server Collector/Sink RPC Tracing Spans Copyright © 2014 NTT DATA Corporation 7
  8. 8. Passing tracing info between threads Ongoing tracing span is stored in ThreadLocal You need to pass tracing info between threads if (header.hasTraceInfo()) {! // If the incoming RPC included tracing info, always continue the trace TraceInfo parentSpan = new TraceInfo(header.getTraceInfo().getTraceId(),! header.getTraceInfo().getParentId());! traceSpan = Trace.startSpan(rpcRequest.toString(), parentSpan).detach();! }! Call call = new Call(header.getCallId(), header.getRetryCount(),! rpcRequest, this, ProtoUtil.convert(header.getRpcKind()),! header.getClientId().toByteArray(), traceSpan);! ! ...! ! if (call.traceSpan != null) {! traceScope = Trace.continueSpan(call.traceSpan);! }! Copyright © 2014 NTT DATA Corporation 8
  9. 9. JIRAs Already available in HBase (HBASE-‐‑‒6449) HDFS (HDFS-‐‑‒5274) Working on YARN (YARN-‐‑‒1418) Copyright © 2014 NTT DATA Corporation 9
  10. 10. Configurations Setting receiver class impl turns on tracing Each receiver impl has its own additional confs ! property! namehbase.trace.spanreceiver.classes/name! valueorg.htrace.impl.HBaseSpanReceiver/value! /property ! property! namehbase.htrace.hbase.collector-quorum/name! value127.0.0.1/value! /property Copyright © 2014 NTT DATA Corporation 10
  11. 11. Tracing from HBase shell trace command start/stop tracing span # You need configuration on client node ! $ hbase shell! trace 'start'! create 'test', 'f'! trace 'stop' Copyright © 2014 NTT DATA Corporation 11
  12. 12. Example: Creating table in HBase Copyright © 2014 NTT DATA Corporation 12
  13. 13. Example: Tracing of putting 200MB of a file to HDFS Copyright © 2014 NTT DATA Corporation 13
  14. 14. Example: Tracing of getting 200MB of a file from HDFS Copyright © 2014 NTT DATA Corporation 14
  15. 15. Example: Zipkin UI Copyright © 2014 NTT DATA Corporation 15
  16. 16. Todo Adding granular tracing spans Sampling and filtering spans Dynamic reconfiguration (HDFS-‐‑‒6956) sink and viewer with less dependency Copyright © 2014 NTT DATA Corporation 16
  17. 17. Copyright © 2011 NTT DATA Corporation Copyright © 2014 NTT DATA Corporation
  • debugger87

    Feb. 7, 2017
  • rajakasturi

    Sep. 9, 2016
  • KenichiTakahashi1

    Jan. 11, 2016
  • tthtlc

    Aug. 20, 2015
  • dengjinbo

    Jul. 31, 2015
  • nagasama

    Jun. 15, 2015
  • psyoblade

    Mar. 4, 2015
  • irtiger

    Feb. 28, 2015
  • hypermin

    Feb. 3, 2015
  • seungbumlee79

    Feb. 3, 2015
  • linekin

    Jan. 14, 2015
  • bingqiangsong

    Dec. 10, 2014
  • ruo91

    Nov. 5, 2014
  • takamasatanaka9

    Oct. 23, 2014
  • bunkertor

    Oct. 21, 2014

HTrace: Tracing in HBase and HDFS HBase Meetup on the Night before Strata/HW @ Google in Chelsea 2014/10/15 NTT DATA Masatake Iwasaki (OSS Professional Services)

Views

Total views

15,653

On Slideshare

0

From embeds

0

Number of embeds

4,599

Actions

Downloads

85

Shares

0

Comments

0

Likes

15

×