The document provides an overview of HTrace, a tracing tool for distributed systems like HBase and HDFS, effective in identifying bottlenecks and enhancing code analysis with low overhead. It describes the structure of tracing spans and their relationships, as well as how to implement and configure tracing for HBase and HDFS. Additionally, it includes examples of tracing operations and outlines future enhancements for the tool.

1. Presentation slide for HBase Meetup on the Night before Strata/HW @ Google in Chelsea
Copyright © 2014 NTT DATA Corporation
HTrace:
Tracing in HBase and HDFS
10/15/2014
Masatake Iwasaki

2. What is HTrace?
https://github.com/cloudera/htrace
Tracing tool for parallel distributed system
like Google's Dapper
Effective for finding bottleneck
Effective for code analysis
Low Overhead
Copyright © 2014 NTT DATA Corporation 2

3. Tracing spans
Span represents traced processing unit and time
Spans have parent-‐‑‒child relationship
Passing tracing info along with RPC
Time
Span Atrace id: 12345
node 1parent: root
Span Btrace id: 12345
RPC
node 2parent: A
RPCRPC
Span Ctrace id: 12345Span D
node 3parent: B
parent: B
trace id: 12345
Copyright © 2014 NTT DATA Corporation 3

4. Starting root span
org.htrace.Trace#startSpan create span
new trace ID is assigned when root span starts
FsShell shell = new FsShell();!
conf.setQuietMode(false);!
shell.setConf(conf);!
int res = 0;!
SpanReceiverHost.getInstance(new HdfsConfiguration());!
TraceScope ts = null;!
try {!
ts = Trace.startSpan(FsShell, Sampler.ALWAYS);!
res = ToolRunner.run(shell, argv);!
} finally {!
shell.close();!
if (ts != null) ts.close();!
}
Copyright © 2014 NTT DATA Corporation 4

5. Starting passive span
Starting child span only when there is parent
For server-‐‑‒side custom tracing span
if (Trace.isTracing()) {!
traceScope = Trace.startSpan(method.getName());!
}
Copyright © 2014 NTT DATA Corporation 5

6. Passing tracing info along with RPC
RPC header has optional field for tracing
RPC with tracing info start span on server-‐‑‒side
message RequestHeader {!
optional uint32 call_id = 1;!
optional RPCTInfo trace_info = 2;!
optional string method_name = 3;!
...!
}!
!
message RPCTInfo {!
optional int64 trace_id = 1;!
optional int64 parent_id = 2;!
}!
Copyright © 2014 NTT DATA Corporation 6

7. Span receivers
Each process loads receiver module
Receivers receive spans from in-‐‑‒process queue
Receivers send spans to collector asynchronously
SpanReceiver
Server
SpanReceiver
RPC
SpanReceiver
Client
Server
Collector/Sink
RPC
Tracing Spans
Copyright © 2014 NTT DATA Corporation 7

8. Passing tracing info between threads
Ongoing tracing span is stored in ThreadLocal
You need to pass tracing info between threads
if (header.hasTraceInfo()) {!
// If the incoming RPC included tracing info, always continue the trace TraceInfo parentSpan = new TraceInfo(header.getTraceInfo().getTraceId(),!
header.getTraceInfo().getParentId());!
traceSpan = Trace.startSpan(rpcRequest.toString(), parentSpan).detach();!
}!
Call call = new Call(header.getCallId(), header.getRetryCount(),!
rpcRequest, this, ProtoUtil.convert(header.getRpcKind()),!
header.getClientId().toByteArray(), traceSpan);!
!
...!
!
if (call.traceSpan != null) {!
traceScope = Trace.continueSpan(call.traceSpan);!
}!
Copyright © 2014 NTT DATA Corporation 8

9. JIRAs
Already available in
HBase (HBASE-‐‑‒6449)
HDFS (HDFS-‐‑‒5274)
Working on
YARN (YARN-‐‑‒1418)
Copyright © 2014 NTT DATA Corporation 9

10. Configurations
Setting receiver class impl turns on tracing
Each receiver impl has its own additional confs
!
property!
namehbase.trace.spanreceiver.classes/name!
valueorg.htrace.impl.HBaseSpanReceiver/value!
/property !
property!
namehbase.htrace.hbase.collector-quorum/name!
value127.0.0.1/value!
/property
Copyright © 2014 NTT DATA Corporation 10

11. Tracing from HBase shell
trace command start/stop tracing span
# You need configuration on client node
!
$ hbase shell!
trace 'start'!
create 'test', 'f'!
trace 'stop'
Copyright © 2014 NTT DATA Corporation 11

12. Example: Creating table in HBase
Copyright © 2014 NTT DATA Corporation 12

13. Example: Tracing of putting 200MB of a file to HDFS
Copyright © 2014 NTT DATA Corporation 13

14. Example: Tracing of getting 200MB of a file from HDFS
Copyright © 2014 NTT DATA Corporation 14

15. Example: Zipkin UI
Copyright © 2014 NTT DATA Corporation 15

16. Todo
Adding granular tracing spans
Sampling and filtering spans
Dynamic reconfiguration (HDFS-‐‑‒6956)
sink and viewer with less dependency
Copyright © 2014 NTT DATA Corporation 16

17. Copyright © 2011 NTT DATA Corporation
Copyright © 2014 NTT DATA Corporation