SlideShare a Scribd company logo
1 of 40
Download to read offline
hosted by
HBaseConAsia2018
HBase and OpenTSDB Practices
at Huawei
Pankaj Kumar, Zhongchaoqiang, Guoyijun, Zhiwei
{Pankaj.kr, zhongchaoqiang, guoyijun, wei.zhi}@huawei.com
hosted by
HBase Tech Lead @ Huawei India
Apache HBase Contributor
5 years of experience in Big Data related projects
$ whoami
3
HBase @ Huawei
Migrated from 1.0.2 version
1.3.1 version +
 Secondary index
 MOB
 Multi split
Migrating to 2.1.x cluster this year
Content
01
02
HBase Practices
OpenTSDB Practices
Accelerate HMaster Startup
Enhanced Replication
Reliable Region Assignment
HBase Practices
1.1 Accelerate HMaster Startup
7
Accelerate HMaster Startup
Problem:
HMaster not available for longer duration on failover/restart
Deployment scenario:
 Large cluster with 500+ nodes
 5000+ Tables and 120000 + regions
 10 namespaces
Discovered problems in multiple areas in Master startups like below
 Slow region locality computation on startup
 Serial region locality calculation
 Too much time spent in region locality calculation
 HMaster aborting due to namespace initialization failure
 Slow SSH/SCP
 Similar to HBASE-14190
 Table info loading was taking too much time
 High Namenode latency
 Many other services creating lots of load in NN
8
Accelerate HMaster Startup
 Slow region locality computation on startup
 Accelerate region Locality computation by computing in parallel
 Detach region locality computation on startup
 Similar solution HBASE-16570
 HMaster aborting due to namespace initialization failure
 Assign system table regions ahead of user table regions
 Assign system tables to HMaster (configure all system tables to
hbase.balancer.tablesOnMaster)
 On cluster/master startup, process old HMaster SSH/SCP ahead of other
RegionServer
 SSH/SCP will replay the WAL and assign the system table regions first
9
Accelerate HMaster Startup
 Table Info Loading on Master startup
Namespace HDFS Path
default t1/.tabledesc/.tableinfo.0000000001
t1/.tabledesc/.tableinfo.0000000002
t1/.tabledesc/.tableinfo.0000000003
t2/.tabledesc/.tableinfo.0000000001
hbase /hbase/data/hbase/acl/.tabledesc/.tableinfo.0000000001
/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
/hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001
Operation Path Result Total RPC to NN
List operation to get path till
all the namespace
/hbase/data/* gets file status for all the
namespaces
1
List operation on each
namespace to get all the
tables in each namespace
/hbase/data/default get all the file status of
all the tables in each
namespace
2 ( = total number of
namespaces in the
cluster)
List operation on each table
to get all the tableinfo files
for the table
/hbase/data/default/
t1/. tabledesc
get all the file status of
all the tableinfo files for
the table
5 (= total number of
tables in the cluster)
Open call for each table’s
latest tableinfo file
/hbase/data/default/
t1/.
tabledesc/.tableinfo.
0000000003
get the stream to
tableinfo file
5 (= total number of
tables in the cluster)
Total RPC 13
Example: Suppose there are two
namespace and total 5 tables with
below path structure in HDFS
Total RPC to NameNode
=
1 + namespace count
+ 2 * table count
10
Accelerate HMaster Startup
Table Info Loading on Master startup
 2011 RPC Calls ( for 10 namespace and 1000 tables in a cluster)
 NN is busy then it will hugely impact the startup of HMaster.
Solution: Reduce number of RPC to Namenode
 HMaster makes a single call to get tableinfo path
 Get LocatedFileStatus of all tableinfo paths based on pattern
(/hbase/data/*/*/.tabledesc/.tableinfo* )
 LocatedFileStatus will also contain block locations of tableinfo file along with
FileStatus details
 DFS client will directly connect to Datanode through FileSystem#open() using
LocatedFileStatus, avoid NN RPC to get the block location of the tableinfo file
Improvement:
In a highly overloaded HDFS cluster, it took around 97 seconds to load 5000
tables info as compared to 224 seconds earlier.
1.2 Enhanced Replication
12
Adaptive call timeout
Problem:
 Replication may timeout when peer cluster is not able to replicate the
entries
 Can be solved by increasing hbase.rpc.timeout at source cluster
 Impact other RPC request
 In Bulkload replication scenario fixed RPC timeout may not
guarantee bigger HFile copy
 Refer HBASE-14937
Solution:
 Source cluster should wait longer
 New configuration parameter hbase.replication.rpc.timeout, default
value will be same as hbase.rpc.timeout
 On each CallTimeOutException increase this replication timeout value
by fixed multiplier
 Increase the replication to certain number of configured times
13
Cross realm support
Problem:
 Replication doesn’t work with Kerberos Cross Real Trust where
principal domain name is not machine hostname
 On new host addition
 Add principal name for the newly added hosts in KDC
 Generate a new keytab file
 Update it across other hosts
 Rigorous task for user to create and replace new keytab file
Solution:
 HBASE- 14866
 Configure the peer cluster principal in replication peer config
 Refer to HBASE-15254 (Open)
 No need to configure in advance, fetch at runtime.
 Make RPC call to peer HBase cluster and fetch the Principal
 Make RPC connection based on this server principal
1.3 Reliable Region Assignment
15
RIT Problem
Problem:
 Region stuck in transition for longer duration due to some fault in cluster
 Zookeeper node version mismatch
 Slow RS response
 Unstable Network
 Client will not be able to perform read/write operation on those regions which are in transition
 Balancer will not run
 Region can’t be recovered until cluster restart
Solution:
 Recover the regions by reassigning them
 Schedule a chore service
 Run periodically and identify the region which stuck in transition from a longer duration (configurable
threshold)
 Recover the region by reassigning them
 New HBCK command to recover regions which are in transition from longer duration
16
Double Assignment Problem
Problem:
 HMaster may assign region to multiple RegionServer in a faulty cluster environment
 Call time out from a overloaded RegionServer
 Old or new client may receive inconsistent data
 Can’t be recovered until cluster restart
Solution:
 Multiply assigned regions should be closed and assign uniquely
 Region server send server load details to HMaster through heartbeat
 Schedule a chore service which run periodically and recover the regions
 Collect each region server load from HMaster memory
 Identify the duplicate regions from the region list
 Validate the duplicate regions with HMaster Assignment Manager in-memory region state
 Close the region from the old region server
 Assign the region
17
Double Assignment Problem
Example:
HMaster
AM’s in-memory region state
r1:RS1 r2:RS3 r2:RS3 r4:RS1 r5:RS2 r6:RS3 r7:RS1 r8:RS2
Double Assignment Recovery Chore
Found region r2 s multiply assigned to RS2 & RS3, so r2 will be closed from RS2 as per AM’s in memory
state
Region Server (RS1)
r1, r4, r7
heartbeatheartbeat heartbeat
Region Server (RS2)
r2, r5, r8
Region Server (RS3)
r3, r6, r2
OpenTSDB Basics
OpenTSDB Improvement
OpenTSDB Improvement
2.1 TSDB Basics
20
Time Series
0
1
2
3
4
5
6
7
8
1 2 3 4 5 6
XX变化曲线图
风力 温度 水位
21
Time Series
……
……
……
Time SeriesData Point
A time series is a series of numeric data points
of some particular metric over time.
- OpenTSDB Document
22
OpenTSDB Schema
sys.cpu.user host=webserver01,cpu=0 1533640130 20
sys.cpu.user host=webserver01,cpu=0 1533640140 25
sys.cpu.user host=webserver01,cpu=0 1533640150 30
sys.cpu.user host=webserver01,cpu=0 1533640160 32
sys.cpu.user host=webserver01,cpu=0 1533640170 35
sys.cpu.user host=webserver01,cpu=0 1533640180 40
metric name tags timestamp
valu
e
OpenTSDB uses a metric name and a group of tags
for identifying time series. Tags are used to identify
different data sources.
23
TSDB Characteristics
• Write Dominate. Read rate is usually a
couple orders of magnitude lower.
• Most queries happens on latest data.
• Most queries are for aggregate analysis
instead of individual data point.
• Primarily Inserts. Updates/deletions are
rarely happens.
24
Basic Functionality For TSDB
• Rollups and Downsampling
• Pre-aggregates and Aggregates
• Interpolation
• Data Life Cycle Management
25
时序数据库单值 vs.多值
Metric TimeStamp DeviceID DeviceType ZoneId Temperature Pressure WaterLine
Engine
20180101 12:00:00 ID001 TypeA 1 66.9 1.33 42.5
20180101 12:00:00 ID002 TypeA 1 68.8 1.28 42.0
20180101 12:00:00 ID003 TypeA 1 67.3 1.35 41.7
20180101 12:01:00 ID001 TypeA 1 67.5 1.30 42.2
Metric TimeStamp DeviceID DeviceType ZoneId Value
Temperature 20180101 12:00:00 ID001 TypeA 1 66.9
Pressure 20180101 12:00:00 ID001 TypeA 1 1.33
WaterLine 20180101 12:00:00 ID001 TypeA 1 42.5
Temperature 20180101 12:01:00 ID002 TypeA 1 68.8
Pressure 20180101 12:01:00 ID002 TypeA 1 1.28
WaterLine 20180101 12:01:00 ID002 TypeA 1 42.0
Tags Field
Tags
Time stampMetric
Time stampMetric Metric Value
Single value model vs. Multi-value model
26
Time Series Storage In HBase
KeyValue KeyValue KeyValue KeyValue KeyValue KeyValue
DataPoint
(T1)
DataPoint
(T7)
Time Series A (20180808-10) (Writing Block)
KeyValue
Compacted DataPoints
Time Series A (20180808-
09)
KeyValue
Compacted DataPoints
Time Series A (20180808-
08)
KeyValue
Compacted DataPoints
Time Series A (20180808-07)
Time Series Separeted into multiple blocks. Each block hold
one hour of data points.
DataPoint
(T2)
DataPoint(T3) DataPoint(T4) DataPoint
(T5)
DataPoint
(T6)
Closed Blocks
27
OpenTSDB Table Design
RowKey Format
Qualifier Format
1BYTE 3BYTES 4BYTES 3BYTES 3BYTES 3BYTES 3BYTES
SALT Metric ID Timestamp Tag Name ID Tag Value ID Tag Name ID Tag Value ID
1 <= N <= 8
2 BYTES
Timestamp ValueLengthValueType
KeyValue KeyValue KeyValue KeyValue KeyValue KeyValue
DataPoint
(T1)
Time Series A (20180808-10) (Writing Block)
DataPoint
(T2)
DataPoint(T3) DataPoint(T4) DataPoint
(T5)
DataPoint(T6)
28
OpenTSDB Compaction
KeyValue KeyValue KeyValue KeyValue KeyValue KeyValue
DataPoint(T1) DataPoint
(T2)
DataPoint
(T3)
DataPoint
(T4)
DataPoint(T5) DataPoint(T6)
KeyValue
DataPoint(TX)
KeyValue KeyValue KeyValue KeyValue KeyValue KeyValue
DataPoint
(T1)
DataPoint(T2) DataPoint(T3) DataPoint
(T4)
DataPoint
(T5)
DataPoint
(T6)
KeyValue
DataPoint(TX)
1. Read All Data Points from the block of Last Hour
2. Compact locally
KeyValue
DataPoints of whole block
3. Write compact row, and delete all exist individual data points
KeyValue
DataPoints of whole block
Delete Marker
2.1 OpenTSDB Improvement
30
Exist OpenTSDB Compaction Flow
OpenTSDB compaction is
helpful for read, and could
decrease total data amount,
but the side effects as follows:
1. OpenTSDB Compaction
requires a read/compact/write
cycle, causing extremely high
traffic to RegionServers.
2. Write compact row and
delete exist individual data
points amplify write I/O.
CompactionQueue
Metric1_Hour1
Metric2_Hour1
Metric3_Hour1
Metric1_Hour2
Metric2_Hour2
Metric3_Hour2
Metric1_Hour3
Metric2_Hour3
Metric3_Hour3
Put
Read and
Remove
Put and Delete
Get
Add RowKey
OpenTSDB
Compact
Logic
HTTPHandler
Compaction
Thread
HBase
TSD
31
HFileHFile
HFile
HFile MemStore
HFile MemStore
Client(TSD)


 1. TSD client read time series data
from Regionserver.
2. TSD write compacted row and
delete marker to RegionServer.
3. HBase internal compaction.
Understanding Write Amplification
Client(TSD)
Single Datapoint Delete Marker Compact Row
32
New OpenTSDB Compaction Flow
New HBase Compaction
implementation for OpenTSDB
that could compact data points
during hbase compaction
running.
Making TSD focus on handing
user read/write requests only.
OpenTSDB
OpenTSDB
Compact
Logic
Read
One Row
Write
Compact
Rows
HBase
Put
Compaction
Thread
HTTP Handler
Region1
Region2
Region3
RegionN
........
HFile
HFile
HFile
New
HFile
33
No More Extra Write Amplification
HFile
HFile MemStore
Client(TSD)

HFile HFile
Compaction
No more extra write
amplication caused by
OpenTSDB data points
compaction.
34
NOTE: TSDs were limited to 300,000 data points per second.
After optimization, write throughtput has been improved
significantly.
Benchmark – Throughput comparison
35
Benchmark – CPU And IO Comparison
36
 Delete old data automatically for
reduce data load.
 HBase Table level TTL is a coarse-
grained mechanism, But different
metrics may have different TTL
requirements.
 A new HBase compaction
implementation for per-metric
level data life cycle management.
Data Life Cycle Management Per Metric
OpenTSDB
OpenTSDB
TTL
Logic
Read One
Row
Write
Unexpired
Rows
HBase
Put
Compaction
Thread
HTTP Handler
Region1
Region2
Region3
RegionN
........
HFile
HFile
HFile
New
HFile
37
Using a two-level thread model design. Receive message, process
message and response to client are all handled by one WorkThread. It
causes the low CPU usage.
TSD
workerThread
RegionServer1WorkerThread
RegionServer2
RegionServer3
workerThread
OpenTSDB RPC Thread Model
Receive Msg
Process Msg
Response Msg
38
Benchmark:
Query latency got at least 3X
improvement for concurrent
queries.
Before After
1 Query 60ms 59ms
10 Concurrent Queries 476ms 135ms
50 Concurrent Queries 2356ms 680ms
Modify the thread model to a Three-Level design. Receiving message and
handling message are finished in different threads. Better CPU usage.
RPC Thread Model Improvement
Boss
Thread
Read Thread
Read Thread
Read Thread
Queue
Worker Thread
write read
Worker Thread
Worker Thread
... ...
Boss
Thread
Worker Thread
Worker Thread
Worker Thread
...
39
Follow Us(关注我们)
微信公众号华为云CloudTable
NoSQL漫谈公有云HBase服务
Copyright©2018 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including, without
limitation, statements regarding the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that could cause actual
results and developments to differ materially from those expressed or implied in the
predictive statements. Therefore, such information is provided for reference purpose
only and constitutes neither an offer nor an acceptance. Huawei may change the
information at any time without notice.
Thank You.

More Related Content

What's hot

HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
HBaseCon 2012 | Solbase - Kyungseog Oh, PhotobucketHBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
HBaseCon 2012 | Solbase - Kyungseog Oh, PhotobucketCloudera, Inc.
 
HBaseConAsia2018 Track3-2: HBase at China Telecom
HBaseConAsia2018 Track3-2:  HBase at China TelecomHBaseConAsia2018 Track3-2:  HBase at China Telecom
HBaseConAsia2018 Track3-2: HBase at China TelecomMichael Stack
 
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - ClouderaHBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - ClouderaCloudera, Inc.
 
Time-Series Apache HBase
Time-Series Apache HBaseTime-Series Apache HBase
Time-Series Apache HBaseHBaseCon
 
HBaseCon 2013: How to Get the MTTR Below 1 Minute and More
HBaseCon 2013: How to Get the MTTR Below 1 Minute and MoreHBaseCon 2013: How to Get the MTTR Below 1 Minute and More
HBaseCon 2013: How to Get the MTTR Below 1 Minute and MoreCloudera, Inc.
 
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQL
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQLHBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQL
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQLCloudera, Inc.
 
HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon
 
HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...
HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...
HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...Michael Stack
 
HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment
HBaseCon 2015: HBase at Scale in an Online and  High-Demand EnvironmentHBaseCon 2015: HBase at Scale in an Online and  High-Demand Environment
HBaseCon 2015: HBase at Scale in an Online and High-Demand EnvironmentHBaseCon
 
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsightOptimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsightHBaseCon
 
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...Cloudera, Inc.
 
HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase
HBaseCon 2015: Blackbird Collections - In-situ  Stream Processing in HBaseHBaseCon 2015: Blackbird Collections - In-situ  Stream Processing in HBase
HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBaseHBaseCon
 
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
HBaseCon 2012 | Building a Large Search Platform on a Shoestring BudgetHBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
HBaseCon 2012 | Building a Large Search Platform on a Shoestring BudgetCloudera, Inc.
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程HBaseCon
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsSupporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsDataWorks Summit
 
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...Cloudera, Inc.
 
Voldemort on Solid State Drives
Voldemort on Solid State DrivesVoldemort on Solid State Drives
Voldemort on Solid State DrivesVinoth Chandar
 
HBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a FlurryHBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a FlurryHBaseCon
 
HBaseCon 2012 | HBase, the Use Case in eBay Cassini
HBaseCon 2012 | HBase, the Use Case in eBay Cassini HBaseCon 2012 | HBase, the Use Case in eBay Cassini
HBaseCon 2012 | HBase, the Use Case in eBay Cassini Cloudera, Inc.
 

What's hot (20)

HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
HBaseCon 2012 | Solbase - Kyungseog Oh, PhotobucketHBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
 
HBaseConAsia2018 Track3-2: HBase at China Telecom
HBaseConAsia2018 Track3-2:  HBase at China TelecomHBaseConAsia2018 Track3-2:  HBase at China Telecom
HBaseConAsia2018 Track3-2: HBase at China Telecom
 
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - ClouderaHBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
 
Time-Series Apache HBase
Time-Series Apache HBaseTime-Series Apache HBase
Time-Series Apache HBase
 
HBaseCon 2013: How to Get the MTTR Below 1 Minute and More
HBaseCon 2013: How to Get the MTTR Below 1 Minute and MoreHBaseCon 2013: How to Get the MTTR Below 1 Minute and More
HBaseCon 2013: How to Get the MTTR Below 1 Minute and More
 
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQL
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQLHBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQL
HBaseCon 2013: How (and Why) Phoenix Puts the SQL Back into NoSQL
 
HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBase
 
HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...
HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...
HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...
 
HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment
HBaseCon 2015: HBase at Scale in an Online and  High-Demand EnvironmentHBaseCon 2015: HBase at Scale in an Online and  High-Demand Environment
HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment
 
Enterprise Grade Streaming under 2ms on Hadoop
Enterprise Grade Streaming under 2ms on HadoopEnterprise Grade Streaming under 2ms on Hadoop
Enterprise Grade Streaming under 2ms on Hadoop
 
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsightOptimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
 
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...
HBaseCon 2012 | HBase Coprocessors – Deploy Shared Functionality Directly on ...
 
HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase
HBaseCon 2015: Blackbird Collections - In-situ  Stream Processing in HBaseHBaseCon 2015: Blackbird Collections - In-situ  Stream Processing in HBase
HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase
 
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
HBaseCon 2012 | Building a Large Search Platform on a Shoestring BudgetHBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsSupporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
 
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
 
Voldemort on Solid State Drives
Voldemort on Solid State DrivesVoldemort on Solid State Drives
Voldemort on Solid State Drives
 
HBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a FlurryHBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a Flurry
 
HBaseCon 2012 | HBase, the Use Case in eBay Cassini
HBaseCon 2012 | HBase, the Use Case in eBay Cassini HBaseCon 2012 | HBase, the Use Case in eBay Cassini
HBaseCon 2012 | HBase, the Use Case in eBay Cassini
 

Similar to HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei

Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsSupporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsDataWorks Summit
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in PinterestHBaseCon
 
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon
 
Domain Name Server
Domain Name ServerDomain Name Server
Domain Name Servervipulvaid
 
IEEE SRDS'12: From Backup to Hot Standby: High Availability for HDFS
IEEE SRDS'12: From Backup to Hot Standby: High Availability for HDFSIEEE SRDS'12: From Backup to Hot Standby: High Availability for HDFS
IEEE SRDS'12: From Backup to Hot Standby: High Availability for HDFSAndré Oriani
 
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay RadiaApache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay RadiaYahoo Developer Network
 
Data correlation using PySpark and HDFS
Data correlation using PySpark and HDFSData correlation using PySpark and HDFS
Data correlation using PySpark and HDFSJohn Conley
 
Apache HBase 1.0 Release
Apache HBase 1.0 ReleaseApache HBase 1.0 Release
Apache HBase 1.0 ReleaseNick Dimiduk
 
Spinnaker VLDB 2011
Spinnaker VLDB 2011Spinnaker VLDB 2011
Spinnaker VLDB 2011sandeep_tata
 
06 coms 525 tcpip - dhcp and dns
06   coms 525 tcpip - dhcp and dns06   coms 525 tcpip - dhcp and dns
06 coms 525 tcpip - dhcp and dnsPalanivel Kuppusamy
 
NonStop Hadoop - Applying the PaxosFamily of Protocols to make Critical Hadoo...
NonStop Hadoop - Applying the PaxosFamily of Protocols to make Critical Hadoo...NonStop Hadoop - Applying the PaxosFamily of Protocols to make Critical Hadoo...
NonStop Hadoop - Applying the PaxosFamily of Protocols to make Critical Hadoo...DataWorks Summit
 
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibabahbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at AlibabaMichael Stack
 
HBase tales from the trenches
HBase tales from the trenchesHBase tales from the trenches
HBase tales from the trencheswchevreuil
 
Nov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars georgeNov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars georgeO'Reilly Media
 

Similar to HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei (20)

Dns
DnsDns
Dns
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsSupporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
 
Introduction
IntroductionIntroduction
Introduction
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
 
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
 
6425 b 10
6425 b 106425 b 10
6425 b 10
 
Domain Name Server
Domain Name ServerDomain Name Server
Domain Name Server
 
IEEE SRDS'12: From Backup to Hot Standby: High Availability for HDFS
IEEE SRDS'12: From Backup to Hot Standby: High Availability for HDFSIEEE SRDS'12: From Backup to Hot Standby: High Availability for HDFS
IEEE SRDS'12: From Backup to Hot Standby: High Availability for HDFS
 
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay RadiaApache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
 
70 640
70 64070 640
70 640
 
Data correlation using PySpark and HDFS
Data correlation using PySpark and HDFSData correlation using PySpark and HDFS
Data correlation using PySpark and HDFS
 
Apache HBase 1.0 Release
Apache HBase 1.0 ReleaseApache HBase 1.0 Release
Apache HBase 1.0 Release
 
Spinnaker VLDB 2011
Spinnaker VLDB 2011Spinnaker VLDB 2011
Spinnaker VLDB 2011
 
Session_2.ppt
Session_2.pptSession_2.ppt
Session_2.ppt
 
06 coms 525 tcpip - dhcp and dns
06   coms 525 tcpip - dhcp and dns06   coms 525 tcpip - dhcp and dns
06 coms 525 tcpip - dhcp and dns
 
NonStop Hadoop - Applying the PaxosFamily of Protocols to make Critical Hadoo...
NonStop Hadoop - Applying the PaxosFamily of Protocols to make Critical Hadoo...NonStop Hadoop - Applying the PaxosFamily of Protocols to make Critical Hadoo...
NonStop Hadoop - Applying the PaxosFamily of Protocols to make Critical Hadoo...
 
DHCP
DHCPDHCP
DHCP
 
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibabahbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
 
HBase tales from the trenches
HBase tales from the trenchesHBase tales from the trenches
HBase tales from the trenches
 
Nov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars georgeNov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars george
 

More from Michael Stack

hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloudhbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on CloudMichael Stack
 
hbaseconasia2019 Recent work on HBase at Pinterest
hbaseconasia2019 Recent work on HBase at Pinteresthbaseconasia2019 Recent work on HBase at Pinterest
hbaseconasia2019 Recent work on HBase at PinterestMichael Stack
 
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltdhbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., LtdMichael Stack
 
hbaseconasia2019 HBase at Didi
hbaseconasia2019 HBase at Didihbaseconasia2019 HBase at Didi
hbaseconasia2019 HBase at DidiMichael Stack
 
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...Michael Stack
 
hbaseconasia2019 HBase at Tencent
hbaseconasia2019 HBase at Tencenthbaseconasia2019 HBase at Tencent
hbaseconasia2019 HBase at TencentMichael Stack
 
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...Michael Stack
 
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...Michael Stack
 
hbaseconasia2019 Pharos as a Pluggable Secondary Index Component
hbaseconasia2019 Pharos as a Pluggable Secondary Index Componenthbaseconasia2019 Pharos as a Pluggable Secondary Index Component
hbaseconasia2019 Pharos as a Pluggable Secondary Index ComponentMichael Stack
 
hbaseconasia2019 OpenTSDB at Xiaomi
hbaseconasia2019 OpenTSDB at Xiaomihbaseconasia2019 OpenTSDB at Xiaomi
hbaseconasia2019 OpenTSDB at XiaomiMichael Stack
 
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Sparkhbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and SparkMichael Stack
 
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBasehbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBaseMichael Stack
 
hbaseconasia2019 Distributed Bitmap Index Solution
hbaseconasia2019 Distributed Bitmap Index Solutionhbaseconasia2019 Distributed Bitmap Index Solution
hbaseconasia2019 Distributed Bitmap Index SolutionMichael Stack
 
hbaseconasia2019 HBase Bucket Cache on Persistent Memory
hbaseconasia2019 HBase Bucket Cache on Persistent Memoryhbaseconasia2019 HBase Bucket Cache on Persistent Memory
hbaseconasia2019 HBase Bucket Cache on Persistent MemoryMichael Stack
 
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACLhbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACLMichael Stack
 
hbaseconasia2019 BDS: A data synchronization platform for HBase
hbaseconasia2019 BDS: A data synchronization platform for HBasehbaseconasia2019 BDS: A data synchronization platform for HBase
hbaseconasia2019 BDS: A data synchronization platform for HBaseMichael Stack
 
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...Michael Stack
 
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...Michael Stack
 
HBaseConAsia2019 Keynote
HBaseConAsia2019 KeynoteHBaseConAsia2019 Keynote
HBaseConAsia2019 KeynoteMichael Stack
 
HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies
HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latenciesHBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies
HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latenciesMichael Stack
 

More from Michael Stack (20)

hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloudhbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on Cloud
 
hbaseconasia2019 Recent work on HBase at Pinterest
hbaseconasia2019 Recent work on HBase at Pinteresthbaseconasia2019 Recent work on HBase at Pinterest
hbaseconasia2019 Recent work on HBase at Pinterest
 
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltdhbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
hbaseconasia2019 Phoenix Practice in China Life Insurance Co., Ltd
 
hbaseconasia2019 HBase at Didi
hbaseconasia2019 HBase at Didihbaseconasia2019 HBase at Didi
hbaseconasia2019 HBase at Didi
 
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...
 
hbaseconasia2019 HBase at Tencent
hbaseconasia2019 HBase at Tencenthbaseconasia2019 HBase at Tencent
hbaseconasia2019 HBase at Tencent
 
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...
 
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...
 
hbaseconasia2019 Pharos as a Pluggable Secondary Index Component
hbaseconasia2019 Pharos as a Pluggable Secondary Index Componenthbaseconasia2019 Pharos as a Pluggable Secondary Index Component
hbaseconasia2019 Pharos as a Pluggable Secondary Index Component
 
hbaseconasia2019 OpenTSDB at Xiaomi
hbaseconasia2019 OpenTSDB at Xiaomihbaseconasia2019 OpenTSDB at Xiaomi
hbaseconasia2019 OpenTSDB at Xiaomi
 
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Sparkhbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and Spark
 
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBasehbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBase
 
hbaseconasia2019 Distributed Bitmap Index Solution
hbaseconasia2019 Distributed Bitmap Index Solutionhbaseconasia2019 Distributed Bitmap Index Solution
hbaseconasia2019 Distributed Bitmap Index Solution
 
hbaseconasia2019 HBase Bucket Cache on Persistent Memory
hbaseconasia2019 HBase Bucket Cache on Persistent Memoryhbaseconasia2019 HBase Bucket Cache on Persistent Memory
hbaseconasia2019 HBase Bucket Cache on Persistent Memory
 
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACLhbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACL
 
hbaseconasia2019 BDS: A data synchronization platform for HBase
hbaseconasia2019 BDS: A data synchronization platform for HBasehbaseconasia2019 BDS: A data synchronization platform for HBase
hbaseconasia2019 BDS: A data synchronization platform for HBase
 
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...
 
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
 
HBaseConAsia2019 Keynote
HBaseConAsia2019 KeynoteHBaseConAsia2019 Keynote
HBaseConAsia2019 Keynote
 
HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies
HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latenciesHBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies
HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies
 

Recently uploaded

Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...
Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...
Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...APNIC
 
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...musaddumba454
 
🍑👄Dehradun Esℂorts Serviℂe☎️9315791090🍑👄 ℂall Girl serviℂe in ☎️Dehradun ℂall...
🍑👄Dehradun Esℂorts Serviℂe☎️9315791090🍑👄 ℂall Girl serviℂe in ☎️Dehradun ℂall...🍑👄Dehradun Esℂorts Serviℂe☎️9315791090🍑👄 ℂall Girl serviℂe in ☎️Dehradun ℂall...
🍑👄Dehradun Esℂorts Serviℂe☎️9315791090🍑👄 ℂall Girl serviℂe in ☎️Dehradun ℂall...Mumbai Escorts
 
Free scottie t shirts Free scottie t shirts
Free scottie t shirts Free scottie t shirtsFree scottie t shirts Free scottie t shirts
Free scottie t shirts Free scottie t shirtsrahman018755
 
一比一定制(Dundee毕业证书)英国邓迪大学毕业证学位证书
一比一定制(Dundee毕业证书)英国邓迪大学毕业证学位证书一比一定制(Dundee毕业证书)英国邓迪大学毕业证学位证书
一比一定制(Dundee毕业证书)英国邓迪大学毕业证学位证书gfhdsfr
 
一比一原版(Princeton毕业证书)普林斯顿大学毕业证如何办理
一比一原版(Princeton毕业证书)普林斯顿大学毕业证如何办理一比一原版(Princeton毕业证书)普林斯顿大学毕业证如何办理
一比一原版(Princeton毕业证书)普林斯顿大学毕业证如何办理C
 
一比一原版(NYU毕业证书)美国纽约大学毕业证如何办理
一比一原版(NYU毕业证书)美国纽约大学毕业证如何办理一比一原版(NYU毕业证书)美国纽约大学毕业证如何办理
一比一原版(NYU毕业证书)美国纽约大学毕业证如何办理Fir
 
一比一原版布兰迪斯大学毕业证如何办理
一比一原版布兰迪斯大学毕业证如何办理一比一原版布兰迪斯大学毕业证如何办理
一比一原版布兰迪斯大学毕业证如何办理A
 
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样Fi
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkklolsDocherty
 
AI Generated 3D Models | AI 3D Model Generator
AI Generated 3D Models | AI 3D Model GeneratorAI Generated 3D Models | AI 3D Model Generator
AI Generated 3D Models | AI 3D Model Generator3DailyAI1
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebJie Liau
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxChloeMeadows1
 
The Rise of Subscription-Based Digital Services.pdf
The Rise of Subscription-Based Digital Services.pdfThe Rise of Subscription-Based Digital Services.pdf
The Rise of Subscription-Based Digital Services.pdfe-Market Hub
 
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理Fir
 
一比一原版(Cranfield毕业证书)英国克兰菲尔德大学毕业证如何办理
一比一原版(Cranfield毕业证书)英国克兰菲尔德大学毕业证如何办理一比一原版(Cranfield毕业证书)英国克兰菲尔德大学毕业证如何办理
一比一原版(Cranfield毕业证书)英国克兰菲尔德大学毕业证如何办理gfhdsfr
 
原版定制(Management毕业证书)新加坡管理大学毕业证原件一模一样
原版定制(Management毕业证书)新加坡管理大学毕业证原件一模一样原版定制(Management毕业证书)新加坡管理大学毕业证原件一模一样
原版定制(Management毕业证书)新加坡管理大学毕业证原件一模一样asdafd
 
Development Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appsDevelopment Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appscristianmanaila2
 
I’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 ShirtI’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 Shirtrahman018755
 

Recently uploaded (20)

Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...
Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...
Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...
 
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
 
🍑👄Dehradun Esℂorts Serviℂe☎️9315791090🍑👄 ℂall Girl serviℂe in ☎️Dehradun ℂall...
🍑👄Dehradun Esℂorts Serviℂe☎️9315791090🍑👄 ℂall Girl serviℂe in ☎️Dehradun ℂall...🍑👄Dehradun Esℂorts Serviℂe☎️9315791090🍑👄 ℂall Girl serviℂe in ☎️Dehradun ℂall...
🍑👄Dehradun Esℂorts Serviℂe☎️9315791090🍑👄 ℂall Girl serviℂe in ☎️Dehradun ℂall...
 
GOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdfGOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdf
 
Free scottie t shirts Free scottie t shirts
Free scottie t shirts Free scottie t shirtsFree scottie t shirts Free scottie t shirts
Free scottie t shirts Free scottie t shirts
 
一比一定制(Dundee毕业证书)英国邓迪大学毕业证学位证书
一比一定制(Dundee毕业证书)英国邓迪大学毕业证学位证书一比一定制(Dundee毕业证书)英国邓迪大学毕业证学位证书
一比一定制(Dundee毕业证书)英国邓迪大学毕业证学位证书
 
一比一原版(Princeton毕业证书)普林斯顿大学毕业证如何办理
一比一原版(Princeton毕业证书)普林斯顿大学毕业证如何办理一比一原版(Princeton毕业证书)普林斯顿大学毕业证如何办理
一比一原版(Princeton毕业证书)普林斯顿大学毕业证如何办理
 
一比一原版(NYU毕业证书)美国纽约大学毕业证如何办理
一比一原版(NYU毕业证书)美国纽约大学毕业证如何办理一比一原版(NYU毕业证书)美国纽约大学毕业证如何办理
一比一原版(NYU毕业证书)美国纽约大学毕业证如何办理
 
一比一原版布兰迪斯大学毕业证如何办理
一比一原版布兰迪斯大学毕业证如何办理一比一原版布兰迪斯大学毕业证如何办理
一比一原版布兰迪斯大学毕业证如何办理
 
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
 
AI Generated 3D Models | AI 3D Model Generator
AI Generated 3D Models | AI 3D Model GeneratorAI Generated 3D Models | AI 3D Model Generator
AI Generated 3D Models | AI 3D Model Generator
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
 
The Rise of Subscription-Based Digital Services.pdf
The Rise of Subscription-Based Digital Services.pdfThe Rise of Subscription-Based Digital Services.pdf
The Rise of Subscription-Based Digital Services.pdf
 
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
 
一比一原版(Cranfield毕业证书)英国克兰菲尔德大学毕业证如何办理
一比一原版(Cranfield毕业证书)英国克兰菲尔德大学毕业证如何办理一比一原版(Cranfield毕业证书)英国克兰菲尔德大学毕业证如何办理
一比一原版(Cranfield毕业证书)英国克兰菲尔德大学毕业证如何办理
 
原版定制(Management毕业证书)新加坡管理大学毕业证原件一模一样
原版定制(Management毕业证书)新加坡管理大学毕业证原件一模一样原版定制(Management毕业证书)新加坡管理大学毕业证原件一模一样
原版定制(Management毕业证书)新加坡管理大学毕业证原件一模一样
 
Development Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appsDevelopment Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of apps
 
I’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 ShirtI’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 Shirt
 

HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei

  • 1. hosted by HBaseConAsia2018 HBase and OpenTSDB Practices at Huawei Pankaj Kumar, Zhongchaoqiang, Guoyijun, Zhiwei {Pankaj.kr, zhongchaoqiang, guoyijun, wei.zhi}@huawei.com
  • 2. hosted by HBase Tech Lead @ Huawei India Apache HBase Contributor 5 years of experience in Big Data related projects $ whoami
  • 3. 3 HBase @ Huawei Migrated from 1.0.2 version 1.3.1 version +  Secondary index  MOB  Multi split Migrating to 2.1.x cluster this year
  • 5. Accelerate HMaster Startup Enhanced Replication Reliable Region Assignment HBase Practices
  • 7. 7 Accelerate HMaster Startup Problem: HMaster not available for longer duration on failover/restart Deployment scenario:  Large cluster with 500+ nodes  5000+ Tables and 120000 + regions  10 namespaces Discovered problems in multiple areas in Master startups like below  Slow region locality computation on startup  Serial region locality calculation  Too much time spent in region locality calculation  HMaster aborting due to namespace initialization failure  Slow SSH/SCP  Similar to HBASE-14190  Table info loading was taking too much time  High Namenode latency  Many other services creating lots of load in NN
  • 8. 8 Accelerate HMaster Startup  Slow region locality computation on startup  Accelerate region Locality computation by computing in parallel  Detach region locality computation on startup  Similar solution HBASE-16570  HMaster aborting due to namespace initialization failure  Assign system table regions ahead of user table regions  Assign system tables to HMaster (configure all system tables to hbase.balancer.tablesOnMaster)  On cluster/master startup, process old HMaster SSH/SCP ahead of other RegionServer  SSH/SCP will replay the WAL and assign the system table regions first
  • 9. 9 Accelerate HMaster Startup  Table Info Loading on Master startup Namespace HDFS Path default t1/.tabledesc/.tableinfo.0000000001 t1/.tabledesc/.tableinfo.0000000002 t1/.tabledesc/.tableinfo.0000000003 t2/.tabledesc/.tableinfo.0000000001 hbase /hbase/data/hbase/acl/.tabledesc/.tableinfo.0000000001 /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001 /hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 Operation Path Result Total RPC to NN List operation to get path till all the namespace /hbase/data/* gets file status for all the namespaces 1 List operation on each namespace to get all the tables in each namespace /hbase/data/default get all the file status of all the tables in each namespace 2 ( = total number of namespaces in the cluster) List operation on each table to get all the tableinfo files for the table /hbase/data/default/ t1/. tabledesc get all the file status of all the tableinfo files for the table 5 (= total number of tables in the cluster) Open call for each table’s latest tableinfo file /hbase/data/default/ t1/. tabledesc/.tableinfo. 0000000003 get the stream to tableinfo file 5 (= total number of tables in the cluster) Total RPC 13 Example: Suppose there are two namespace and total 5 tables with below path structure in HDFS Total RPC to NameNode = 1 + namespace count + 2 * table count
  • 10. 10 Accelerate HMaster Startup Table Info Loading on Master startup  2011 RPC Calls ( for 10 namespace and 1000 tables in a cluster)  NN is busy then it will hugely impact the startup of HMaster. Solution: Reduce number of RPC to Namenode  HMaster makes a single call to get tableinfo path  Get LocatedFileStatus of all tableinfo paths based on pattern (/hbase/data/*/*/.tabledesc/.tableinfo* )  LocatedFileStatus will also contain block locations of tableinfo file along with FileStatus details  DFS client will directly connect to Datanode through FileSystem#open() using LocatedFileStatus, avoid NN RPC to get the block location of the tableinfo file Improvement: In a highly overloaded HDFS cluster, it took around 97 seconds to load 5000 tables info as compared to 224 seconds earlier.
  • 12. 12 Adaptive call timeout Problem:  Replication may timeout when peer cluster is not able to replicate the entries  Can be solved by increasing hbase.rpc.timeout at source cluster  Impact other RPC request  In Bulkload replication scenario fixed RPC timeout may not guarantee bigger HFile copy  Refer HBASE-14937 Solution:  Source cluster should wait longer  New configuration parameter hbase.replication.rpc.timeout, default value will be same as hbase.rpc.timeout  On each CallTimeOutException increase this replication timeout value by fixed multiplier  Increase the replication to certain number of configured times
  • 13. 13 Cross realm support Problem:  Replication doesn’t work with Kerberos Cross Real Trust where principal domain name is not machine hostname  On new host addition  Add principal name for the newly added hosts in KDC  Generate a new keytab file  Update it across other hosts  Rigorous task for user to create and replace new keytab file Solution:  HBASE- 14866  Configure the peer cluster principal in replication peer config  Refer to HBASE-15254 (Open)  No need to configure in advance, fetch at runtime.  Make RPC call to peer HBase cluster and fetch the Principal  Make RPC connection based on this server principal
  • 14. 1.3 Reliable Region Assignment
  • 15. 15 RIT Problem Problem:  Region stuck in transition for longer duration due to some fault in cluster  Zookeeper node version mismatch  Slow RS response  Unstable Network  Client will not be able to perform read/write operation on those regions which are in transition  Balancer will not run  Region can’t be recovered until cluster restart Solution:  Recover the regions by reassigning them  Schedule a chore service  Run periodically and identify the region which stuck in transition from a longer duration (configurable threshold)  Recover the region by reassigning them  New HBCK command to recover regions which are in transition from longer duration
  • 16. 16 Double Assignment Problem Problem:  HMaster may assign region to multiple RegionServer in a faulty cluster environment  Call time out from a overloaded RegionServer  Old or new client may receive inconsistent data  Can’t be recovered until cluster restart Solution:  Multiply assigned regions should be closed and assign uniquely  Region server send server load details to HMaster through heartbeat  Schedule a chore service which run periodically and recover the regions  Collect each region server load from HMaster memory  Identify the duplicate regions from the region list  Validate the duplicate regions with HMaster Assignment Manager in-memory region state  Close the region from the old region server  Assign the region
  • 17. 17 Double Assignment Problem Example: HMaster AM’s in-memory region state r1:RS1 r2:RS3 r2:RS3 r4:RS1 r5:RS2 r6:RS3 r7:RS1 r8:RS2 Double Assignment Recovery Chore Found region r2 s multiply assigned to RS2 & RS3, so r2 will be closed from RS2 as per AM’s in memory state Region Server (RS1) r1, r4, r7 heartbeatheartbeat heartbeat Region Server (RS2) r2, r5, r8 Region Server (RS3) r3, r6, r2
  • 20. 20 Time Series 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 XX变化曲线图 风力 温度 水位
  • 21. 21 Time Series …… …… …… Time SeriesData Point A time series is a series of numeric data points of some particular metric over time. - OpenTSDB Document
  • 22. 22 OpenTSDB Schema sys.cpu.user host=webserver01,cpu=0 1533640130 20 sys.cpu.user host=webserver01,cpu=0 1533640140 25 sys.cpu.user host=webserver01,cpu=0 1533640150 30 sys.cpu.user host=webserver01,cpu=0 1533640160 32 sys.cpu.user host=webserver01,cpu=0 1533640170 35 sys.cpu.user host=webserver01,cpu=0 1533640180 40 metric name tags timestamp valu e OpenTSDB uses a metric name and a group of tags for identifying time series. Tags are used to identify different data sources.
  • 23. 23 TSDB Characteristics • Write Dominate. Read rate is usually a couple orders of magnitude lower. • Most queries happens on latest data. • Most queries are for aggregate analysis instead of individual data point. • Primarily Inserts. Updates/deletions are rarely happens.
  • 24. 24 Basic Functionality For TSDB • Rollups and Downsampling • Pre-aggregates and Aggregates • Interpolation • Data Life Cycle Management
  • 25. 25 时序数据库单值 vs.多值 Metric TimeStamp DeviceID DeviceType ZoneId Temperature Pressure WaterLine Engine 20180101 12:00:00 ID001 TypeA 1 66.9 1.33 42.5 20180101 12:00:00 ID002 TypeA 1 68.8 1.28 42.0 20180101 12:00:00 ID003 TypeA 1 67.3 1.35 41.7 20180101 12:01:00 ID001 TypeA 1 67.5 1.30 42.2 Metric TimeStamp DeviceID DeviceType ZoneId Value Temperature 20180101 12:00:00 ID001 TypeA 1 66.9 Pressure 20180101 12:00:00 ID001 TypeA 1 1.33 WaterLine 20180101 12:00:00 ID001 TypeA 1 42.5 Temperature 20180101 12:01:00 ID002 TypeA 1 68.8 Pressure 20180101 12:01:00 ID002 TypeA 1 1.28 WaterLine 20180101 12:01:00 ID002 TypeA 1 42.0 Tags Field Tags Time stampMetric Time stampMetric Metric Value Single value model vs. Multi-value model
  • 26. 26 Time Series Storage In HBase KeyValue KeyValue KeyValue KeyValue KeyValue KeyValue DataPoint (T1) DataPoint (T7) Time Series A (20180808-10) (Writing Block) KeyValue Compacted DataPoints Time Series A (20180808- 09) KeyValue Compacted DataPoints Time Series A (20180808- 08) KeyValue Compacted DataPoints Time Series A (20180808-07) Time Series Separeted into multiple blocks. Each block hold one hour of data points. DataPoint (T2) DataPoint(T3) DataPoint(T4) DataPoint (T5) DataPoint (T6) Closed Blocks
  • 27. 27 OpenTSDB Table Design RowKey Format Qualifier Format 1BYTE 3BYTES 4BYTES 3BYTES 3BYTES 3BYTES 3BYTES SALT Metric ID Timestamp Tag Name ID Tag Value ID Tag Name ID Tag Value ID 1 <= N <= 8 2 BYTES Timestamp ValueLengthValueType KeyValue KeyValue KeyValue KeyValue KeyValue KeyValue DataPoint (T1) Time Series A (20180808-10) (Writing Block) DataPoint (T2) DataPoint(T3) DataPoint(T4) DataPoint (T5) DataPoint(T6)
  • 28. 28 OpenTSDB Compaction KeyValue KeyValue KeyValue KeyValue KeyValue KeyValue DataPoint(T1) DataPoint (T2) DataPoint (T3) DataPoint (T4) DataPoint(T5) DataPoint(T6) KeyValue DataPoint(TX) KeyValue KeyValue KeyValue KeyValue KeyValue KeyValue DataPoint (T1) DataPoint(T2) DataPoint(T3) DataPoint (T4) DataPoint (T5) DataPoint (T6) KeyValue DataPoint(TX) 1. Read All Data Points from the block of Last Hour 2. Compact locally KeyValue DataPoints of whole block 3. Write compact row, and delete all exist individual data points KeyValue DataPoints of whole block Delete Marker
  • 30. 30 Exist OpenTSDB Compaction Flow OpenTSDB compaction is helpful for read, and could decrease total data amount, but the side effects as follows: 1. OpenTSDB Compaction requires a read/compact/write cycle, causing extremely high traffic to RegionServers. 2. Write compact row and delete exist individual data points amplify write I/O. CompactionQueue Metric1_Hour1 Metric2_Hour1 Metric3_Hour1 Metric1_Hour2 Metric2_Hour2 Metric3_Hour2 Metric1_Hour3 Metric2_Hour3 Metric3_Hour3 Put Read and Remove Put and Delete Get Add RowKey OpenTSDB Compact Logic HTTPHandler Compaction Thread HBase TSD
  • 31. 31 HFileHFile HFile HFile MemStore HFile MemStore Client(TSD)    1. TSD client read time series data from Regionserver. 2. TSD write compacted row and delete marker to RegionServer. 3. HBase internal compaction. Understanding Write Amplification Client(TSD) Single Datapoint Delete Marker Compact Row
  • 32. 32 New OpenTSDB Compaction Flow New HBase Compaction implementation for OpenTSDB that could compact data points during hbase compaction running. Making TSD focus on handing user read/write requests only. OpenTSDB OpenTSDB Compact Logic Read One Row Write Compact Rows HBase Put Compaction Thread HTTP Handler Region1 Region2 Region3 RegionN ........ HFile HFile HFile New HFile
  • 33. 33 No More Extra Write Amplification HFile HFile MemStore Client(TSD)  HFile HFile Compaction No more extra write amplication caused by OpenTSDB data points compaction.
  • 34. 34 NOTE: TSDs were limited to 300,000 data points per second. After optimization, write throughtput has been improved significantly. Benchmark – Throughput comparison
  • 35. 35 Benchmark – CPU And IO Comparison
  • 36. 36  Delete old data automatically for reduce data load.  HBase Table level TTL is a coarse- grained mechanism, But different metrics may have different TTL requirements.  A new HBase compaction implementation for per-metric level data life cycle management. Data Life Cycle Management Per Metric OpenTSDB OpenTSDB TTL Logic Read One Row Write Unexpired Rows HBase Put Compaction Thread HTTP Handler Region1 Region2 Region3 RegionN ........ HFile HFile HFile New HFile
  • 37. 37 Using a two-level thread model design. Receive message, process message and response to client are all handled by one WorkThread. It causes the low CPU usage. TSD workerThread RegionServer1WorkerThread RegionServer2 RegionServer3 workerThread OpenTSDB RPC Thread Model Receive Msg Process Msg Response Msg
  • 38. 38 Benchmark: Query latency got at least 3X improvement for concurrent queries. Before After 1 Query 60ms 59ms 10 Concurrent Queries 476ms 135ms 50 Concurrent Queries 2356ms 680ms Modify the thread model to a Three-Level design. Receiving message and handling message are finished in different threads. Better CPU usage. RPC Thread Model Improvement Boss Thread Read Thread Read Thread Read Thread Queue Worker Thread write read Worker Thread Worker Thread ... ... Boss Thread Worker Thread Worker Thread Worker Thread ...
  • 40. Copyright©2018 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice. Thank You.