The State of HBase Replication

1
The State of HBase Replication
Jean-Daniel Cryans
May 5th, 2014

©2014 Cloudera, Inc. All rights reserved.
About me
2
• Software Engineer at Cloudera, Storage team
• Apache HBase committer since 2008, PMC
member

Motivation for HBase Replication
• Even though HBase is:
3

• distributed;
3

• distributed;
• fault-tolerant;
3

• distributed;
• fault-tolerant;
• highly available; and
3

• distributed;
• fault-tolerant;
• highly available; and
• almost magic.
3

The Current State
• It’s production-ready.
4

The Current State
• It’s used to replicate data between thousands
of nodes across continents.
4

The Current State
• It’s used to replicate data between thousands
of nodes across continents.
• It’s used for Disaster Recovery, geo-
distributed serving, and more.
4

5
Agenda
• Four Years of Replication
• Use Cases in Production
• Roadmap

Design
• Clusters are distinct
• Pull VS push
• Sync VS Async
6

Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
7
Master
20 RS
Slave
15 RS

• .META. operations aren’t replicated
7
Master
20 RS
Slave
15 RS

• Regions can be different
7
Master
20 RS
Slave
15 RS

• Regions can be different
• Security has to be conﬁgured for each cluster
7
Master
20 RS
Slave
15 RS

Push instead of Pull
8
MySQL
Master
MySQL
Slave
Get binlog
Apply locally
MySQL Replication uses Pull
Cluster A Cluster B

Push instead of Pull
9
RS RSreplicate entries
Apply to cluster
HBase Replication uses Push
Cluster A Cluster B

Async instead of Sync
10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
Synchronous Replication

10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
Put
2
3
1

10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
Put
2
3
1
Ack Ack
Put
5
6
4
78

11
Asynchronous Replication

11
Cluster A
RS
HLog
MemStore
Put
Ack
2
3
1
4

11
Cluster A
RS
HLog
MemStore
Put
Ack
2
3
1
4
Cluster B
RS
HLog
MemStore
Ack
Put
3
4
2
5
HLog
Tailing
Thread
1

First Release - 0.90.0
• Simple master-slave (only one)
• Disabled by default
• Uses ZK as a metadata store
12

Original Implementation
13
replicateLogEntries()Replication
Source
ZooKeeper
Watcher
Region Server on
Master Cluster
Replication
Sink
HTable
Put
Delete
Region Server on
Slave Cluster

First Lesson Learned
• HDFS doesn’t support tailing ﬁles being
written to. It requires:
• open()
• seek()// go where we stopped last time
• while (not EOF || enoughData)
• read()
• close()
• repeat
14

Second Lesson Learned
• Single threaded, non-batched ZK is slow
• ZK didn’t have an atomic move operation
• Doubles # ops needed, race conditions
15

Second Lesson Learned
• Single threaded, non-batched ZK is slow
• ZK didn’t have an atomic move operation
• Doubles # ops needed, race conditions
15
/hbase
/replication
/RS1
/1
/hlog1
/hlog2
...
/hbase
/replication
/RS2
/1-RS1
/hlog1
1. create new hlog2
2. delete old hlog2

Second Release - 0.92.0
• Cyclic replication
• Multi-slave (scope LOCAL or GLOBAL)
• Enable / disable peer
• Special conﬁgurations
16

Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X

Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X

Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
Put Row X

Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
Put Row X
Row X is from 1
Don’t replicate!

Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X

Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X

Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X Put Row X

Enable / Disable Peers
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread

> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
Is the peer enabled?

19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog

19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
HLog

19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
HLog
HLog

19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
HLog
HLog
HLog

19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
HLog
HLog
HLog
HLog
HLog

Special Conﬁgurations
• KEEP_DELETED_CELLS
• Must be used on slaves with replication when
deleting data.
20

Special Conﬁgurations
• KEEP_DELETED_CELLS
• Must be used on slaves with replication when
deleting data.
• MIN_VERSION
• With TTL, makes it easy to conﬁgure a slave that
contains only the last few days of data.
20

Third Lesson Learned
• It’s easy to DDOS yourself.
• Replication was using the normal handlers...
• ... and using them to write back!
21
Handler1: Put
Handler2: Delete
Handler3: Replicate
Handler4: Get
Handler5: Put
Replicated Put goes in the queue

Fourth Lesson Learned
• Instinctively, what would something called
stop_replication do?
22

Fourth Lesson Learned
• Instinctively, what would something called
stop_replication do?
• Good intentions, bad outcomes, HBASE-8861
22
start/stop_replication
X

Third Release - 0.96.0 / 0.98.0
• Replication enabled by default!
• Completely refactored for readability/
extensibility (Chris Trezzo)
• ReplicationSyncUp tool (HBASE-9047)
• Throttling (HBASE-9501)
• Finer grained replication controls
(HBASE-8751)
23

ReplicationSyncUp Tool
• Works on an offline cluster
• Can ﬁnish replicating the queues in ZK
• Useful to ﬁnish draining a master cluster
24
HBase
HDFS
ZooKeeper
HBase
HDFS
ZooKeeper
ReplicationSyncUp

Finer Grained Replication Controls
> set_peer_tableCFs '2', "table1;
table2:cf1,cf2; table3:cfA,cfB"
• Meaning: enable replication to peer #2 for:
• All of table1
• cf1 and cf2 from table2
• cfA and cfB from table3
25

26
Agenda
• Roadmap

Flurry
• Two data centers, coast to coast
• Three clusters, in master-master pairs
• 1200 nodes
• 800 nodes
• 30 nodes
• Replication traffic: 2Gbps
• Latency between DCs: 85ms
27

Opower
• Two clusters, same data center
• Master: tens of nodes
• Slave: tens of nodes
• Replication traffic: 1GB/day
• Bulk load replication traffic: 180GB/day
• Recent use case
28

Lily HBase Indexer
• Collaboration between NGData & Cloudera.
• NGData are the creators of the Lily data
management platform.
• Lily HBase Indexer
• Service which acts as a HBase replication listener.
• Custom sink writes to SolrCloud.
• Integrates Cloudera Morphlines library for ETL of
rows.
29

30
Agenda
• Roadmap

Stop Relying on Permanent Znodes
• Current rule is to never rely on znodes to
survive cluster restarts, upgrades, etc.
• State data should be kept in an HBase table.
• Notiﬁcation done through a new mechanism
• See: https://issues.apache.org/jira/browse/
HBASE-10295
31

Deﬁne a Replication Interface
• Replication is somewhat extendable but it
lacks stable interfaces.
• The HBase Indexer is such an extension and it
required surgery every time a committer
sneezed.
• See: https://issues.apache.org/jira/browse/
HBASE-10504
32

Distributed Counters
• Incrementing consists of:
33

1.Taking a lock;
33

1.Taking a lock;
2.Get’ing the current value; and
33

1.Taking a lock;
3.Put’ing the newly incremented value.
33

1.Taking a lock;
• This breaks in Master-Master because the
Puts are overwriting each other.
33

1.Taking a lock;
• This breaks in Master-Master because the
Puts are overwriting each other.
• See https://issues.apache.org/jira/browse/
HBASE-2804
33

More Tooling
• Replication management console, one shell to
rule all the clusters!
• Replication bootstrapping tool.
• Tool that can move queues between region
servers.
• Tool that can throttle replication on a live
cluster.
34

Questions?
• Or ping me async:
• @jdcryans
• jdcryans@cloudera.com
• jdcryans on #hbase irc.freenode.net
35

The State of HBase Replication

More Related Content

What's hot

Similar to The State of HBase Replication

More from HBaseCon

Recently uploaded

The State of HBase Replication