Your SlideShare is downloading. ×
0
1
The State of HBase Replication
Jean-Daniel Cryans
May 5th, 2014
©2014 Cloudera, Inc. All rights reserved.
About me
2
• Software Engineer at Cloudera, Storage team
• Apache HBase committe...
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
3
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
3
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
• fault-...
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
• fault-...
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
• fault-...
©2014 Cloudera, Inc. All rights reserved.
Motivation for HBase Replication
• Even though HBase is:
• distributed;
• fault-...
©2014 Cloudera, Inc. All rights reserved.
The Current State
• It’s production-ready.
4
©2014 Cloudera, Inc. All rights reserved.
The Current State
• It’s production-ready.
• It’s used to replicate data between...
©2014 Cloudera, Inc. All rights reserved.
The Current State
• It’s production-ready.
• It’s used to replicate data between...
©2014 Cloudera, Inc. All rights reserved.
5
Agenda
• Four Years of Replication
• Use Cases in Production
• Roadmap
©2014 Cloudera, Inc. All rights reserved.
Design
• Clusters are distinct
• Pull VS push
• Sync VS Async
6
©2014 Cloudera, Inc. All rights reserved.
Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
7
Master
20 RS
Slave
15 RS
©2014 Cloudera, Inc. All rights reserved.
Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
• .META. operations aren’t...
©2014 Cloudera, Inc. All rights reserved.
Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
• .META. operations aren’t...
©2014 Cloudera, Inc. All rights reserved.
Clusters are Distinct
• HBase doesn’t span DCs, HDFSs
• .META. operations aren’t...
©2014 Cloudera, Inc. All rights reserved.
Push instead of Pull
8
MySQL
Master
MySQL
Slave
Get binlog
Apply locally
MySQL R...
©2014 Cloudera, Inc. All rights reserved.
Push instead of Pull
9
RS RSreplicate entries
Apply to cluster
HBase Replication...
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
S...
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
P...
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
10
Cluster A Cluster B
RS
HLog
MemStore
RS
HLog
MemStore
P...
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
11
Asynchronous Replication
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
11
Asynchronous Replication
Cluster A
RS
HLog
MemStore
Put...
©2014 Cloudera, Inc. All rights reserved.
Async instead of Sync
11
Asynchronous Replication
Cluster A
RS
HLog
MemStore
Put...
©2014 Cloudera, Inc. All rights reserved.
First Release - 0.90.0
• Simple master-slave (only one)
• Disabled by default
• ...
©2014 Cloudera, Inc. All rights reserved.
Original Implementation
13
replicateLogEntries()Replication
Source
ZooKeeper
Wat...
©2014 Cloudera, Inc. All rights reserved.
First Lesson Learned
• HDFS doesn’t support tailing files being
written to. It re...
©2014 Cloudera, Inc. All rights reserved.
Second Lesson Learned
• Single threaded, non-batched ZK is slow
• ZK didn’t have...
©2014 Cloudera, Inc. All rights reserved.
Second Lesson Learned
• Single threaded, non-batched ZK is slow
• ZK didn’t have...
©2014 Cloudera, Inc. All rights reserved.
Second Release - 0.92.0
• Cyclic replication
• Multi-slave (scope LOCAL or GLOBA...
©2014 Cloudera, Inc. All rights reserved.
Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Cyclic Replication
17
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
Put Row ...
©2014 Cloudera, Inc. All rights reserved.
Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X
©2014 Cloudera, Inc. All rights reserved.
Multi-Slave
18
Cluster
1
Cluster
2
Cluster
3
Put Row X
Put Row X Put Row X
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
Tailing
Thread
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
...
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
...
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
...
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
...
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
...
©2014 Cloudera, Inc. All rights reserved.
Enable / Disable Peers
> disable_peer ‘2’
19
Cluster 1
RS
HLog
Cluster 2
RSHLog
...
©2014 Cloudera, Inc. All rights reserved.
Special Configurations
• KEEP_DELETED_CELLS
• Must be used on slaves with replica...
©2014 Cloudera, Inc. All rights reserved.
Special Configurations
• KEEP_DELETED_CELLS
• Must be used on slaves with replica...
©2014 Cloudera, Inc. All rights reserved.
Third Lesson Learned
• It’s easy to DDOS yourself.
• Replication was using the n...
©2014 Cloudera, Inc. All rights reserved.
Fourth Lesson Learned
• Instinctively, what would something called
stop_replicat...
©2014 Cloudera, Inc. All rights reserved.
Fourth Lesson Learned
• Instinctively, what would something called
stop_replicat...
©2014 Cloudera, Inc. All rights reserved.
Third Release - 0.96.0 / 0.98.0
• Replication enabled by default!
• Completely r...
©2014 Cloudera, Inc. All rights reserved.
ReplicationSyncUp Tool
• Works on an offline cluster
• Can finish replicating the...
©2014 Cloudera, Inc. All rights reserved.
Finer Grained Replication Controls
> set_peer_tableCFs '2', "table1;
table2:cf1,...
©2014 Cloudera, Inc. All rights reserved.
26
Agenda
• Four Years of Replication
• Use Cases in Production
• Roadmap
©2014 Cloudera, Inc. All rights reserved.
Flurry
• Two data centers, coast to coast
• Three clusters, in master-master pai...
©2014 Cloudera, Inc. All rights reserved.
Opower
• Two clusters, same data center
• Master: tens of nodes
• Slave: tens of...
©2014 Cloudera, Inc. All rights reserved.
Lily HBase Indexer
• Collaboration between NGData & Cloudera.
• NGData are the c...
©2014 Cloudera, Inc. All rights reserved.
30
Agenda
• Four Years of Replication
• Use Cases in Production
• Roadmap
©2014 Cloudera, Inc. All rights reserved.
Stop Relying on Permanent Znodes
• Current rule is to never rely on znodes to
su...
©2014 Cloudera, Inc. All rights reserved.
Define a Replication Interface
• Replication is somewhat extendable but it
lacks ...
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
33
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
33
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
2.Get’ing the ...
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
2.Get’ing the ...
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
2.Get’ing the ...
©2014 Cloudera, Inc. All rights reserved.
Distributed Counters
• Incrementing consists of:
1.Taking a lock;
2.Get’ing the ...
©2014 Cloudera, Inc. All rights reserved.
More Tooling
• Replication management console, one shell to
rule all the cluster...
©2014 Cloudera, Inc. All rights reserved.
Questions?
• Or ping me async:
• @jdcryans
• jdcryans@cloudera.com
• jdcryans on...
Upcoming SlideShare
Loading in...5
×

The State of HBase Replication

2,477

Published on

Speaker: Jean-Daniel Cryans (Cloudera)

HBase Replication has come a long way since its inception in HBase 0.89 almost four years ago. Today, master-master and cyclic replication setups are supported; many bug fixes and new features like log compression, per-family peers configuration, and throttling have been added; and a major refactoring has been done. This presentation will recap the work done during the past four years, present a few use cases that are currently in production, and take a look at the roadmap.

Published in: Software, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,477
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
146
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "The State of HBase Replication"

  1. 1. 1 The State of HBase Replication Jean-Daniel Cryans May 5th, 2014
  2. 2. ©2014 Cloudera, Inc. All rights reserved. About me 2 • Software Engineer at Cloudera, Storage team • Apache HBase committer since 2008, PMC member
  3. 3. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: 3
  4. 4. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; 3
  5. 5. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; • fault-tolerant; 3
  6. 6. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; • fault-tolerant; • highly available; and 3
  7. 7. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; • fault-tolerant; • highly available; and • almost magic. 3
  8. 8. ©2014 Cloudera, Inc. All rights reserved. Motivation for HBase Replication • Even though HBase is: • distributed; • fault-tolerant; • highly available; and • almost magic. 3
  9. 9. ©2014 Cloudera, Inc. All rights reserved. The Current State • It’s production-ready. 4
  10. 10. ©2014 Cloudera, Inc. All rights reserved. The Current State • It’s production-ready. • It’s used to replicate data between thousands of nodes across continents. 4
  11. 11. ©2014 Cloudera, Inc. All rights reserved. The Current State • It’s production-ready. • It’s used to replicate data between thousands of nodes across continents. • It’s used for Disaster Recovery, geo- distributed serving, and more. 4
  12. 12. ©2014 Cloudera, Inc. All rights reserved. 5 Agenda • Four Years of Replication • Use Cases in Production • Roadmap
  13. 13. ©2014 Cloudera, Inc. All rights reserved. Design • Clusters are distinct • Pull VS push • Sync VS Async 6
  14. 14. ©2014 Cloudera, Inc. All rights reserved. Clusters are Distinct • HBase doesn’t span DCs, HDFSs 7 Master 20 RS Slave 15 RS
  15. 15. ©2014 Cloudera, Inc. All rights reserved. Clusters are Distinct • HBase doesn’t span DCs, HDFSs • .META. operations aren’t replicated 7 Master 20 RS Slave 15 RS
  16. 16. ©2014 Cloudera, Inc. All rights reserved. Clusters are Distinct • HBase doesn’t span DCs, HDFSs • .META. operations aren’t replicated • Regions can be different 7 Master 20 RS Slave 15 RS
  17. 17. ©2014 Cloudera, Inc. All rights reserved. Clusters are Distinct • HBase doesn’t span DCs, HDFSs • .META. operations aren’t replicated • Regions can be different • Security has to be configured for each cluster 7 Master 20 RS Slave 15 RS
  18. 18. ©2014 Cloudera, Inc. All rights reserved. Push instead of Pull 8 MySQL Master MySQL Slave Get binlog Apply locally MySQL Replication uses Pull Cluster A Cluster B
  19. 19. ©2014 Cloudera, Inc. All rights reserved. Push instead of Pull 9 RS RSreplicate entries Apply to cluster HBase Replication uses Push Cluster A Cluster B
  20. 20. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 10 Cluster A Cluster B RS HLog MemStore RS HLog MemStore Synchronous Replication
  21. 21. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 10 Cluster A Cluster B RS HLog MemStore RS HLog MemStore Put 2 3 1 Synchronous Replication
  22. 22. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 10 Cluster A Cluster B RS HLog MemStore RS HLog MemStore Put 2 3 1 Ack Ack Put 5 6 4 78 Synchronous Replication
  23. 23. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 11 Asynchronous Replication
  24. 24. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 11 Asynchronous Replication Cluster A RS HLog MemStore Put Ack 2 3 1 4
  25. 25. ©2014 Cloudera, Inc. All rights reserved. Async instead of Sync 11 Asynchronous Replication Cluster A RS HLog MemStore Put Ack 2 3 1 4 Cluster B RS HLog MemStore Ack Put 3 4 2 5 HLog Tailing Thread 1
  26. 26. ©2014 Cloudera, Inc. All rights reserved. First Release - 0.90.0 • Simple master-slave (only one) • Disabled by default • Uses ZK as a metadata store 12
  27. 27. ©2014 Cloudera, Inc. All rights reserved. Original Implementation 13 replicateLogEntries()Replication Source ZooKeeper Watcher Region Server on Master Cluster Replication Sink HTable Put Delete Region Server on Slave Cluster
  28. 28. ©2014 Cloudera, Inc. All rights reserved. First Lesson Learned • HDFS doesn’t support tailing files being written to. It requires: • open() • seek()// go where we stopped last time • while (not EOF || enoughData) • read() • close() • repeat 14
  29. 29. ©2014 Cloudera, Inc. All rights reserved. Second Lesson Learned • Single threaded, non-batched ZK is slow • ZK didn’t have an atomic move operation • Doubles # ops needed, race conditions 15
  30. 30. ©2014 Cloudera, Inc. All rights reserved. Second Lesson Learned • Single threaded, non-batched ZK is slow • ZK didn’t have an atomic move operation • Doubles # ops needed, race conditions 15 /hbase /replication /RS1 /1 /hlog1 /hlog2 ... /hbase /replication /RS2 /1-RS1 /hlog1 1. create new hlog2 2. delete old hlog2
  31. 31. ©2014 Cloudera, Inc. All rights reserved. Second Release - 0.92.0 • Cyclic replication • Multi-slave (scope LOCAL or GLOBAL) • Enable / disable peer • Special configurations 16
  32. 32. ©2014 Cloudera, Inc. All rights reserved. Cyclic Replication 17 Cluster 1 Cluster 2 Cluster 3 Put Row X
  33. 33. ©2014 Cloudera, Inc. All rights reserved. Cyclic Replication 17 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X
  34. 34. ©2014 Cloudera, Inc. All rights reserved. Cyclic Replication 17 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X Put Row X
  35. 35. ©2014 Cloudera, Inc. All rights reserved. Cyclic Replication 17 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X Put Row X Row X is from 1 Don’t replicate!
  36. 36. ©2014 Cloudera, Inc. All rights reserved. Multi-Slave 18 Cluster 1 Cluster 2 Cluster 3 Put Row X
  37. 37. ©2014 Cloudera, Inc. All rights reserved. Multi-Slave 18 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X
  38. 38. ©2014 Cloudera, Inc. All rights reserved. Multi-Slave 18 Cluster 1 Cluster 2 Cluster 3 Put Row X Put Row X Put Row X
  39. 39. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread
  40. 40. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread Is the peer enabled?
  41. 41. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog Is the peer enabled?
  42. 42. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog HLog Is the peer enabled?
  43. 43. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog HLog HLog Is the peer enabled?
  44. 44. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog HLog HLog HLog Is the peer enabled?
  45. 45. ©2014 Cloudera, Inc. All rights reserved. Enable / Disable Peers > disable_peer ‘2’ 19 Cluster 1 RS HLog Cluster 2 RSHLog Tailing Thread HLog HLog HLog HLog HLog Is the peer enabled?
  46. 46. ©2014 Cloudera, Inc. All rights reserved. Special Configurations • KEEP_DELETED_CELLS • Must be used on slaves with replication when deleting data. 20
  47. 47. ©2014 Cloudera, Inc. All rights reserved. Special Configurations • KEEP_DELETED_CELLS • Must be used on slaves with replication when deleting data. • MIN_VERSION • With TTL, makes it easy to configure a slave that contains only the last few days of data. 20
  48. 48. ©2014 Cloudera, Inc. All rights reserved. Third Lesson Learned • It’s easy to DDOS yourself. • Replication was using the normal handlers... • ... and using them to write back! 21 Handler1: Put Handler2: Delete Handler3: Replicate Handler4: Get Handler5: Put Replicated Put goes in the queue
  49. 49. ©2014 Cloudera, Inc. All rights reserved. Fourth Lesson Learned • Instinctively, what would something called stop_replication do? 22
  50. 50. ©2014 Cloudera, Inc. All rights reserved. Fourth Lesson Learned • Instinctively, what would something called stop_replication do? • Good intentions, bad outcomes, HBASE-8861 22 start/stop_replication X
  51. 51. ©2014 Cloudera, Inc. All rights reserved. Third Release - 0.96.0 / 0.98.0 • Replication enabled by default! • Completely refactored for readability/ extensibility (Chris Trezzo) • ReplicationSyncUp tool (HBASE-9047) • Throttling (HBASE-9501) • Finer grained replication controls (HBASE-8751) 23
  52. 52. ©2014 Cloudera, Inc. All rights reserved. ReplicationSyncUp Tool • Works on an offline cluster • Can finish replicating the queues in ZK • Useful to finish draining a master cluster 24 HBase HDFS ZooKeeper HBase HDFS ZooKeeper ReplicationSyncUp
  53. 53. ©2014 Cloudera, Inc. All rights reserved. Finer Grained Replication Controls > set_peer_tableCFs '2', "table1; table2:cf1,cf2; table3:cfA,cfB" • Meaning: enable replication to peer #2 for: • All of table1 • cf1 and cf2 from table2 • cfA and cfB from table3 25
  54. 54. ©2014 Cloudera, Inc. All rights reserved. 26 Agenda • Four Years of Replication • Use Cases in Production • Roadmap
  55. 55. ©2014 Cloudera, Inc. All rights reserved. Flurry • Two data centers, coast to coast • Three clusters, in master-master pairs • 1200 nodes • 800 nodes • 30 nodes • Replication traffic: 2Gbps • Latency between DCs: 85ms 27
  56. 56. ©2014 Cloudera, Inc. All rights reserved. Opower • Two clusters, same data center • Master: tens of nodes • Slave: tens of nodes • Replication traffic: 1GB/day • Bulk load replication traffic: 180GB/day • Recent use case 28
  57. 57. ©2014 Cloudera, Inc. All rights reserved. Lily HBase Indexer • Collaboration between NGData & Cloudera. • NGData are the creators of the Lily data management platform. • Lily HBase Indexer • Service which acts as a HBase replication listener. • Custom sink writes to SolrCloud. • Integrates Cloudera Morphlines library for ETL of rows. 29
  58. 58. ©2014 Cloudera, Inc. All rights reserved. 30 Agenda • Four Years of Replication • Use Cases in Production • Roadmap
  59. 59. ©2014 Cloudera, Inc. All rights reserved. Stop Relying on Permanent Znodes • Current rule is to never rely on znodes to survive cluster restarts, upgrades, etc. • State data should be kept in an HBase table. • Notification done through a new mechanism • See: https://issues.apache.org/jira/browse/ HBASE-10295 31
  60. 60. ©2014 Cloudera, Inc. All rights reserved. Define a Replication Interface • Replication is somewhat extendable but it lacks stable interfaces. • The HBase Indexer is such an extension and it required surgery every time a committer sneezed. • See: https://issues.apache.org/jira/browse/ HBASE-10504 32
  61. 61. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 33
  62. 62. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 33
  63. 63. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 2.Get’ing the current value; and 33
  64. 64. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 2.Get’ing the current value; and 3.Put’ing the newly incremented value. 33
  65. 65. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 2.Get’ing the current value; and 3.Put’ing the newly incremented value. • This breaks in Master-Master because the Puts are overwriting each other. 33
  66. 66. ©2014 Cloudera, Inc. All rights reserved. Distributed Counters • Incrementing consists of: 1.Taking a lock; 2.Get’ing the current value; and 3.Put’ing the newly incremented value. • This breaks in Master-Master because the Puts are overwriting each other. • See https://issues.apache.org/jira/browse/ HBASE-2804 33
  67. 67. ©2014 Cloudera, Inc. All rights reserved. More Tooling • Replication management console, one shell to rule all the clusters! • Replication bootstrapping tool. • Tool that can move queues between region servers. • Tool that can throttle replication on a live cluster. 34
  68. 68. ©2014 Cloudera, Inc. All rights reserved. Questions? • Or ping me async: • @jdcryans • jdcryans@cloudera.com • jdcryans on #hbase irc.freenode.net 35
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×