This document discusses potential future changes to the topology and architecture of HBase clusters. It describes drivers for changes like supporting clusters with over 1 million regions and improved high availability. Specific proposals discussed include co-locating the HMaster and metadata, splitting or not splitting the metadata region, compacting the in-memory metadata, removing dependencies on ZooKeeper, implementing multiple active masters, and partitioning the master responsibilities. The talk encourages joining online discussions to provide input on these proposals.

1. HBase 2.0
Сluster topology evolution
Mikhail Antonov, WANdisco
@m_a_antonov

2. Evolution drivers
- Scaling cluster to 1M+ regions
- Stricter distributed state
management
- Improved HA
- Simplified deployment

3. This talk...
… is:
- Prospects of potential
changes being
discussed and/or
prototyped
- Encouragement to join
discussions on mail lists
& JIRA
… is not:
- New features overview
- Cluster configuration
guide
- About lessons learned

4. Bird’s-eye view
- Meta
- Co-located, for easier state management
- Splittable, for better scalability
- Compacted & replicated, for both + HA?
- Re-wiring internal communications
- Mostly refactoring ZK-related workflows
- Multi-master and ZK-less client
- Multiple active masters
- Client never talking to ZooKeeper
- Partitioned master?

5. Co-located HMaster & meta - on or off?
Co-location pros:
- Simpler and faster state
management
- Less chances to lose
any of the two
Co-location cons:
- prevents meta from ever
splitting
- More chances to lose
both

6. Meta - to split or not to split?
..Split:
- No memory constraints
- Better IOPS scaling
- Avoid write amplification
- As per original design -
Root->Meta->User regions
.. Not to split:
- soooo much simpler?

7. Compact in-memory meta, maybe?
- Make meta a single compact in-memory structure
- Each entry in it could be ~300-500 bytes
- Then every node can hold a replica of it in memory
- Better HA for meta, better IOPS?

8. Going ZK-less
- ZK-less region assignments are already there
- Permanent table state was just moved out of ZK
Next?
- Get remaining permanent state (replication) out of ZK
- Make clients not to talk to ZooKeeper?

9. Multiple active masters
- … Instead of Active - Backups
- Because you don’t want master to be down for too long
(region assignments & recovery, load balancing, splits /
merges are put on hold)
- Each master maintains copy of cluster state (meta etc)
- Masters agree among them on global operations
- Each client chooses one of the masters to talk to

10. Making master failures less critical
- By allowing RegionServers to take some subset of
master duties (e.g. serving status requests)
- Merge/splits, balancing, assignment & recovery would
still require master intervention

11. Partitioned master?
- Whole tablespace is split into partitions
- Several masters, each one is responsible for its partition
- Would require big rework of load-balancer,
AssignmentManager and other centralized services

12. Join the discussion!
- HBASE-11165 (Scaling to 1M+ regions)
- HBASE-11467 (ZK-less client + multi-master)