Hbase
•Download as ODP, PDF•
0 likes•256 views
This document discusses several key concepts in HBase including its architecture, data model, caching mechanisms, compaction process, block size considerations, failover process, and region splitting. It addresses questions around HBase's master-slave model, location of the meta table, row key design principles, the roles of memstore and blockcache, how minor and major compactions work, tradeoffs of large versus small block sizes, and how the system handles failures or administrative actions that involve moving regions.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Hbase
Similar to Hbase (20)
Recently uploaded
Recently uploaded (20)
Hbase
- 7. Flow
- 8. Hbase Compaction ● Minor Compaction and Major compaction ? ● hbase.store.compaction.ratio Ratio used in compaction file selection algorithm (default 1.2f). ● hbase.hstore.compaction.min (.90 hbase.hstore.compactionThreshold) (files) Minimum number of StoreFiles per Store to be selected for a compaction to occur (default 2).
- 9. Hbase Compaction ● hbase.hstore.compaction.max (files) Maximum number of StoreFiles to compact per minor compaction (default 10). ● hbase.hstore.compaction.min.size (bytes) Any StoreFile smaller than this setting with automatically be a candidate for compaction. Defaults to hbase.hregion.memstore.flush.size (128 mb). ● hbase.hstore.compaction.max.size (.92) (bytes) Any StoreFile larger than this setting with automatically be excluded from compaction (default Long.MAX_VALUE). ● The minor compaction StoreFile selection logic is size based, and selects a file for compaction when the file <= sum(smaller_files) * hbase.hstore.compaction.ratio.
- 11. Hbase failover ● What if one RegionServer goes down ? ● A table is disable and reenabled ? ● A cluster is stopped and restarted ? ● The HBase balancer decides to move a region to balance data sizes across RegionServers ?
- 12. Hbase Splitting ● Presplitting ● Auto splitting ● Min (R^2 * “hbase.hregion.memstore.flush.size”, “hbase.hregion.max.filesize”), 512MB, 1152MB, 2GB, 3.2GB, 4.6GB, 6.2GB,
- 13. Hbase Splitting
Editor's Notes
- Blockcache explain LruBlockCache SlabCache BucketCache Multi-Level Caching
- Explain Memstore Explain problem with more column family(more small files) ? More minor compaction.
- Smaller block size for random read. Larger block size for sequential read. Hbase block size for bigger data.
- 1st case is same as 4th 2nd case – round robin 3rd case – Will check the Meta table and try to assign to same region server.
- Explain Memstore Explain problem with more column family(more small files) ? More minor compaction.
- 1. RegionServer decides locally to split the region, and prepares the split. As a first step, it creates a znode in zookeeper under /hbase/region-in-transition/region-name in SPLITTING state. 2. The Master learns about this znode, since it has a watcher for the parent region-in-transition znode. 3. RegionServer creates a sub-directory named “.splits” under the parent’s region directory in HDFS. 4. RegionServer closes the parent region, forces a flush of the cache and marks the region as offline in its local data structures. At this point, client requests coming to the parent region will throw NotServingRegionException. The client will retry with some backoff. 5. RegionServer create the region directories under .splits directory, for daughter regions A and B, and creates necessary data structures. Then it splits the store files, in the sense that it creates two Reference files per store file in the parent region. Those reference files will point to the parent regions files. 6. RegionServer creates the actual region directory in HDFS, and moves the reference files for each daughter. 7. RegionServer sends a Put request to the .META. table, and sets the parent as offline in the .META. table and adds information about daughter regions. At this point, there won’t be individual entries in .META. for the daughters. Clients will see the parent region is split if they scan .META., but won’t know about the daughters until they appear in .META.. Also, if this Put to .META. succeeds, the parent will be effectively split. If the RegionServer fails before this RPC succeeds, Master and the next region server opening the region will clean dirty state about the region split. After the .META. update, though, the region split will be rolled-forward by Master. 8. RegionServer opens daughters in parallel to accept writes. 9. RegionServer adds the daughters A and B to .META. together with information that it hosts the regions. After this point, clients can discover the new regions, and issue requests to the new region. Clients cache the .META. entries locally, but when they make requests to the region server or .META., their caches will be invalidated, and they will learn about the new regions from .META.. 10. RegionServer updates znode /hbase/region-in-transition/region-name in zookeeper to state SPLIT, so that the master can learn about it. The balancer can freely re-assign the daughter regions to other region servers if it chooses so. 11. After the split, meta and HDFS will still contain references to the parent region. Those references will be removed when compactions in daughter regions rewrite the data files. Garbage collection tasks in the master periodically checks whether the daughter regions still refer to parents files. If not, the parent region will be removed.