Downloaded 723 times
![Hbase运维碎碎念Flush/Compaction/SplitCompactSplitThread.javalock.lock();try { if(!this.server.isStopped()) { // Don't interrupt us while we are working byte [] midKey = r.compactStores(); if (r.getLastCompactInfo() != null) { // compaction aborted?this.server.getMetrics().addCompaction(r.getLastCompactInfo()); } if (shouldSplitRegion() && midKey != null && !this.server.isStopped()) { split(r, midKey); } } } finally {lock.unlock();}](https://image.slidesharecdn.com/hbase-110627050457-phpapp01/85/Hbase-27-320.jpg)
![每次insert一个KV时，数据（byte[]数组）复制到curChunk，nextFreeOffset随之增长](https://image.slidesharecdn.com/hbase-110627050457-phpapp01/85/Hbase-36-320.jpg)
![Hbase运维碎碎念压力测试 $ ./hbaseorg.apache.hadoop.hbase.PerformanceEvaluationUsage: java org.apache.hadoop.hbase.PerformanceEvaluation \ [--miniCluster] [--nomapred] [--rows=ROWS] <command> <nclients>Options:miniCluster Run the test on an HBaseMiniClusternomapred Run multiple clients using threads (rather than use mapreduce) rows Rows each client runs. Default: One millionflushCommits Used to determine if the test should flush the table. Default: falsewriteToWAL Set writeToWAL on puts. Default: TrueCommand:filterScan Run scan test using a filter to find a specific row based on it's value (make sure to use --rows=20)randomRead Run random read testrandomSeekScan Run random seek and scan 100 testrandomWrite Run random write test scan Run scan test (read every row) scanRange10 Run random seek scan with both start and stop row (max 10 rows) scanRange100 Run random seek scan with both start and stop row (max 100 rows) scanRange1000 Run random seek scan with both start and stop row (max 1000 rows) scanRange10000 Run random seek scan with both start and stop row (max 10000 rows)sequentialRead Run sequential read testsequentialWrite Run sequential write testArgs:nclients Integer. Required. Total number of clients (and HRegionServers) running: 1 <= value <= 500Examples: To run a single evaluation client: $ bin/hbaseorg.apache.hadoop.hbase.PerformanceEvaluationsequentialWrite 1](https://image.slidesharecdn.com/hbase-110627050457-phpapp01/85/Hbase-47-320.jpg)
Uploaded byhaiyuan ning
Hbase运维碎碎念
该文档是关于HBase运维的总结，涉及Java、HDFS和HBase的基本概念和操作，包括JVM配置、垃圾回收、内存管理、数据flush和合并等技术细节。文中还讨论了监控、性能调优和容灾策略，并提供了一些实际操作建议和参数设定的指导。文档强调了对HBase运行和管理所需的知识背景和实践经验的重要性。
Hbase运维碎碎念
- 1.
- 2.
- 3.
- 4. Hbase运维碎碎念tmpwatch运行一段时间后，发现jps无法列举java进程了，崩溃。。。Why？Java需要将pid信息写入到 /tmp/hsperfdata_usernametmpwatch定期清理/tmp修改sudo vi/tmp/cron.daily/tmpwatch/usr/sbin/tmpwatch “$flags” -x /tmp/hsperfdata_* …
- 5.
- 6. Hbase运维碎碎念JVM Heap简单来说，Heap分为年轻代(Young)/年老代(Old/ Tenured)/持久代(Perm)-Xms:Heap初始尺寸-Xmx: Heap最大尺寸-Xmn: 年轻代尺寸-XX:NewRatio: 设置Young与Old的大小比例，-server时默认为1:2-XX:SurvivorRatio: 设置Eden与Survivor的比例，默认为32。-XX:MaxPermSize： 持久代大小，默认64M。-XX:NewSize： 年轻代大小-XX:MaxNewSize: 年轻代最大尺寸-Xss: 每个线程的stack尺寸-XX:MinHeapFreeRatio：空余堆内存小于40%时，JVM就会增大堆。 -XX:MaxHeapFreeRatio：空余堆内存大于70%时，JVM会减少堆。
- 7. Hbase运维碎碎念JVM Heap年轻代分为Eden，Survior 1，Survior2新对象在Eden区分配内存。GC时，将Eden和有对象的Survior区(From Space)的所有对象复制到另外一个Survior(To Space)，然后清空Eden和原Sruvior。当一个对象在from和to之间复制的次数超过一定阀值(-XX:MaxTenuringThreshold)后，进入到年老代。如果是CMS GC，这个阀值默认为0，也就是经过一次copy后就进入年老代。 -XX:+PrintTenuringDistribution可以查看详细统计信息。持久代不GC，除非CMS GC且显式设置 XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
- 8. Hbase运维碎碎念JVM GCGC分为对Young的GC（minor）和同时对Young和Old的GC（Major）Serial GC，单线程GC，适合小Heap单CPU。-XX:+UseSerialGCParallelGC，也称Throughput GC。与第一种GC不同在于Young区是多线程的，但Old区仍然单线程。-XX:+UseParallelGC。Java 6.0已经支持Old区并行GC： -XX:+UseParallelOldGC。GC的线程数可以通过-XX:ParallelGCThreads=<N>调整。通过-XX:MaxGCPauseMillis=<N>可以提示最大GC暂停时间，设置此参数会导致JVM自动调整Heap大小和一些JVM相关参数，慎用。Concurrent GC，也称CMS GC，可以在Old区的回收同时，运行应用程序。-XX:+UseConcMarkSweepGC参数启动该GC。该GC还可以开启增量模式-XX:+CMSIncrementalMode
- 9.
- 10. Hbase运维碎碎念JVM GCConcurrent GC并发GC更消耗CPU，假如有N个CPU，GC可使用1<= K <= ceil(N/4) 。并发GC可能造成Heap碎片，Cloudera推出了MemStore-Local Allocation Buffers特性来改善碎片问题。如果并发GC进行中，有新对象要进入年老代会导致Concurrent Mode Failure，此时只能stop the world，改用串行Full GC。如果新的对象要进入年老代，但年老代没有足够大小的连续空间，会导致Promotion Failuer，此时只能stop the world，进行Full GC-XX:CMSInitiatingOccupancyFraction=<N> 可以控制启动CMC GC的阀值，N表示年老代的大小百分比。如果Concurrent Mode Failure频繁出现，可以考虑降低其值。
- 11. Hbase运维碎碎念JVM GC不管哪种GC，年轻代的GC都会阻塞线程。推荐年轻代使用Parallel NewGC，年老代使用CMS GC-XX:+UseParNewGC –XX:+UseConcMarkSweepGC禁止程序显式GC-XX:+DisableExplicitGC打印GC Log-verbose:gc -Xloggc:filename -XX:+PrintGCDetails -XX:+PrintGCTimeStamps打印年老代FreeListSpace的状态 -XX:PrintFLSStatistics=1打印CMS的信息-XX:PrintCMSStatistics=1
- 12. Hbase运维碎碎念NUMA从JDK6u2开始，增加了-XX:+UseNUMA，在 UseParallelGC时，JVM会将年轻代根据节点数切分为不同的pool。cat /sys/devices/system/node/node0/meminfoNode0 MemTotal: 12382196 kBNode 0 MemFree: 9739412 kBNode 0 MemUsed: 2642784 kBNode 0 Active: 1805868 kBNode 0 Inactive: 548076 kBNode 0 HighTotal: 0 kBNode 0 HighFree: 0 kBNode 0 LowTotal: 12382196 kBNode 0 LowFree: 9739412 kBNode 0 Dirty: 396 kBNode 0 Writeback: 0 kBNode 0 FilePages: 1765192 kBNode 0 Mapped: 23236 kBNode 0 AnonPages: 589204 kBNode 0 PageTables: 3672 kBNode 0 NFS_Unstable: 0 kBNode 0 Bounce: 0 kBNode 0 Slab: 141612 kBNode 0 HugePages_Total: 0Node 0 HugePages_Free: 0
- 13. Hbase运维碎碎念NUMA$ numastat node0 node1numa_hit 1675519925 1659271911numa_miss 0 0numa_foreign 0 0interleave_hit 110847 112840local_node 1675317828 1658259635other_node 202097 1012276
- 14.
- 15. Hbase运维碎碎念NUMA关闭NUMA？vi /etc/grub.conf…Title RedHat Enterprise Linux Server (2.6.18-164.el5) root (hd0,0) kernel /vmlinuz-2.6.18-164.el5 ro root=LABEL=/ numa=off console=tty0 console=ttyS1,115200 initrd /initrd-2.6.18-164.el5.img
- 16. Hbase运维碎碎念Large PageOS设置假设使用2G的大页面内存（2M页面）echo 1000> /proc/sys/vm/nr_hugepagesecho 2147483647 > /proc/sys/kernel/shmmaxJVM设置-XX:+UseLargePages-Xmx整个JVM Heap必须全部使用大页内存，或者全部不用。
- 17.
- 18. Hbase运维碎碎念ulimitsudo vi /etc/security/limits.conf*soft nproc 65536* hard nproc 65536* soft nofile 65536* hard nofile 65536Swapecho 0 > /proc/sys/vm/swappiness
- 19.
- 20.
- 21.
- 22. Hbase运维碎碎念DataNodedfs.datanode.max.xcievers默认256，建议至少4096+，太小可能导致频繁报错：hdfs.DFSClient: Could notobtain block blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes contain current block.
- 23.
- 24.
- 25.
- 26.
- 27. Hbase运维碎碎念Flush/Compaction/SplitCompactSplitThread.javalock.lock();try { if(!this.server.isStopped()) { // Don't interrupt us while we are working byte [] midKey = r.compactStores(); if (r.getLastCompactInfo() != null) { // compaction aborted?this.server.getMetrics().addCompaction(r.getLastCompactInfo()); } if (shouldSplitRegion() && midKey != null && !this.server.isStopped()) { split(r, midKey); } } } finally {lock.unlock();}
- 28.
- 29.
- 30. Hbase运维碎碎念Flush/Compaction/Splithbase.regionserver.maxlogs默认32，当Hlog的数量超过32个时会造成flush所有的region，不管它的memstore是否满。hbase.regionserver.global.memstore.upperLimit默认0.4，表示region server上所有的memstore占用的内存总和最多为MaxHeap的40%，超过则会加锁刷磁盘，一直要等到某个memstore刷到磁盘，且memstore总和下去了，才会继续， Flush是串行操作，所以对memstore多或写非常频繁的场景要特别注意。hbase.regionserver.global.memstore.lowerLimit默认0.35，当所有MemStore的大小超过MaxHeap的35%时，开始持续Flush，以尽量避免到upperLimit导致锁。
- 31. Hbase运维碎碎念Minor CompactionMinor Compaction只合并StoreFile，不清理历史版本和删除记录hbase.hstore.compaction.max默认10，一次minorcompaction最多只处理10个StoreFile
- 32. Hbase运维碎碎念Major CompactionMajor Compaction合并所有StoreFile，清理历史版本和删除记录hbase.hregion.majorcompaction默认86400000毫秒=1天，同一个region两次major compaction操作的时间间隔。如果一次minor compaction选中的StoreFile是这个region的所有StoreFile，minor compaction会自动升级为major compaction手工触发：$hbase shellhbase(main):015:0> compact 'test'0 row(s) in 0.2220 secondshbase(main):016:0> major_compact 'test'0 row(s) in 0.3240 secondsSo，如果没有delete，major compaction可以不用太频繁的执行。
- 33.
- 34. Hbase运维碎碎念MSLABMemStore-Local Allocation Buffer每个MemStore都有一个MemStoreLAB实例
- 35.
- 36.
- 37.
- 38.
- 39.
- 40. Hbase运维碎碎念Regionserverzookeeper.session.timeout默认3分钟，也就是rs超过3分钟没有给zk消息，Master就认为rs挂了。如果gc的过程中阻塞时间超过了3分钟，那就杯具了，so。。。hfile.block.cache.size默认0.2，全局公用的hfile的cache，最多占用MaxHeap的20%。当数据在memstore中读取不到时，就会从这个cache里获取，当从此cache中获取不到时，就需要读取文件。当cache的block达到了这个值的85%时，即会启动evict（日志中会出现“Block cache LRUeviction”），将cache清除到75%大小，可通过日志中的“LRU Stats: ”来观察cache的命中率
- 41.
- 42.
- 43.
- 44.
- 45. Hbase运维碎碎念备份恢复导出到本地或者HDFShbaseorg.apache.hadoop.hbase.mapreduce.Driver export \Tablename/desctination导入回Hbasehbaseorg.apache.hadoop.hbase.mapreduce.Driver import \Tablename /source导入TSV文件（以Tab分割的文本文件）hbaseorg.apache.hadoop.hbase.mapreduce.Driverimporttsv \ -Dimporttsv.columns=a,b,cTablename /source如果是csv文件，只需要加参数-Dimporttsv.separator=,即可
- 46. Hbase运维碎碎念备份恢复Hbase集群间CopyTablehbaseorg.apache.hadoop.hbase.mapreduce.CopyTable …或者hbaseorg.apache.hadoop.hbase.mapreduce.Drivercopytable …Hadoop集群间Copy文件hadoopdistcp-p -update "hdfs://A:8020/user/foo/bar" "hdfs://B:8020/user/foo/baz“Mozilla开发的Backup工具对运行的Hbase进行做distcp会导致不一致，Mozilla为此开发了一个Backup工具http://blog.mozilla.com/data/2011/02/04/migrating-hbase-in-the-trenches/
- 47. Hbase运维碎碎念压力测试 $ ./hbaseorg.apache.hadoop.hbase.PerformanceEvaluationUsage:java org.apache.hadoop.hbase.PerformanceEvaluation \ [--miniCluster] [--nomapred] [--rows=ROWS] <command> <nclients>Options:miniCluster Run the test on an HBaseMiniClusternomapred Run multiple clients using threads (rather than use mapreduce) rows Rows each client runs. Default: One millionflushCommits Used to determine if the test should flush the table. Default: falsewriteToWAL Set writeToWAL on puts. Default: TrueCommand:filterScan Run scan test using a filter to find a specific row based on it's value (make sure to use --rows=20)randomRead Run random read testrandomSeekScan Run random seek and scan 100 testrandomWrite Run random write test scan Run scan test (read every row) scanRange10 Run random seek scan with both start and stop row (max 10 rows) scanRange100 Run random seek scan with both start and stop row (max 100 rows) scanRange1000 Run random seek scan with both start and stop row (max 1000 rows) scanRange10000 Run random seek scan with both start and stop row (max 10000 rows)sequentialRead Run sequential read testsequentialWrite Run sequential write testArgs:nclients Integer. Required. Total number of clients (and HRegionServers) running: 1 <= value <= 500Examples: To run a single evaluation client: $ bin/hbaseorg.apache.hadoop.hbase.PerformanceEvaluationsequentialWrite 1