Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

hbaseconasia2017: HBase Practice At XiaoMi

1,214 views

Published on

Zheng Hu

We'll share some HBase experience at XiaoMi:
1. How did we tuning G1GC for HBase Clusters.
2. Development and performance of Async HBase Client.

hbaseconasia2017 hbasecon hbase xiaomi https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

Published in: Technology
  • You won't get rich, but we do pay. ✄✄✄ https://t.cn/A6ybKmr1
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Did you try ⇒ www.HelpWriting.net ⇐?. They know how to do an amazing essay, research papers or dissertations.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sie können Hilfe bekommen bei ⇒ www.WritersHilfe.com ⇐. Erfolg und Grüße!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

hbaseconasia2017: HBase Practice At XiaoMi

  1. 1. HBase Practice At Xiaomi huzheng@xiaomi.com
  2. 2. About This Talk ● Async HBase Client ○ Why Async HBase Client ○ Implementation ○ Performance ● How do we tuning G1GC for HBase ○ CMS vs G1 ○ Tuning G1GC ○ G1GC in XiaoMi HBase Cluster
  3. 3. Part-1 Async HBase Client
  4. 4. Why Async HBase Client ? Request-1 Response-1 Request-2 Response-2 Request-3 Response-3 Request-4 Response-4 Request-1 Request-2 Response-1 Request-3 Response2 Response-3 Request-4 Response-4 RPC-1 RPC-2 RPC-3 RPC-4 RPC-1 RPC-2 RPC-3 RPC-4 Blocking Client (Single Thread) Non-Blocking Client(Single Thread)
  5. 5. Fault Amplification When Using Blocking Client RegionServer RegionServer RegionServer Handler-1 Handler-2 Handler-3 Services RegionServer RegionServer RegionServer Handler-1 Handler-1 Handler-1 Services Get Stuck Get Stuck Get Stuck All of handlers are blocked if a region server blocked when using blocking client Get Stuck Availability: 66% Availability: 0%
  6. 6. Why Async HBase Client ? ● Region Server / Master STW GC ● Slow RPC to HDFS ● Region Server Crash ● High Load ● Network Failure BTW: HBase may also suffer from fault amplification when accessing HDFS, so AsyncDFSClient ?
  7. 7. Async HBase Client VS OpenTSDB/asynchbase Async HBase Client OpenTSDB/asynchbase(1.8) HBase Version >=2.0.0 Both 0.9x and 1.x.x Table API Every API In Blocking API Parts of Blocking API HBase Admin Supported Not Supported Implementation Included In HBase Project Independent Project Based On PB protocol Coprocessor Supported Not Supported
  8. 8. Async HBase Client Example
  9. 9. RpcChannel BlockingRpcChannel BlockingRpcClientNettyRpcClient ClientService.Interface ClientService.BlockingInterface AsyncTable Table Async HBase Client Architecture Async Client Sync Client Other AbstractRpcClient
  10. 10. Performance Test ○ Test Async RPC by CompletableFuture.get() (Based on XiaoMi HBase0.98) ○ Proof that latency of async client is at least the same as blocking hbase client.
  11. 11. Part-2 HBase + G1GC Tuning
  12. 12. CMS VS G1 OldGen GC (Before) OldGen GC (After) CMS Old Gen GC
  13. 13. CMS VS G1 Mixed GC (Before) Mixed GC (After) G1 Old Gen GC
  14. 14. CMS VS G1 ● STW Full GC ○ CMS can only compact fragments when full GC, so theoretically you can not avoid full GC. ○ G1 will compact fragments incrementally by multiple mixed GC. so it provides the ability to avoid full GC. ● Heap Size ○ G1 is more suitable for huge heap than CMS
  15. 15. Pressure Test ● Pre-load data ○ A new table with 400 regions ○ 100 millions rows whose value is 1000 bytes ● Pressure test for G1GC tuning ○ 40 write clients + 20 read clients ○ 1 hour for each JVM option changed ● HBase configuration ○ 0.3 <= global memstore limit <= 0.45 ○ hfile.block.cache.size = 0.1 ○ hbase.hregion.memstore.flush.size = 256 MB ○ hbase.bucketcache.ioengine = offheap
  16. 16. Test Environment RegionServer RegionServer RegionServer RegionServer RegionServer HMaster ● Java: JDK 1.8.0_111 ● Heap: 30G Heap + 30G OFF-Heap ● CPU: 24 Core ● DISK: 4T x 12 HDD ● Network Interface: 10Gb/s 5 Region Server HBase Cluster
  17. 17. Initial JVM Options -Xmx30g -Xms30g -XX:MaxDirectMemorySize=30g -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=90 -XX:G1NewSizePercent=1 -XX:InitiatingHeapOccupancyPercent=30 -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16 -XX:MaxTenuringThreshold=1 -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=32 -XX:G1OldCSetRegionThresholdPercent=5 -verbose:gc -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1 Options to print gc log
  18. 18. Important G1 Options MaxGCPauseMillis Sets a target for the maximum GC pause time. Soft Limit. (value: 90) G1NewSizePercent Minimum size for young generation. (value: 1) InitiatingHeapOccupancyPercent The Java heap occupancy threshold that triggers a concurrent GC cycle. (value: 65) MaxTenuringThreshold Maximum value for tenuring threshold. (value: 1) G1HeapRegionSize Sets the size of a G1 region. (value: 32m) G1MixedGCCountTarget The target number of mixed garbage collections after a marking cycle to collect old regions. (value: 32) G1OldCSetRegionThresholdPercent Sets an upper limit on the number of old regions to be collected during a mixed garbage collection cycle. (value: 5)
  19. 19. 15 min period
  20. 20. 15 min period Real heap usage is much higher than IHOP (Bad) 70 Mixed GC cycles in 15 min Too Frequently (Bad) GC few garbage for one mixed gc cycle (Bad) Young gen adaptivity(Good)
  21. 21. GC cost 4.2%~6.7% time (Bad)
  22. 22. Tuning #1 ● How do we tuning ? ○ IHOP > MaxMemstoreSize%Heap + L1CacheSize%Heap + Delta(~ 10%) ○ Bucket Cache is offheap, need NO consideration when tuning IHOP ● Next Tuning ○ MemstoreSize = 45% , L1CacheSize ~ 10%, Delta ~ 10% ○ Increase InitiatingHeapOccupancyPercent to 65
  23. 23. IHOP works as expected (Good) 6 Mixed GC cycles in 15 min(Better) Mixed GC reclaim more garbage(Good)
  24. 24. Time in GC decrease(Good) [4.2%~6.7%] -> [1.8%~5.5%]
  25. 25. Mixed GC takes 130ms? (Bad) Only 17 mixed GC ? But we set G1MixedGCCountTarget=32 (Bad) Most mixed GC take > 90ms (Bad)
  26. 26. Tuning #2 Let’s analyze our mixed gc logs
  27. 27. Tuning #2 Let’s analyze our mixed gc logs Cleanup 550-502=48 old regions every mixed gc Stop mixed gc util G1HeapWastePercent
  28. 28. Tuning #2 ● How do we tuning ? ○ G1OldCSetRegionThresholdPercent is 5. ○ 48 regions * 32MB(=1536MB) <= 30(g) * 1024 * 0.05 (=1536MB) ○ Try to decrease G1OldCSetRegionThresholdPercent for reducing mixed gc time. ● Next Tuning ○ Decrease G1OldCSetRegionThresholdPercent to 2
  29. 29. Less mixed gc take > 90ms (Better) More mixed gc(~22 times) (Better)
  30. 30. 33,554,432< 54,067,080 ? Many objects with age =1 in Eden gen will be moved into old gen directly , which lead to old gen increasing so fast. finally mix gc occur frequently. (Bad) Another Problem
  31. 31. Tuning #3 ● How do we tuning ? ○ Total Survivor size >= 30(G) * 1024 * 0.01 / 8 = 38.4 MB ○ Age-1-Size is about 50 MB ~ 60MB ● Next Tuning ○ Set SurvivorRatio from 8 (default) to 4
  32. 32. 5 Mixed GC cycles in 15 min(Better) (6 -> 5)
  33. 33. Only ~3 mixed gc in the cycle take > 90ms (Better)
  34. 34. Time in GC decrease(Better) [1.8%~5.5%] -> [1.6%~4.7%]
  35. 35. Conclusion -Xmx30g -Xms30g -XX:MaxDirectMemorySize=30g -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=90 -XX:G1NewSizePercent=1 -XX:InitiatingHeapOccupancyPercent=65 (Before 30) -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16 -XX:MaxTenuringThreshold=1 -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=32 -XX:G1OldCSetRegionThresholdPercent=2 (Before 5) -XX:SurvivorRatio=4 (Before 8)
  36. 36. Conclusion ● Current state ○ Young generation adaptivity (Good) ○ About 5 mixed gc cycles in 15 mins (Good) ○ Few(~3) mixed gc take more than 90ms (Good) ○ Overall time in gc is [1.6%~4.7%] (Good) ● Next tuning ○ Let’s increase our test pressure ○ 1 billion rows whose value is 1000 bytes (100 million rows before) ○ 200 write clients + 100 read clients (40 write clients + 20 read clients before)
  37. 37. Let’s analyze our mixed gc logs 2 continuous mixed gc happen in 101ms (Bad) 4 continuous mixed gc happen in 1 second (Bad)
  38. 38. ● Problem ○ Multiple continuous mixed gc happen in short time interval ○ A RPC lifetime may cross multiple continuous mixed gc which lead to the RPC become very slow ● Reason ○ Mixed GC consume the expected MaxGCPauseMillis, and some mixed gc take even longer time ○ G1 adjust the young gen to 1% ○ But because of our high QPS, young gen is consumed quickly. ○ We are in a mixed gc cycle. ○ Finally, G1 trigger multiple continuous mixed gc frequently Tuning #4
  39. 39. ● Typical G1 GC tuning method ○ Increase MaxGCPauseMillis to meet our mixed gc cost ? ■ But young gc will also spend more time. ● Trick ○ Increase G1NewSizePercent to enlarge the young gen size ■ Young GC will not be affected. ■ More young gen during mixed gc for avoiding trigger mixed gc by running out of young gen. ● Next tuning ○ Increase our G1NewSizePercent to 4 Tuning #4
  40. 40. Let’s analyze our mixed gc logs 2 continuous mixed gc interval >= 1 sec(Better)
  41. 41. Conclusion ● Initial IHOP ○ BucketCache OFF-Heap ■ IHOP > MaxMemstore%Heap + MaxL1CacheSize%Heap + Delta ○ BucketCache On-Heap ■ IHOP > MaxMemstore%Heap + MaxL1CacheSize%Heap + MaxL2CacheSize%Heap + Delta ● Young GC Cost ○ Choose the right G1NewSizePercent & MaxGCPauseMillis ● Mixed GC Cost ○ G1MixedGCCountTarget (Mixed GC cycle) ○ G1OldCSetRegionThresholdPercent (Single Mixed GC time) ○ Take care of your survivorRatio.
  42. 42. G1GC In XiaoMi Cluster 12G + 12G 12G + 12G 12G + 12G 50G heap + 50G offheap CMS to G1 12G + 12G ● Less Full GC,Better availability ● Less region servers. One Node with 128G Mem One Node with 128G Mem
  43. 43. G1GC Options In XiaoMi Cluster -Xmx50g -Xms50g -XX:MaxDirectMemorySize=50g -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis={50/90/500} for SSD/HDD/offline cluster -XX:G1NewSizePercent={2/5} for normal/heavy load cluster -XX:InitiatingHeapOccupancyPercent=65 -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16 -XX:MaxTenuringThreshold=1 -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64 -XX:G1OldCSetRegionThresholdPercent=5 -verbose:gc -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
  44. 44. Thank You !

×