Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Five major tips to maximize performance on a 200+ SQL HBase/Phoenix cluster

859 views

Published on

Slides presented at PhoenixCon 2016.

Published in: Software
  • Be the first to comment

Five major tips to maximize performance on a 200+ SQL HBase/Phoenix cluster

  1. 1. Five major tips to maximize performance on a 200+ SQL HBase/Phoenix cluster Masayasu “Mas” Suzuki Shinji Nagasaka Takanari Tamesue Sony Corporation
  2. 2. 2 Who we are, and why we chose HBase/Phoenix  We are DevOps members from Sony’s News Suite team – http://socialife.sony.net/  HBase/Phoenix was chosen because of – Scalability, – SQL compatibility, and – secondary indexing support
  3. 3. 3 Our use case Internet Sony News Suite Server Architecture Application Server HBase Phoenix EventHandler HTTP SQL (READ) SQL (WRITE) Fetcher HTTP End user Outside content providers Main use case is caching contents temporarily
  4. 4. 4 Basic test design  Query response time is measured as shown in red  Query read/write ratio is 6 to 1  12 different types of queries using eight separate indexes Application Server HBase Phoenix EventHandler SQL (READ) SQL (WRITE) Fetcher
  5. 5. 5 Table schema  A table with 1.2 billion records were created  Each record is around 1.0 Kbytes – Raw data is around 1.7 KBytes each – Gzip is used to compress column pt and hence the total comes out to be around 1.0 Kbytes  id is the primary key – Two MD5 hashed values are concatenated to create id • Example: df461a2bda4002aaaa8117d4e43ee737_cfcd208495d565ef66e7dff9f98764da CHAR(65) id VARCHAR ai VARCHAR ao DECIMAL b DECIMAL c CHAR(5) cl CHAR(2) lg DECIMAL lw DECIMAL u VARBINARY pt 1adf… TR DSATE... 82122... 9071.9 true es 823.199 0.1243 (binary) 9d0a… FB Adad... 54011… 122114.5 true ja 23.632 5.22 (binary) c5ae... KW 4 of … 20011… 3253.55 false fr 0.343 2.77 (binary) ea4a... AB p7mj… 67691… 8901.0 true en 76.21 23.11 (binary) 1.2 billion records
  6. 6. 6 Split points  Because it was impossible to store all 1.2 billion records on one single node, we manually split the tables by defining the split points  Split points were set so that each divided block, or region file, would be nearly equal in size – This was possible because we knew a. the exact range of our primary keys, and b. the hashed values of our primary keys would be uniformly distributed CREATE TABLE IF NOT EXISTS TBL_1200M_IDX_LZ4_VER1_SPLT200_PTBIN_INT2DEC ( id CHAR(65) NOT NULL, ai VARCHAR, ao VARCHAR, b DECIMAL, c DECIMAL, cl CHAR(5), lg CHAR(2), lw DECIMAL, u DECIMAL, p_t VARBINARY, CONSTRAINT my_pk PRIMARY KEY ( id ) ) COMPRESSION='LZ4', VERSIONS='1', MAX_FILESIZE=26843545600 SPLIT ON ( '0148','0290','03d8','0520','0668','07b0','08f8','0a40', …,'fef8' );
  7. 7. 7 Distribution of region file per RegionServer  If split points can be evenly set, then data allocation can be evened out Different color denotes different tables 200 RegionServer Total size per node
  8. 8. 8 Queries  Ratio of R/W queries is 6 to 1  Sample READ queries  Sample WRITE queries  Constants (ex. in the above example, 228343239, or the value of b) were randomly generated to simulate current production environment SELECT id FROM TBL_1200M_IDX_LZ4_VER1_SPLT200_PTBIN_INT2DEC WHERE b=228343239 AND cl='false'; SELECT id FROM TBL_1200M_IDX_LZ4_VER1_SPLT200_PTBIN_INT2DEC WHERE ai=‘AB' AND cl='false' AND c>0 AND c<1417648603068; /* Written as a Java PreparedStatement */ UPSERT INTO TBL_1200M_IDX_LZ4_VER1_SPLT200_PTBIN_INT2DEC (id,p_t,c,lw,u) VALUES (?,?,?,?,?)
  9. 9. 9 Queries – Details Query No. Name Read/Write Percentage generated Description Randomly generated part 1 Id READ 25% Search using primary key Id (primary key) 2 IdCnt READ 10% Count using primary key Id (primary key) 3 IdOr READ 10% Search using “OR” of ten primary keys Id (primary key) 4 AiAoU READ 5% Search using columns Ai, Ao, and U Ai, Ao, U 5 AiCCl READ 5% Search using columns Ai, C, and Cl Ai, C, Cl 6 AiLwCl READ 5% Search using columns Ai, Lw, and Cl Ai, Lw, Cl 7 AiULg READ 5% Search using columns Ai, U, and Lg Ai, U, Lg 8 BCl READ 5% Search using columns B and Cl B, Cl 9 BLg READ 5% Search using columns B and Lg B, Lg 10 CLg READ 5% Search using columns C and Lg C, Lg 11 LwLg READ 5% Search using columns Lw and Lg Lw, Lg 12 PtCLwU WRITE 15% Upsert binary data Pt and upsert columns C, Lw, and U Id (primary key), Pt, C, Lw, U
  10. 10. 10 Secondary indexes  Following eight indexes were created  Eight indexes are designed to be orthogonal indexes  Split points were manually set for index tables so that each region file would be similar in size Index No. Name Index type Description 1 AiAoU CHAR/CHAR/DECIMAL For use in search using columns Ai, Ao, and U 2 AiCCl CHAR/DECIMAL/CHAR For use in search using columns Ai, C, and Cl 3 AiLwCl CHAR/DECIMAL/CHAR For use in search using columns Ai, Lw, and Cl 4 AiULg CHAR/DECIMAL/CHAR For use in search using columns Ai, U, and Lg 5 BCl DECIMAL/CHAR For use in search using columns B and Cl 6 BLg DECIMAL/CHAR For use in search using columns B and Lg 7 CLg DECIMAL/CHAR For use in search using columns C and Lg 8 LwLg DECIMAL/CHAR For use in search using columns Lw and Lg
  11. 11. 11 Test environment HBase Clusters Zookeepers HMasters Zookeeper 1 Zookeeper 2 Zookeeper 3 HMaster (Main) HMaster (Secondary) HMaster (Secondary Backup) RegionServer sRegionServer 1 Clients Client 1 Client 2 Client 100 ・・・ disk RegionServer 1 disk RegionServer 200 disk SYSTEM.CATALOG (Meta data for Phoenix Plug-in) ・・・ 100 clients (100 x c4.xlarge) 3 Zookeepers (3 x m3.xlarge) 3 HMasters (3 x m3.xlarge) 200 RegionServers (199 x r3.xlarge) (1 x c4.8xlarge)
  12. 12. 12 Tools used  Tools were especially useful for – Pinpointing the bottlenecks in resource usage – Determining when and where an error occurred within the cluster – Verifying the effect of solutions applied – Managing multiple nodes seamlessly without having to manage them separately Tools used Purpose Analysis of resource usage per AWS instance (ex. CPU usage, network traffic, disk utilization, Java stats) Analysis of status of HBase and Hadoop layers (ex. number of regions, store files, requests) Analysis of distribution of each HBase table over the cluster (ex. number and size of region files per node) Fabric Remotely control multiple nodes via SSH
  13. 13. 13 Performance test apparatus & results  Test apparatus  Test results Specs Number of records 1.2 billion records (1KB each) Number of indexes 8 orthogonal indexes Servers 3 Zookeepers (Zookeeper 3.4.5, m3.xlarge x 3) 3 HMaster servers (hadoop 2.5.0, hbase 0.98.6, Phoenix 4.3.0, m3.xlarge x 3) 200 RegionServers (hadoop 2.5.0, hbase 0.98.6, Phoenix 4.3.0, r3.xlarge x 199, c4.8xlarge x 1) Clients 100 x c4.xlarge Results Number of queries 51,053 queries/sec Response time (average) 46 ms
  14. 14. 14 Cost  Total: $325,236 (per year, “All Upfront” pricing)  This is a preliminary setup! – There is room for further spec/cost optimization Node Type Instance Type Quantity Cost (per year) HBase:ZooKeeper m3.xlarge 3 $ 4,284 Hadoop:Name Node HBase:Hmaster m3.xlarge 3 $ 4,284 Hadoop:Data Node HBase:RegionServer r3.xlarge 199 $ 307,455 HBase:RegionServer (for housing meta table SYSTEM.CATALOG) c4.8xlarge 1 $ 9,213
  15. 15. 15 Five major tips to maximize performance using HBase/Phoenix Ordered by effectiveness
  16. 16. 16 Tips 1 – Use SQL hint clause when using an index  Response without hint clause  Response with hint clause 0 50 100 150 200 250 300 350 400 Id IdCnt IdOr AiAoU AiCCl AiLwCl AiULg BCl BLg CLg LwLg PtCLwU 0.08 0.5 1 1.5 2 2.5 [ms] Queries using primary key Write query Queries using index Elapsed time [hours] Performance improved by 6 times 0 50 100 150 200 250 300 350 400 Id IdCnt IdOr AiAoU AiCCl AiLwCl AiULg BCl BLg CLg LwLg PtCLwU 0.08 0.5 1 1.5 2 2.5 [ms] Queries using primary key Write query Queries using index Elapsed time [hours]
  17. 17. 17 Tips 1 – Use SQL hint clause when using an index  Major possible cause (yet to be verified) – When the index is used, an extra RPC is issued to verify latest meta/statistics – Using hint clause may reduce this RPC (still hypothesis)  Other possible solutions – Changing “UPDATE_CACHE_FREQUENCY” (available from Phoenix 4.7) may resolve this issue (we have not tried this yet) From Phoenix website … https://phoenix.apache.org/#Altering “When a SQL statement is run which references a table, Phoenix will by default check with the server to ensure it has the most up to date table metadata and statistics. This RPC may not be necessary when you know in advance that the structure of a table may never change.”
  18. 18. 18 Tips 2 – Use memories aggressively  In early stages of our testing, disk utilization and iowait of RegionServers were extremely high Test period Test period iowait
  19. 19. 19 Tips 2 – Use memories aggressively  Issue was most critical during major compaction and index creation  Initially, we thought we had enough memory – Total size of data (includes all tables/indexes and mirrored data in Hadoop layer) • More than 1,360 GB – Total available memory combined on RegionServers (then) • Around 1,500 GB (m3.2xlarge(30GiB) x 50 nodes)  But this left very little margin for computation intensive tasks  We decided to allocate memory at least 3 times the size of data for added protection and performance (has worked thus far)
  20. 20. 20 Tips 3 – Manually split the region file but don’t over split them  A single table is too big to be placed and managed by one single node  We wanted to know whether we should split in a “more finer” way or in a “more coarser” way
  21. 21. 21 Tips 3 – Manually split the region file but don’t over split them  Comparison between 200 and 4002 split points – 200 RegionServers were used in both cases Don’t over split region files 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 0 2 4 6 8 10 12 14 16 Volumeprocessed[queries/sec] Elapsed time [H] SplitPoint = 200 SplitPoint = 4002 0 100 200 300 400 500 600 700 0 2 4 6 8 10 12 14 16 Responsetime[ms] Elapsed time [H] SplitPoint = 200 SplitPoint = 4002
  22. 22. 22 Tips 4 – Scale-out instead of scale-up  Comparison of RegionServers running c3.4xlarge and c3.8xlarge – c3.8xlarge is twice the spec of c3.4xlarge – Combined computing power of “100 nodes of c3.4xlarge” is equal to “50 nodes of c3.8xlarge”, but former scores better 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 0 2 4 6 8 Volumeprocessed[queries/sec] Elapsed time [H] c3.4xlarge x 100 c3.8xlarge x 50 0 20 40 60 80 100 120 140 160 180 200 0 2 4 6 8 Responsetime[ms] Elapsed time [H] c3.4xlarge x 100 c3.8xlarge x 50 Scale-out!
  23. 23. 23 Tips 5 – Avoid running power intense tasks simultaneously  For example, do not run major compaction together with index creation  Also, performance impact from major compaction can be lessened by running them in smaller units 26,142 29,980 24000 25000 26000 27000 28000 29000 30000 31000 Volumeprocessed[queries/sec] 91 ms 80 ms 72 74 76 78 80 82 84 86 88 90 92 94 Responsetime[ms] Major compaction for nine tables done simultaneously Major compaction for nine tables done separately Major compaction for nine tables done simultaneously Major compaction for nine tables done separately 13% increase in volume processed 9% faster
  24. 24. 24 Items of very limited or no success
  25. 25. 25 First and foremost  Please understand that these are lessons learned through our tests on our environment  Any one or all of these items may prove useful in your environment
  26. 26. 26 Items of limited success – Changing GC algorithm  RegionServers’ GC algorithm were changed and tested  Performance is more even with G1  Performance of G1 is on average, 2% lower than CMS 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 0 2 4 6 8 10 Volumeprocessed[queries/sec] Elapsed time [H] CMS G1 0 20 40 60 80 100 120 140 160 180 200 0 2 4 6 8 10 Responsetime[ms] Elapsed time [H] CMS G1
  27. 27. 27 Items of limited success – Changing Java heap size  RegionServers’ Java heap size were changed and tested  Maximum physical memory is 30.5 GiB (r3.xlarge)  When heap was set to 26.0 GB, system crashed after five hours 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 0 4 8 12 16 Volumeprocessed[queries/sec] Elapsed time [H] JavaHeap = 20.5GB JavaHeap = 23.0GB JavaHeap = 26.0GB 0 20 40 60 80 100 120 140 160 180 200 0 4 8 12 16 Responsetime[ms] Elapsed time [H] JavaHeap = 20.5GB JavaHeap = 23.0GB JavaHeap = 26.0GB
  28. 28. 28 Items of limited success – Changing disk file format  RegionServers’ disk file format was changed and tested  The newer xfs tend to score slightly better when compared at its highs 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 0 4 8 12 16 Volumeprocessed[queries/sec] Elapsed time [H] ext4 xfs 0 20 40 60 80 100 120 140 160 180 200 0 4 8 12 16 Responsetime[ms] Elapsed time [H] ext4 xfs
  29. 29. 29 Closing comments
  30. 30. 30 Five major tips to maximize performance on HBase/Phoenix Ordered by effectiveness (Most effective on the very top) – An extra RPC is issued when the client runs a SQL statement that uses a secondary index – Using SQL hint clause can mitigate this – From Ver. 4.7, changing “UPDATE_CACHE_FREQUENCY” may also work (we have yet to test this) – A memory rich node should be selected for use in RegionServers so as to minimize disk access – More nodes running in parallel yield better results than fewer but powerful nodes running in parallel – As an example, running major compaction and index creation simultaneously should be avoided Tips 1. Use SQL hint clause when using a secondary index Tips 2. Use memories aggressively Tips 3. Manually split the region file if you can but never over split them Tips 4. Scale-out instead of scale-up Tips 5. Avoid running power intensive tasks simultaneously
  31. 31. 31 Special Thanks  Takafumi Suzuki – Thank you very much for the countless and invaluable discussions – We owe the success of this project to you!  Thank you very much!
  32. 32. “Sony” is a registered trademark of Sony Corporation. Names of Sony products and services are the registered trademarks and/or trademarks of Sony Corporation or its Group companies. Other company names and product names are the registered trademarks and/or trademarks of the respective companies.

×