Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
www.vmcd.org 
HBASE Performance test by YCSB 
----------------------------------------------------------------------------...
www.vmcd.org 
<name> 
HBase DB Binding</<repositories> 
<repository> 
<id> 
central</<name> 
Maven Repository Switchboard<...
www.vmcd.org 
</dependency> 
<dependency> 
<groupId> 
org.apache.hadoop</<artifactId>hadoop-hdfs</artifactId> 
<version>2....
www.vmcd.org 
<execution> 
<phase> 
package</<goals> 
<goal>single</goal> 
</goals> 
</execution> 
</executions> 
</plugin...
www.vmcd.org 
Run test scripts: 
$ls -l |grep .sh 
for mode in a b c d e f g h i j k do for thread in `seq 1 128` do /home...
www.vmcd.org 
Workload Summary 
Total 11 modes to run test script on 1~128 threads 
Workload all (ops/sec a-k mode) 
0 
50...
www.vmcd.org 
Workload A AverageLatency (us) : read50% update50% 
0 
5000 
10000 
15000 
20000 
25000 
1 
5 
9 
13 
17 
21...
www.vmcd.org 
Workload B AverageLatency (us) : read95% update5% 
0 
5000 
10000 
15000 
20000 
25000 
30000 
1 
5 
9 
13 
...
www.vmcd.org 
Workload C AverageLatency (us) : read100% 
0 
5000 
10000 
15000 
20000 
25000 
30000 
1 
5 
9 
13 
17 
21 
...
www.vmcd.org 
Workload D AverageLatency (us) : read95% insert5% 
0 
1000 
2000 
3000 
4000 
5000 
6000 
7000 
1 
5 
9 
13 ...
www.vmcd.org 
Workload E AverageLatency (us) : scan95% insert5% 
0 
10000 
20000 
30000 
40000 
50000 
60000 
70000 
80000...
www.vmcd.org 
Workload F AverageLatency (us) : read50% readmodify50% 
0 
2000 
4000 
6000 
8000 
10000 
12000 
1 
5 
9 
13...
www.vmcd.org 
Workload G AverageLatency (us) : read5% update95% 
0 
2000 
4000 
6000 
8000 
10000 
12000 
1 
5 
9 
13 
17 ...
www.vmcd.org 
Workload H AverageLatency (us) : update50% insert50% 
0 
500 
1000 
1500 
2000 
2500 
3000 
1 
5 
9 
13 
17 ...
www.vmcd.org 
Workload I AverageLatency (us) : read30% insert70% 
0 
2000 
4000 
6000 
8000 
10000 
12000 
1 
5 
9 
13 
17...
www.vmcd.org 
Workload J AverageLatency (us) : read20% update30% insert50% 
0 
2000 
4000 
6000 
8000 
10000 
12000 
14000...
www.vmcd.org 
Workload K AverageLatency (us) : update20% insert80% 
0 
1000 
2000 
3000 
4000 
5000 
6000 
7000 
8000 
1 
...
www.vmcd.org 
HBASE ENV: 3 regionservers and 10 hdfs datanodes 
#hbase version 2014-09-03 22:46:34,973 INFO [main] util.Ve...
www.vmcd.org 
Datanode OS system information : #cat /etc/issue CentOS release 6.5 (Final) 
#cat /proc/cpuinfo |grep proces...
www.vmcd.org 
Block device characteristics VPD page (SBC): Nominal rotation rate: 7200 rpm : 3.5 inch 
=== START OF INFORM...
Upcoming SlideShare
Loading in …5
×

HBASE Performane Test

1,116 views

Published on

Hbase performance test by YCSB including latency and ops (operations/s)

Published in: Technology
  • Be the first to comment

HBASE Performane Test

  1. 1. www.vmcd.org HBASE Performance test by YCSB ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Bulid YCSB From github We use thumbtack modify version https://github.com/thumbtack-technology/ycsb Modify pom.xml – using CDH platform version: <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 maven-v4_0_0.xsd"> <modelVersion> 4.0.0</<parent> <groupId> com.yahoo.ycsb</<artifactId> root</<version> 0.1.4</</parent> <artifactId>hbase-binding</artifactId>
  2. 2. www.vmcd.org <name> HBase DB Binding</<repositories> <repository> <id> central</<name> Maven Repository Switchboard</<url>http://repo1.maven.org/maven2</url> </repository> <repository> <id>maven-hadoop</id> <name> Hadoop Releases</<url>https://repository.cloudera.com/content/repositories/releases/</url> </repository> <repository> <id>cloudera-repos</id> <name> Cloudera Repos</<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> </repository> </repositories> <dependencies> <dependency> <groupId> org.apache.hbase</<artifactId>hbase-client</artifactId> <version>0.96.1.1-cdh5.0.0</version> <!-- <version>0.96.1.1-hadoop2</version> -->
  3. 3. www.vmcd.org </dependency> <dependency> <groupId> org.apache.hadoop</<artifactId>hadoop-hdfs</artifactId> <version>2.3.0-cdh5.0.0</version> </dependency> <dependency> <groupId> com.yahoo.ycsb</<artifactId> core</<version>${project.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId> org.apache.maven.plugins</<artifactId>maven-assembly-plugin</artifactId> <version>${maven.assembly.version}</version> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <appendAssemblyId> false</</configuration> <executions>
  4. 4. www.vmcd.org <execution> <phase> package</<goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> Run mvn clean package – build HBase DB Binding successfully Copy core-site.xml hbase-site.xml hdfs-site.xml to /home/hadoop/ycsb/hbase/src/main/conf Load 50 millions data to usertable: YCSB Client 0.1 Command line: -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=family -p recordcount=50000000 -s -threads 10 -p measurementtype=timeseries -p timeseries.granularity=2000 –load [OVERALL], Reconnections, 0.0 [OVERALL], RunTime(ms), 823765.0 [OVERALL], Operations, 4.9999968E7 [OVERALL], Throughput(ops/sec), 60696.883213052264
  5. 5. www.vmcd.org Run test scripts: $ls -l |grep .sh for mode in a b c d e f g h i j k do for thread in `seq 1 128` do /home/hadoop/ycsb/bin/ycsb run hbase -P workloads/workload"$mode" -p columnfamily=f1 -p table=test2 -s -threads "$thread" |tee workload"$mode"_t"$thread" done sleep 60 done -rw-rw-r-- 1 hadoop 58901 Sep 10 23:27 workloada_t1 … -rw-rw-r-- 1 hadoop 58901 Sep 10 23:27 workloadk_t120 -rw-rw-r-- 1 hadoop 58943 Sep 10 23:28 workloadk_t121 -rw-rw-r-- 1 hadoop 58998 Sep 10 23:28 workloadk_t122 -rw-rw-r-- 1 hadoop 59037 Sep 10 23:28 workloadk_t123 -rw-rw-r-- 1 hadoop 59087 Sep 10 23:29 workloadk_t124 -rw-rw-r-- 1 hadoop 59120 Sep 10 23:29 workloadk_t125 -rw-rw-r-- 1 hadoop 59176 Sep 10 23:29 workloadk_t126 -rw-rw-r-- 1 hadoop 59223 Sep 10 23:30 workloadk_t127 -rw-rw-r-- 1 hadoop 59266 Sep 10 23:30 workloadk_t128 -rw-rw-r-- 1 hadoop 54592 Sep 11 09:43 load50m.data
  6. 6. www.vmcd.org Workload Summary Total 11 modes to run test script on 1~128 threads Workload all (ops/sec a-k mode) 0 50000 100000 150000 200000 250000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 mode_a mode_b mode_c mode_d mode_e mode_f mode_g mode_h mode_i mode_j mode_k
  7. 7. www.vmcd.org Workload A AverageLatency (us) : read50% update50% 0 5000 10000 15000 20000 25000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read_50% update_50%
  8. 8. www.vmcd.org Workload B AverageLatency (us) : read95% update5% 0 5000 10000 15000 20000 25000 30000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read_95% update_5%
  9. 9. www.vmcd.org Workload C AverageLatency (us) : read100% 0 5000 10000 15000 20000 25000 30000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read100% read100%
  10. 10. www.vmcd.org Workload D AverageLatency (us) : read95% insert5% 0 1000 2000 3000 4000 5000 6000 7000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read 95% insert5%
  11. 11. www.vmcd.org Workload E AverageLatency (us) : scan95% insert5% 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 scan95% insert5%
  12. 12. www.vmcd.org Workload F AverageLatency (us) : read50% readmodify50% 0 2000 4000 6000 8000 10000 12000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read50% readmodify50%
  13. 13. www.vmcd.org Workload G AverageLatency (us) : read5% update95% 0 2000 4000 6000 8000 10000 12000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read5% update95%
  14. 14. www.vmcd.org Workload H AverageLatency (us) : update50% insert50% 0 500 1000 1500 2000 2500 3000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 update50% insert50%
  15. 15. www.vmcd.org Workload I AverageLatency (us) : read30% insert70% 0 2000 4000 6000 8000 10000 12000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read30% insert70%
  16. 16. www.vmcd.org Workload J AverageLatency (us) : read20% update30% insert50% 0 2000 4000 6000 8000 10000 12000 14000 16000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read20% update30% insert50%
  17. 17. www.vmcd.org Workload K AverageLatency (us) : update20% insert80% 0 1000 2000 3000 4000 5000 6000 7000 8000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 update20% insert80%
  18. 18. www.vmcd.org HBASE ENV: 3 regionservers and 10 hdfs datanodes #hbase version 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: HBase 0.96.1.1-cdh5.0.0 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: Subversion file: ///var/lib/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.0.0-Packaging-HBase-2014-03-27_22-53-27/hbase-0.96.1.1-cdh5.0.0 -r Unknown 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: Compiled by jenkins on Thu Mar 27 23:02:13 PDT 2014 export HBASE_OPTS="-Xloggc:/home/hadoop/logs/gc -XX:+UseConcMarkSweepGC -XX:PermSize=96m -XX:MaxPermSize=256m -Xss1m" export HBASE_REGIONSERVER_OPTS="$OPTS -XX:+HeapDumpOnOutOfMemoryError -Xms32g -Xmx32g -Xmn16g -XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80" #export HBASE_REGIONSERVER_OPTS="$OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -Xms32g -Xmx32g -Xmn12g -XX:SurvivorRatio=10 -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSClassUnloadingEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:CMSFullGCsBeforeCompaction=0 -XX:+DisableExplicitGC -Dsun.net.client.defaultConnectTimeout=10000 -Dsun.net.client.defaultReadTimeout=30000" export HBASE_MASTER_OPTS="-Xms4g -Xmx4g" export HBASE_LOG_DIR=/home/hadoop/logs/hbase export HBASE_MANAGES_ZK=false
  19. 19. www.vmcd.org Datanode OS system information : #cat /etc/issue CentOS release 6.5 (Final) #cat /proc/cpuinfo |grep processor wc -l 24 #cat /proc/meminfo |more MemTotal: 99035868 kB Mount system info: /dev/sdb1 on data1 type ext4 (rw) /dev/sdc1 on data2 type ext4 (rw) /dev/sdd1 on data3 type ext4 (rw) /dev/sde1 on data4 type ext4 (rw) /dev/sdf1 on data5 type ext4 (rw) /dev/sdg1 on data6 type ext4 (rw) /dev/sdh1 on data7 type ext4 (rw) /dev/sdi1 on data8 type ext4 (rw) /dev/sdj1 on data9 type ext4 (rw) /dev/sdk1 on data10 type ext4 (rw) /dev/sdl1 on data11 type ext4 (rw) /dev/sdm1 on data12 type ext4 (rw) #sg_vpd /dev/sde --page=0xb1
  20. 20. www.vmcd.org Block device characteristics VPD page (SBC): Nominal rotation rate: 7200 rpm : 3.5 inch === START OF INFORMATION SECTION === Device Model: ST2000NM0033-9ZM175 Serial Number: Z1X1YV4Q LU WWN Device Id: 5 000c50 06672b0c2 Firmware Version: SN03 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ACS-2 (revision not indicated) Local Time is: Wed Sep 3 15:53:55 2014 CST SMART support is: Available - device has SMART capability. SMART support is: Enabled

×