Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jee conf

1,631 views

Published on

The main topic of slides is building high availability high throughput system for receiveing and saving different kind of information with horizontal scalling possibility using HBase, Flume and Grizzly hosted on Amazon EC2 low cost instances. Talk describes HBase HA cluster setup process with useful hints and EC2 pitfalls, Flume setup process with providing comparasion between standalone and embedded Flume versions and show difference and usecases of both versions. A lot of attention payed to Flume2Hbase streaming features with tweaks and different approaches for speeding up this process.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Jee conf

  1. 1. 1 How to manage 10K+ request per second in HA manner ● Moisieienko Valerii ● JeeConf 2015
  2. 2. 2 Initial System Requirements ● High throughput (3K+ RPS 50/50 write/read) ● Scalability (5-10...50 instances) ● High availability (fall is unacceptable) ● Low cost (~ 500$/month)
  3. 3. 3 Low Cost?!
  4. 4. 4 Solution E L B E L B HTTPHTTP Amazon EC2 FLUMEFLUME GRIZZLYGRIZZLY HBASE
  5. 5. 5 EC2: Instances HDFS cluster Endpoint cluster vCPU 2 2 RAM 8 4 Storage type EBS (Magnetic) Instance-based Count 3 2
  6. 6. 6 Let's go deep: HDFS 1200 1200 1500 3072 512 512Server1,Heap(MB) 1200 1200 3072 512 2100 Server2,Heap(MB) 1200 1500 3072 512 1800 Server3,Heap(MB) HDFS Namenode (active) HDFS Namenode (backup) HDFS Datanode Other/Free
  7. 7. 7 Let's go deep: HBase 1200 1200 1500 3072 512 512Server1,Heap(MB) 1200 1200 3072 512 2100 Server2,Heap(MB) 1200 1500 3072 512 1800 Server3,Heap(MB) HBase Master (active) HBase Master (backup) HBase Regionserver Zookeeper Other/Free
  8. 8. 8 HBase: GC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=60 -XX:NewSize=64m -XX:MaxNewSize=64m
  9. 9. 9 HBase: Compression ● Around 40% safe data occupied space ● create 'table_test',{NAME => 'family_test',VERSIONS => '1', COMPRESSION => 'SNAPPY'} 0 20 40 60 80 100 120 Without compression Snappy GB
  10. 10. 10 Flume:Overview SOURCESOURCECHANNEL FLUMEFLUME HBASE NetcatMemoryAsyncHBaseSink SINKSINK
  11. 11. 11 Flume: HBase Sink Configuration agent.sinks.h1.type = org.apache.flume.sink.hbase.AsyncHBaseSink agent.sinks.h1.zookeeperQuorum = server1,server2,server3 agent.sinks.h1.channel = m1 agent.sinks.h1.table = users agent.sinks.h1.columnFamily = data agent.sinks.h1.batchSize = 100 agent.sinks.h1.coalesceIncrements = true agent.sinks.h1.enableWal = true agent.sinks.h1.serializer = ? extends org.apache.flume.sink.hbase.AsyncHbaseEventSeri alizer
  12. 12. 12 Flume: AsyncHbaseEventSerializer void initialize(byte[] table, byte[] cf); void setEvent(Event event); List<PutRequest> getActions(); List<AtomicIncrementRequest> getIncrements(); void cleanUp();
  13. 13. 13 AWS: VPC ● IP control ● Elastic IP permanency ● Internal DNS via Route 53 ● Security
  14. 14. 14 AWS: Storage Type Options Pro Contra Instance-based ●Direct hardware attach ●No network costs ●Fast ●Fixed volume, based on instance type ●Erase in case of instance reboot EBS ●Volume flexible configuration ●Magnetic/SSD choice ●Persistent ●Instance independent ●Pay only for occupied volume ●Network attach ●Pay for I/O requests S3 ●Fast via Amazon API ●Cheap ●Cloud storage ●Very slow when instance attached
  15. 15. 15 AWS: EBS Pro Contra Instance-based ●Direct hardware attach ●No network costs ●Fast ●Fixed volume, based on instance type ●Erase in case of instance reboot EBS ●Volume flexible configuration ●Magnetic/SSD choice ●Persistent ●Instance independent ●Pay only for occupied volume ●Network attach ●Pay for I/O requests S3 ●Fast via Amazon API ●Cheap ●Cloud storage ●Very slow when instance attached
  16. 16. 16 Results 1 Avg Max Write RPS 4K 6K Read RPS 2K 3K Read response time 15 ms 30 ms Costs: ~400 $/month
  17. 17. 17 That's all?!
  18. 18. 18 Load ● Traffic through endpoint x2 ● HBase data volume x5 ● HBase request count x2
  19. 19. 19 Major Compaction 2015-03-28 11:11:15,894 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in data of users,4012ea33-46b9- 42cd-9094-f2a939991fbf,1421554452486.c9e5790f5b45f4306fe860a5ec5480d7. into 136b805bebbe44319f032b7d6d89a5dc, size=331.7 M; total size for store is 331.7 M 2015-03-28 11:11:15,894 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=users,4012ea33-46b9-42cd-9094- f2a939991fbf,1421554452486.c9e5790f5b45f4306fe860a5ec5480d7., storeName=data, fileCount=3, fileSize=334.2 M, priority=4, time=181415339541360; duration=17sec
  20. 20. 20 Major Compaction: jitter ● Default: 0.3-0.5 (depends on version) ● 14 HRegions on 3 servers → ~2-3 simultaneous major compactions per day <property> <name> hbase.hregion.majorcompaction.jitter </name> <value>0.8</value> </property>
  21. 21. 21 Major Compaction: IO ● ~ 40% CPU IO wait time during compactions
  22. 22. 22 Major Compaction: Magnetic vs GP2 Magnetic GP2(SSD) IOPS ~200 ~1300 Cost 0,05$/GB 0,10$/GB
  23. 23. 23 HBase: Read Performance Tweaks ● hbase-site.xml <property> <name>hbase.regionserver.handler.count</name> <value>30</value> </property> <property> <name>hbase.regionserver.global.memstore.lowerLimit</name> <value>0.45</value> </property> <property> <name>hbase.regionserver.global.memstore.upperLimit</name> <value>0.55</value> </property> ● hdfs-site.xml <property> <name>dfs.datanode.handler.count</name> <value>30</value> </property>
  24. 24. 24 Flume: Standalone GRIZZLY ENDPOINT HTTPHTTPPUT OBJECT PUT OBJECT FLUMEFLUME JSONJSON HBASE
  25. 25. 25 Flume: Go Embedded GRIZZLY ENDPOINT DATA OBJECT DATA OBJECT EMBEDDED FLUME EMBEDDED FLUME PUT OBJECT PUT OBJECT HBASE HTTPHTTP
  26. 26. 26 Flume: Standalone vs Embedded Pro Contra Flume Standalone ●No / few code ●Variety data sources ●Scalability ●Data transformation ●Network overhead Flume Embedded ●No network overhead ●Less data transformation ●Full process control ●A lot of code ●Complexity ●Own data source handling
  27. 27. 27 Final Solution E L B E L B HTTPHTTP Amazon EC2 GRIZZLY+EMBEDDED FLUME HBASE
  28. 28. 28 Results 2 Avg Max Write RPS 7K 9K Read RPS 3K 5K Read response time 8 ms 27 ms Costs: ~420 $/month
  29. 29. 29 Conclusions ● AWS: VPC + EBS is a great deal for HDFS/HBase cluster ● Don't use big NewSize for HBase ● Use Snappy ● Use async batch puts where possible ● Use SSD for HBase data ● Remember about HBase compaction ● Embedded Flume could be a good solution
  30. 30. 30 Thank you for attention Questions? valeramoiseenko@gmail.com

×