Load Balancer Outline inHBase ( CDH 3 update 0 )               Hisayoshi TAMAKI                       2011/6/16         Tw...
Load Balancer Outline in HBase           1. Bulk assignmentLoad       2. Inline region placementBalancer           3. Clus...
Load Balancer Outline in    HBase   dedicated thread for the load balancing    › Step 1 : Thread sleep during period at w...
Step 1package org.apache.hadoop.hbase.master.HMasterprivate static Thread getAndStartBalancerChore(final HMaster master) {...
Load Balancer Outline in    HBase   dedicated thread for the load balancing    › Step 1 : Thread sleep during period at w...
Step 2@Overridepublic boolean balance() { if (!this.balanceSwitch) return false; synchronized (this.balancer) {     List<R...
Step 2org.apache.hadoop.hbase.master.LoadBalancerpublic List<RegionPlan> balanceCluster(   Map<HServerInfo,List<HRegionInf...
this chart does notStep 2 : Planning flow                                                          cover all steps.       ...
this chart does notStep 2 : Planning flow                                                                     cover all st...
this chart does notStep 2 : Planning flow                                                                     cover all st...
this chart does notStep 2 : Planning flow                                                                     cover all st...
this chart does notStep 2 : Planning flow                                                                     cover all st...
this chart does notStep 2 : Planning flow                                                                     cover all st...
this chart does notStep 2 : Planning flow                                                                     cover all st...
Load Balancer Outline in    HBase   dedicated thread for the load balancing    › Step 1 : Thread sleep during period at w...
Step 3@Overridepublic boolean balance() { if (!this.balanceSwitch) return false; synchronized (this.balancer) {     List<R...
Questions?
   Ted Yu’s Blog ( by Zhihong Yu )    http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.html
Upcoming SlideShare
Loading in …5
×

Load balancer Outline in HBase (cdh3u0)

1,939 views

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,939
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
22
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • 5min\n
  • \n
  • &amp;#x7D50;&amp;#x8AD6;&amp;#x3068;&amp;#x3057;&amp;#x3066;&amp;#x306F;&amp;#x3001;&amp;#x5168;&amp;#x3066;&amp;#x304C;&amp;#x5E73;&amp;#x5747;&amp;#x5316;&amp;#x3055;&amp;#x308C;&amp;#x308B;\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 1/10 comitter\n
  • Load balancer Outline in HBase (cdh3u0)

    1. 1. Load Balancer Outline inHBase ( CDH 3 update 0 ) Hisayoshi TAMAKI 2011/6/16 Twitter : @hisayosh
    2. 2. Load Balancer Outline in HBase 1. Bulk assignmentLoad 2. Inline region placementBalancer 3. Cluster-wide load balancing
    3. 3. Load Balancer Outline in HBase dedicated thread for the load balancing › Step 1 : Thread sleep during period at which the region balancer runs in the HMaster › Step 2 : Generate a global load balancing plan › Step 3 : Assignment Manager class executes the plan
    4. 4. Step 1package org.apache.hadoop.hbase.master.HMasterprivate static Thread getAndStartBalancerChore(final HMaster master) { String name = master.getServerName() + "-BalancerChore"; int period = master.getConfiguration().getInt("hbase.balancer.period", 300000); // Start up the load balancer chore Chore chore = new Chore(name, period, master) { @Override protected void chore() { master.balance(); hbase-site.xml } hbase.balancer.period }; return Threads.setDaemonThreadRunning(chore); } Period at which the region balancer runs in the Master.
    5. 5. Load Balancer Outline in HBase dedicated thread for the load balancing › Step 1 : Thread sleep during period at which the region balancer runs in the HMaster › Step 2 : Generate a global load balancing plan › Step 3 : Assignment Manager class executes the plan
    6. 6. Step 2@Overridepublic boolean balance() { if (!this.balanceSwitch) return false; synchronized (this.balancer) { List<RegionPlan> plans = this.balancer.balanceCluster(assignments); if (plans != null && !plans.isEmpty()) { for (RegionPlan plan: plans) { LOG.info("balance " + plan); this.assignmentManager.balance(plan); } return true;}
    7. 7. Step 2org.apache.hadoop.hbase.master.LoadBalancerpublic List<RegionPlan> balanceCluster( Map<HServerInfo,List<HRegionInfo>> clusterState) {}
    8. 8. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 return Get Server number YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
    9. 9. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 5 5 5 1 0 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
    10. 10. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 4 4 4 1 0 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
    11. 11. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 4 4 4 3 1 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
    12. 12. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 3 3 3 3 1 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
    13. 13. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 3 3 3 3 3 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
    14. 14. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 4 3 3 3 3 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
    15. 15. Load Balancer Outline in HBase dedicated thread for the load balancing › Step 1 : Thread sleep during period at which the region balancer runs in the HMaster › Step 2 : Generate a global load balancing plan › Step 3 : Assignment Manager class executes the plan
    16. 16. Step 3@Overridepublic boolean balance() { if (!this.balanceSwitch) return false; synchronized (this.balancer) { List<RegionPlan> plans = this.balancer.balanceCluster(assignments); if (plans != null && !plans.isEmpty()) { for (RegionPlan plan: plans) { LOG.info("balance " + plan); this.assignmentManager.balance(plan); } package org.apache.hadoop.hbase.master.AssignmentManager return true;}
    17. 17. Questions?
    18. 18.  Ted Yu’s Blog ( by Zhihong Yu ) http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.html

    ×