Load balancer Outline in HBase (cdh3u0)

  • 1,513 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,513
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
18
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • 5min\n
  • \n
  • 結論としては、全てが平均化される\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 1/10 comitter\n

Transcript

  • 1. Load Balancer Outline inHBase ( CDH 3 update 0 ) Hisayoshi TAMAKI 2011/6/16 Twitter : @hisayosh
  • 2. Load Balancer Outline in HBase 1. Bulk assignmentLoad 2. Inline region placementBalancer 3. Cluster-wide load balancing
  • 3. Load Balancer Outline in HBase dedicated thread for the load balancing › Step 1 : Thread sleep during period at which the region balancer runs in the HMaster › Step 2 : Generate a global load balancing plan › Step 3 : Assignment Manager class executes the plan
  • 4. Step 1package org.apache.hadoop.hbase.master.HMasterprivate static Thread getAndStartBalancerChore(final HMaster master) { String name = master.getServerName() + "-BalancerChore"; int period = master.getConfiguration().getInt("hbase.balancer.period", 300000); // Start up the load balancer chore Chore chore = new Chore(name, period, master) { @Override protected void chore() { master.balance(); hbase-site.xml } hbase.balancer.period }; return Threads.setDaemonThreadRunning(chore); } Period at which the region balancer runs in the Master.
  • 5. Load Balancer Outline in HBase dedicated thread for the load balancing › Step 1 : Thread sleep during period at which the region balancer runs in the HMaster › Step 2 : Generate a global load balancing plan › Step 3 : Assignment Manager class executes the plan
  • 6. Step 2@Overridepublic boolean balance() { if (!this.balanceSwitch) return false; synchronized (this.balancer) { List<RegionPlan> plans = this.balancer.balanceCluster(assignments); if (plans != null && !plans.isEmpty()) { for (RegionPlan plan: plans) { LOG.info("balance " + plan); this.assignmentManager.balance(plan); } return true;}
  • 7. Step 2org.apache.hadoop.hbase.master.LoadBalancerpublic List<RegionPlan> balanceCluster( Map<HServerInfo,List<HRegionInfo>> clusterState) {}
  • 8. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 return Get Server number YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
  • 9. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 5 5 5 1 0 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
  • 10. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 4 4 4 1 0 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
  • 11. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 4 4 4 3 1 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
  • 12. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 3 3 3 3 1 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
  • 13. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 3 3 3 3 3 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
  • 14. this chart does notStep 2 : Planning flow cover all steps. needed to fill all Region Server : 5 to min and none NO Balancer Region Num : 16 left to drain all Start MIN = 3 to MAX MAX = 4 1 2 3 4 5 return Get Server number 4 3 3 3 3 YES Region we need Plan NO more to fill NO numServers ! =0? MIN ? YES YESMIN = region / region server grab one from eachMAX = region mod region server == 0 ? return null most loaded untilmin : min + 1 enough Walk down least server NO loaded, filling each outside to the MIN range ? YES we still have NO Walk down most return null regions to dish loaded, pruning out? return each to the MAX Region YES Plan Walk down least return loaded, assigning to assign underloaded to Region each to fill up to MIN MAX Plan
  • 15. Load Balancer Outline in HBase dedicated thread for the load balancing › Step 1 : Thread sleep during period at which the region balancer runs in the HMaster › Step 2 : Generate a global load balancing plan › Step 3 : Assignment Manager class executes the plan
  • 16. Step 3@Overridepublic boolean balance() { if (!this.balanceSwitch) return false; synchronized (this.balancer) { List<RegionPlan> plans = this.balancer.balanceCluster(assignments); if (plans != null && !plans.isEmpty()) { for (RegionPlan plan: plans) { LOG.info("balance " + plan); this.assignmentManager.balance(plan); } package org.apache.hadoop.hbase.master.AssignmentManager return true;}
  • 17. Questions?
  • 18.  Ted Yu’s Blog ( by Zhihong Yu ) http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.html