Successfully reported this slideshow.
Your SlideShare is downloading. ×

Multi-Tenant HBase Cluster - HBaseCon2018-final

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Multi-Tenant HBase Cluster - HBaseCon2018-final

  1. 1. © 2018 Bloomberg Finance L.P. All rights reserved. Me to We: Moving to a Multi-Tenant HBase Cluster HBaseCon 2018 June 18, 2018 Biju Nair
  2. 2. © 2018 Bloomberg Finance L.P. All rights reserved. Outline • Need for multi-tenancy • HBase features for multi-tenancy • Experience with multi-tenancy features • Summary
  3. 3. © 2018 Bloomberg Finance L.P. All rights reserved. Bloomberg in a nutshell The Bloomberg Terminal delivers a diverse array of information on a single platform to facilitate financial decision- making.
  4. 4. © 2018 Bloomberg Finance L.P. All rights reserved. Bloomberg by the numbers • Founded in 1981 • 325,000 subscribers in 170 countries • Over 19,000 employees in 192 locations • More News reporters than The New York Times + Washington Post + Chicago Tribune
  5. 5. © 2018 Bloomberg Finance L.P. All rights reserved. Bloomberg technology by the numbers • 5,000+ software engineers • 150+ technologists and data scientists devoted to machine learning • One of the largest private networks in the world • 120 billion pieces of data from the financial markets each day, with a peak of more than 10 million messages/second • 2 million news stories ingested / published each day (500+ news stories ingested/second) • News content from 125K+ sources • Over 1 billion messages and Instant Bloomberg (IB) chats handled daily
  6. 6. © 2018 Bloomberg Finance L.P. All rights reserved. HBase at Bloomberg • Started with v0.94.6 • 2B+ reads per day • 1B+ writes per day • 51+ TB of compressed data stored in HBase
  7. 7. © 2018 Bloomberg Finance L.P. All rights reserved. One Cluster Per Application Photo By: Todd Pruden, Fort Hood Public Affairs
  8. 8. © 2018 Bloomberg Finance L.P. All rights reserved. Consolidation
  9. 9. © 2018 Bloomberg Finance L.P. All rights reserved. Multiple Applications Per Cluster
  10. 10. © 2018 Bloomberg Finance L.P. All rights reserved. Streamline Use of Clusters
  11. 11. © 2018 Bloomberg Finance L.P. All rights reserved. Cost • 550 Wh – 4,800 KWh/Year • Greenhouse gas emissions – 8,755 miles of driving a car • CO2 emissions from – 0.535 of 1 year home electricity use • Sequestered by – 4.2 acres of U.S. forests in 1 year https://www.zdnet.com/article/toolkit-calculate-datacenter-server-power-usage/ https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator
  12. 12. © 2018 Bloomberg Finance L.P. All rights reserved. Namespace & ACL
  13. 13. © 2018 Bloomberg Finance L.P. All rights reserved. Namespace & ACL • hbase> create_namespace ‘PH‘ • create ' PH:red', 'f1' • hbase> grant ‘phadmin', 'RWXCA', '@PH‘ • hbase> revoke ‘phadmin', '@PH‘ • Updates to hbase-site.xml — hbase.security.authorization — hbase.coprocessor.region.classes — hbase.coprocessor.master.classes — hbase.coprocessor.regionserver.classes — hbase.security.exec.permission.checks
  14. 14. © 2018 Bloomberg Finance L.P. All rights reserved. Region Server Groups
  15. 15. © 2018 Bloomberg Finance L.P. All rights reserved. Architecture HMaster RS1 RS2 RS3 RS4 RS5 RS6 RS7 RS8 system ZK
  16. 16. © 2018 Bloomberg Finance L.P. All rights reserved. Architecture HMaster RS1 RS2 RS3 RS4 RS5 RS6 RS7 RS8 system ZK
  17. 17. © 2018 Bloomberg Finance L.P. All rights reserved. Internals HMaster RSGroupAdminEndpoint RSGroupBasedLoadBalancer FilterBy Group LoadBalancer RSGroupInfoManager RSGroup Table RSGroup on ZK Achieving HBase multitenancy with regionserver groups and favored nodes – DataWorks Summit 2017
  18. 18. © 2018 Bloomberg Finance L.P. All rights reserved. RSgroup Creation <property> <name>hbase.coprocessor.master.classes</name> <value>org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint</value> </property> <property> <name>hbase.master.loadbalancer.class</name> <value>org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer</value </property>
  19. 19. © 2018 Bloomberg Finance L.P. All rights reserved. RSgroup Creation • add_rsgroup ‘derivatives’ • move_servers_rsgroup ‘derivatives’,[‘rs1:16020’,’rs2:16021’] • move_tables_rsgroup ‘derivatives’,[‘tbl1’,’tbl2’] • list_rsgroups ‘derivatives’ • move_servers_namespaces_rsgroup ‘derivatives’,[‘ns1’,’ns2’]* • create_namespace 'ns',{'hbase.rsgroup.name' => ’derivatives'}
  20. 20. © 2018 Bloomberg Finance L.P. All rights reserved.
  21. 21. © 2018 Bloomberg Finance L.P. All rights reserved. Storage Quota
  22. 22. © 2018 Bloomberg Finance L.P. All rights reserved. Internals Master 1 Create Quota 3 Compute Space 4 Update Current Use Region Server 2 Report region size 5 Read Current Use 6 Enforce Policy HBase Quota
  23. 23. © 2018 Bloomberg Finance L.P. All rights reserved. Enable Quota <property> <name>hbase.quota.enabled</name> <value>true</value> </property>
  24. 24. © 2018 Bloomberg Finance L.P. All rights reserved. Space Quota Creation • set_quota TYPE => SPACE, TABLE => 'tb1', LIMIT => '100G', POLICY => NO_INSERTS • set_quota TYPE => SPACE, NAMESPACE => 'ns', LIMIT => '150T', POLICY => NO_WRITES • Other Policies: NO_WRITES_COMPACTIONS, DISABLE, NONE • Table level quota takes precedence • Preserve quota definition - hbase.quota.remove.on.table.delete
  25. 25. © 2018 Bloomberg Finance L.P. All rights reserved. CPU/Network Quota
  26. 26. © 2018 Bloomberg Finance L.P. All rights reserved. Internals Master Create Quota Region Server Verify current use and restrict HBase Quota
  27. 27. © 2018 Bloomberg Finance L.P. All rights reserved. Query Throttling • set_quota TYPE => THROTTLE, USER => 'u1', LIMIT => '10req/sec' • set_quota TYPE => THROTTLE, NAMESPACE => 'ns1', LIMIT => '10req/sec' • set_quota TYPE => THROTTLE, THROTTLE_TYPE => WRITE, TABLE => 't1', LIMIT => '10M/sec' • set_quota USER => 'u1', GLOBAL_BYPASS => true
  28. 28. © 2018 Bloomberg Finance L.P. All rights reserved. Meta-Data Quota
  29. 29. © 2018 Bloomberg Finance L.P. All rights reserved. Internals Master Create Quota Verify Quota Use Region Server Region Split HBase Quota
  30. 30. © 2018 Bloomberg Finance L.P. All rights reserved. Meta-Data Quota Creation • create_namespace 'ns1', {'hbase.namespace.quota.maxregions'=>'10'} • create_namespace 'ns2', {'hbase.namespace.quota.maxtables'=>'2’} • alter_namespace 'ns3', {METHOD => 'set', 'hbase.namespace.quota.maxtables'=>'5’} • alter_namespace 'ns3', {METHOD => 'unset', NAME => 'hbase.namespace.quota.maxtables'}
  31. 31. © 2018 Bloomberg Finance L.P. All rights reserved. Request Queues
  32. 32. © 2018 Bloomberg Finance L.P. All rights reserved. Request Queues • Number of request handler per region server — hbase.regionserver.handler.count • Number of queues per request handler — hbase.ipc.server.callqueue.handler.factor • read vs write queue ratio — hbase.ipc.server.callqueue.read.ratio • get vs scan queue ratio —hbase.ipc.server.callqueue.scan.ratio
  33. 33. © 2018 Bloomberg Finance L.P. All rights reserved. Rubber meets the road
  34. 34. © 2018 Bloomberg Finance L.P. All rights reserved. RSGroup - Performance 678189 687062 667110 760310 758847 815204 Run 1 Run 1 Run 3 Write Performance Avg Avg Avg RG 444978 481672 520719 619840 606806 603353 Run 1 Run 1 Run 3 Write Performance Min Min Min RG
  35. 35. © 2018 Bloomberg Finance L.P. All rights reserved. RSGroup – Data Locality 0.00% 8.16% 1.87% 23.81% 10.01% 2.66% 14.76% 4.90% 15.43% 14.71% 16.53% 15.99% 6.32% 21.72% No Fail Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Non local blocks Shared Cluster RS Group
  36. 36. © 2018 Bloomberg Finance L.P. All rights reserved.
  37. 37. © 2018 Bloomberg Finance L.P. All rights reserved. RSGroup • Replication doesn’t recognize RSgroup — Vote for HBASE-15531 J • RSes in groups should be distributed across racks • Group for SYSTEM tables should be large to withstand failures • Failure of RSes in a group requires master restart for recovery — HBASE-20728 • Reverting RSgroup feature requires additional steps • RSgroup uses SLB and hence supports region replication
  38. 38. © 2018 Bloomberg Finance L.P. All rights reserved. RSGroup • Altering namespace to set RSGroup is not supported in 1.x • Failure of all RSes in SYSTEM RSgroup makes cluster unavailable — HBASE-20632 • JIRA’s — HBASE-20566 — HBASE-20606 — HBASE-20607 — HBASE-20666
  39. 39. © 2018 Bloomberg Finance L.P. All rights reserved. Space Quota Limit, 10 Actual, 210.9 0 50 100 150 200 250 300 350 Run 1 - def Run 2 - def Run3 - def Run 4 - 60 sec Run 5 - 30 sec StoragesizeinGB Space Quota - Limit vs Actual
  40. 40. © 2018 Bloomberg Finance L.P. All rights reserved. Space Quota • Space Quota Properties — hbase.regionserver.quotas.policy.refresher.chore.period — hbase.regionserver.quotas.policy.refresher.report.percent — hbase.regionserver.quotas.region.size.reporting.chore.period — hbase.master.quotas.observer.chore.period • HBASE-18133 — Low-latency space quota size reports • HBASE-20588 — LIMIT => NONE doesn’t take effect • HBASE-20662 — Disabled table due to space violation doesn’t get re-enabled
  41. 41. © 2018 Bloomberg Finance L.P. All rights reserved. Meta-Data Quota
  42. 42. © 2018 Bloomberg Finance L.P. All rights reserved. Quota • Limit on region count can lead to large regions • HBASE-20705 — Having RPC Quota on a table prevents Space quota to be recreated/removed • Setting HDFS quota on a namespace can bring down RS
  43. 43. © 2018 Bloomberg Finance L.P. All rights reserved. Summary
  44. 44. © 2018 Bloomberg Finance L.P. All rights reserved. HBase Multi-Tenancy Story is Good HBase Feature Multi-Tenant Cluster NS* Table User Namespace + ACL Logical grouping of tenants X X X Regionserver Groups Allocate CPU/Memory/Network per tenant X X Space Quota Guarantee storage space per tenant X X Request Throttling Control CPU/Network use X X X Meta-Data Quota Control number of objects created X * NS - Namespace
  45. 45. © 2018 Bloomberg Finance L.P. All rights reserved. Improvements • HBASE-15531 • SYSTEM table availability • Memory Quota
  46. 46. © 2018 Bloomberg Finance L.P. All rights reserved. Acknowledgement • HBASE-6721 – Region server groups • HBASE-16961 – Storage quota • HBASE-11598 – Network/CPU Quota • HBASE-8410 – Meta-Data Quota • HBASE-8015 – Namespace and ACL • HBASE-11355 – Request Queue
  47. 47. © 2018 Bloomberg Finance L.P. All rights reserved. Thank You! Reference: http://hbase.apache.org Connect with Hadoop Team: hadoop@bloomberg.net
  48. 48. © 2018 Bloomberg Finance L.P. All rights reserved. We are hiring! Questions? https://www.bloomberg.com/careers

×