Apache HBase in the Enterprise Data Hub at Cerner

380 views

Published on

Swarnim Kulkarni (Cerner)
Cerner has been an active consumer of HBase for a very long time, storing petabytes of healthcare data in its multiple isolated HBase clusters. This talk will walk through the design of Cerner's enterprise data hub with a focus on the multi-tenant HBase as a service offering within the hub.

Published in: Software
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
380
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Apache HBase in the Enterprise Data Hub at Cerner

  1. 1. HBase  in  the  Enterprise  Data  Hub   It’s  all  about  co-­‐exis0ng  peacefully!   Swarnim  Kulkarni   May  24th  2016  
  2. 2. innova8on  
  3. 3. innova8on  noun   in·∙no·∙va·∙0on  ˌi-­‐nə-­‐ˈvā-­‐shən   1.    the  introduc0on  of  something  new   2.    a  new  idea,  method  or  device  
  4. 4. Chart  Search  
  5. 5. Sepsis  
  6. 6. Millennium  +  
  7. 7. Popula8on  Health  
  8. 8. HBase  sits  at  the   core  of  a  lot  of  these   solu8ons!  
  9. 9. Each  solu8on  built   with  specific   business  problem  in   mind  
  10. 10. CHART     SEARCH   SEPSIS   MILLENNIUM  +   POP  HEALTH  
  11. 11. CHART     SEARCH   SEPSIS   MILLENNIUM  +   POP  HEALTH  
  12. 12. Prevents  assets  from  being  shared   High  investment   High  maintenance  costs   High  barrier  of  entry  for  innova8on    
  13. 13. Enter     The  Data  Hub  
  14. 14. Bob   Tim  
  15. 15. Bob  
  16. 16. = + +
  17. 17. Tim  
  18. 18. = + +
  19. 19. Innova8ons   Source  of  data   Raw  data  
  20. 20. Data  Hub  Owner   Data  Hub  Consumers   Workers  
  21. 21. “An  Enterprise  Data  Hub  is  a  centralized   loca0on  to  store  all  data,  for  as  long  as   needed,  in  its  original  fidelity;  with   flexibility  to  run  variety  of  enterprise   workloads  including  batch  processing  and   interac0ve  SQL  -­‐  with  robust  security,   audi0ng  and  management.”  
  22. 22. “An  Enterprise  Data  Hub  is  a  centralized   loca-on  to  store  all  data,  for  as  long  as   needed,  in  its  original  fidelity;  with   flexibility  to  run  variety  of  enterprise   workloads  including  batch  processing  and   interac0ve  SQL  -­‐  with  robust  security,   audi0ng  and  management.”  
  23. 23. “An  Enterprise  Data  Hub  is  a  centralized   loca-on  to  store  all  data,  for  as  long  as   needed,  in  its  original  fidelity;  with   flexibility  to  run  variety  of  enterprise   workloads  including  batch  processing  and   interac0ve  SQL  -­‐  with  robust  security,   audi0ng  and  management.”  
  24. 24. “An  Enterprise  Data  Hub  is  a  centralized   loca-on  to  store  all  data,  for  as  long  as   needed,  in  its  original  fidelity;  with   flexibility  to  run  variety  of  enterprise   workloads  including  batch  processing  and   interac0ve  SQL  -­‐  with  robust  security,   audi0ng  and  management.”  
  25. 25. “An  Enterprise  Data  Hub  is  a  centralized   loca-on  to  store  all  data,  for  as  long  as   needed,  in  its  original  fidelity;  with   flexibility  to  run  variety  of  enterprise   workloads  including  batch  processing  and   interac0ve  SQL  -­‐  with  robust  security,   audi-ng  and  management.”  
  26. 26. Mul8-­‐tenant   Secure  and  Compliant   Ac8ve  archive  of  all  data   Low  barrier  of  entry   Hadoop  as  a  Service   Data  Hub  is….  
  27. 27. So  why  provide  HBase   as  a  service?  
  28. 28. Maximize  cluster   u8liza8on   Minimum  resource  guarantee   sharing   Low  barrier  of  entry   HBase  as  a  Service   Knowledge   Data  Data  
  29. 29. Requirements  
  30. 30. Support  mul8-­‐tenant  environment  for  mul8ple   users   Isolated  deployment   Set  quota  per  consumer   ACLs  for  project  level  administra8on   Appropriate  security  for  user  authen8ca8on  
  31. 31. Architecture  
  32. 32. Smaller  HBase  clusters  part  of  the  bigger  “hub”   Scale  as  per  need   Easier  management  
  33. 33. Resource  Management  
  34. 34. Making  sure  that  you  have  sufficient   resources  before  you  start  the  job!  
  35. 35. CGroups   •  Control  Groups   •  Police  and  limit  CPU,  Disk  I/O  and  Memory  usage   •  Resource  guarantee  by  sta8c  par88oning     •  Very  useful  in  case  of  conten8on      
  36. 36. •  Value  for  resource-­‐limits  driven  by  individual  tenants  depending   on  the  workload   •  Defaults  to  70-­‐30  usage  for  memory  and  CPU  between   (NodeManager  +  Datanode)  and  RegionServer   •  Higher  memory  on  RS  (at  least  4  GB)  helps  avoid  swapping   and  happier  GC   CPU   CPU   CPU   MEMORY   MEMORY   MEMORY   SEPSIS   M+   POP  HEALTH  
  37. 37. What  we  learned   •  Recommend  tenants  to  give  atleast  4  GB  of  memory  to   the  regionserver  for  smoother  opera8on   •  Coopera8ve  memory  (ex:  via  JVM  heaps,  max.  container   sizes)  works  bejer  than  cgroup  limits   •  Disable  swapping  on  HBase  nodes   •  Disable  HDFS  load  balancer   •  CMS  GC  performs  way  bejer   •  Limit  number  of  containers  running  on  nodes  to  maximize   performance  
  38. 38. Property   Value   yarn.nodemanager.container-­‐ executor.class   org.apache.hadoop.yarn.server.node manager.LinuxContainerExecutor   yarn.nodemanager.linux-­‐container-­‐ executor.group   hadoop   yarn.nodemanager.linux-­‐container-­‐ executor.resources-­‐handler.class   org.apache.hadoop.yarn.server.node manager.u8l.CgroupsLCEResourcesHa ndler   yarn.nodemanager.linux-­‐container-­‐ executor.cgroups.hierarchy   /yarn   yarn.nodemanager.linux-­‐container-­‐ executor.cgroups.mount   true   hjps://hadoop.apache.org/docs/current/hadoop-­‐yarn/hadoop-­‐yarn-­‐site/NodeManagerCgroups.html  
  39. 39. Request  queues   •  Priori8ze  variety  of  workflows  that  need  access  to  HBase   •  Important  to  meet  SLAs  for  tenants   •  “FIFO”  vs  “Deadline”  queue  type   •  “Deadline”(default)  proved  to  work  prejy  well  for  most  of   the  cases   •  Cannot  be  set  per  tenant  but  has  to  be  set  per  cluster  –   support  coming  soon  
  40. 40. Quotas   •  Makes  sure  that  no  single  tenant  abuses  the  system   •  Peaceful  coexistence   •  Promotes  a  pay-­‐per-­‐use  model.  Could  buy  a  higher   throjle  limit  by  contribu8ng  more  number  of  nodes   •  Usually  we  set  throjle  on  a  per  namespace  basis  but   could  also  set  on  the  per  table/user  basis  if  needed  
  41. 41. Tenant  Isola8on  
  42. 42. Namespace   •  Logical  grouping  of  HBase  tables   •  Analogous  to  databases   •  Provides  tenants  with  individual  space  to  operate  on   •  Tied  to  the  AD  group  used  when  onboarding   •  Could  also  apply  quotas(max  regions/tables)  per   namespace(HBASE-­‐8410)  but  not  using  that  feature  for   now  
  43. 43. Security  
  44. 44. Authen8ca8on   •  Cluster  secured  by  Kerberos   •  Disallow  impersona8on   •  Require  kinit  to  first  authen8cate  with  the  KDC  before   accessing  the  cluster  
  45. 45. ACLs   •  Provides  the  authoriza8on  piece   •  Set  per  namespace  and  8ed  to  the  AD  groups   •  Required  proper8es  
  46. 46. ACLs   <ac0on>  -­‐  Determines  the  type  of  ac8on  –  grant  or  revoke     <en0ty>    -­‐  Determines  en8ty  to  grant  access  to  –  user  or  groups     <level>        -­‐  Determines  the  access  level  –  RWXCA     <scope>    -­‐  Determines  the  scope  for  access  –  namespace,  table,  column   family  or  Cell     Must  be  a  super  user  to  run  these  commands     (Determined  by  hbase.supersuser)  
  47. 47. Example  
  48. 48. ~33%  of  all  US  healthcare  data  
  49. 49. Millennium  +   4  CDH  5.5.2  clusters  (soon  to  be  5)   1245  tables   113,982  total  regions   673  regionservers   895  TB  of  data!  (unreplicated)  
  50. 50. Popula8on  Health   900  tables   2  million  requests/sec/day   780  regionservers   115,000  regions   700  TB  of  data!  (unreplicated)  
  51. 51. Lots  of  common  data….   •  Reference  Data   •  Provider  informa8on   •  Insurance  informa8on   •  SNOMED/  ICD9  /  ICD10  data   •  Ac8vity  Data   •  Visits   •  Procedures   •  Vitals  
  52. 52. Crawler   M I L L E N N I U M Collector   HBase  Cluster   Crawler   Collector   HBase  Cluster   M+   Pop.  H   Makes  sense  to  be  hosted  in  the  Data  hub  instead!  
  53. 53. Onboarding  –  Capacity  planning   M+   Popula8on   Health  
  54. 54. Onboarding  -­‐  Deployment   M+   Pop.  Health   DATA  HUB   Tenants  could  choose  to   modify  the  cgroup   configura8on  depending   on  expected  workloads   or  just  stay  with  defaults  
  55. 55. Onboarding  -­‐  Isola8on   •  Create  AD  groups   •  poph_users,  pop_admins   •  mplus_users,  mplus_admins   •  Create  namespaces  (as  super  user)   hbase(main):001:0> create_namespace ’mplus' 0 row(s) in 0.5650 seconds hbase(main):001:0> create_namespace ’poph' 0 row(s) in 0.7262 seconds
  56. 56. Onboarding  -­‐  Quotas     •  Setup  quotas   hbase(main):001:0> set_quota TYPE => THROTTLE, NAMESPACE => ’mplus', LIMIT => ’10000 req/sec' 0 row(s) in 0.7255 seconds hbase(main):001:0> set_quota TYPE => THROTTLE, NAMESPACE => ’poph', LIMIT => ’1500 req/sec' 0 row(s) in 0.5677 seconds  
  57. 57. Onboarding  -­‐  Security   •  Kerberos  secured.     •  Require  kinit  for  regular  users  to  access  the  cluster   •  Deploy  keytabs  for  service  users   •  Setup  ACLs   hbase(main):001:0> grant '@poph_users', 'RWCX', ’@poph’ 0 row(s) in 0.3250 seconds   hbase(main):001:0> grant '@poph_admins', 'RWACX', ’@poph’ 0 row(s) in 0.4332 seconds
  58. 58. Future  
  59. 59. What  we  can  do  bejer   •  Namespace  quota  support  (HBASE-­‐8410)   •  Limit  tables/regions  per  namespace   •  Region  Server  Groups  (HBASE-­‐6721)   •  Pin  namespace/tables  to  subset  of  regionservers   •  Advanced  namespace  security  (HBASE-­‐9206)   •  Higher  flexibility  to  admins  and  tenants  for  namespace   management    
  60. 60. @CernerEng   hjp://engineering.cerner.com/  

×