HBase
in
the
Enterprise
Data
Hub
It’s
all
about
co-­‐exis0ng
peacefully!
Swarnim
Kulkarni
May
24th
2016

innova8on

innova8on
noun
in·∙no·∙va·∙0on
ˌi-­‐nə-­‐ˈvā-­‐shən
1.
the
introduc0on
of
something
new
2.
a
new
idea,
method
or
device

Chart
Search

Sepsis

Millennium
+

Popula8on
Health

HBase
sits
at
the
core
of
a
lot
of
these
solu8ons!

Each
solu8on
built
with
specific
business
problem
in
mind

CHART
SEARCH
SEPSIS
MILLENNIUM
+
POP
HEALTH

CHART
SEARCH
SEPSIS
MILLENNIUM
+
POP
HEALTH

Prevents
assets
from
being
shared
High
investment
High
maintenance
costs
High
barrier
of
entry
for
innova8on

Enter
The
Data
Hub

Bob
Tim

Bob

= +
+

Tim

= +
+

Innova8ons
Source
of
data
Raw
data

Data
Hub
Owner
Data
Hub
Consumers
Workers

“An
Enterprise
Data
Hub
is
a
centralized
loca0on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi0ng
and
management.”

“An
Enterprise
Data
Hub
is
a
centralized
loca-on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi0ng
and
management.”

“An
Enterprise
Data
Hub
is
a
centralized
loca-on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi0ng
and
management.”

“An
Enterprise
Data
Hub
is
a
centralized
loca-on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi0ng
and
management.”

“An
Enterprise
Data
Hub
is
a
centralized
loca-on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi-ng
and
management.”

Mul8-­‐tenant
Secure
and
Compliant
Ac8ve
archive
of
all
data
Low
barrier
of
entry
Hadoop
as
a
Service
Data
Hub
is….

So
why
provide
HBase
as
a
service?

Maximize
cluster
u8liza8on
Minimum
resource
guarantee
sharing
Low
barrier
of
entry
HBase
as
a
Service
Knowledge
Data
Data

Requirements

Support
mul8-­‐tenant
environment
for
mul8ple
users
Isolated
deployment
Set
quota
per
consumer
ACLs
for
project
level
administra8on
Appropriate
security
for
user
authen8ca8on

Architecture

Smaller
HBase
clusters
part
of
the
bigger
“hub”
Scale
as
per
need
Easier
management

Resource
Management

Making
sure
that
you
have
sufficient
resources
before
you
start
the
job!

CGroups
• Control
Groups
• Police
and
limit
CPU,
Disk
I/O
and
Memory
usage
• Resource
guarantee
by
sta8c
par88oning
• Very
useful
in
case
of
conten8on

• Value
for
resource-­‐limits
driven
by
individual
tenants
depending
on
the
workload
• Defaults
to
70-­‐30
usage
for
memory
and
CPU
between
(NodeManager
+
Datanode)
and
RegionServer
• Higher
memory
on
RS
(at
least
4
GB)
helps
avoid
swapping
and
happier
GC
CPU
CPU
CPU
MEMORY
MEMORY
MEMORY
SEPSIS
M+
POP
HEALTH

What
we
learned
• Recommend
tenants
to
give
atleast
4
GB
of
memory
to
the
regionserver
for
smoother
opera8on
• Coopera8ve
memory
(ex:
via
JVM
heaps,
max.
container
sizes)
works
bejer
than
cgroup
limits
• Disable
swapping
on
HBase
nodes
• Disable
HDFS
load
balancer
• CMS
GC
performs
way
bejer
• Limit
number
of
containers
running
on
nodes
to
maximize
performance

Property
Value
yarn.nodemanager.container-­‐
executor.class
org.apache.hadoop.yarn.server.node
manager.LinuxContainerExecutor
yarn.nodemanager.linux-­‐container-­‐
executor.group
hadoop
yarn.nodemanager.linux-­‐container-­‐
executor.resources-­‐handler.class
org.apache.hadoop.yarn.server.node
manager.u8l.CgroupsLCEResourcesHa
ndler
yarn.nodemanager.linux-­‐container-­‐
executor.cgroups.hierarchy
/yarn
yarn.nodemanager.linux-­‐container-­‐
executor.cgroups.mount
true
hjps://hadoop.apache.org/docs/current/hadoop-­‐yarn/hadoop-­‐yarn-­‐site/NodeManagerCgroups.html

Request
queues
• Priori8ze
variety
of
workflows
that
need
access
to
HBase
• Important
to
meet
SLAs
for
tenants
• “FIFO”
vs
“Deadline”
queue
type
• “Deadline”(default)
proved
to
work
prejy
well
for
most
of
the
cases
• Cannot
be
set
per
tenant
but
has
to
be
set
per
cluster
–
support
coming
soon

Quotas
• Makes
sure
that
no
single
tenant
abuses
the
system
• Peaceful
coexistence
• Promotes
a
pay-­‐per-­‐use
model.
Could
buy
a
higher
throjle
limit
by
contribu8ng
more
number
of
nodes
• Usually
we
set
throjle
on
a
per
namespace
basis
but
could
also
set
on
the
per
table/user
basis
if
needed

Tenant
Isola8on

Namespace
• Logical
grouping
of
HBase
tables
• Analogous
to
databases
• Provides
tenants
with
individual
space
to
operate
on
• Tied
to
the
AD
group
used
when
onboarding
• Could
also
apply
quotas(max
regions/tables)
per
namespace(HBASE-­‐8410)
but
not
using
that
feature
for
now

Security

Authen8ca8on
• Cluster
secured
by
Kerberos
• Disallow
impersona8on
• Require
kinit
to
first
authen8cate
with
the
KDC
before
accessing
the
cluster

ACLs
• Provides
the
authoriza8on
piece
• Set
per
namespace
and
8ed
to
the
AD
groups
• Required
proper8es

ACLs
<ac0on>
-­‐
Determines
the
type
of
ac8on
–
grant
or
revoke
<en0ty>
-­‐
Determines
en8ty
to
grant
access
to
–
user
or
groups
<level>
-­‐
Determines
the
access
level
–
RWXCA
<scope>
-­‐
Determines
the
scope
for
access
–
namespace,
table,
column
family
or
Cell
Must
be
a
super
user
to
run
these
commands
(Determined
by
hbase.supersuser)

Example

~33%
of
all
US
healthcare
data

Millennium
+
4
CDH
5.5.2
clusters
(soon
to
be
5)
1245
tables
113,982
total
regions
673
regionservers
895
TB
of
data!
(unreplicated)

Popula8on
Health
900
tables
2
million
requests/sec/day
780
regionservers
115,000
regions
700
TB
of
data!
(unreplicated)

Lots
of
common
data….
• Reference
Data
• Provider
informa8on
• Insurance
informa8on
• SNOMED/
ICD9
/
ICD10
data
• Ac8vity
Data
• Visits
• Procedures
• Vitals

Crawler
M
I
L
L
E
N
N
I
U
M
Collector
HBase
Cluster
Crawler
Collector
HBase
Cluster
M+
Pop.
H
Makes
sense
to
be
hosted
in
the
Data
hub
instead!

Onboarding
–
Capacity
planning
M+
Popula8on
Health

Onboarding
-­‐
Deployment
M+
Pop.
Health
DATA
HUB
Tenants
could
choose
to
modify
the
cgroup
configura8on
depending
on
expected
workloads
or
just
stay
with
defaults

Onboarding
-­‐
Isola8on
• Create
AD
groups
• poph_users,
pop_admins
• mplus_users,
mplus_admins
• Create
namespaces
(as
super
user)
hbase(main):001:0> create_namespace ’mplus'
0 row(s) in 0.5650 seconds
hbase(main):001:0> create_namespace ’poph'
0 row(s) in 0.7262 seconds

Onboarding
-­‐
Quotas
• Setup
quotas
hbase(main):001:0> set_quota TYPE => THROTTLE, NAMESPACE
=> ’mplus', LIMIT => ’10000 req/sec'
0 row(s) in 0.7255 seconds
hbase(main):001:0> set_quota TYPE => THROTTLE, NAMESPACE
=> ’poph', LIMIT => ’1500 req/sec'
0 row(s) in 0.5677 seconds

Onboarding
-­‐
Security
• Kerberos
secured.
• Require
kinit
for
regular
users
to
access
the
cluster
• Deploy
keytabs
for
service
users
• Setup
ACLs
hbase(main):001:0> grant '@poph_users', 'RWCX', ’@poph’
0 row(s) in 0.3250 seconds
hbase(main):001:0> grant '@poph_admins', 'RWACX', ’@poph’
0 row(s) in 0.4332 seconds

Future

What
we
can
do
bejer
• Namespace
quota
support
(HBASE-­‐8410)
• Limit
tables/regions
per
namespace
• Region
Server
Groups
(HBASE-­‐6721)
• Pin
namespace/tables
to
subset
of
regionservers
• Advanced
namespace
security
(HBASE-­‐9206)
• Higher
flexibility
to
admins
and
tenants
for
namespace
management

@CernerEng
hjp://engineering.cerner.com/