Multitenancy At Bloomberg - HBase and Oozie

© 2017 Bloomberg Finance L.P. All rights reserved.
HBase and Oozie
Multitenancy at Bloomberg
DataWorks Summit
June 14th
, 2017
Clay Baenziger
Hadoop Infrastructure
hadoop@bloomberg.net

About Bloomberg
Technology Introduction
HBase Multitenancy
●Workloads, Resources, Access
Oozie Self-Service
●Deployments
HBase and Oozie
●Scheduled Compactions, Exporting HBase Snapshots
What we will cover today

About Bloomberg
Bloomberg quickly and accurately delivers business and
financial information, news and insight around the world.
A Sense of Scale:
●550 exchange feeds and over 100 billion market data messages a
day
●400 million emails and 17 million IM’s daily across the Bloomberg
Professional Service
●Over 2,700 journalists and analysts in over 120 countries
●Producing more than 5,000 stories a day
●Reaching over 360 million homes worldwide

About Bloomberg
Project JIRAs Project JIRAs Project JIRAs
Phoenix 24 HBase 20 Spark 9
Zookeeper 8 HDFS 6 Bigtop 3
Oozie 4 Storm 2 Hive 2
Hadoop 2 YARN 2 Kafka 2
Flume 1 HAWQ 1 Total 86
Apache Solr: 3 core committers (one PMC member) – commits in every release since 4.6
(Reporter or assignee from our Foundational Services group and affiliated projects)

Technology Intro – Apache HBase
What?
●Distributed database designed to host very large tables – billions of
rows by millions of columns
●Block cache, bloom filters and time-line consistency for highly available,
real-time queries
●Sharded, versioned, non-relational database modeled after Google's
Bigtable – compacting log structured merge tree design
●Supports exports and backups – by global administrators

Technology Intro – Apache Oozie
What?
•Oozie is a workflow scheduler system to manage Apache Hadoop jobs.
•Oozie workflow jobs are Directed Acyclical Graphs (DAGs) of actions.
•Oozie workflows can be templated with properties.
•Oozie coordinator jobs are reoccurring Oozie workflow jobs triggered by time and data
availability.
•Oozie is integrated with the rest of the Hadoop stack supporting several types of Hadoop
jobs, security tokens as well as providing system specific jobs out of the box.
•Oozie is a scalable, highly available and extensible system.

HBase Multitenancy
Why?
Larger capacity reserve:
●Can handle request spikes
●Can lose a rack
●Higher per-machine usage
●Multi-cluster Hadoop support is evolving
Why Not?
●Isolation
●Easier to understand

Write Heavy
●Memstore Heavy
●Compactions Optimized for HFile Size
●Flush Size Tuning
Read Heavy
●Cache Heavy
●Compactions Optimized for Minimal HFiles
●Read Replicas
Mixed Read/Write
●SSDs
●Read Replicas
HBase Workloads

●Availability:
●Data Bugs
●Thread Death/Starvation
●User Code Bugs
●
HBase Contested Resources
Storage:
●Memstore
●HDFS
●Cache
●Input/Outut:
●Latency
●Queues
●Ingest

●Data Bugs:
●Can isolate tenants with Region Server Groups – HBASE-6721
●Thread Death/Starvation:
●Master becomes a zombie if filesystem object closes – HBASE-17287
●“Region Server Too Busy” – Request Quotas
●Garbage Collection “Bombs” – HBASE-18023 - “Log multi-* requests for
more than threshold number of rows”
●User Code Bugs (Coprocessors)
●”Coprocessors - Uses, Abuses, Solutions” Esther Kundin and Clay Baenziger
– HBase Con East, September 26th, 2016
●Can run only approved coprocessors – HBASE-16700 – “Allow for
Coprocessor Whitelisting”
HBase Resources – Availability

●Memstore (Flushes)
●Affects HFile Quantity
●Can Block Writes
●Compacting Memstore
●
●HDFS
●Denial-of-Service – HBASE-16961 - “FileSystem Quotas”
●
●Cache
●Multiple workloads thrashing the cache
HBase Resources – Storage

Latency
●Caching – Off-Heaping
●De-Prioritizing Scanners
Queues – Monitoring!
●Replication Queue
●Handler Queue
Ingest
●Splits
●Compactions
●Bulk-Load
HBase Resources – Input/Output

●Kerberos - Need a way to get identity:
●LDAP Group Traversal – HADOOP-12291
●Namespaces
●Grants
●Multiple Clusters:
●Spark-Hbase Connector (https://github.com/hortonworks-spark/shc PR#120)
●Oozie Delegation Token Acquistion – OOZIE-1646 – “HBase Table Copy
between two HBase servers doesn't work with Kerberos”
HBase Access

Oozie and Self-Service
Self-Service:
“The serving of oneself with goods or services” – Merriam-Webster.COM
• Automation – pipelines – workflows; re-occurrence – coordinators
•Job Status – callback URLs to notify of job progress
•Job IDs – Map/Reduce or YARN IDs for action’s internal sub-jobs’ ID’s
•Authentication – delegation tokens (no keytabs!)

Separates Continuous Integration from Continuous Deployment
Leverages:
●Git – OOZIE-2877
●Maven – OOZIE-2878
Axioms – write a deployment workflow for your job (and its workflow)
Must follow principals of:
●Idempotency – On re-runs, results in same state as first run
●Cleanliness – Removes old artifacts (state)
●Separation of Configuration from Code – deploy same workflow to
development and production with only workflow property differences
●See also slides to ”Cluster Continuous Delivery with Oozie” – ApacheCon
North America - Big Data, May 18th, 2017
Oozie Deployments

HBase and Oozie
Why?
Self-Service – No DBA!
Regularly schedulable
Runs as the project user with no keytab
Logs and reporting
●Can record HBase errors with logs organized by each Oozie job
●Can report success of job with proactive callbacks
●Can verify performance of job with SLA subsystem
●Can provide infrastructure insight into what’s running

●
●
HBase Scheduled Compactions
Compactions are I/O and processing heavy – can be:
●Detrimental to read or write performance
●Lead to split storms or rebalancing
●Good to plan for impact – schedule them
●Can also go region-by-region to lessen impact
●Can poll to know when compaction is complete
●A simple Java action:https://tinyurl.com/oozie-hbase-compaction

●
●
HBase Scheduled Compactions
Key Oozie-isms:
●Java action needs to know how to get configuration:
conf.addResource(new Path("file:///",
System.getProperty("oozie.action.conf.xml")));
●Need to pass delegation tokens from Oozie:
if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") !=
null)
{ conf.set("mapreduce.job.credentials.binary",
System.getenv("HADOOP_TOKEN_FILE_LOCATION")); }
●Need to use Oozie’s Credentials Action Authentication
●Must pass in properties manually – no jobxml support – OOZIE-2947

Backup Requirements:
●Live backups (cannot disable table or take HBase offline)
●Self-service (non-HBase user can backup/restore their own data)
●Automatable procedure (Oozie)
●Works on a secure cluster (hdfs:///hbase is non-world readable)
●Backup location may not be running Hbase
●Does not require significant architectural “baggage” addition to HBase
●
●
HBase Snapshot Export

Table Snapshot
Snapshot
initiated
Oozie workflow submitted
(as namespace admin)
HBase Snapshot Export
HBase Export
Snapshot
Perm. check
Oozie Export
Snapshot Action
Namespace Admin
User
HDFS
Create “dropbox” directory

HBase and Oozie
Clay Baenziger
https://github.com/bloomberg
hadoop@bloomberg.net

HBase and Oozie
DataWorks Summit
June 14th
, 2017
Clay Baenziger
Hadoop@Bloomberg.NET

Multitenancy At Bloomberg - HBase and Oozie

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Multitenancy At Bloomberg - HBase and Oozie

Similar to Multitenancy At Bloomberg - HBase and Oozie (20)

More from DataWorks Summit

More from DataWorks Summit (20)

Recently uploaded

Recently uploaded (20)

Multitenancy At Bloomberg - HBase and Oozie