The document provides an extensive overview of HBase, a non-relational, distributed column-oriented database, detailing its architecture, data model, and common operational challenges. It discusses best practices for deployment, data representation, write and read paths, performance tuning, and tools like Apache Phoenix for SQL support. The document also addresses troubleshooting techniques, monitoring, and the importance of proper key design for optimal data retrieval.

1. HBase from the Trenches
Avinash Ramineni
Email: avinash@clairvoyantsoft.com
LinkedIn: https://www.linkedin.com/in/avinashramineni

2. Agenda
• Intro to HBase
– Overview
– Data Model
– Architecture
• Common Problems
• Best Practices
• Tools and Utilities

3. Intro to HBase
• Non-relational distributed column-oriented
database
– Modeled after Google’s BigTable
• Billions of Rows and Millions of Columns
• Sparse, consistent, distributed sorted map
• Built on top of HDFS
• Tight integration with MapReduce
• Supports Random CRUD Operations

4. Intro to HBase
• Fault Tolerant
• Horizontally Scalable
• Real-time Random read-write access to data
stored in HDFS
• Millions of queries / second
• Support for transactions at a single row level
• Bloom filters
• Automatic Sharding
• Implemented in Java

5. Data Model
• Data is stored in Tables
• Tables contain rows
– Rows are referenced by a unique key - Rowkey
• Rows are made of columns which are grouped
in column families
• Rows are sorted
• Everything is stored as a sequence of bytes
• All entries are versioned and timestamped

6. Data Represenation

7. HBase Cluster
• HBase Master
• Zookeeper
• Region Servers
• HDFS - Data Nodes

8. Component View

9. Logical View

10. HBase API
• API is simple
• Operations
– Get,Put,Delete,Scan,MapReduce
• Connection
• Create this instance only once per application and
share it during its runtime
• Htable
– Zookeeper
• HBase:meta

11. Column Families
• All columns that are accesed together need to be
grouped into a Column Family
• No need to access or load data that is not used
• At the column family we can define the settings
like
– compression, version retention policy, cache priority
– Understand the data, access pattern and group
column family
• Column Family and Column Qualifiers are stored
as bytes
– Avoid being verbose

12. HBase Write Path

13. HBase Compactions
• HDFS does not support updates
– HFiles are immutable
– New HFiles are created
• Minor Compactions
– Small HFiles are merged into larger Hfiles
– Deletes are not applied
• Major Compactions
– Hfiles with in column family are merged into Single
Hfile
– Deletes are applied

14. Rowkey
• Immutable
• Get it right the first time before a lot of data is
loaded
• What if we got it wrong ?
– New table and load the data
– If TTL set..let the data expire

15. Secondary Indexes
• Querying / Accessing records other than by
Rowkey
• Map Reduce jobs to populate index table
– Periodic update
• Build a secondary index with dual writes
• Co-processors

16. Region Hotspotting
• Client traffic not equally distributed across the region
servers
– Performance degradation
– Region unavailability
• Poor rowkey design
– Monotonically increasing RowKey
• Timeseries or Sequence
– Salting
• Read Vs Writes
• GET ?
– Hashing
• Salt with one-way hash of rowkey

17. Short Circuit Reads
• Region Servers are co-located with data nodes
• HMaster assigns Regions keeping data locality
in to consideration (mostly)
• dfs.client.read.short-circuit
– Region Servers read the data directly from HDFS
rather than going through Datanode
• Locality Loss

18. Pre-Splitting
• Region splitting
– Grows untill it needs to be split
– Region at a time is served by only 1 Region Server
• Pre-split a table into regions at table creation
time
– Uniformly distribute write load across region servers
– Understand the keyspace
• Risk of uneven load distribution
• Auto splitting
– Constant size region split policy
– IncreasingToUpperBoundRegionSplitPolicy

19. Bulk Loading
• Native API
– Disable WAL
• MapReduce Job to generate Hfile
– Load using completebulkload / importTSV tool
• Loads into relevant region
– Faster than going through normal write path
• No writes to WAL and Memstore
• No flushing and compacting

20. Troubleshooting
• ulimit -n
– Limits on number of files and processs
• HBase is database and needs to open a number
of files
• dfs.datanode.max.transfer.threadsrr.
• Network
• OS Parameters

21. You are Dead Exception
• Region Servers going down
– Zookeeper
• Distributed co-ordinated service
– HBase Master asks the region server to shutdown
– Garbage Collection
– Zookeeper session timeout

22. Performance Tuning
• Compression
– Reduces data stored on disk and transferred
– Compression speed over ratio
• Load Balancing - Balancer
• Merging Regions
• Batch Writes
– Client Write Buffer
– AutoFlush
• MemStore-local allocation buffers
– Garbage Collection Issues

23. Tuning
• Heavy Writes
– Flushes, compacting,splitting increase IO and degrade
cluster performance
• Keep Region sizes larger
• Keep Hfile size large
• Heavy Sequential Reads
• Higher block size
• Avoid Caching on table
• Heavy Random Reads
• Higher Blocklevel cache
• Lower Memstore limit
• Smaller block size

24. Apache Phoenix
• SQL over Hbase
– Compiles into Hbase Scans
– Orchetrates parallel execution
– Aggregate queries
• JDBC API’s over Native HBase API.
• Salting Buckets PreSplitting
• Trafodion
– Transactional SQL on HBase

25. Hannibal
• Monitor and maintain HBase Clusters
• How well regions are balanced over the
cluster?
• How well regions are split for each table
• How regions evolve over time
• How long compactions take
• Integration with HUE

26. Hannibal

27. Hannibal

28. Hannibal

29. Operational Aspects
• Metrics
– Master
• Cluster requests,split time,split size
– RegionServer
• Blockcache,memstore,compaction,store,IO
• Htrace
– Trace tool for parallel distributed system
• Monitoring
– Nagios
– Hannibal
– Ganglia
– Graphite
– OpenTSDB
• Backup
– Export,CopyTable,Snapshot

30. Questions?
avinash@clairvoyantsoft.com