Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Chicago Data Summit: Apache HBase: An Introduction

Apache HBase is an open source distributed data-store capable of managing billions of rows of semi-structured data across large clusters of commodity hardware. HBase provides real-time random read-write access as well as integration with Hadoop MapReduce, Hive, and Pig for batch analysis. In this talk, Todd will provide an introduction to the capabilities and characteristics of HBase, comparing and contrasting it with traditional database systems. He will also introduce its architecture and data model, and present some example use cases.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Chicago Data Summit: Apache HBase: An Introduction

  1. 1. Apache HBase: an introduction <ul><li>Todd Lipcon </li></ul><ul><li>[email_address] </li></ul><ul><li>@tlipcon @cloudera </li></ul><ul><li>April 26th, 2011 </li></ul>
  2. 2. <ul><li>Software Engineer at </li></ul><ul><li>Committer and PMC member on Apache HBase, HDFS, MapReduce, and Thrift </li></ul><ul><li>Previously: systems programming, operations, large scale data analysis </li></ul><ul><li>I love data and data systems </li></ul>Introductions
  3. 3. Outline <ul><li>What is HBase? </li></ul><ul><li>HBase Architecture 101 </li></ul><ul><li>HBase vs Other Technologies </li></ul><ul><li>Use Cases </li></ul><ul><li>Questions </li></ul>
  4. 4. Apache HBase HBase is an open source , distributed , sorted map datastore modeled after Google’s BigTable
  5. 5. Open Source <ul><li>Apache 2.0 License </li></ul><ul><li>Committers and contributors from diverse organizations </li></ul><ul><ul><li>Cloudera, Facebook, StumbleUpon, Trend Micro, etc. </li></ul></ul>
  6. 6. Distributed <ul><li>Store and access data on 1-700 commodity servers </li></ul><ul><li>Automatic failover based on Apache ZooKeeper </li></ul><ul><li>Linear scaling of capacity and IOPS by adding servers </li></ul>
  7. 7. Sorted Map Datastore <ul><li>Not a relational database (very light “schema”) </li></ul><ul><li>Tables consist of rows, each of which has a primary key (row key) </li></ul><ul><li>Each row may have any number of columns, like a Map<byte[], byte[]> </li></ul><ul><li>Rows are stored in sorted order </li></ul>
  8. 8. Sorted Map Datastore (logical view as “records”) A single cell might have different values at different timestamps Different rows may have different sets of columns (table is sparse ) Different types of data separated into different “ column families” Implicit PRIMARY KEY in RDBMS terms Data is all byte[] in HBase Row key Data cutting info: { ‘height’: ‘9ft’, ‘state’: ‘CA’} roles: { ‘ASF’: ‘director’, ‘Hadoop’: ‘founder’} tlipcon info: { ‘height’: ‘5ft7, ‘state’: ‘CA’} roles: { ‘Hadoop’: ‘committer’@ts=2010, ‘ Hadoop’: ‘PMC’@ts=2011, ‘ Hive’: ‘contributor’ }
  9. 9. Sorted Map Datastore (physical view as “cells”) Sorted on disk by Row key, Col key, descending timestamp Milliseconds since unix epoch Row key Column key Timestamp Cell value cutting info:height 1273516197868 9ft cutting info:state 1273871824184 CA cutting roles:ASF 1273871823022 director cutting roles:Hadoop 1273746289103 founder tlipcon info:height 1273878447049 5ft7 tlipcon info:state 1273616297446 CA tlipcon roles:Hadoop 1300062064923 PMC tlipcon roles:Hadoop 1293388212294 Committer tlipcon roles:Hive 1273616297446 contributor
  10. 10. Column Families <ul><li>Different sets of columns may have different properties and access patterns </li></ul><ul><li>Configurable by column family: </li></ul><ul><ul><li>Compression (none, gzip, LZO) </li></ul></ul><ul><ul><li>Version retention policies </li></ul></ul><ul><ul><li>Cache priority </li></ul></ul><ul><li>CFs stored separately on disk: access one without wasting IO on the other. </li></ul>
  11. 11. HBase API <ul><li>get(row) </li></ul><ul><li>put(row, Map<column, value>) </li></ul><ul><li>scan(key range, filter) </li></ul><ul><li>increment(row, columns) </li></ul><ul><li>… (checkAndPut, delete, etc…) </li></ul><ul><li>MapReduce/Hive </li></ul>
  12. 12. Accessing HBase <ul><li>Java API (thick client) </li></ul><ul><li>REST/HTTP </li></ul><ul><li>Apache Thrift (any language) </li></ul><ul><li>Hive/Pig for analytics </li></ul>
  13. 13. High Level Architecture HBase HDFS ZooKeeper Java Client MapReduce Hive/Pig Thrift/REST Gateway Your Application
  14. 14. HBase Terms <ul><li>Region </li></ul><ul><ul><li>- A subset of a table's rows, like a range partition </li></ul></ul><ul><ul><li>- Automatically sharded </li></ul></ul><ul><li>RegionServer (slave) </li></ul><ul><ul><li>Serves data for reads and writes </li></ul></ul><ul><li>Master </li></ul><ul><ul><li>Responsible for coordinating the slaves </li></ul></ul><ul><ul><li>Assigns regions, detects failures of Region Servers Controls some admin functions </li></ul></ul>
  15. 15. HBase Daemons RegionServer HDFS HMaster RegionServer RegionServer … HMaster ZK Peer ZK Peer ZK Peer ZK Quorum Client Client finds RegionServer addresses in ZooKeeper Client reads and writes rows by directly accessing the RegionServers Master assigns regions and achieves load balancing
  16. 16. HBase Deployment (big cluster) HDFS NameNode Secondary NameNode MapReduce JobTracker ZooKeeper ZooKeeper ZooKeeper HMaster HMaster RegionServer DataNode TaskTracker RegionServer DataNode TaskTracker RegionServer DataNode TaskTracker RegionServer DataNode TaskTracker RegionServer DataNode TaskTracker 3 or 5 nodes ZK HMaster with one standby 40+ slaves with HBase, HDFS, and MR slave processes
  17. 17. HBase Deployment (small cluster / POC) NameNode SecondaryNameNode HMaster JobTracker ZooKeeper RegionServer DataNode TaskTracker RegionServer DataNode TaskTracker RegionServer DataNode TaskTracker RegionServer DataNode TaskTracker RegionServer DataNode TaskTracker 5+ slaves with HBase, HDFS, and MR slave processes The proverbial basket full of eggs
  18. 18. HBase vs other systems
  19. 19. HBase vs just HDFS If you have neither random write nor random read, stick to HDFS! Plain HDFS/MR HBase Write pattern Append-only Random write, bulk incremental Read pattern Full table scan, partition table scan Random read, small range scan, or table scan Hive (SQL) performance Very good 4-5x slower Structured storage Do-it-yourself / TSV / SequenceFile / Avro / ? Sparse columnar data model Max data size 30+ PB ~1PB
  20. 20. HBase vs RDBMS RDBMS HBase Data layout Row-oriented Column-oriented Transactions Multi-row ACID Single row only Query language SQL get/put/scan/etc * Security Authentication/Authorization Work in progress Indexes On arbitrary columns Row-key only Max data size TBs ~1PB Read/write throughput limits 1000s queries/second Millions of queries/second
  21. 21. HBase vs other “NoSQL” <ul><li>Favors Consistency over Availability (but availability is good in practice!) </li></ul><ul><li>Great Hadoop integration (very efficient bulk loads, MapReduce analysis) </li></ul><ul><li>Ordered range partitions (not hash) </li></ul><ul><li>Automatically shards/scales (just turn on more servers) </li></ul><ul><li>Sparse column storage (not key-value) </li></ul>
  22. 22. HBase in Numbers <ul><li>Largest cluster : 700 nodes, ~700TB </li></ul><ul><li>Most clusters: 5-20 nodes, 100GB-4TB </li></ul><ul><li>Writes : 1-3ms, 1k-10k writes/sec per node </li></ul><ul><li>Reads : 0-3ms cached, 10-30ms disk </li></ul><ul><ul><li>10-40k reads / second / node from cache </li></ul></ul><ul><li>Cell size : 0-3MB preferred </li></ul>
  23. 23. Use cases
  24. 24. Audit Logging <ul><li>Online service requires per-user audit logs </li></ul><ul><li>Row key userid_timestamp allows efficient range-scan lookups to fetch per-user history </li></ul><ul><li>Server-side Filter mechanism allows efficient queries </li></ul><ul><li>MapReduce for analytic questions about user behavior </li></ul>
  25. 25. Facebook Analytics <ul><li>Realtime counters of URLs shared, links “liked”, impressions generated </li></ul><ul><li>20 billion events/day (200K events/sec) </li></ul><ul><li>~30 second latency from click to count </li></ul><ul><li>Heavy use of incrementColumnValue API for consistent counters </li></ul><ul><li>Tried MySQL, Cassandra, settled on HBase </li></ul>
  26. 26. OpenTSDB <ul><li>Scalable time-series store and metrics collector </li></ul><ul><li>Thousands of machines each generating hundreds of operational metrics </li></ul><ul><li>Thousands of writes/second </li></ul><ul><li>Web interface to fetch and display graphs of metrics over time for selected hosts </li></ul>
  27. 27. Powered By HBase … and others
  28. 28. Use HBase if… <ul><ul><li>You need random write, random read, or both ( but not neither ) </li></ul></ul><ul><ul><li>You need to do many thousands of operations per second on multiple TB of data </li></ul></ul><ul><ul><li>Your access patterns are well-known and simple </li></ul></ul>
  29. 29. Don’t use HBase if… <ul><ul><li>You only append to your dataset, and tend to read the whole thing </li></ul></ul><ul><ul><li>You primarily do ad-hoc analytics (ill defined access patterns) </li></ul></ul><ul><ul><li>Your data easily fits on one beefy node </li></ul></ul>
  30. 30. Resources <ul><li>Download CDH3 ( </li></ul><ul><li>Cloudera HBase training (1 st chapter free online) </li></ul><ul><li> </li></ul><ul><li> #hbase </li></ul><ul><li>Coming soon: HBase: The Definitive Guide by Lars George </li></ul>
  31. 31. Questions? <ul><li>[email_address] </li></ul><ul><li>(feedback? yes!) </li></ul><ul><li>(hiring? yes!) </li></ul>