October 2013 HUG: HBase 0.96
•
7 likes•1,992 views
The next major version - 0.96- of Apache HBase have several new features. The "Singularity", because you will have to start and stop your cluster to upgrade to 0.96. 0.96 requires Apache Hadoop 1.0.0 at least, and supported on Hadoop 2.0.0 as well. 0.96 uses protobufs all the time. All of its serializations to ZooKeeper, to the filesystem, and over rpc are protobufs. It runs on JDK7. Metrics have been edited and converted to use Hadoop Metrics2. It has HBase Snapshots and PrefixTreeCompression, etc. This presentation captures a high-level overview of what's new in HBase 0.96.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to October 2013 HUG: HBase 0.96
Similar to October 2013 HUG: HBase 0.96 (20)
More from Yahoo Developer Network
More from Yahoo Developer Network (20)
Recently uploaded
Recently uploaded (20)
October 2013 HUG: HBase 0.96
- 1. 0.96.0 Bay Area Hadoop User Group, October 16th, 2013
- 2. Michael Stack <stack@apache.org> • • • • 0.96.0 Release Manager Chair of Apache HBase PMC* Apache Hadoop PMC Engineer at Cloudera in San Francisco * Project Management Committee
- 3. HBase?
- 5. "...open source, distributed, scalable, consistent, low latency, random access non-relational database..."
- 6. Inspiration A Google Technology described in a 2006 paper, by Chang et al.?
- 7. ●Apache Top-level Project ○hbase.apache.org ●Up out of Apache Hadoop contrib ●Project goal: “Billions of rows X millions of columns on clusters of ‘commodity hardware” ●HBase persists all data to HDFS ●Uses Apache ZooKeeper ○Cluster coordination
- 8. When would I use it?
- 10. SCA LI NG!
- 11. Who uses it?
- 14. Diverse team* COMMITTERS! Preferably ALIVE! * http://hbase.apache.org/team-list.html
- 16. • Release every month • Each more stable • & more performant • Some features… • • Currently at 0.94.12 Wire compatible between releases
- 20. (Self-)Migration
- 21. Downstreamers ● Minimal API disturbance – None? – Last-minute feedback Hive, Sqoop, OpenTSDB Deprecations ● ●
- 22. Stats ● >2k issues fixed >1500 in 0.96.x only Currently 6th Release Candidate – ● ● Branched 7months ago ● 18months in the making
- 23. Requirements ● Hadoop 1.0.3+ ● Hadoop 2.1.0-beta+ ● Must choose one
- 24. Big Themes ● ● ● ● Stability Operability – Insight, tools Scalability Evolvability
- 28. • • • • HBase Dedicated meta WAL Don't put WAL replicas on local node – 33% of reads have to timeout Lowered ZK timeout – 30s instead of 180s Watcher script kills znode – • Detection time approaches 0 Faster assignment
- 29. • HDFS HDFS-4721 Speed up lease/block recovery when DN fails and a block goes into recovery – • HDFS-3703 Decrease the datanode failure detection time – • Do not recover on STALE DNs Avoid reading STALE DNs HDFS-3912 Detecting and avoiding stale datanodes for writing
- 30. Coming... ● Faster WAL replay/Distributed WAL Replay – No intermediate files – No wait on NN Committed ● Experimental Regions online immediately for Writes ● ● – ● Read older consistent view “Favored Nodes”
- 33. One rationale for pb: http://goo.gl/N0HO6n
- 34. • • • • System tables Filesystem Up in zookeeper Over the wire
- 35. RPC • • Implements Protobuf Service ● Specification! Data on the side oEncoding oCompression PB DATA
- 36. Scalability • • • • e.g. Replicating 1k to 1k & heading north HBASE-8778 Region assigments scan table directory making them slow for huge tables HBASE-9208 ReplicationLogCleaner slow at large scale HBASE-8877 Reentrant row locks
- 38. Snapshots • • • By Table oSnapshot, clone, restore, export Inexpensive oJust metadata Good for... oBackups oReplication oOffline processing
- 39. Integration Tests • Cluster test module • "Borrows" test types from all over o Netflix "ChaosMonkey" o Apache Accumulo linked-list dataloss checker o Standalone or cluster o Sizeable x data x runtime hbase-it/src/test/java//org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java hbase-it/src/test/java//org/apache/hadoop/hbase/mapreduce/IntegrationTestImportTsv.java hbase-it/src/test/java//org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java hbase-it/src/test/java//org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java hbase-it/src/test/java//org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java
- 40. StochasticLoadBalancer • Region Count • Locality • Movement Cost • Table Count • Regions/Table/RegionServer • Read/Write Counts • Memstore Size • Storefile Size
- 41. Tracing • Review HDFS-5274 Add Tracing to HDFS!
- 42. Namespaces • Grouping of tables – Like database in mysql • System/User hbase:meta Quota Coming – Security by ns – Grouping on cluster by ns – • •
- 43. Metrics2 ● Radical revamp ● Module of Interfaces – H1 and H2 Impls modules ● Categories/Naming/Patterns
- 44. API ● Client/Dev ● Hadoop Annotations – ● Stable/Evolving/Private Cell Interface – KeyValue deprecated
- 46. Miscellaneous • X-Row (in-region) Transactions • Hardened Assignment • Hardened Replication • New UI • Online Merge • Finer grained ACLs • More Coprocessor hooks
- 47. More Misc. • Maven modularized • Client-side Types • Revamped defaults • Compactions o Pluggable o Smarter triggers • Windows!
- 48. 0.96.1, 0.96.2, etc. ● ● ● ● Bug fixes Performance fixes ONLY! No features!
- 50. • Right after 0.96.0 – Month or two • Rolling upgrade from 0.96.0 • In-line Cell-tags • Quota/Groupings • Reverse Scan
- 51. 1.0.0?