HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase

8,213 views

Published on

Presented by: Enis Soztutar, Hortonworks

Published in: Technology

HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase

  1. 1. © Hortonworks Inc. 2011 HBase and HDFSUnderstanding file system usage in HBase Enis Söztutar enis [ at ] apache [dot] org @enissoz Page 1
  2. 2. © Hortonworks Inc. 2011 About Me Page 2 Architecting the Future of Big Data • In the Hadoop space since 2007 • Committer and PMC Member in Apache HBase and Hadoop • Working at Hortonworks as member of Technical Staff • Twitter: @enissoz
  3. 3. © Hortonworks Inc. 2011 Motivation • HBase as a database depends on FileSystem for many things • HBase has to work over HDFS, linux & windows • HBase is the most advanced user of HDFS • For tuning for IO performance, you have to understand how HBase does IO Page 3 Architecting the Future of Big Data MapReduce Large files Few random seek Batch oriented High throughput Failure handling at task level Computation moves to data HBase Large files A lot of random seek Latency sensitive Durability guarantees with sync Computation generates local data Large number of open files
  4. 4. © Hortonworks Inc. 2011 Agenda • Overview of file types in Hbase • Durability semantics • IO Fencing / Lease recovery • Data locality – Short circuit reads (SSR) – Checksums – Block Placement • Open topics Page 4 Architecting the Future of Big Data
  5. 5. © Hortonworks Inc. 2011 HBase file types Architecting the Future of Big Data Page 5
  6. 6. © Hortonworks Inc. 2011 Overview of file types • Mainly three types of files in Hbase – Write Ahead Logs (a.k.a. WALs, logs) – Data files (a.k.a. store files, hfiles) – References / symbolic or logical links (0 length files) • Every file is 3-way replicated Page 6 Architecting the Future of Big Data
  7. 7. © Hortonworks Inc. 2011 Overview of file types /hbase/.archive /hbase/.logs/ /hbase/.logs/server1,60020,1370043265148/ /hbase/.logs/server1,60020,1370043265148/server1%2C60020%2C1370043265148.1370050467720 /hbase/.logs/server1,60020,1370043265105/server1%2C60020%2C1370043265105.1370046867591 … /hbase/.oldlogs /hbase/usertable/0711fd70ce0df641e9440e4979d67995/family/449e2fa173c14747b9d2e5.. /hbase/usertable/0711fd70ce0df641e9440e4979d67995/family/9103f38174ab48aa898a4b.. /hbase/table1/565bfb6220ca3edf02ac1f425cf18524/f1/49b32d3ee94543fb9055.. /hbase/.hbase-snapshot/usertable_snapshot/0ae3d2a93d3cf34a7cd30../family/12f114.. … Page 7 Architecting the Future of Big Data Write Ahead Logs Data files Links
  8. 8. © Hortonworks Inc. 2011 Data Files (HFile) • Immutable once written • Generated by flush or compactions (sequential writes) • Read randomly (preads), or sequentially • Big in size (flushsize -> tens of GBs) • All data is in blocks (Hfile blocks not to be confused by HDFS blocks) • Data blocks have target size: – BLOCKSIZE in column family descriptor – 64K by default – Uncompressed and un-encoded size • Index blocks (leaf, intermediate, root) have target size: – hfile.index.block.max.size, 128K by default • Bloom filter blocks have target size: – io.storefile.bloom.block.size, 128K by default Page 8 Architecting the Future of Big Data
  9. 9. © Hortonworks Inc. 2011 Data Files (HFile version 2.x) Page 9 Architecting the Future of Big Data
  10. 10. © Hortonworks Inc. 2011 Data Files • IO happens at block boundaries – Random reads => disk seek + read whole block sequentially – Read blocks are put into the block cache – Leaf index blocks and bloom filter blocks also go to the block cache • Use smaller block sizes for faster random-access – Smaller read + faster in-block search – Block index becomes bigger, more memory consumption • Larger block sizes for faster scans • Think about how many key values will fit in an average block • Try compression and Data Block Encoding (PREFIX, DIFF, FAST_DIFF, PREFIX_TREE) – Minimizes file sizes + on disk block sizes Page 10 Architecting the Future of Big Data Key length Value length Row length Row key Family length Family Column qualifier Timesta mp KeyType Value Int (4) Int (4) Short(2) Byte[] byte Byte[] Byte[] Long(8) byte Byte[]
  11. 11. © Hortonworks Inc. 2011 Reference Files / Links • When region is split, “reference files” are created referring to the top or bottom half of the parent store file according to splitkey • HBase does not delete data/WAL files just “archives” them /hbase/.oldlogs /hbase/.archive • Logs/hfiles are kept until TTL, and replication or snapshots are not referring to them – (hbase.master.logcleaner.ttl, 10min) – (hbase.master.hfilecleaner.ttl, 5min) • HFileLink: kind of hard / soft links that is application specific • HBase snapshots are logical links to files (with backrefs) Page 11 Architecting the Future of Big Data
  12. 12. © Hortonworks Inc. 2011 Write Ahead Logs • One logical WAL per region / one physical per regionserver • Rolled frequently – hbase.regionserver.logroll.multiplier (0.95) – hbase.regionserver.hlog.blocksize (default file system block size) • Chronologically ordered set of files, only last one is open for writing • Exceeding hbase.regionserver.maxlogs (32) will cause force flush • Old log files are deleted as a whole • Every edit is appended • Sequential writes from WAL, sync very frequently (hundreds of times per sec) • Only sequential reads from replication, and crash recovery • One log file per region server limits the write throughput per Region Server Page 12 Architecting the Future of Big Data
  13. 13. © Hortonworks Inc. 2011 Durability (as in ACID) Architecting the Future of Big Data Page 13
  14. 14. © Hortonworks Inc. 2011 Overview of Write Path 1. Client sends the operations over RPC (Put/Delete) 2. Obtain row locks 3. Obtain the next mvcc write number 4. Tag the cells with the mvcc write number 5. Add the cells to the memstores (changes not visible yet) 6. Append a WALEdit to WAL, do not sync 7. Release row locks 8. Sync WAL 9. Advance mvcc, make changes visible Page 14 Architecting the Future of Big Data
  15. 15. © Hortonworks Inc. 2011 Durability • 0.94 and before: – HTable property “DEFERRED_LOG_FLUSH” and – Mutation.setWriteToWAL(false) • 0.94 and 0.96: Page 15 Architecting the Future of Big Data Durability Semantics USE_DEFAULT Use global hbase default, OR table default (SYNC_WAL) SKIP_WAL Do not write updates to WAL ASYNC_WAL Write entries to WAL asynchronously (hbase.regionserver.optionallogflushinterval, 1 sec default) SYNC_WAL Write entries to WAL, flush to datanodes FSYNC_WAL Write entries to WAL, fsync in datanodes
  16. 16. © Hortonworks Inc. 2011 Durability • 0.94 Durability setting per Mutation (HBASE-7801) / per table (HBASE- 8375) • Allows intermixing different durability settings for updates to the same table • Durability is chosen from the mutation, unless it is USE_DEFAULT, in which case Table’s Durability is used • Limit the amount of time an edit can live in the memstore (HBASE-5930) – hbase.regionserver.optionalcacheflushinterval – Default 1hr – Important for SKIP_WAL – Cause a flush if there are unflushed edits that are older than optionalcacheflushinterval Page 16 Architecting the Future of Big Data
  17. 17. © Hortonworks Inc. 2011 Durability Page 17 Architecting the Future of Big Data public enum Durability { USE_DEFAULT, SKIP_WAL, ASYNC_WAL, SYNC_WAL, FSYNC_WAL } Per Table: HTableDescriptor htd = new HTableDescriptor("myTable"); htd.setDurability(Durability.ASYNC_WAL); admin.createTable(htd); Shell: hbase(main):007:0> create 't12', 'f1', DURABILITY=>'ASYNC_WAL’ Per mutation: Put put = new Put(rowKey); put.setDurability(Durability.ASYNC_WAL); table.put(put);
  18. 18. © Hortonworks Inc. 2011 Durability (Hflush / Hsync) • Hflush() : Flush the data packet down the datanode pipeline. Wait for ack’s. • Hsync() : Flush the data packet down the pipeline. Have datanodes execute FSYNC equivalent. Wait for ack’s. • hflush is currently default, hsync() usage in HBase is not implemented (HBASE-5954). Also not optimized (2x slow) and only Hadoop 2.0. • hflush does not lose data, unless all 3 replicas die without syncing to disk (datacenter power failure) • Ensure that log is replicated 3 times hbase.regionserver.hlog.tolerable.lowreplication defaults to FileSystem default replication count (3 for HDFS) Page 18 Architecting the Future of Big Data public interface Syncable { public void hflush() throws IOException; public void hsync() throws IOException; }
  19. 19. © Hortonworks Inc. 2011 Page 19 Architecting the Future of Big Data
  20. 20. © Hortonworks Inc. 2011 IO Fencing Fencing is the process of isolating a node of a computer cluster or protecting shared resources when a node appears to be malfunctioning Page 20 Architecting the Future of Big Data
  21. 21. © Hortonworks Inc. 2011 IO Fencing Page 21 Architecting the Future of Big Data Region1Client Region Server A (dying) WAL Region1 Region Server B Append+sync ack edit edit WAL Append+sync ack Master Zookeeper RegionServer A znode deleted assign Region1 Region Server A Region 2 … … … YouAreDeadException abort RegionServer A session timeout -- B RegionServer A session timeout Client
  22. 22. © Hortonworks Inc. 2011 IO Fencing • Split Brain • Ensure that a region is only hosted by a single region server at any time • If master thinks that region server no longer hosts the region, RS should not be able to accept and sync() updates • Master renames the region server logs directory on HDFS: – Current WAL cannot be rolled, new log file cannot be created – For each WAL, before replaying recoverLease() is called – recoverLease => lease recovery + block recovery – Ensure that WAL is closed, and all data is visible (file length) • Guarantees for region data files: – Compactions => Remove files + add files – Flushed => Allowed since resulting data is idempotent • HBASE-2231, HBASE-7878, HBASE-8449 Page 22 Architecting the Future of Big Data
  23. 23. © Hortonworks Inc. 2011 Data Locality Short circuit reads, checksums, block placement Architecting the Future of Big Data Page 23
  24. 24. © Hortonworks Inc. 2011 HDFS local reads (short circuit reads) • Bypasses the datanode layer and directly goes to the OS files • Hadoop 1.x implementation: – DFSClient asks for local paths for a block to the local datanode – Datanode checks whether the user has permission – Client gets the path for the block, opens the file with FileInputStream hdfs-site.xml dfs.block.local-path-access.user = hbase dfs.datanode.data.dir.perm = 750 hbase-site.xml dfs.client.read.shortcircuit = true Page 24 Architecting the Future of Big Data RegionServer Hadoop FileSystem DFSClient Datanode OS Filesystem (ext3) Disks Disks Disks HBase Client RPC RPC BlockReader
  25. 25. © Hortonworks Inc. 2011 HDFS local reads (short circuit reads) • Hadoop 2.0 implementation (HDFS-347) – Keep the legacy implementation – Use Unix Domain sockets to pass the File Descriptor (FD) – Datanode opens the block file and passes FD to the BlockReaderLocal running in Regionserver process – More secure than previous implementation – Windows also supports domain sockets, need to implement native APIs • Local buffer size dfs.client.read.shortcircuit.buffer.size – BlockReaderLocal will fill this whole buffer everytime HBase will try to read an HfileBlock – dfs.client.read.shortcircuit.buffer.size = 1MB vs 64KB Hfile block size – SSR buffer is a direct buffer (in Hadoop 2, not in Hadoop 1) – # regions x # stores x #avg store files x # avg blocks per file x SSR buffer size – 10 regions x 2 x 4 x (1GB / 64MB) x 1 MB = 1.28GB non-heap memory usage Page 25 Architecting the Future of Big Data
  26. 26. © Hortonworks Inc. 2011 Checksums • HDFS checksums are not inlined. • Two files per block, one for data, one for checksums (HDFS-2699) • Random positioned read causes 2 seeks • HBase checksums comes with 0.94 (HDP 1.2+). HBASE-5074. Page 26 Architecting the Future of Big Data blk_123456789 .blk_123456789.meta : Data chunk (dfs.bytes-per-checksum, 512 bytes) : Checksum chunk (4 bytes)
  27. 27. © Hortonworks Inc. 2011 Checksums Page 27 Architecting the Future of Big Data • HFile version 2.1 writes checksums per Hfile block • HDFS checksum verification is bypassed on block read, will be done by HBase • If checksum fails, we go back to reading checksums from HDFS for “some time” • Due to double checksum bug(HDFS-3429) in remote reads in Hadoop 1, not enabled by default for now. Benchmark it yourself hbase.regionserver.checksum.verify = true hbase.hstore.bytes.per.checksum = 16384 hbase.hstore.checksum.algorithm = CRC32C Never set this: dfs.client.read.shortcircuit.skip.checksum = false HFile : Hfile data block chunk : Checksum chunk Hfile block : Block header
  28. 28. © Hortonworks Inc. 2011 Rack 1 / Server 1 DataNode Default Block Placement Policy Page 28 Architecting the Future of Big Data b1 RegionServer Region A Region B StoreFile StoreFile StoreFile StoreFile StoreFile b2 b2 b9 b1 b1 b2 b3 b2 b1 b2b1 Rack N / Server M DataNode b2 b1 b1 Rack L / Server K DataNode b2 b1 Rack X / Server Y DataNode b1b2 b2 b3 RegionServer RegionServer RegionServer
  29. 29. © Hortonworks Inc. 2011 Data locality for HBase • Poor data locality when the region is moved: – As a result of load balancing – Region server crash + failover • Most of the data won’t be local unless the files are compacted • Idea (from Facebook): Regions have affiliated nodes (primary, secondary, tertiary), HBASE-4755 • When writing a data file, give hints to the NN that we want these locations for block replicas (HDFS-2576) • LB should assign the region to one of the affiliated nodes on server crash – Keep data locality – SSR will still work • Reduces data loss probability Page 29 Architecting the Future of Big Data
  30. 30. © Hortonworks Inc. 2011 Rack X / Server Y RegionServer Rack L / Server K RegionServer Rack N / Server M RegionServer Rack 1 / Server 1 Default Block Placement Policy Page 30 Architecting the Future of Big Data RegionServer Region A StoreFile StoreFile StoreFile Region B StoreFile StoreFile DataNode b1 b2 b2 b9 b1 b1 b2 b3 b2 b1 b2b1 DataNode b1 b2 b2 b9b1 b2 b1 DataNode b1 b2 b2 b9 b2 b1 DataNode b1 b2 b3 b2 b1
  31. 31. © Hortonworks Inc. 2011 Other considerations • HBase riding over Namenode HA – Both Hadoop 1 (NFS based) and Hadoop 2 HA (JQM, etc) – Heavily tested with full stack HA • Retry HDFS operations • Isolate FileSystem usage from HBase internals • Hadoop 2 vs Hadoop 1 performance – Hadoop 2 is coming! • HDFS snapshots vs HBase snapshots – HBase DOES NOT use HDFS snapshots – Need hardlinks – Super flush API • HBase security vs HDFS security – All files are owned by HBase principal – No ACL’s in HDFS. Allowing a user to read Hfiles / snapshots directly is hard Page 31 Architecting the Future of Big Data
  32. 32. © Hortonworks Inc. 2011 Open Topics • HDFS hard links – Rethink how we do snapshots, backups, etc • Parallel writes for WAL – Reduce latency on WAL syncs • SSD storage, cache – SSD storage type in Hadoop or local filesystem – Using SSD’s as a secondary cache – Selectively places tables / column families on SSD • HDFS zero-copy reads (HDFS-3051, HADOOP-8148) • HDFS inline checksums (HDFS-2699) • HDFS Quorum reads (HBASE-7509) Page 32 Architecting the Future of Big Data
  33. 33. © Hortonworks Inc. 2011 Thanks Questions? Architecting the Future of Big Data Page 33 Enis Söztutar enis [ at ] apache [dot] org @enissoz

×