Successfully reported this slideshow.
Your SlideShare is downloading. ×

HBase Internals And Operations

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

HBase Internals And Operations

  1. 1. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Internals and Operations SRECon19 Asia/Pacific June 13, 2019 Biju Nair Software Engineer bnair10@bloomberg.net
  2. 2. © 2019 Bloomberg Finance L.P. All rights reserved. Agenda • Introduction to HBase • Operating HBase • Questions
  3. 3. © 2019 Bloomberg Finance L.P. All rights reserved. Bloomberg in a nutshell The Bloomberg Terminal delivers a diverse array of information on a single platform to facilitate financial decision-making.
  4. 4. © 2019 Bloomberg Finance L.P. All rights reserved. Bloomberg technology by the numbers • 5,000+ software engineers • 150+ technologists and data scientists devoted to machine learning • One of the largest private networks in the world • 120 billion pieces of data from the financial markets each day, with a peak of more than 10 million messages/second • 2 million news stories ingested / published each day (500+ news stories ingested/second) • News content from 125K+ sources • Over 1 billion messages and Instant Bloomberg (IB) chats handled daily
  5. 5. © 2019 Bloomberg Finance L.P. All rights reserved. HBase at Bloomberg • Started with v0.94.6 • >2 billion reads per day • >1 billion writes per day • 51+ TB of compressed data stored in HBase
  6. 6. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Principles • Ordered Key Value Store • Distributed shared nothing
  7. 7. © 2019 Bloomberg Finance L.P. All rights reserved. Key Value Key-9999 Value-a Key-9998 Value-b Key-9997 Value-c Key-9996 Value-d … … Key-9995 Value-e Key-9994 Value-f
  8. 8. © 2019 Bloomberg Finance L.P. All rights reserved. Ordered Key Value Key-9999 Value-a Key-9998 Value-b Key-9997 Value-c Key-9996 Value-d … … Key-9995 Value-e Key-9994 Value-a Lexicographicorder Key-9993 Value-g
  9. 9. © 2019 Bloomberg Finance L.P. All rights reserved. Ordered Key Value Key-9999 Value-a Key-9998 Value-b Key-9997 Value-c Key-9996 Value-d … … Key-9995 Value-e Key-9994 Value-a Lexicographicorder Key-9993 Value-g Region
  10. 10. © 2019 Bloomberg Finance L.P. All rights reserved. Distributed Ordered Key Value Key-199 Value Key-198 Value Key-197 Value Key-499 Value … Key-498 Value Key-497 Value ordered … Key-299 Value … Key-298 Value Key-297 Value ordered … … … Key-599 Value … Key-598 Value Key-597 Value … Key-399 Value … Key-398 Value Key-397 Value … Key-999 Value … Key-998 Value Key-997 Value … Region Region
  11. 11. © 2019 Bloomberg Finance L.P. All rights reserved. Distributed Ordered Key Value RegionServer Key-199 Value Key-198 Value Key-197 Value Key-499 Value … Key-498 Value Key-497 Value … Key-299 Value … Key-298 Value Key-297 Value … … … Key-599 Value … Key-598 Value Key-597 Value … Key-399 Value … Key-398 Value Key-397 Value … Key-999 Value … Key-998 Value Key-997 Value … Region Region RegionServer RegionServer RegionServer RegionServer RegionServer
  12. 12. © 2019 Bloomberg Finance L.P. All rights reserved. Table Row View Key Value Column Id ValueRow Id Timestamp
  13. 13. © 2019 Bloomberg Finance L.P. All rights reserved. Table Row View R11|col1|1234567 Value-A R11|col2|1234567 Value-B R11|col3|1234567 Value-C R11|col4|1234567 Value-D R11 Value-A Value-B Value-C Col1 Col2 Col3 Value-D Col4
  14. 14. © 2019 Bloomberg Finance L.P. All rights reserved. Versioning R11|col1|1234567 Value-A1 R11|col1|1234566 Value-A R11|col2|1234567 Value-B R11|col3|1234567 Value-CC R11|col3|1234563 Value-C R11|col4|1234567 Value-DD R11|col4|1234560 Value-D1 R11|col4|1234557 Value-D Descendingorder
  15. 15. © 2019 Bloomberg Finance L.P. All rights reserved. cf2cf1 Column Family R11|col1|1234567 Value-A R11|col2|1234567 Value-B R11|col3|1234567 Value-C R11 Value-A Value-B Value-C cf1:col1 cf1:col2 cf1:col3 Value-1 cf2:col1 R11|col1|1234567 Value-1 R11|col2|1234567 Value-2 R11|col3|1234567 Value-3 Value-2 cf2:col2 Value-3 cf2:col3
  16. 16. © 2019 Bloomberg Finance L.P. All rights reserved. ACIDity • Atomic at row level • Consistent to a point in time before the request • Isolation through MVCC (reads) and row locks (mutations) • Durability is guaranteed for all successful mutations
  17. 17. © 2019 Bloomberg Finance L.P. All rights reserved. Namespace Namespace: Blue Namespace: Yellow ACL ACL ACL ACL ACL
  18. 18. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Write HBase Region Server File System WAL Write 1
  19. 19. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Write HBase RS Memstore File System WAL Write 1 2
  20. 20. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Write HBase RS Memstore File System WAL t1 t1 t1 Store files Write 1 2 3 Block Block Block Idx BlmData
  21. 21. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Write HBase RS Memstore DFSClient WAL t1 t1 t1 Store files Write 1 2 3
  22. 22. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Write HBase RS Memstore DFSClient WAL t1 t1 t1 Store files Write 1 2 3 HDFS DataNode WAL t1 t1 t1
  23. 23. © 2019 Bloomberg Finance L.P. All rights reserved. Compaction File System K-x K-x D-1 Store files K-xK-xK-x
  24. 24. © 2019 Bloomberg Finance L.P. All rights reserved. Compaction File System Store files K-x Compaction File System K-x K-x D-1 Store files K-xK-xK-x
  25. 25. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Read HBase RS Memstore File System WAL Read 1 Block Cache Block t1 t1 t1 Store files
  26. 26. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Read HBase RS Memstore File System WAL Read 1 Block Cache 2 Block t1 t1 t1 Store files
  27. 27. © 2019 Bloomberg Finance L.P. All rights reserved. HDFS Read HBase/ DFSClient HDFS File System Data TCP
  28. 28. © 2019 Bloomberg Finance L.P. All rights reserved. HDFS Short-Circuit Read HBase/ DFSClient HDFS File System Data TCP Open File Pass FD
  29. 29. © 2019 Bloomberg Finance L.P. All rights reserved. Large Read Cache HBase Memstore File System WAL t1 t1 t1 Store files Read 1 Block Cache 2 Block
  30. 30. © 2019 Bloomberg Finance L.P. All rights reserved. Large Read Cache HBase Memstore File System WAL t1 t1 t1 Store files Read 1 Block Cache 3 Block Off-heap Cache Block 2
  31. 31. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Complete HMaster RS1 RS2 RS3 RS4 RS5 RS6 system
  32. 32. © 2019 Bloomberg Finance L.P. All rights reserved. HMaster HBase Complete HMaster RS1 RS2 RS3 RS4 RS5 RS6 system
  33. 33. © 2019 Bloomberg Finance L.P. All rights reserved. HMaster HBase Complete HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK
  34. 34. © 2019 Bloomberg Finance L.P. All rights reserved. HMaster HBase Complete HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK
  35. 35. © 2019 Bloomberg Finance L.P. All rights reserved. HMaster HBase Complete HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK HBase Client
  36. 36. © 2019 Bloomberg Finance L.P. All rights reserved. Region Server Failure HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK T1 r1 HBase Client
  37. 37. © 2019 Bloomberg Finance L.P. All rights reserved. Region Server Failure HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK T1 r1 HBase Client
  38. 38. © 2019 Bloomberg Finance L.P. All rights reserved. Region Server Failure HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK T1 r1 HBase Client
  39. 39. © 2019 Bloomberg Finance L.P. All rights reserved. Region Replication HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK T1 r1 HBase Client T1 r1
  40. 40. © 2019 Bloomberg Finance L.P. All rights reserved. Region Replication HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK T1 r1T1 r1 HBase Client
  41. 41. © 2019 Bloomberg Finance L.P. All rights reserved. Region Replication HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK T1 r1T1 r1 HBase Client https://www.youtube.com/watch?v=l6S-Vbs9WsU
  42. 42. © 2019 Bloomberg Finance L.P. All rights reserved. Load Balancing HMaster RS1 RS2 RS3 RS4 RS5 RS6 system ZK HBase Client
  43. 43. © 2019 Bloomberg Finance L.P. All rights reserved. Load Balancing HMaster Balancer/AM RS1 RS2 RS3 RS4 RS5 RS6 system ZK HBase Client
  44. 44. © 2019 Bloomberg Finance L.P. All rights reserved. Balancer • Region Count Cost • Primary Region Count Cost • Table Skew Cost • Locality Cost • Rack Locality Cost • Region Replica Host Cost • Region Replica Rack Cost • Read Request Cost • Write Request Cost • Memstore Size Cost • Storefile Size Cost • Move Cost
  45. 45. © 2019 Bloomberg Finance L.P. All rights reserved. Other Features • HBase Replication • HBase multi-tenancy support — https://www.youtube.com/watch?v=bZjz2G38Ju0 • HBase Co-processors and Filters — https://www.slideshare.net/Hadoop_Summit/hbase-coprocessors-uses-abuses-solutions
  46. 46. © 2019 Bloomberg Finance L.P. All rights reserved. NameNode RS4 HMaster ZQuorum ZQuorum Operator View HMaster RS1 RS2 RS4 RS5 RS6 HBase ClientZK Quorum DN1 DN2 DN3 DN4 DN5 DN6 NameNode JVM OS JVM OS JVM OS JVM OS JVM OS JVM OS HW HW HW HW HW HW
  47. 47. © 2019 Bloomberg Finance L.P. All rights reserved. ZooKeeper Availability • ZK Quorum • One leader and remaining followers — stat — ruok — mntr • Test for availability — e.g., List children of a znode
  48. 48. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Availability • Master Availability - http://hmaster-node:16010/jmx — "name" : "Hadoop:service=HBase,name=Master,sub=Server","tag.isActiveMaster" : "true" • Dead RegionServers — "name" : "Hadoop:service=HBase,name=Master,sub=Server", "numDeadRegionServers" : 0 • Region In Transition — "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger","ritCount" : 0 • Test for availability — e.g., Query system table by listing tables
  49. 49. © 2019 Bloomberg Finance L.P. All rights reserved. HDFS Availability • Namenode Availability - http://namenode-host:50070/jmx —"name" : "Hadoop:service=NameNode,name=FSNamesystem", "tag.HAState" : "active" • Dead Datanodes — "name" : "Hadoop:service=NameNode,name=NameNodeInfo", "DeadNodes" : "{}“ • Missing Blocks — "name" : "Hadoop:service=NameNode,name=NameNodeInfo", "NumberOfMissingBlocks" : 0 • Percentage Used — "name" : "Hadoop:service=NameNode,name=NameNodeInfo", "PercentUsed" : 59 • Under replicated blocks — "name" : "Hadoop:service=NameNode,name=FSNamesystemState","UnderReplicatedBlocks":0 • Test for availability — e.g., Append data to a test file
  50. 50. © 2019 Bloomberg Finance L.P. All rights reserved. HBase Performance • RegionServer JMX metrics - http://rs-node:60300/jmx — "name“:"Hadoop:service=HBase,name=RegionServer,sub=Server“ • Blockcache hit ratio • Request counts • Request response time • Compaction related metrics • Region count • Flush related metrics • Percentage of files local • Split related metrics — "name" : "Hadoop:service=HBase,name=RegionServer,sub=Tables", • Table level metrics https://www.slideshare.net/MichaelStack4/hbaseconasia2018-track31-serving-billions-of-queries-in-millisecond-latencies
  51. 51. © 2019 Bloomberg Finance L.P. All rights reserved. JVM • GC – JMX Metrics — "name" : "java.lang:type=GarbageCollector,name=ParNew", — "name" : "java.lang:type=GarbageCollector,name=ConcurrentMarkSweep", • GC Logging — -verbose:gc — -XX:+PrintHeapAtGC — -XX:+PrintGCDetails — -XX:+PrintGCTimeStamps — -XX:+PrintGCDateStamps — -XX:+PrintGCApplicationStoppedTime — -XX:+PrintClassHistogram — -XX:+PrintGCApplicationConcurrentTime — -XX:+PrintTenuringDistribution — -Xloggc:
  52. 52. © 2019 Bloomberg Finance L.P. All rights reserved. OS/HW • Memory • CPU • Disk • Networking
  53. 53. © 2019 Bloomberg Finance L.P. All rights reserved. Logs • ZooKeeper Log • HDFS — Namenode log — Datanode log • HBase — Master log — RegionServer log • OS — Syslog
  54. 54. © 2019 Bloomberg Finance L.P. All rights reserved. Interacting with HBase • HBase shell — DDL: create namespace/table, alter — Security: grant, revoke — DML: get, put, scan — Tools: assign, compact, balance — General: status • HBase admin API • HBase client API
  55. 55. © 2019 Bloomberg Finance L.P. All rights reserved. Data Backup / Restore • Snapshot — hbase shell > snapshot 'table', 'table_mmddyy’ • Restore from snapshot — hbase shell > restore_snapshot 'table_mmddyy' • Export Snapshot — $ hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot • CopyTable — hbase org.apache.hadoop.hbase.mapreduce.CopyTable
  56. 56. © 2019 Bloomberg Finance L.P. All rights reserved. Thank You! Acknowledgement: Apache HBase Community Reference: http://hbase.apache.org Connect with Hadoop Team: hadoop@bloomberg.net
  57. 57. © 2019 Bloomberg Finance L.P. All rights reserved. We are hiring! Questions? https://www.bloomberg.com/careers

×