Understanding what is a region for HBase, why those transitions, how to troubleshoot and fix potential problems that may arise from this important HBase internal operation.

2. •
•
•
•
•
•
•
•

3. •
•
•
•
•

4. •
•
•
•

5. •
•
•
•
•
•
•

6. •
•

7. •

9. •
•
// Fix up assignment manager status
status.setStatus("Starting assignment manager");
this.assignmentManager.joinCluster();
…………

10. •

11. •
•
•
•
•
•
•
•
•

12. •
•
•
•
•
•

13. •
•
2017-04-12 09:05:46,518 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {a3a747cf03c1249c9d394ec7114b4c17 state=OFFLINE, ts=1492013146322,
server=host-10-17-81-106.coe.cloudera.com,60020,1491568321924} to {a3a747cf03c1249c9d394ec7114b4c17 state=PENDING_OPEN, ts=1492013146518,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599}
017-04-12 09:05:48,169 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {a3a747cf03c1249c9d394ec7114b4c17 state=PENDING_OPEN, ts=1492013146518,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599} to {a3a747cf03c1249c9d394ec7114b4c17 state=OPENING, ts=1492013148169,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599}
2017-04-12 09:05:48,920 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {a3a747cf03c1249c9d394ec7114b4c17 state=OPENING, ts=1492013148169,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599} to {a3a747cf03c1249c9d394ec7114b4c17 state=OPEN, ts=1492013148920,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599}
2017-04-12 09:05:46,749 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Open test,,1488813550988.a3a747cf03c1249c9d394ec7114b4c17.
2017-04-12 09:05:48,529 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined a3a747cf03c1249c9d394ec7114b4c17; next sequenceid=1734
2017-04-12 09:05:48,534 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks for test,,1488813550988.a3a747cf03c1249c9d394ec7114b4c17.
2017-04-12 09:05:48,787 INFO org.apache.hadoop.hbase.MetaTableAccessor: Updated row test,,1488813550988.a3a747cf03c1249c9d394ec7114b4c17. with
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599

14. •
•
2017-04-12 09:05:46,313 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning 3 region(s) to host-10-17-81-106.coe.cloudera.com,60020,1492013113599
2017-04-12 09:05:46,518 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {752c827664cedb16bd3b1b7ecbda25b7 state=OFFLINE, ts=1492013146322,
server=host-10-17-81-106.coe.cloudera.com,60020,1491568321924} to {752c827664cedb16bd3b1b7ecbda25b7 state=PENDING_OPEN, ts=1492013146518,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599}
2017-04-12 09:05:46,819 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {752c827664cedb16bd3b1b7ecbda25b7 state=PENDING_OPEN, ts=1492013146518,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599} to {752c827664cedb16bd3b1b7ecbda25b7 state=OPENING, ts=1492013146819,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599}
2017-04-12 09:05:47,998 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {752c827664cedb16bd3b1b7ecbda25b7 state=OPENING, ts=1492013146819,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599} to {752c827664cedb16bd3b1b7ecbda25b7 state=CLOSED, ts=1492013147998,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599}
2017-04-12 09:05:48,006 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {752c827664cedb16bd3b1b7ecbda25b7 state=CLOSED, ts=1492013148004,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599} to {752c827664cedb16bd3b1b7ecbda25b7 state=OFFLINE, ts=1492013148006,
server=host-10-17-81-106.coe.cloudera.com,60020,1492013113599}
2017-04-12 09:05:46,556 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Open test3-clone2,,1491211108318.752c827664cedb16bd3b1b7ecbda25b7.
2017-04-12 09:05:47,881 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Could not initialize all stores for the region=test3-clone2,,1491211108318.752c827664cedb16bd3b1b7ecbda25b7.
2017-04-12 09:05:47,902 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=test3-clone2,,1491211108318.752c827664cedb16bd3b1b7ecbda25b7.,
starting to roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink
locations=[hdfs://nameservice1/hbase/data/default/test3/99e0261dda05c9f5ed7a803201f030e8/t/de84a1ab669346d381057e0903df9186,
hdfs://nameservice1/hbase/.tmp/data/default/test3/99e0261dda05c9f5ed7a803201f030e8/t/de84a1ab669346d381057e0903df9186,
hdfs://nameservice1/hbase/mobdir/data/default/test3/99e0261dda05c9f5ed7a803201f030e8/t/de84a1ab669346d381057e0903df9186,
hdfs://nameservice1/hbase/archive/data/default/test3/99e0261dda05c9f5ed7a803201f030e8/t/de84a1ab669346d381057e0903df9186]
at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:949)
at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:824)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:799)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6480)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6441)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6412)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6368)

15. •
•
•
•
•
•
•
•
•

16. •
•
•
•
•
•
•
•

17. •
•
•
•
•
•
•

18. •
2017-04-20 10:44:20,269 INFO org.apache.hadoop.hbase.wal.WALSplitter: Splitting wal:
hdfs://nameservice1/hbase/WALs/host-10-17-81-107.coe.cloudera.com,60020,1492709053764-splitting/host-10-17-81-107.coe.cloudera.com%2C60020%2C1492709053764.null0.1492709058838,
length=83
2017-04-20 10:44:20,269 INFO org.apache.hadoop.hbase.wal.WALSplitter: DistributedLogReplay = false
2017-04-20 10:44:24,868 INFO org.apache.hadoop.hbase.wal.WALSplitter: 3 split writers finished; closing...
2017-04-20 10:44:25,549 INFO org.apache.hadoop.hbase.wal.WALSplitter: Rename
hdfs://nameservice1/hbase/data/default/test3-clone3/7f57bf2ad2e9a3fad7368fbc07bdeb53/recovered.edits/0000000000000000562.temp to
hdfs://nameservice1/hbase/data/default/test3-clone3/7f57bf2ad2e9a3fad7368fbc07bdeb53/recovered.edits/0000000000000000562
2017-04-20 10:44:25,643 INFO org.apache.hadoop.hbase.wal.WALSplitter: Rename
hdfs://nameservice1/hbase/data/default/test3-clone/70586538b14a7a2fb8e7806ce2b21ff9/recovered.edits/0000000000000000586.temp to
hdfs://nameservice1/hbase/data/default/test3-clone/70586538b14a7a2fb8e7806ce2b21ff9/recovered.edits/0000000000000000586
2017-04-20 10:44:25,758 INFO org.apache.hadoop.hbase.wal.WALSplitter: Rename
hdfs://nameservice1/hbase/data/default/test/a3a747cf03c1249c9d394ec7114b4c17/recovered.edits/0000000000000001740.temp to
hdfs://nameservice1/hbase/data/default/test/a3a747cf03c1249c9d394ec7114b4c17/recovered.edits/0000000000000001740
2017-04-20 10:44:26,000 INFO org.apache.hadoop.hbase.wal.WALSplitter: Rename
hdfs://nameservice1/hbase/data/default/test3-clone/ded870d3604dbd3df0b956a347c7790b/recovered.edits/0000000000000000584.temp to
hdfs://nameservice1/hbase/data/default/test3-clone/ded870d3604dbd3df0b956a347c7790b/recovered.edits/0000000000000000584
2017-04-20 10:44:26,058 INFO org.apache.hadoop.hbase.wal.WALSplitter: Rename
hdfs://nameservice1/hbase/data/default/test2/fcf33a401d0092a34b8b68a76184af07/recovered.edits/0000000000000000055.temp to
hdfs://nameservice1/hbase/data/default/test2/fcf33a401d0092a34b8b68a76184af07/recovered.edits/0000000000000000056
2017-04-20 10:44:26,058 INFO org.apache.hadoop.hbase.wal.WALSplitter: Processed 6 edits across 5 regions; edits skipped=0; log
file=hdfs://nameservice1/hbase/WALs/host-10-17-81-107.coe.cloudera.com,60020,1492709053764-splitting/host-10-17-81-107.coe.cloudera.com%2C60020%2C1492709053764.null0.1492709058
838, length=83, corrupted=false, progress failed=false

19. •
•
•
•
•
•
•
•
•