HDFS metadata (fsimage and edits) difference CDH3 and CDH4
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

HDFS metadata (fsimage and edits) difference CDH3 and CDH4

on

  • 752 views

HDFS fsimage and edits difference in CDH3 and CDH4

HDFS fsimage and edits difference in CDH3 and CDH4

Statistics

Views

Total Views
752
Views on SlideShare
746
Embed Views
6

Actions

Likes
4
Downloads
6
Comments
0

1 Embed 6

https://twitter.com 6

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

HDFS metadata (fsimage and edits) difference CDH3 and CDH4 Presentation Transcript

  • 1. fsimage and edits in CDH3 and CDH4Tatsuo Kawasakitatsuo@cloudera.com
  • 2. objectiveHDFS metadata (fsimage and edits) management is differentbetween CDH3 and CDH4.This presentation introduces a these difference.Please let me know if you find any issue.
  • 3. HDFS metada in CDH3[root@localhost ~]# ls -ltr /var/lib/hadoop-0.20/cache/hadoop/dfs/name/current/total 1100-rw-r--r-- 1 hdfs hdfs 101 Jan 30 00:21 VERSION-rw-r--r-- 1 hdfs hdfs 8 Jan 30 00:21 fstime-rw-r--r-- 1 hdfs hdfs 57248 Jan 30 00:21 fsimage-rw-r--r-- 1 hdfs hdfs 1048580 Jan 31 16:16 editsafter checkpoint[root@localhost ~]# ls -ltr /var/lib/hadoop-0.20/cache/hadoop/dfs/name/current/total 84-rw-r--r-- 1 hdfs hdfs 101 Feb 5 14:37 VERSION-rw-r--r-- 1 hdfs hdfs 8 Feb 5 14:37 fstime-rw-r--r-- 1 hdfs hdfs 66760 Feb 5 14:37 fsimage-rw-r--r-- 1 hdfs hdfs 4 Feb 5 14:37 edits
  • 4. timeline (CDH3) NameNode CheckPoint CheckPoint put file start Done t0 t1 t2 t3 t4 rename fsimage fsimage.ckpt fsimage Update edits edits Update metadata in edits.new edits memory rename create fstime fstime get transfer update timeSecondary NameNode fsimage fsimage.ckpt merge edits
  • 5. Secondary NN Web UI (CDH3)
  • 6. HDFS metadata in CDH4After formatting HDFS-bash-4.1$ ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/total 1040-rw-r--r-- 1 hdfs hdfs 1048576 Feb 5 01:35 edits_inprogress_0000000000000000001-rw-rw-r-- 1 hdfs hdfs 119 Feb 5 01:33 fsimage_0000000000000000000-rw-rw-r-- 1 hdfs hdfs 62 Feb 5 01:33 fsimage_0000000000000000000.md5-rw-r--r-- 1 hdfs hdfs 2 Feb 5 01:35 seen_txid-rw-rw-r-- 1 hdfs hdfs 202 Feb 5 01:33 VERSION-bash-4.1$ hexdump -C /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/seen_txid00000000 31 0a |1.|00000002 Transaction ID is included in seen_txid
  • 7. try to add new file[training@localhost ~]$ hadoop fs -put /etc/hosts hosts[training@localhost ~]$
  • 8. oiv - fsimage viewer-bash-4.1$ hdfs oiv -i /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/fsimage_0000000000000000000 -o aaa-bash-4.1$ cat aaadrwxr-xr-x - hdfs supergroup 0 1969-12-31 19:00 / ‘hosts’ file has not written in fsimage before checkpointing
  • 9. oev – edits viewer -bash-4.1$ hdfs oev -i /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/edits_0000000000000000020-0000000000000000027 -o bbb cat bbb <?xml version="1.0" encoding="UTF-8"?> <EDITS> <EDITS_VERSION>-40</EDITS_VERSION> <RECORD> <OPCODE>OP_START_LOG_SEGMENT</OPCODE> <DATA> start <TXID>20</TXID> </DATA> Transaction ID </RECORD> <RECORD> <OPCODE>OP_SET_GENSTAMP</OPCODE> <DATA> <TXID>21</TXID> <GENSTAMP>1003</GENSTAMP> </DATA> </RECORD> Transaction ID <RECORD> <OPCODE>OP_ADD</OPCODE>Put <DATA> <TXID>22</TXID>Transaction <LENGTH>0</LENGTH> <PATH>/user/training/hosts._COPYING_</PATH> <REPLICATION>1</REPLICATION> <MTIME>1360046220628</MTIME> <ATIME>1360046220628</ATIME> <BLOCKSIZE>67108864</BLOCKSIZE> <CLIENT_NAME>DFSClient_NONMAPREDUCE_1911533003_1</CLIENT_NAME> <CLIENT_MACHINE>127.0.0.1</CLIENT_MACHINE> <PERMISSION_STATUS> <USERNAME>training</USERNAME> <GROUPNAME>supergroup</GROUPNAME> <MODE>420</MODE> </PERMISSION_STATUS> </DATA> </RECORD>
  • 10. oev – edits viewer (cont) ファイル名edits_0000000000000000020-0000000000000000027 <RECORD> <OPCODE>OP_SET_GENSTAMP</OPCODE> <DATA> <TXID>23</TXID> <GENSTAMP>1004</GENSTAMP> </DATA> </RECORD> <RECORD> <OPCODE>OP_UPDATE_BLOCKS</OPCODE> <DATA> <TXID>24</TXID> <PATH>/user/training/hosts._COPYING_</PATH> <BLOCK> <BLOCK_ID>-3498739165311848505</BLOCK_ID> <NUM_BYTES>0</NUM_BYTES> <GENSTAMP>1004</GENSTAMP> </BLOCK> </DATA> </RECORD> <RECORD> <OPCODE>OP_CLOSE</OPCODE> <DATA> <TXID>25</TXID> <LENGTH>0</LENGTH> <PATH>/user/training/hosts._COPYING_</PATH> <REPLICATION>1</REPLICATION> <MTIME>1360046220735</MTIME> <ATIME>1360046220628</ATIME> <BLOCKSIZE>67108864</BLOCKSIZE> <CLIENT_NAME></CLIENT_NAME> <CLIENT_MACHINE></CLIENT_MACHINE> <BLOCK> <BLOCK_ID>-3498739165311848505</BLOCK_ID> <NUM_BYTES>83</NUM_BYTES> <GENSTAMP>1004</GENSTAMP> </BLOCK>
  • 11. oev – edits viewer (cont) ファイル名:edits_0000000000000000020-0000000000000000027 <PERMISSION_STATUS> <USERNAME>training</USERNAME> <GROUPNAME>supergroup</GROUPNAME> <MODE>420</MODE> </PERMISSION_STATUS> </DATA> </RECORD> <RECORD> <OPCODE>OP_RENAME_OLD</OPCODE> <DATA> <TXID>26</TXID> <LENGTH>0</LENGTH> <SRC>/user/training/hosts._COPYING_</SRC> <DST>/user/training/hosts</DST> End <TIMESTAMP>1360046220738</TIMESTAMP> </DATA> Transaction ID </RECORD> <RECORD> <OPCODE>OP_END_LOG_SEGMENT</OPCODE> <DATA> <TXID>27</TXID> </DATA> </RECORD> </EDITS>
  • 12. After checkpointing-bash-4.1$ ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/total 1376-rw-r--r-- 1 hdfs hdfs 1317 Feb 5 01:36 edits_0000000000000000001-0000000000000000019-rw-r--r-- 1 hdfs hdfs 471 Feb 5 01:37 edits_0000000000000000020-0000000000000000027-rw-r--r-- 1 hdfs hdfs 30 Feb 5 01:38 edits_0000000000000000028-0000000000000000029-rw-r--r-- 1 hdfs hdfs 30 Feb 5 01:39 edits_0000000000000000030-0000000000000000031-rw-r--r-- 1 hdfs hdfs 30 Feb 5 01:40 edits_0000000000000000032-0000000000000000033-rw-r--r-- 1 hdfs hdfs 30 Feb 5 01:41 edits_0000000000000000034-0000000000000000035(略)-rw-r--r-- 1 hdfs hdfs 30 Feb 5 02:53 edits_0000000000000000178-0000000000000000179-rw-r--r-- 1 hdfs hdfs 30 Feb 5 02:54 edits_0000000000000000180-0000000000000000181-rw-r--r-- 1 hdfs hdfs 30 Feb 5 02:55 edits_0000000000000000182-0000000000000000183-rw-r--r-- 1 hdfs hdfs 30 Feb 5 02:56 edits_0000000000000000184-0000000000000000185-rw-r--r-- 1 hdfs hdfs 30 Feb 5 02:58 edits_0000000000000000186-0000000000000000187-rw-r--r-- 1 hdfs hdfs 1048576 Feb 5 02:58 edits_inprogress_0000000000000000188-rw-rw-r-- 1 hdfs hdfs 119 Feb 5 01:33 fsimage_0000000000000000000-rw-rw-r-- 1 hdfs hdfs 62 Feb 5 01:33 fsimage_0000000000000000000.md5-rw-r--r-- 1 hdfs hdfs 1211 Feb 5 02:58 fsimage_0000000000000000187-rw-r--r-- 1 hdfs hdfs 62 Feb 5 02:58 fsimage_0000000000000000187.md5-rw-r--r-- 1 hdfs hdfs 4 Feb 5 02:58 seen_txid Transaction ID is-bash-4.1$ hexdump -C /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/seen_txid included in00000000 31 38 38 0a |188.|00000004 seen_txid
  • 13. oiv - fsimage viewer-bash-4.1$ hdfs oiv -i /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/fsimage_0000000000000000187 -o aaa-bash-4.1$ cat aaadrwxr-xr-x - hdfs supergroup 0 2013-02-05 01:35 /drwxr-xr-x - hdfs supergroup 0 2013-02-05 01:35 /userdrwxr-xr-x - mapred supergroup 0 2013-02-05 01:35 /vardrwxrwxrwt - hdfs supergroup 0 2013-02-05 01:37 /user/training-rw-r--r-- 1 training supergroup 83 2013-02-05 01:37 /user/training/hostsdrwxr-xr-x - mapred supergroup 0 2013-02-05 01:35 /var/libdrwxr-xr-x - mapred supergroup 0 2013-02-05 01:35 /var/lib/hadoop-hdfsdrwxr-xr-x - mapred supergroup 0 2013-02-05 01:35 /var/lib/hadoop-hdfs/cachedrwxr-xr-x - mapred supergroup 0 2013-02-05 01:35 /var/lib/hadoop-hdfs/cache/mapreddrwxr-xr-x - mapred supergroup 0 2013-02-05 01:35 /var/lib/hadoop-hdfs/cache/mapred/mapreddrwx------ - mapred supergroup 0 2013-02-05 01:35 /var/lib/hadoop- ‘hosts’file hashdfs/cache/mapred/mapred/system been-rw------- 1 mapred supergroup written in HDFS 2013-02-05 01:35 /var/lib/hadoop- 4 afterhdfs/cache/mapred/mapred/system/jobtracker.info checkpoint
  • 14. oev – edits viewer -bash-4.1$ hdfs oev -i /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/edits_inprogress_0000000000000000188 -o bbb -bash-4.1$ cat bbb <?xml version="1.0" encoding="UTF-8"?> <EDITS> <EDITS_VERSION>-40</EDITS_VERSION> Transaction ID <RECORD> <OPCODE>OP_START_LOG_SEGMENT</OPCODE> <DATA> <TXID>188</TXID> </DATA> </RECORD> </EDITS>
  • 15. timeline (CDH4)-1 NameNode t0 (10 times transaction) t1 (22 times transaction) t2 fsimage_0 create new edits_inprogress using new Transaction ID edits_inprogress_1 edits_inprogress_11 edits_inprogress_33 edits_1-10 edits_1-10 finalize and rename edits_11-32 (transaction 1-10) finalize and renameSecondary NameNode (transaction 11-22) trigger a log roll: 1) NN Startup 2) saveNameSpace 3) SecondaryNN CheckPoint 4) storage directroy becomes available 5) admin operation
  • 16. timeline (CDH4)-2 NameNode CheckPoint CheckPoint start Done t4 t5 t6 fsimage_0 fsimage_0 fsimage_ckpt_33 fsimage_33 rename edits_inprogress_33 edits_inprogress_34 edits_1-10 edits_1-10 edits_11-32 edits_11-32 transfer edits_33-Secondary NameNode 33 get fsimage_ckpt_33 merge
  • 17. parameters (CDH4)fsimage_0 The number of image checkpoint files that will be retainedfsimage_33 dfs.namenode.num.checkpoints.retainededits_inprogress_34edits_1-10edits_11-32 The number of extra transaction which should be retainededits_33-33 dfs.namenode.num.extra.edits.retained interval dfs.namenode.checkpoint.period transcations dfs.namenode.checkpoint.txns Secondary NameNode Poll NameNode every seconds dfs.namenode.checkpoint.check.period *fstime is no longer necessary since it’s all encapsulated in the transaction IDs
  • 18. Secondary NN web UI (CDH4)
  • 19. reference• HDFS paramters • http://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop- project-dist/hadoop-hdfs/hdfs-default.xml• HDFS-1073 • https://issues.apache.org/jira/secure/attachment/12478323/hdfs 1073.pdf• O’Reilly Hadoop: The definitive Guide, 3rd edition