Downloaded 17 times
Uploaded byKoos van Strien
Block replication on HDFS
Files in Hadoop are broken into blocks that are replicated across multiple Data Nodes for redundancy. The document shows how log files a.txt, b.txt and e.txt are each broken into multiple blocks that are stored on different Data Nodes, with each block replicated at least 3 times to ensure the data is not lost if a Node fails.
Block replication on HDFS
- 1. Files are brokenup in ‘blocks’ Log file a.txt Block 1 Essentials of Hadoop for Big Data Block 2 Log file b.txt Block 3 Block 4 1 Log file e.txt Block 5
- 2. Blocks are replicated(at least 3x) throughout the cluster on Data Nodes Block 5 Node 1 Node 2 Node 3 Node 4 Block 4 Block 3 Block 2 Block 1 Essentials of Hadoop for Big Data 2
- 3. Blocks are replicated(at least 3x) throughout the cluster on Data Nodes Block 5 Block 1 Block 2 Block 1 Essentials of Hadoop for Big Data Block 3 Block 1 Block 2 Block 4 Block 4 Block 3 Node 2 Node 1 Block 5 Block 4 Block 5 Block 1 Block 2 Block 3 Block 5 Node 3 Node 4 Block 2 Block 3 Block 4 3