Download as PDF, PPTX
Uploaded byKai Sasaki
図でわかるHDFS Erasure Coding
Erasure coding in HDFS provides redundancy for data blocks while using less storage space compared to simple replication. It works by splitting files into data blocks and parity blocks striped across multiple data nodes. When reading data, erasure coding reconstructs missing or corrupted blocks from the parity blocks to maintain data reliability even if some blocks are lost. Erasure coding reduces storage overhead compared to replication and improves read performance by allowing reconstruction of missing blocks in parallel from different nodes.
図でわかるHDFS Erasure Coding
- 1. 図でわかる HDFS Erasure Coding KaiSasaki Treasure Data Inc.
- 2. Who am I 佐々木海(Kai Sasaki) Software Engineer at Treasure Data Inc. http://www.treasuredata.com Hadoop, Spark, DL4J
- 3. Agenda • Erasure Coding •Under the Namespace • Writing Side • Reading Side
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21. INode and BlockInfo BlockInfo INode BlockInfoBlockInfo… … Block Block Block
- 22. INode and BlockInfo BlockInfo INode BlockInfoBlockInfo… … Block Block Block BlockGroup
- 23.
- 24.
- 25. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup indexGroupId 4bit 60bit
- 26. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup indexGroupId 4bit 60bit index 0 index 2 index 1 Saving memory
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40. Data BlockGroup 0 Data Block ParityBlock Stripe
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59.
- 60. まとめ • Namespace ->Saving memory BlockInfoStriped, BlockIdManager • Writing Side -> Saving diskspace usage INodeFile • Reading Side -> Saving reading time DFSStripedInputStream
- 61.