図でわかるHDFS Erasure Coding
•
11 likes•4,776 views
Erasure coding in HDFS provides redundancy for data blocks while using less storage space compared to simple replication. It works by splitting files into data blocks and parity blocks striped across multiple data nodes. When reading data, erasure coding reconstructs missing or corrupted blocks from the parity blocks to maintain data reliability even if some blocks are lost. Erasure coding reduces storage overhead compared to replication and improves read performance by allowing reconstruction of missing blocks in parallel from different nodes.
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to 図でわかるHDFS Erasure Coding
Similar to 図でわかるHDFS Erasure Coding (11)
More from Kai Sasaki
More from Kai Sasaki (20)
Recently uploaded
Recently uploaded (20)
図でわかるHDFS Erasure Coding
- 1. 図でわかる HDFS Erasure Coding Kai Sasaki Treasure Data Inc.
- 2. Who am I 佐々木 海(Kai Sasaki) Software Engineer at Treasure Data Inc. http://www.treasuredata.com Hadoop, Spark, DL4J
- 3. Agenda • Erasure Coding • Under the Namespace • Writing Side • Reading Side
- 20. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo…
- 21. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block
- 22. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block BlockGroup
- 25. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit
- 26. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit index 0 index 2 index 1 Saving memory
- 27. Writing Side
- 28. Data 0
- 29. Data64KB 0
- 39. Data BlockGroup 0 Data Block Parity Block
- 40. Data BlockGroup 0 Data Block Parity Block Stripe
- 50. Reading Side
- 51. BlockGroup 0 0
- 60. まとめ • Namespace -> Saving memory BlockInfoStriped, BlockIdManager • Writing Side -> Saving diskspace usage INodeFile • Reading Side -> Saving reading time DFSStripedInputStream
- 61. ありがとうございました