図でわかるHDFS Erasure Coding

1. 図でわかる  HDFS Erasure Coding Kai Sasaki Treasure Data Inc.

2. Who am I 佐々木海(Kai Sasaki) Software Engineer at Treasure Data Inc.  http://www.treasuredata.com Hadoop, Spark, DL4J

3. Agenda • Erasure Coding • Under the Namespace • Writing Side • Reading Side

4. Erasure Coding

5. Replication Block HDFS

6. Replication Block Block Block Block HDFS

7. Replication Block Block Block Block HDFS

8. Replication Block Block Block Block HDFS Capacity Overhead   x3

9. Replication Block Block Block Block HDFS Redundancy 2 Capacity Overhead   x3

10. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block RS-6-3

11. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9

15. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block BlockRedundancy 3

16. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block Capacity Overhead   x1.5 Redundancy 3

17. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block BlockGroup

18. Under the Namespace

19. INode and BlockInfo BlockInfo INode

20. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo…

21. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block

22. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block BlockGroup

23. BlockInfo … Block Block Block BlockGroup

24. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup

25. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit

26. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit index 0 index 2 index 1 Saving memory

27. Writing Side

28. Data 0

29. Data64KB 0

30. Data BlockGroup 64KB 0

31. Data BlockGroup 0

38. Data BlockGroup 0 Data Block

39. Data BlockGroup 0 Data Block Parity Block

40. Data BlockGroup 0 Data Block Parity Block Stripe

47. Data BlockGroup 0 0

48. Data BlockGroup 0 0

49. Data BlockGroup 0 0 Saving diskspace usage

50. Reading Side

51. BlockGroup 0 0

52. BlockGroup 0 0 200kb 500kb

59. BlockGroup 0 0 200kb 500kb Saving reading time

60. まとめ • Namespace -> Saving memory  BlockInfoStriped, BlockIdManager • Writing Side -> Saving diskspace usage  INodeFile • Reading Side -> Saving reading time  DFSStripedInputStream

61. ありがとうございました

図でわかるHDFS Erasure Coding

図でわかるHDFS Erasure Coding

Recommended

More Related Content

Viewers also liked

Viewers also liked(20)

Similar to 図でわかるHDFS Erasure Coding

Similar to 図でわかるHDFS Erasure Coding(11)

More from Kai Sasaki

More from Kai Sasaki(20)

Recently uploaded

Recently uploaded(20)

図でわかるHDFS Erasure Coding