Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

10

Share

Download to read offline

図でわかるHDFS Erasure Coding

Download to read offline

Illustration of HDFS Erasure Coding

図でわかるHDFS Erasure Coding

  1. 1. 図でわかる
 HDFS Erasure Coding Kai Sasaki Treasure Data Inc.
  2. 2. Who am I 佐々木 海(Kai Sasaki) Software Engineer at Treasure Data Inc.
 http://www.treasuredata.com Hadoop, Spark, DL4J
  3. 3. Agenda • Erasure Coding • Under the Namespace • Writing Side • Reading Side
  4. 4. Erasure Coding
  5. 5. Replication Block HDFS
  6. 6. Replication Block Block Block Block HDFS
  7. 7. Replication Block Block Block Block HDFS
  8. 8. Replication Block Block Block Block HDFS Capacity Overhead 
 x3
  9. 9. Replication Block Block Block Block HDFS Redundancy 2 Capacity Overhead 
 x3
  10. 10. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block RS-6-3
  11. 11. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  12. 12. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  13. 13. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  14. 14. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  15. 15. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block BlockRedundancy 3
  16. 16. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block Capacity Overhead 
 x1.5 Redundancy 3
  17. 17. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block BlockGroup
  18. 18. Under the Namespace
  19. 19. INode and BlockInfo BlockInfo INode
  20. 20. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo…
  21. 21. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block
  22. 22. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block BlockGroup
  23. 23. BlockInfo … Block Block Block BlockGroup
  24. 24. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup
  25. 25. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit
  26. 26. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit index 0 index 2 index 1 Saving memory
  27. 27. Writing Side
  28. 28. Data 0
  29. 29. Data64KB 0
  30. 30. Data BlockGroup 64KB 0
  31. 31. Data BlockGroup 0
  32. 32. Data BlockGroup 0
  33. 33. Data BlockGroup 0
  34. 34. Data BlockGroup 0
  35. 35. Data BlockGroup 0
  36. 36. Data BlockGroup 0
  37. 37. Data BlockGroup 0
  38. 38. Data BlockGroup 0 Data Block
  39. 39. Data BlockGroup 0 Data Block Parity Block
  40. 40. Data BlockGroup 0 Data Block Parity Block Stripe
  41. 41. Data BlockGroup 0
  42. 42. Data BlockGroup 0
  43. 43. Data BlockGroup 0
  44. 44. Data BlockGroup 0
  45. 45. Data BlockGroup 0
  46. 46. Data BlockGroup 0
  47. 47. Data BlockGroup 0 0
  48. 48. Data BlockGroup 0 0
  49. 49. Data BlockGroup 0 0 Saving diskspace usage
  50. 50. Reading Side
  51. 51. BlockGroup 0 0
  52. 52. BlockGroup 0 0 200kb 500kb
  53. 53. BlockGroup 0 0 200kb 500kb
  54. 54. BlockGroup 0 0 200kb 500kb
  55. 55. BlockGroup 0 0 200kb 500kb
  56. 56. BlockGroup 0 0 200kb 500kb
  57. 57. BlockGroup 0 0 200kb 500kb
  58. 58. BlockGroup 0 0 200kb 500kb
  59. 59. BlockGroup 0 0 200kb 500kb Saving reading time
  60. 60. まとめ • Namespace -> Saving memory
 BlockInfoStriped, BlockIdManager • Writing Side -> Saving diskspace usage
 INodeFile • Reading Side -> Saving reading time
 DFSStripedInputStream
  61. 61. ありがとうございました
  • blue_sky1

    Jun. 1, 2020
  • lalaguozhe

    Nov. 29, 2018
  • RyousukeNakazato

    Nov. 17, 2017
  • terumba

    Jan. 4, 2016
  • usuyama

    Nov. 12, 2015
  • mitsunorikomatsu

    Nov. 11, 2015
  • shuichiroaiba

    Nov. 11, 2015
  • yasuhidesakai

    Nov. 10, 2015
  • marna3

    Nov. 10, 2015
  • RuiGAO6

    Nov. 10, 2015

Illustration of HDFS Erasure Coding

Views

Total views

4,675

On Slideshare

0

From embeds

0

Number of embeds

1,929

Actions

Downloads

68

Shares

0

Comments

0

Likes

10

×