Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

図でわかるHDFS Erasure Coding

4,226 views

Published on

Illustration of HDFS Erasure Coding

Published in: Data & Analytics
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

図でわかるHDFS Erasure Coding

  1. 1. 図でわかる
 HDFS Erasure Coding Kai Sasaki Treasure Data Inc.
  2. 2. Who am I 佐々木 海(Kai Sasaki) Software Engineer at Treasure Data Inc.
 http://www.treasuredata.com Hadoop, Spark, DL4J
  3. 3. Agenda • Erasure Coding • Under the Namespace • Writing Side • Reading Side
  4. 4. Erasure Coding
  5. 5. Replication Block HDFS
  6. 6. Replication Block Block Block Block HDFS
  7. 7. Replication Block Block Block Block HDFS
  8. 8. Replication Block Block Block Block HDFS Capacity Overhead 
 x3
  9. 9. Replication Block Block Block Block HDFS Redundancy 2 Capacity Overhead 
 x3
  10. 10. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block RS-6-3
  11. 11. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  12. 12. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  13. 13. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  14. 14. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  15. 15. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block BlockRedundancy 3
  16. 16. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block Capacity Overhead 
 x1.5 Redundancy 3
  17. 17. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block BlockGroup
  18. 18. Under the Namespace
  19. 19. INode and BlockInfo BlockInfo INode
  20. 20. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo…
  21. 21. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block
  22. 22. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block BlockGroup
  23. 23. BlockInfo … Block Block Block BlockGroup
  24. 24. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup
  25. 25. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit
  26. 26. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit index 0 index 2 index 1 Saving memory
  27. 27. Writing Side
  28. 28. Data 0
  29. 29. Data64KB 0
  30. 30. Data BlockGroup 64KB 0
  31. 31. Data BlockGroup 0
  32. 32. Data BlockGroup 0
  33. 33. Data BlockGroup 0
  34. 34. Data BlockGroup 0
  35. 35. Data BlockGroup 0
  36. 36. Data BlockGroup 0
  37. 37. Data BlockGroup 0
  38. 38. Data BlockGroup 0 Data Block
  39. 39. Data BlockGroup 0 Data Block Parity Block
  40. 40. Data BlockGroup 0 Data Block Parity Block Stripe
  41. 41. Data BlockGroup 0
  42. 42. Data BlockGroup 0
  43. 43. Data BlockGroup 0
  44. 44. Data BlockGroup 0
  45. 45. Data BlockGroup 0
  46. 46. Data BlockGroup 0
  47. 47. Data BlockGroup 0 0
  48. 48. Data BlockGroup 0 0
  49. 49. Data BlockGroup 0 0 Saving diskspace usage
  50. 50. Reading Side
  51. 51. BlockGroup 0 0
  52. 52. BlockGroup 0 0 200kb 500kb
  53. 53. BlockGroup 0 0 200kb 500kb
  54. 54. BlockGroup 0 0 200kb 500kb
  55. 55. BlockGroup 0 0 200kb 500kb
  56. 56. BlockGroup 0 0 200kb 500kb
  57. 57. BlockGroup 0 0 200kb 500kb
  58. 58. BlockGroup 0 0 200kb 500kb
  59. 59. BlockGroup 0 0 200kb 500kb Saving reading time
  60. 60. まとめ • Namespace -> Saving memory
 BlockInfoStriped, BlockIdManager • Writing Side -> Saving diskspace usage
 INodeFile • Reading Side -> Saving reading time
 DFSStripedInputStream
  61. 61. ありがとうございました

×