Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Erasure Coding Costs and Benefits

576 views

Published on

Costs and benefits of using erasure coding to increase reliability while reducing redundancy.

Published in: Technology
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Erasure Coding Costs and Benefits

  1. 1. Erasure Coding Costs and Bene
  2. 2. ts John D. Cook1 Robert Primmer2 Ab de Kwant2 1Singular Value Consulting 2Hitachi Data Systems March 28, 2014 John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  3. 3. ts
  4. 4. Two ways to store a 12 GB video John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  5. 5. ts
  6. 6. Two ways to store a 12 GB video Replication John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  7. 7. ts
  8. 8. Two ways to store a 12 GB video Replication Store an extra copy. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  9. 9. ts
  10. 10. Two ways to store a 12 GB video Replication Store an extra copy. Disk usage: 24 GB, 100% overhead. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  11. 11. ts
  12. 12. Two ways to store a 12 GB video Replication Store an extra copy. Disk usage: 24 GB, 100% overhead. Simplest example of erasure coding John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  13. 13. ts
  14. 14. Two ways to store a 12 GB video Replication Store an extra copy. Disk usage: 24 GB, 100% overhead. Simplest example of erasure coding Split into two halves, A and B. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  15. 15. ts
  16. 16. Two ways to store a 12 GB video Replication Store an extra copy. Disk usage: 24 GB, 100% overhead. Simplest example of erasure coding Split into two halves, A and B. Store A, B, and A B. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  17. 17. ts
  18. 18. Two ways to store a 12 GB video Replication Store an extra copy. Disk usage: 24 GB, 100% overhead. Simplest example of erasure coding Split into two halves, A and B. Store A, B, and A B. Disk usage: 18 GB, 50% overhead. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  19. 19. ts
  20. 20. Two ways to store a 12 GB video Replication Store an extra copy. Disk usage: 24 GB, 100% overhead. Simplest example of erasure coding Split into two halves, A and B. Store A, B, and A B. Disk usage: 18 GB, 50% overhead. Can recover B, for example, with (A B) A. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  21. 21. ts
  22. 22. Two ways to store a 12 GB video Replication Store an extra copy. Disk usage: 24 GB, 100% overhead. Simplest example of erasure coding Split into two halves, A and B. Store A, B, and A B. Disk usage: 18 GB, 50% overhead. Can recover B, for example, with (A B) A. NB: = . John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  23. 23. ts
  24. 24. Another way to store a 12 GB video John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  25. 25. ts
  26. 26. Another way to store a 12 GB video Split into three equal parts: A, B, C. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  27. 27. ts
  28. 28. Another way to store a 12 GB video Split into three equal parts: A, B, C. Store A, B, C. and A B C. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  29. 29. ts
  30. 30. Another way to store a 12 GB video Split into three equal parts: A, B, C. Store A, B, C. and A B C. Disk usage: 16 GB, 33% overhead. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  31. 31. ts
  32. 32. Another way to store a 12 GB video Split into three equal parts: A, B, C. Store A, B, C. and A B C. Disk usage: 16 GB, 33% overhead. Can recover B, for example, with (A B C) A C. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  33. 33. ts
  34. 34. Another way to store a 12 GB video Split into three equal parts: A, B, C. Store A, B, C. and A B C. Disk usage: 16 GB, 33% overhead. Can recover B, for example, with (A B C) A C. Call this 3 + 1 encoding. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  35. 35. ts
  36. 36. Another look at 3 + 1 encoding John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  37. 37. ts
  38. 38. Another look at 3 + 1 encoding 2 664 1 0 0 0 1 0 0 0 1 1 1 1 3 775 2 4 x1 x2 x3 3 5 = 2 664 y1 y2 y3 y4 3 775 John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  39. 39. ts
  40. 40. Another look at 3 + 1 encoding 2 664 1 0 0 0 1 0 0 0 1 1 1 1 3 775 2 4 x1 x2 x3 3 5 = 2 664 y1 y2 y3 y4 3 775 To recover, erase missing element of y and corresponding row of matrix. Solve linear system. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  41. 41. ts
  42. 42. Another look at 3 + 1 encoding 2 664 1 0 0 0 1 0 0 0 1 1 1 1 3 775 2 4 x1 x2 x3 3 5 = 2 664 y1 y2 y3 y4 3 775 To recover, erase missing element of y and corresponding row of matrix. Solve linear system. Addition is (XOR). Multiplication by 1 is identity. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  43. 43. ts
  44. 44. 2 + 2 encoding John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  45. 45. ts
  46. 46. 2 + 2 encoding Split 12 GB
  47. 47. le into equal halves A and B. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  48. 48. ts
  49. 49. 2 + 2 encoding Split 12 GB
  50. 50. le into equal halves A and B. Store A, B, (3 A) (2 B), and (2 A) (3 B). John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  51. 51. ts
  52. 52. 2 + 2 encoding Split 12 GB
  53. 53. le into equal halves A and B. Store A, B, (3 A) (2 B), and (2 A) (3 B). Can recover from the loss on any two disks. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  54. 54. ts
  55. 55. 2 + 2 encoding Split 12 GB
  56. 56. le into equal halves A and B. Store A, B, (3 A) (2 B), and (2 A) (3 B). Can recover from the loss on any two disks. Total disk: 24 GB. Overhead: 100%. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  57. 57. ts
  58. 58. 2 + 2 encoding Split 12 GB
  59. 59. le into equal halves A and B. Store A, B, (3 A) (2 B), and (2 A) (3 B). Can recover from the loss on any two disks. Total disk: 24 GB. Overhead: 100%. Same disk use as replication, but probability of loss O(p3) rather than O(p2). John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  60. 60. ts
  61. 61. 2 + 2 encoding again John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  62. 62. ts
  63. 63. 2 + 2 encoding again 2 664 1 0 0 1 3 2 2 3 3 775 x1 x2 = 2 664 y1 y2 y3 y4 3 775 John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  64. 64. ts
  65. 65. 2 + 2 encoding again 2 664 1 0 0 1 3 2 2 3 3 775 x1 x2 = 2 664 y1 y2 y3 y4 3 775 Operate on pairs of bits. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  66. 66. ts
  67. 67. 2 + 2 encoding again 2 664 1 0 0 1 3 2 2 3 3 775 x1 x2 = 2 664 y1 y2 y3 y4 3 775 Operate on pairs of bits. Arithmetic carried out in GF(22). John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  68. 68. ts
  69. 69. 2 + 2 encoding again 2 664 1 0 0 1 3 2 2 3 3 775 x1 x2 = 2 664 y1 y2 y3 y4 3 775 Operate on pairs of bits. Arithmetic carried out in GF(22). Addition is XOR. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  70. 70. ts
  71. 71. 2 + 2 encoding again 2 664 1 0 0 1 3 2 2 3 3 775 x1 x2 = 2 664 y1 y2 y3 y4 3 775 Operate on pairs of bits. Arithmetic carried out in GF(22). Addition is XOR. Multiplication is more complicated. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  72. 72. ts
  73. 73. 2 + 2 encoding again 2 664 1 0 0 1 3 2 2 3 3 775 x1 x2 = 2 664 y1 y2 y3 y4 3 775 Operate on pairs of bits. Arithmetic carried out in GF(22). Addition is XOR. Multiplication is more complicated. Recover by striking missing elements of y and corresponding rows of matrix. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  74. 74. ts
  75. 75. 2 + 2 encoding again 2 664 1 0 0 1 3 2 2 3 3 775 x1 x2 = 2 664 y1 y2 y3 y4 3 775 Operate on pairs of bits. Arithmetic carried out in GF(22). Addition is XOR. Multiplication is more complicated. Recover by striking missing elements of y and corresponding rows of matrix. You multiply matrix times vector in the usual way, but with GF(4) arithmetic. More on that soon. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  76. 76. ts
  77. 77. m + n erasure coding John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  78. 78. ts
  79. 79. m + n erasure coding Divide data into m fragments and add n parity fragments. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  80. 80. ts
  81. 81. m + n erasure coding Divide data into m fragments and add n parity fragments. Construct an m + n encoding scheme using Reed-Solomon codes. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  82. 82. ts
  83. 83. m + n erasure coding Divide data into m fragments and add n parity fragments. Construct an m + n encoding scheme using Reed-Solomon codes. Can recover from the loss of any n fragments. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  84. 84. ts
  85. 85. m + n erasure coding Divide data into m fragments and add n parity fragments. Construct an m + n encoding scheme using Reed-Solomon codes. Can recover from the loss of any n fragments. There is an m + n code for all positive m and n. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  86. 86. ts
  87. 87. m + n erasure coding Divide data into m fragments and add n parity fragments. Construct an m + n encoding scheme using Reed-Solomon codes. Can recover from the loss of any n fragments. There is an m + n code for all positive m and n. Construct an (m + n) m matrix. Encode and decode as before. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  88. 88. ts
  89. 89. m + n erasure coding Divide data into m fragments and add n parity fragments. Construct an m + n encoding scheme using Reed-Solomon codes. Can recover from the loss of any n fragments. There is an m + n code for all positive m and n. Construct an (m + n) m matrix. Encode and decode as before. Arithmetic carried out in GF(q) where q = 2r m + n. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  90. 90. ts
  91. 91. m + n erasure coding Divide data into m fragments and add n parity fragments. Construct an m + n encoding scheme using Reed-Solomon codes. Can recover from the loss of any n fragments. There is an m + n code for all positive m and n. Construct an (m + n) m matrix. Encode and decode as before. Arithmetic carried out in GF(q) where q = 2r m + n. Encode data in blocks of r bits. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  92. 92. ts
  93. 93. m + n erasure coding Divide data into m fragments and add n parity fragments. Construct an m + n encoding scheme using Reed-Solomon codes. Can recover from the loss of any n fragments. There is an m + n code for all positive m and n. Construct an (m + n) m matrix. Encode and decode as before. Arithmetic carried out in GF(q) where q = 2r m + n. Encode data in blocks of r bits. RAID 5 = m + 1; RAID 6 = m + 2. Next: What is GF(q) and where does matrix come from? John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  94. 94. ts
  95. 95. Finite
  96. 96. elds John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  97. 97. ts
  98. 98. Finite
  99. 99. elds GF(q) has q elements. John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  100. 100. ts
  101. 101. Finite
  102. 102. elds GF(q) has q elements. There exists a
  103. 103. eld of order q if and only if q = pr . John D. Cook, Robert Primmer, Ab de Kwant Erasure Coding Costs and Bene
  104. 104. ts
  105. 105. Finite
  106. 106. elds GF(q) has q elements. There exists a
  107. 107. eld of order q if and only if q = pr . Elements are polynomials in r

×