SlideShare a Scribd company logo
Progressive Data Retrieval for Distributed Networked
                                     Storage
Abstract—

We propose decentralized progressive data retrieval (PDR) mechanism for data
reconstruction in a network of Byzantine and crash-stop nodes. The scheme
progressively retrieves stored data, such that it achieves the minimum
communication cost possible. In particular, PDR gracefully adapts the cost of
successful data retrieval to the number of Byzantine and crash-stop storage nodes.
At the core of PDR is an incremental Reed-Solomon decoding (IRD) procedure
that is highly computation efficient for data reconstruction. IRD’s computation
efficiency arises from its ability to utilize intermediate computation results. In
addition, we provide an in-depth analysis of PDR and compare it to decentralized
erasure coding and decentralized fountain coding algorithms for distributed storage
systems. Moreover, our implementation results show that PDR has up to 35 times
lower computation time over the state-of the- art error-erasure decoding scheme for
distributed storage systems. In our analysis, we also show that the code structure of
PDR and the number of available storage nodes are independent of each other, and
they can be used to control both the data dissemination and retrieval complexity.

Reasons for the proposal:

A crash-stop or erasure node in a networked storage system is one that becomes
unavailable because it can neither transmit nor receive data. Byzantine nodes are
those nodes that fail in an arbitrary manner and cannot be trusted. They are more
pertinent than erasure nodes given the prevalence of cheap storage devices,
software bugs, and malicious attacks. Crash-stop nodes can be identified, via keep-
alive messages for instance. However, the identity of Byzantine nodes can be
known, to a data collector requiring data reconstruction, only via an error-erasure
decoding algorithm. Hence a Efficient method to overcome these issues has to be
designed .

Existing system & demerits :

Reed-Solomon (RS) codes have been used for storage in stand-alone storage
systems . However, our work concerns RS codes for networked storage. Even
though RS codes can be applied to achieve minimum communication overhead,
their centralized nature and high coordination overhead make them unsuitable .

Proposed system:

We propose a storage-optimal and decentralized coding scheme that disperses
contents of a data file, and retrieves it efficiently in networked storage systems.
Our RS coding scheme provides better performance than the decentralized erasure
and fountain coding schemes. The key novelty of our solution lies in a progressive
data retrieval procedure, which retrieves just enough data from live storage nodes
and performs decoding incrementally. As a result, both communication and
computation costs are minimized, and adapt to the degree of errors, due to
Byzantine nodes, in the system. In this paper, coded symbols and storage symbols
are used interchangeably.

An     erasure code can be defined by the following two primitives:
For the decode primitive, if any k of the n coded symbols suffice to reconstruct the
original data, the code has optimal reception efficiency and is referred to as
(n,k)Maximum Distance Separable (MDS) erasure code.

More Related Content

Similar to Progressive data retrieval for distributed

JPD1406 Enabling Data Integrity Protection in Regenerating-Coding-Based Clou...
JPD1406  Enabling Data Integrity Protection in Regenerating-Coding-Based Clou...JPD1406  Enabling Data Integrity Protection in Regenerating-Coding-Based Clou...
JPD1406 Enabling Data Integrity Protection in Regenerating-Coding-Based Clou...
chennaijp
 
enabling data integrity protection in regenerating-coding-based cloud storage...
enabling data integrity protection in regenerating-coding-based cloud storage...enabling data integrity protection in regenerating-coding-based cloud storage...
enabling data integrity protection in regenerating-coding-based cloud storage...
swathi78
 
Enabling data integrity protection in regenerating coding-based cloud storage...
Enabling data integrity protection in regenerating coding-based cloud storage...Enabling data integrity protection in regenerating coding-based cloud storage...
Enabling data integrity protection in regenerating coding-based cloud storage...
Papitha Velumani
 
Erasure codes fast 2012
Erasure codes fast 2012Erasure codes fast 2012
Erasure codes fast 2012
Accenture
 
Postponed Optimized Report Recovery under Lt Based Cloud Memory
Postponed Optimized Report Recovery under Lt Based Cloud MemoryPostponed Optimized Report Recovery under Lt Based Cloud Memory
Postponed Optimized Report Recovery under Lt Based Cloud Memory
IJARIIT
 
Privacy preserving public auditing for regenerating-code-based cloud storage
Privacy preserving public auditing for regenerating-code-based cloud storagePrivacy preserving public auditing for regenerating-code-based cloud storage
Privacy preserving public auditing for regenerating-code-based cloud storage
Nagamalleswararao Tadikonda
 
Parallel and distributed system projects for java and dot net
Parallel and distributed system projects for java and dot netParallel and distributed system projects for java and dot net
Parallel and distributed system projects for java and dot net
redpel dot com
 
Big Data: RDBMS vs. Hadoop vs. Spark
Big Data: RDBMS vs. Hadoop vs. SparkBig Data: RDBMS vs. Hadoop vs. Spark
Big Data: RDBMS vs. Hadoop vs. Spark
Graisy Biswal
 
wcn.pptx
wcn.pptxwcn.pptx
wcn.pptx
RahulRam100
 
A distributed fault intrusion tolerant sensor
A distributed fault intrusion tolerant sensorA distributed fault intrusion tolerant sensor
A distributed fault intrusion tolerant sensor
IMPULSE_TECHNOLOGY
 
Titles with Abstracts_2023-2024_Cloud Computing.pdf
Titles with Abstracts_2023-2024_Cloud Computing.pdfTitles with Abstracts_2023-2024_Cloud Computing.pdf
Titles with Abstracts_2023-2024_Cloud Computing.pdf
info751436
 
Iaetsd mapreduce streaming over cassandra datasets
Iaetsd mapreduce streaming over cassandra datasetsIaetsd mapreduce streaming over cassandra datasets
Iaetsd mapreduce streaming over cassandra datasets
Iaetsd Iaetsd
 
Review and Performance Comparison of Distributed Wireless Reprogramming Proto...
Review and Performance Comparison of Distributed Wireless Reprogramming Proto...Review and Performance Comparison of Distributed Wireless Reprogramming Proto...
Review and Performance Comparison of Distributed Wireless Reprogramming Proto...
IOSR Journals
 
P2P Cache Resolution System for MANET
P2P Cache Resolution System for MANETP2P Cache Resolution System for MANET
P2P Cache Resolution System for MANET
IJCSIS Research Publications
 
Storage Area Networks Unit 1 Notes
Storage Area Networks Unit 1 NotesStorage Area Networks Unit 1 Notes
Storage Area Networks Unit 1 Notes
Sudarshan Dhondaley
 
Dynamic control of coding for progressive packet arrivals in dtns
Dynamic control of coding for progressive packet arrivals in dtnsDynamic control of coding for progressive packet arrivals in dtns
Dynamic control of coding for progressive packet arrivals in dtns
JPINFOTECH JAYAPRAKASH
 
Compare Array vs Host vs Hypervisor vs Network-Based Replication
Compare Array vs Host vs Hypervisor vs Network-Based ReplicationCompare Array vs Host vs Hypervisor vs Network-Based Replication
Compare Array vs Host vs Hypervisor vs Network-Based Replication
MaryJWilliams2
 
Secure computing for java and dot net
Secure computing for java and dot netSecure computing for java and dot net
Secure computing for java and dot net
redpel dot com
 
A novel authenticated cipher for rfid systems
A novel authenticated cipher for rfid systemsA novel authenticated cipher for rfid systems
A novel authenticated cipher for rfid systems
ijcisjournal
 
A big-data architecture for real-time analytics
A big-data architecture for real-time analyticsA big-data architecture for real-time analytics
A big-data architecture for real-time analytics
ramikaurraminder
 

Similar to Progressive data retrieval for distributed (20)

JPD1406 Enabling Data Integrity Protection in Regenerating-Coding-Based Clou...
JPD1406  Enabling Data Integrity Protection in Regenerating-Coding-Based Clou...JPD1406  Enabling Data Integrity Protection in Regenerating-Coding-Based Clou...
JPD1406 Enabling Data Integrity Protection in Regenerating-Coding-Based Clou...
 
enabling data integrity protection in regenerating-coding-based cloud storage...
enabling data integrity protection in regenerating-coding-based cloud storage...enabling data integrity protection in regenerating-coding-based cloud storage...
enabling data integrity protection in regenerating-coding-based cloud storage...
 
Enabling data integrity protection in regenerating coding-based cloud storage...
Enabling data integrity protection in regenerating coding-based cloud storage...Enabling data integrity protection in regenerating coding-based cloud storage...
Enabling data integrity protection in regenerating coding-based cloud storage...
 
Erasure codes fast 2012
Erasure codes fast 2012Erasure codes fast 2012
Erasure codes fast 2012
 
Postponed Optimized Report Recovery under Lt Based Cloud Memory
Postponed Optimized Report Recovery under Lt Based Cloud MemoryPostponed Optimized Report Recovery under Lt Based Cloud Memory
Postponed Optimized Report Recovery under Lt Based Cloud Memory
 
Privacy preserving public auditing for regenerating-code-based cloud storage
Privacy preserving public auditing for regenerating-code-based cloud storagePrivacy preserving public auditing for regenerating-code-based cloud storage
Privacy preserving public auditing for regenerating-code-based cloud storage
 
Parallel and distributed system projects for java and dot net
Parallel and distributed system projects for java and dot netParallel and distributed system projects for java and dot net
Parallel and distributed system projects for java and dot net
 
Big Data: RDBMS vs. Hadoop vs. Spark
Big Data: RDBMS vs. Hadoop vs. SparkBig Data: RDBMS vs. Hadoop vs. Spark
Big Data: RDBMS vs. Hadoop vs. Spark
 
wcn.pptx
wcn.pptxwcn.pptx
wcn.pptx
 
A distributed fault intrusion tolerant sensor
A distributed fault intrusion tolerant sensorA distributed fault intrusion tolerant sensor
A distributed fault intrusion tolerant sensor
 
Titles with Abstracts_2023-2024_Cloud Computing.pdf
Titles with Abstracts_2023-2024_Cloud Computing.pdfTitles with Abstracts_2023-2024_Cloud Computing.pdf
Titles with Abstracts_2023-2024_Cloud Computing.pdf
 
Iaetsd mapreduce streaming over cassandra datasets
Iaetsd mapreduce streaming over cassandra datasetsIaetsd mapreduce streaming over cassandra datasets
Iaetsd mapreduce streaming over cassandra datasets
 
Review and Performance Comparison of Distributed Wireless Reprogramming Proto...
Review and Performance Comparison of Distributed Wireless Reprogramming Proto...Review and Performance Comparison of Distributed Wireless Reprogramming Proto...
Review and Performance Comparison of Distributed Wireless Reprogramming Proto...
 
P2P Cache Resolution System for MANET
P2P Cache Resolution System for MANETP2P Cache Resolution System for MANET
P2P Cache Resolution System for MANET
 
Storage Area Networks Unit 1 Notes
Storage Area Networks Unit 1 NotesStorage Area Networks Unit 1 Notes
Storage Area Networks Unit 1 Notes
 
Dynamic control of coding for progressive packet arrivals in dtns
Dynamic control of coding for progressive packet arrivals in dtnsDynamic control of coding for progressive packet arrivals in dtns
Dynamic control of coding for progressive packet arrivals in dtns
 
Compare Array vs Host vs Hypervisor vs Network-Based Replication
Compare Array vs Host vs Hypervisor vs Network-Based ReplicationCompare Array vs Host vs Hypervisor vs Network-Based Replication
Compare Array vs Host vs Hypervisor vs Network-Based Replication
 
Secure computing for java and dot net
Secure computing for java and dot netSecure computing for java and dot net
Secure computing for java and dot net
 
A novel authenticated cipher for rfid systems
A novel authenticated cipher for rfid systemsA novel authenticated cipher for rfid systems
A novel authenticated cipher for rfid systems
 
A big-data architecture for real-time analytics
A big-data architecture for real-time analyticsA big-data architecture for real-time analytics
A big-data architecture for real-time analytics
 

More from IMPULSE_TECHNOLOGY

17
1717
16
1616
15
1515
25
2525
24
2424
23
2323
22
2222
21
2121
20
2020
19
1919
18
1818
16
1616
15
1515
14
1414
13
1313
12
1212
11
1111
10
1010
9
99
8
88

More from IMPULSE_TECHNOLOGY (20)

17
1717
17
 
16
1616
16
 
15
1515
15
 
25
2525
25
 
24
2424
24
 
23
2323
23
 
22
2222
22
 
21
2121
21
 
20
2020
20
 
19
1919
19
 
18
1818
18
 
16
1616
16
 
15
1515
15
 
14
1414
14
 
13
1313
13
 
12
1212
12
 
11
1111
11
 
10
1010
10
 
9
99
9
 
8
88
8
 

Recently uploaded

Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Dr. Vinod Kumar Kanvaria
 
Top five deadliest dog breeds in America
Top five deadliest dog breeds in AmericaTop five deadliest dog breeds in America
Top five deadliest dog breeds in America
Bisnar Chase Personal Injury Attorneys
 
The Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collectionThe Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collection
Israel Genealogy Research Association
 
A Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in EducationA Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in Education
Peter Windle
 
The History of Stoke Newington Street Names
The History of Stoke Newington Street NamesThe History of Stoke Newington Street Names
The History of Stoke Newington Street Names
History of Stoke Newington
 
Life upper-Intermediate B2 Workbook for student
Life upper-Intermediate B2 Workbook for studentLife upper-Intermediate B2 Workbook for student
Life upper-Intermediate B2 Workbook for student
NgcHiNguyn25
 
Hindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdfHindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdf
Dr. Mulla Adam Ali
 
PIMS Job Advertisement 2024.pdf Islamabad
PIMS Job Advertisement 2024.pdf IslamabadPIMS Job Advertisement 2024.pdf Islamabad
PIMS Job Advertisement 2024.pdf Islamabad
AyyanKhan40
 
Azure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHatAzure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHat
Scholarhat
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
Nguyen Thanh Tu Collection
 
A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
Jean Carlos Nunes Paixão
 
CACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdfCACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdf
camakaiclarkmusic
 
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptxC1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
mulvey2
 
South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)
Academy of Science of South Africa
 
Introduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp NetworkIntroduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp Network
TechSoup
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
Nicholas Montgomery
 
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
IreneSebastianRueco1
 
Digital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments UnitDigital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments Unit
chanes7
 
Smart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICTSmart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICT
simonomuemu
 
World environment day ppt For 5 June 2024
World environment day ppt For 5 June 2024World environment day ppt For 5 June 2024
World environment day ppt For 5 June 2024
ak6969907
 

Recently uploaded (20)

Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
 
Top five deadliest dog breeds in America
Top five deadliest dog breeds in AmericaTop five deadliest dog breeds in America
Top five deadliest dog breeds in America
 
The Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collectionThe Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collection
 
A Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in EducationA Strategic Approach: GenAI in Education
A Strategic Approach: GenAI in Education
 
The History of Stoke Newington Street Names
The History of Stoke Newington Street NamesThe History of Stoke Newington Street Names
The History of Stoke Newington Street Names
 
Life upper-Intermediate B2 Workbook for student
Life upper-Intermediate B2 Workbook for studentLife upper-Intermediate B2 Workbook for student
Life upper-Intermediate B2 Workbook for student
 
Hindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdfHindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdf
 
PIMS Job Advertisement 2024.pdf Islamabad
PIMS Job Advertisement 2024.pdf IslamabadPIMS Job Advertisement 2024.pdf Islamabad
PIMS Job Advertisement 2024.pdf Islamabad
 
Azure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHatAzure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHat
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
 
A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
 
CACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdfCACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdf
 
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptxC1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
 
South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)South African Journal of Science: Writing with integrity workshop (2024)
South African Journal of Science: Writing with integrity workshop (2024)
 
Introduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp NetworkIntroduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp Network
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
 
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
RPMS TEMPLATE FOR SCHOOL YEAR 2023-2024 FOR TEACHER 1 TO TEACHER 3
 
Digital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments UnitDigital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments Unit
 
Smart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICTSmart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICT
 
World environment day ppt For 5 June 2024
World environment day ppt For 5 June 2024World environment day ppt For 5 June 2024
World environment day ppt For 5 June 2024
 

Progressive data retrieval for distributed

  • 1. Progressive Data Retrieval for Distributed Networked Storage Abstract— We propose decentralized progressive data retrieval (PDR) mechanism for data reconstruction in a network of Byzantine and crash-stop nodes. The scheme progressively retrieves stored data, such that it achieves the minimum communication cost possible. In particular, PDR gracefully adapts the cost of successful data retrieval to the number of Byzantine and crash-stop storage nodes. At the core of PDR is an incremental Reed-Solomon decoding (IRD) procedure that is highly computation efficient for data reconstruction. IRD’s computation efficiency arises from its ability to utilize intermediate computation results. In addition, we provide an in-depth analysis of PDR and compare it to decentralized erasure coding and decentralized fountain coding algorithms for distributed storage systems. Moreover, our implementation results show that PDR has up to 35 times lower computation time over the state-of the- art error-erasure decoding scheme for distributed storage systems. In our analysis, we also show that the code structure of PDR and the number of available storage nodes are independent of each other, and they can be used to control both the data dissemination and retrieval complexity. Reasons for the proposal: A crash-stop or erasure node in a networked storage system is one that becomes unavailable because it can neither transmit nor receive data. Byzantine nodes are those nodes that fail in an arbitrary manner and cannot be trusted. They are more pertinent than erasure nodes given the prevalence of cheap storage devices, software bugs, and malicious attacks. Crash-stop nodes can be identified, via keep- alive messages for instance. However, the identity of Byzantine nodes can be
  • 2. known, to a data collector requiring data reconstruction, only via an error-erasure decoding algorithm. Hence a Efficient method to overcome these issues has to be designed . Existing system & demerits : Reed-Solomon (RS) codes have been used for storage in stand-alone storage systems . However, our work concerns RS codes for networked storage. Even though RS codes can be applied to achieve minimum communication overhead, their centralized nature and high coordination overhead make them unsuitable . Proposed system: We propose a storage-optimal and decentralized coding scheme that disperses contents of a data file, and retrieves it efficiently in networked storage systems. Our RS coding scheme provides better performance than the decentralized erasure and fountain coding schemes. The key novelty of our solution lies in a progressive data retrieval procedure, which retrieves just enough data from live storage nodes and performs decoding incrementally. As a result, both communication and computation costs are minimized, and adapt to the degree of errors, due to Byzantine nodes, in the system. In this paper, coded symbols and storage symbols are used interchangeably. An erasure code can be defined by the following two primitives:
  • 3. For the decode primitive, if any k of the n coded symbols suffice to reconstruct the original data, the code has optimal reception efficiency and is referred to as (n,k)Maximum Distance Separable (MDS) erasure code.