Bit Level Preservation


Published on

Assessing and Mitigating Bit-Level Threats, Digital Preservation 2012, Washington, DC.

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Text is licensed under the Creative Commons Attribution-Share Alike 3.0 United States License. To view a copy of this license, visit or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Note, images on pages 3, 34 , 37 are under separate licenses: Curatorial error image © AtomSmasher, Distributed under CC-BY-SA Physical infrastructure failure image, produced by NSF, included under CC-BY-SA Media Failure, knob, and graph images obtained from Wikimedia Commons, distributed under CC-BY-SA Insider attack image obtained from under CC-BY-NC
  • Unknown: organizational disruptions are likely, but how risky are they?
  • Areas for research: What is the impact and likelihood of each risk? What is the cost of each control?
  • This is licensed under the Creative Commons Attribution-Share Alike 3.0 United States License. To view a copy of this license, visit or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
  • Poll the Audience: To get a sense of where everyone’s practices could we get a show of hands for the following questions: How many of your organizations maintain two or more additional copies of your content? … One additional copy? How many of your organizations are tracking fixity information for content you intend to preserve? - Now, how many of you have audit procedures in place for that content? Lastly, how many of you have procedures to repair digital content when an audit discovers fixity discontinuities? - For those of you with organizations not engaging in these practices why is that the case? Panel then Audience questions: What have been the biggest challenges related to the topics you all presented for each of your organizations? [Open the question to the audience] - What kinds of rules and models should we be putting in place for fiddling with the Knobs in Micah’s slides? [Open the question to the audience] What do we need to be able to make it easier to balance these tradeoffs? [Open the question to the audience]
  • Bit Level Preservation

    1. 1. Prepared for Digital Preservation 2012 Washington, DC July 2012Assessing and Mitigating Bit- Level Preservation Risks NDSA Infrastructure Working Group
    2. 2. INTRODUCTION:A Framework for AddressingBit-Level Preservation Risk Micah Altman <> Director of Research, MIT Libraries Mark Evans <> Digital Archiving Practice Manager, Tessela Inc.
    3. 3. Threats to BitsPhysical & Hardware Software Insider & External Attacks Organizational Failure Media Curatorial Error
    4. 4. Do you know where your data are?
    5. 5. Encoding Priscilla Caplan <>Assistant Director for Digital Library Services, Florida Virtual Campus
    6. 6. Compression• Many types of compression: – Format based file compression, e.g. JPEG2000 – Tape hardware compression at the drive – NAS compression via appliance or storage device – Data deduplication• Is it lossless?• Is it transparent?• Is it proprietary?• What is effect on error recovery?
    7. 7. Compression Tradeoffs• Tradeoffs – Space savings allows more copies at same cost – But makes files more sensitive to data corruption• Erasure coding in cloud storage – Massively more reliable – But dependent on proprietary index
    8. 8. Encryption• Two contexts: – Archiving encrypted content – Archive encrypting content• Reasons to encrypt: – Prevent unauthorized access • Especially in Cloud and on tape – To enforce DRM – Legal requirements (HIPAA, state law) • Though only required for transmission, not “at rest”
    9. 9. Encryption Concerns• Increased file size• Performance penalty• Additional expense• Makes files more sensitive to data corruption• May complicate format migration• May complicate legitimate access• Risk of loss of encryption keys• Difficulty of enterprise level key management• Obsolescence of encryption formats• Obsolescence of PKI infrastructure
    10. 10. Redundancy & Diversity Andrea Goethals <> Manager of Digital Preservation and Repository Services, Harvard University
    11. 11. Failures WILL happen• Real problem: failures you can’t recover from!• A few mitigating concepts: redundancy & diversity
    12. 12. Redundancy (multiple duplicates)• Ecology – Redundancy hypothesis = species redundancy enhances ecosystem resiliency• Digital preservation – Example: Multiple copies of content
    13. 13. Diversity (variations)• Finance – Portfolio effect = diversification of assets stabilizes financial portfolios• Ecology – Response diversity = diversification stabilizes ecosystem processes• Digital preservation – Examples: different storage media, storage locations with different geographic threats
    14. 14. What can “fail”? What can’t?• Likely candidates – Storage component faults • Latent sector errors (physical problems) • Silent data corruption (higher-level, usually SW problems) • Whole disks – Organizational disruptions (changes in finances, priorities, staffing)
    15. 15. Data loss risks Redundancy & diversity controls (impact & likelihood?) (costs?)Environmental factors Replication to different data centers e.g. temperature, vibrations affecting multiple devices in same data centerShared component faults Replication to different data centers or e.g. power connections, cooling, redundant components, replication SCSI controllers, software bugs software systemsLarge-scale disasters Replication to different geographic areas e.g. earthquakesMalicious attacks Distinct security zones e.g. wormsHuman error Different administrative control e.g. accidental deletionsOrganizational faults Different organizational control e.g. budget cuts
    16. 16. BIT-LEVEL FIXITY Karen Cariani <> Director WGBH Media Library and Archives, WGBH Educational Foundation John Spencer <> President, BMS/Chase LLC
    17. 17. Bit-Level Fixity• Fixity is a “property” and a “process” (as defined from the 2008 PREMIS data dictionary)• It is a “property”, where a message digest (usually referred to as a checksum) is created as a validation tool to ensure bit- level accuracy when migrating a digital file from one carrier to another
    18. 18. Bit-Level Fixity• It is also a “process”, in that fixity must be integrated into every digital preservation workflow• Fixity is common in digital repositories, as it is easily put in the ingest and refresh migration cycles• Fixity of digital files is a cornerstone of archival best practices
    19. 19. So what’s the problem?• While bit-level fixity solutions are readily available, there remains a large constituency of content creators that place minimal (or zero) value on this procedure• Legacy IT environments, focused on business processes, are not “standards- driven”, more so by vendors, budgets, and poorly defined archival workflow strategies
    20. 20. So what’s the problem?• A vast majority of commercial digital assets are stored “dark” (i.e. data tape or even worse, random HDDs), with no fixity strategy in place• For private companies, individuals, and content creators with digital assets, bit- level fixity remains a mystery – a necessary outreach effort remains
    21. 21. So what’s the problem?• Major labels, DIY artists, indie labels, amateur and semi-professional archivists, photographers, oral histories, and born- digital films usually ignore the concept of fixity• All of the these constituencies need guidance to engage fixity into their daily workflow or suffer the consequences when the asset is needed NOW to monetize…
    22. 22. Overview:Auditing & Repair Micah Altman <> Director of Research, MIT Libraries
    23. 23. Audit [aw-dit]: An independent evaluation of records and activities to assess a system of controlsFixity mitigates risk only if used for auditing.
    24. 24. Functions of Storage Auditing• Detect corruption/deletion of content• Verify compliance with storage/replication policies• Prompt repair actions
    25. 25. Bit-Level Audit Design Choices• Audit regularity and coverage: on-demand (manually); on object access; on event; randomized sample; scheduled/comprehensive• Fixity check & comparison algorithms• Auditing scope: integrity of object; integrity of collection; integrity of network; policy compliance; public/transparent auditing• Trust model• Threat model
    26. 26. Repair Auditing mitigates risk only if used for repair.Design Elements•Repair latency: – Detection to start of repair – Repair duration•Repair algorithm
    27. 27. LOCKSS Auditing & RepairDecentralized, peer-2-peer, tamper-resistant replication & repair
    28. 28. DuraCloud 1.0* Auditing & Repair Storage replicated across cloud providers * Version 2.0 release notes and documentation indicate the addition of regular bit integrity check, and the ability to sync changes from a primary copy for premium users. Detailed features and pecific algorithms do not yet appear fully documented
    29. 29. iRODS Auditing & RepairRules-based federated storage grid
    30. 30. Auditing & RepairTRAC-Aligned policy auditing as a overlay network
    31. 31. Summary: Micah Altman <> Director of Research, MIT Libraries
    32. 32. Methods for Mitigating Risk Physical: Formats File File Media, Transforms: Systems: Hardware, compression, transforms, Environment encoding, deduplication, encryption redundancyDiversification Number Fixity Audit Repair of copies of copies
    33. 33. How can we choose?• Clearly state decision problem• Model connections between choices &outcomes• Empirically calibrate and validate
    34. 34. The Problem Keeping risk of object loss fixed -- what choices minimize $? Are we there“Dual problem” yet?Keeping $ fixed, what choices minimize risk?ExtensionFor specific cost functions for loss of object:Loss(object_i), of all lost objectsWhat choices minimize:Total cost= preservation cost+ sum(E(Loss)) cost
    35. 35. Modeling
    36. 36. Measurements• Media – MBTF theoretical and actual• File transformations: – compression ratio – partial recoverability• Filesystem transformations: – Deduplication – Compression ratio• Diversification – Single points of failure – Correlated failures• Copies, Audit, Repair – Simulation models – Audit studies
    37. 37. Questions* What techniques are you using? What models guide the “knobs”? Contact the NDSA Infrastructure Working Group:* Thanks to our moderator: Trevor Owens <>,Digital Archivist, Library of Congress