0
Prepared for
CNI Fall Membership Meeting
December 2013

A National Agenda for
Digital Stewardship
Presenters:
Micah Altman...
This Talk
Who are the NDSA?
Why develop an agenda for digital stewardship?
What should national priorities be?
… digital c...
Collaborators & Co-Conspirators
• The 150+ institutional members of NDSA, and the
10000+ hours contributed by their repres...
Who are the NDSA?

A National Agenda for Digital Stewardship

5
About the NDSA
• Founded in 2010, the National Digital Stewardship Alliance
(NDSA) is a consortium of institutions that ar...
Recent
Outputs

Working
Groups

NDSA Initiatives

Extending Knowledge
Tools for Practice
• Preservation Storage Survey
• L...
Why develop an
agenda for digital
stewardship?

A National Agenda for Digital Stewardship

8
Why a national agenda for digital
stewardship?
• Effective digital stewardship is vital for:
– maintaining authentic publi...
Why now?
Climate
Strong trends towards:
• More production of
digital content
• More publishing, filtering
and access
• Mor...
Isn’t digital preservation a solved
problem?

• Why not put everything in Amazon?
• Amazon claims reliability of 99.999999...
What Was Accomplished?
The
National Agenda for Digital Stewardship
identifies high-impact opportunities to
advance:
• the ...
How was this accomplished it?
• Contributed community effort

- Development: contributions from the (now 150+) institution...
National priorities for…

Digital Content

A National Agenda for Digital Stewardship

14
Digital Content Areas
•
•
•
•

Web and Social Media
Electronic Records
Moving Image and Recorded Sound
Research Data

A Na...
Digital Content Areas
• Across all areas, content size, value and
selection represent a core challenge
• Important to deve...
Raising Awareness
and Articulating Value
• Content area webinars
• Follow up blog posts, interviews
• Content Case Studies...
National priorities for…

Technical
Infrastructure
A National Agenda for Digital Stewardship

18
2014 Technical Infrastructure
Priorities
• File Format Action Plan Development
• Interoperability and Portability in Stora...
Technical Infrastructure
File Format Action Plan Development
–Stewardship organizations are amassing large collections of ...
•

•

•

Interoperability and Portability in
Storage Architectures

As stewardship organizations manage increasingly large...
Integration of Digital Forensics Tools
• Digital Forensics tools are essential for working across the
range of heterogeneo...
Ensuring Content Integrity
•

•

•

Digital preservation is possible through a chain of migration
of current hardware and ...
National priorities for…

Organizational
Development
A National Agenda for Digital Stewardship

24
Organizational Roles, Policies,
and Practices
Identifies need to increase
cross‐organizational cooperation to
increase the...
“People who work together will win,
whether it be against complex football
defenses, or the problems of modern
society.”
-...
2014 Priorities for Cross-organizational
Cooperation
1) Provision networked preservation services
– network of preservatio...
National priorities for…

Research

A National Agenda for Digital Stewardship

28
Research Priorities
• Applied Research for Cost Modeling and
Audit Modeling
• Understanding Information Equivalence &
Sign...
What does the discipline believe?
• Our digital evidence base erodes
• There are multiple threats to information –
diversi...
How have we learned…
The Limitspreservation practices and outcomes are
of Case Studies
Most current evidence for digital
b...
Simple question?
• If you have 1000 files (bitstreams), and you’d
like to have 99.99% chance of accessing them
in 20 years...
What are some threats?

Physical & Hardware

Insider &
External
Attacks

Software

Organizational
Failure
A National Agend...
Amazon’s Unrealistic Nine Nines
• What are the units? - Collection? Object? Bit?
• How many of these do you have?
• Seems ...
The Problem Restated
Keeping risk of object loss fixed
-- what choices minimize $?

Are we there
yet?

“Dual problem”
Keep...
Physical:
Media,
Hardware,
Environment

Diversification
of copies

File
Transforms:
compression,
encoding,
encryption

For...
Media
characteristics

Bit Corruption

Modeling

Threat
characteristics

Correlations

Format
Characteristics
Logical Scop...
What Else do We Need To Know?
• What is the expected future value of a specified collection of digital
content?
• What con...
How do we learn?
• Apply existing research methodologies from other fields
-- especially fields involving observation rese...
What’s next?

A National Agenda for Digital Stewardship

40
A National Preservation Agenda for
2015 and Beyond
• Drafts and update process starts this winter
• Community review proce...
Moving Digital Preservation Forward
NDSA has a commitment to:
• Facilitating broad collaboration
• Promoting dissemination...
Want more information?
Contact NDSA for…
• Briefings, webinars, and consultations on the
Agenda or other NDSA work
• Assis...
More Information

digitalpreservation.gov/ndsa/na
tionalagenda
ndsa@loc.gov
A National Agenda for Digital Stewardship

44
A National Agenda for Digital Stewardship
Upcoming SlideShare
Loading in...5
×

A National Agenda for Digital Stewardship

930

Published on

This was presented at the 2013 CNI Fall Member meeting:

http://www.cni.org/events/membership-meetings/upcoming-meeting/fall-2013/

Digital stewardship is vital for the authenticity of public records, the reliability of scientific evidence, and the enduring accessibility to our cultural heritage. Knowledge of ongoing research, practice, and organizational collaborations has been distributed widely across disciplines, sectors, and communities of practice. The National Agenda for Digital Stewardship annually integrates the perspective of dozens of experts and hundreds of institutions, convened through the Library of Congress, to identify the highest-impact opportunities to advance the state of the art; the state of practice; and the state of collaboration within the next 3-5 years. This talk discusses key highlights from the inaugural report and related ongoing work by the National Digital Stewardship Alliance.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
930
On Slideshare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • This work. by the National Digital Stewardship Alliance (digitalpreservation.gov/ndsa/) is licensed under the Creative Commons Attribution-Share Alike 3.0 United States License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/us/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
  • 10 Minutes
  • 8 Minutes
  • 10 Minutes
  • 5 Minutes
  • 10 Minutes
  • 5 Minutes
  • Transcript of "A National Agenda for Digital Stewardship "

    1. 1. Prepared for CNI Fall Membership Meeting December 2013 A National Agenda for Digital Stewardship Presenters: Micah Altman, <escience@mit.edu>, MIT Michelle Gallinger, Library of Congress Abigail Grotke, Library of Congress Trevor Owens, Library of Congress
    2. 2. This Talk Who are the NDSA? Why develop an agenda for digital stewardship? What should national priorities be? … digital content … technical infrastructure … organizational roles … research areas What’s next?
    3. 3. Collaborators & Co-Conspirators • The 150+ institutional members of NDSA, and the 10000+ hours contributed by their representatives to NDSA working groups, meetings and reports • National Agenda Authors: Micah Altman, Jefferson Bailey, Karen Cariani, Jim Corridan, Jonathan Crabtree, Blaine Dessy, Michelle Gallinger, Andrea Goethals, Abigail Grotke, Cathy Hartman, Butch Lazorchak, Jane Mandelbaum, Carol Minton Morris, Trevor Owens, Meg Phillips, John Spencer, Helen Tibbo, Tyler Walters, Kate Wittenberg, Kate Zwaard A National Agenda for Digital Stewardship 4
    4. 4. Who are the NDSA? A National Agenda for Digital Stewardship 5
    5. 5. About the NDSA • Founded in 2010, the National Digital Stewardship Alliance (NDSA) is a consortium of institutions that are committed to the long-term preservation of digital information. • Our mission is to establish, maintain, and advance the capacity to preserve our nation's digital resources for the benefit of present and future generations. • NDSA member institutions represent all sectors, and include universities, consortia, professional associations, commercial enterprises, and government agencies at the federal, state, and local levels. The Library of Congress provides organizational support and substantive collaboration as Secretariat. • Based on collaborative community effort -- there are no fees for NDSA membership. Each member institution commits to to NDSA principles, and contributes efforts to working groups, reports, surveys, meetings and other NDSA initiatives. A National Agenda for Digital Stewardship 6
    6. 6. Recent Outputs Working Groups NDSA Initiatives Extending Knowledge Tools for Practice • Preservation Storage Survey • Levels of Preservation • Web Harvesting Survey • Digital Preservation in a Box • Preservation Staffing Survey • Digital Preservation on • Geospatial Selection & Wikipedia Appraisal report • Content case studies Dissemination • NDSA Interview Series • National agenda for digital stewardship • NDSA Innovation Awards • NDSA Social Media A National Agenda for Digital Stewardship 7
    7. 7. Why develop an agenda for digital stewardship? A National Agenda for Digital Stewardship 8
    8. 8. Why a national agenda for digital stewardship? • Effective digital stewardship is vital for: – maintaining authentic public records – growing a reliable scientific evidence base – providing durable access to our cultural heritage • Knowledge of ongoing research, practice, and organizational collaborations is distributed widely across disciplines, sectors, and communities of practice A National Agenda for Digital Stewardship 9
    9. 9. Why now? Climate Strong trends towards: • More production of digital content • More publishing, filtering and access • More learners and collaborators • More attention to public information A National Agenda for Digital Stewardship Weather 10
    10. 10. Isn’t digital preservation a solved problem? • Why not put everything in Amazon? • Amazon claims reliability of 99.999999999% (Longer odds than winning powerball, being struck by lightning, and finding alien life, combined) A National Agenda for Digital Stewardship 11
    11. 11. What Was Accomplished? The National Agenda for Digital Stewardship identifies high-impact opportunities to advance: • the state of the art • the state of practice • the state of collaboration A National Agenda for Digital Stewardship 12
    12. 12. How was this accomplished it? • Contributed community effort - Development: contributions from the (now 150+) institutional members through working group participation, workshop discussion, commentary - Writing: LC Staff, chairs of NDSA working groups, coordination committee - Reviewing: expert reviewers in the preservation community • Integrating diverse perspectives from multiple disciplines & sectors • The persistence, organization, and commitment of the Library of Congress in its role as Secretariat A National Agenda for Digital Stewardship 13
    13. 13. National priorities for… Digital Content A National Agenda for Digital Stewardship 14
    14. 14. Digital Content Areas • • • • Web and Social Media Electronic Records Moving Image and Recorded Sound Research Data A National Agenda for Digital Stewardship 15
    15. 15. Digital Content Areas • Across all areas, content size, value and selection represent a core challenge • Important to develop theoretically grounded and empirically tested models of information valuation. A National Agenda for Digital Stewardship 16
    16. 16. Raising Awareness and Articulating Value • Content area webinars • Follow up blog posts, interviews • Content Case Studies • Content Matters blog posts A National Agenda for Digital Stewardship 17
    17. 17. National priorities for… Technical Infrastructure A National Agenda for Digital Stewardship 18
    18. 18. 2014 Technical Infrastructure Priorities • File Format Action Plan Development • Interoperability and Portability in Storage Architectures • Integration of Digital Forensics Tools • Ensuring Content Integrity A National Agenda for Digital Stewardship 19
    19. 19. Technical Infrastructure File Format Action Plan Development –Stewardship organizations are amassing large collections of digital materials suggests a need to monitor the heterogeneous digital files the organizations are managing. –Need for tools and services for creating file-format action plans is needed to make timely execution of file format plans a reality for data stewards. –The digital preservation community would further benefit from organizations sharing their assessments of institutional risk and their plans for mitigating that risk and addressing file format problems with specific plans. A National Agenda for Digital Stewardship 20
    20. 20. • • • Interoperability and Portability in Storage Architectures As stewardship organizations manage increasingly large and complex data sets, the need for interoperability at various levels within the technical hardware and software stacks that make digital preservation becomes increasingly important. Interoperability of storage devices, hardware, data tape, and file systems software and would help alleviate bottlenecks in the interrelationship between distinct functions in workflows. Need for establishing and promoting technical means by which lower levels of the technology stack can directly integrate without requiring extensive computation and processing at higher levels. A National Agenda for Digital Stewardship 21
    21. 21. Integration of Digital Forensics Tools • Digital Forensics tools are essential for working across the range of heterogeneous kinds of digital materials coming under stewardship • Projects like BitCurator are pulling together the suite of tools to do this work and developing processes and workflows. • We are now at the point of implementation, it’s time for organizations to start implementing and sharing information about their work • The result of this work, will be large sets of heterogeneous digital files which will then push for the development of tools to work with these kinds of data at scale. A National Agenda for Digital Stewardship 22
    22. 22. Ensuring Content Integrity • • • Digital preservation is possible through a chain of migration of current hardware and software systems to yet-to-beestablished future infrastructures. Essential to develop guidance on how to plan for and manage these changes Abstract requirements for fixity are useful as principals, but when applied universally can actually be detrimental in some digital preservation system architectures. Need for best practices for fixity in particular system designs and configurations. Need for the development of standards, practices and strategies that directly address migration, in particular, around end-to-end fixity checking A National Agenda for Digital Stewardship 23
    23. 23. National priorities for… Organizational Development A National Agenda for Digital Stewardship 24
    24. 24. Organizational Roles, Policies, and Practices Identifies need to increase cross‐organizational cooperation to increase the impact and leverage investments made by individual institutions. A National Agenda for Digital Stewardship 25
    25. 25. “People who work together will win, whether it be against complex football defenses, or the problems of modern society.” - NFL coach, Vince Lombardi A National Agenda for Digital Stewardship 26
    26. 26. 2014 Priorities for Cross-organizational Cooperation 1) Provision networked preservation services – network of preservation service providers with specialized services rather than every organization performing all aspects of digital preservation 2) Collaborate on shepherding and promotion of standards – digital preservation community representation on the relevant standards bodies rather than each organization needing to participate in every body 3) Share digital preservation training and staffing resources A National Agenda for Digital Stewardship 27
    27. 27. National priorities for… Research A National Agenda for Digital Stewardship 28
    28. 28. Research Priorities • Applied Research for Cost Modeling and Audit Modeling • Understanding Information Equivalence & Significance • Policy Research on Trust Frameworks • Preservation at Scale • The Evidence Base for Digital Preservation A National Agenda for Digital Stewardship 29
    29. 29. What does the discipline believe? • Our digital evidence base erodes • There are multiple threats to information – diversifying against them is crucial • Lifecycle analysis is critical for better longterm management of information • Better practices are needed A National Agenda for Digital Stewardship 30
    30. 30. How have we learned… The Limitspreservation practices and outcomes are of Case Studies Most current evidence for digital based on local case studies and convenience samples • Case studies are useful for: – – – – existence proofs raising awareness of problems process tracing hypothesis generation, • Case studies are not enough to – – – – advance our scientific knowledge create robust predictive models test causal hypotheses strongly guide decision making. • Systematic Evidence is needed both to support – general selection of digital preservation practices and method – applications of selected digital preservation methods in a specific operational context. A National Agenda for Digital Stewardship 31
    31. 31. Simple question? • If you have 1000 files (bitstreams), and you’d like to have 99.99% chance of accessing them in 20 years. How do you store them? A National Agenda for Digital Stewardship 32
    32. 32. What are some threats? Physical & Hardware Insider & External Attacks Software Organizational Failure A National Agenda for Digital Stewardship Curatorial Error 33
    33. 33. Amazon’s Unrealistic Nine Nines • What are the units? - Collection? Object? Bit? • How many of these do you have? • Seems to be entirely theoretical – – – – MBTF + Independence * enough replicas No details for estimate provided No historical reliability statistics provided No service reliability auditing provided • Reasons to Doubt Theoretical Calculations – Storage manufacture hardware MTBF (mean time between failures) is inaccurate… – Failures across hardware replicas are not independent – Many potential correlated failure modes not addressed: • software failure (e.g. a bug in the AWS software for its control backplane) • legal threats (leading to account lock-out — such as this, deletion, or content removal); • institutional threats (such as a change in Amazon’s business model) • Process threats (someone hits the delete button by mistake; forgets to pay the bill; or AWS rejects the payment) • Amazon SLA’s do not incorporate or reflect “design” reliability claims even slighltly: – No claim to reliability in SLA’s (or uptime, availability, response time…) – Sole recovery for breach is limited to refund of fees for periods the service was unavailable A National Agenda for Digital Stewardship 34
    34. 34. The Problem Restated Keeping risk of object loss fixed -- what choices minimize $? Are we there yet? “Dual problem” Keeping $ fixed, what choices minimize risk? For specific cost functions for loss of object: risk Extension Loss(object_i), of all lost objects What choices minimize: Total cost= preservation cost+ sum(E(Loss)) cost
    35. 35. Physical: Media, Hardware, Environment Diversification of copies File Transforms: compression, encoding, encryption Formats Number of copies Verification Replication Local Storage Methods for Mitigating Bit-LevelRisk Fixity File Systems: transforms, deduplication, redundancy Audit Repair
    36. 36. Media characteristics Bit Corruption Modeling Threat characteristics Correlations Format Characteristics Logical Scope of Corruption Corruption File/encoding Characteristics Filesystem Characteristics Detection Auditing Frequency Auditing Algorithm Probability of Successful Repair Repair Repair Algorithm Repair Frequency Repair duration
    37. 37. What Else do We Need To Know? • What is the expected future value of a specified collection of digital content? • What content is already being effectively stewarded by other organizations? • How much is the expected future cost of preserving that content? • How often do different threats to information manifest – storage hardware or media failures – software errors cause information loss – stored information becomes inaccessible because of obsolete formats, or loss of other contextual knowledge – that human error or maliciousness causes loss content in an information system • What is the reliability of current digital preservation networks and services? • How successful are other proposed strategies for replication, monitoring, certification, and auditing at preventing loss due to these threats? A National Agenda for Digital Stewardship 38
    38. 38. How do we learn? • Apply existing research methodologies from other fields -- especially fields involving observation research on humans and human systems • Some useful methodologies: – probability-based surveys (e.g. of information management practice and outcomes) – replicable simulation experiments tied to theoretically grounded models of information management and risk; – creation of testbeds and test-corpuses which can be used to systematically compare new practices, tools, and methods; – field experiments, in which randomized interventions are applied and evaluated in real operational environments. A National Agenda for Digital Stewardship 39
    39. 39. What’s next? A National Agenda for Digital Stewardship 40
    40. 40. A National Preservation Agenda for 2015 and Beyond • Drafts and update process starts this winter • Community review process late spring • An update will be presented in July at Digital Preservation 2014 A National Agenda for Digital Stewardship 41
    41. 41. Moving Digital Preservation Forward NDSA has a commitment to: • Facilitating broad collaboration • Promoting dissemination and engagement • Regular updates and revisions of the National Agenda and core NDSA surveys A National Agenda for Digital Stewardship 42
    42. 42. Want more information? Contact NDSA for… • Briefings, webinars, and consultations on the Agenda or other NDSA work • Assistance in gathering comments on National policies and programs • Assistance in recruiting experts for review and discussion panels; grant review • Referrals to content stewards in specific areas A National Agenda for Digital Stewardship 43
    43. 43. More Information digitalpreservation.gov/ndsa/na tionalagenda ndsa@loc.gov A National Agenda for Digital Stewardship 44
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×