Repositories and digital preservation

UKOLN is supported by: Repositories and Digital Preservation Michael Day Research and Development Team Leader UKOLN, University of Bath RSP 'Goes back to' School, Matfen Hall, Northumberland, 14-16 September 2009

Presentation outline
General context: repositories and digital preservation
Digital preservation overview
Tools:
Preservation Planning (Plato)
Repository audit (TRAC, DRAMBORA)
Repositories and the curation of research data
Roles and responsibilities
Infrastructures
Curation challenges

The repository context (1)
Repository content is one part of a much wider digital preservation problem
No major specific digital preservation requirements
Repositories are part of the evolving structure of scholarly communication
Preservation needs to be considered in the same conceptual space as things like e-journals (e.g., Portico)
There is a commonly-held view that e-prints are just duplicates of the conventional literature created for immediate access and will not therefore need preservation

Repositories can contain many types of materials, but the main focus has usually been on “e-prints” or theses
Broadly text-based (analogues of traditional papers)
This simplifies preservation requirements
Compared with complex multimedia objects, “the preservation of e-prints is relatively straightforward from a technical point of view” (Pinfield and James, 2003)
A large percentage of repository content has (until very recently) been made up of a relatively limited number of formats, e.g.:
PDF, HTML, MS Word, RTF, TeX, PostScript

Repositories are beginning to consider their role in preserving a wider range of content types
Maintain accurate records of the whole research process or lifecycle (digital curation)
Includes: research data (simulations, materials, the results of high-throughput instrumentation, open science, etc.), Web pages, Web 2.0 content (blogs, etc.), learning objects, images, time-based media, etc.
For example: KeepIt project exemplars - research papers, science data, arts, teaching materials and theses: http://preservation.eprints.org/keepit/
This makes defining preservation requirements for repositories more difficult

Repositories need to consider carefully their longer-term objectives and ambitions
Clifford Lynch: “An institutional repository needs to be a service with continuity behind it … Institutions need to recognise that they are making commitments for the long term”
This is dependent on institutional support
Shared infrastructures
Bilateral, regional, national, international
Distributed approaches possible (e.g., SHERPA DP)
A potential key role for national or research libraries (as with DARE in the Netherlands)

Integration of preservation services with repository software
Some experimentation in the PRESERV project: http://preservation.eprints.org/
Used third-party registries of format information (DROID and PRONOM from The National Archives) to characterise and validate repository content and to analyse risks
RSP briefing paper on preservation and storage formats: http://www.rsp.ac.uk/pubs/briefingpapers-docs/technical-preservformats.pdf

Digital preservation basics
An ongoing approach to managing digital content based on:
The identification and adoption of appropriate preservation strategies
Creation or Ingest stages are normally the best time to ensure that data are fit-for-purpose and “preservable”
The collection and management of appropriate metadata
Capture of explicit and implicit knowledge, contexts
The ongoing monitoring of technical contexts and the application of preservation planning techniques
Continual monitoring of the organisation (audit)

Technical challenges
Digital media
Currently magnetic or optical tape and disks, some devices (e.g., memory sticks)
Uncertain lifetimes
Hardware and software dependence
Most digital objects are dependent on particular configurations of hardware and software
Relatively short obsolescence cycles

Conceptual challenges (1)
What is an digital object?
Some are analogues of traditional objects, e.g. meeting minutes, research papers (e-prints)
Others are not, e.g. Web pages, GIS, 3D models of chemical structures
Complexity
Dynamic nature

Three layers:
Physical: the bits stored on a particular medium
Logical: defines how the bits are used by a software application, based on data types (e.g. ASCII); in order to understand (or preserve) the bits, we need to know how to process this
Conceptual: things that we deal with in the real world
From: Ken Thibodeau, “Overview of technological approaches to digital preservation and challenges in coming years.” In: The state of digital preservation: an international perspective (CLIR, 2002): http://www.clir.org/

On which of these layers should preservation activities focus?
We need to preserve the ability to reproduce the objects, not just the bits (would a printout do?)
In fact, we can change the bits and logical representation and still reproduce an authentic conceptual object (e.g. converting into PDF)
Increased focus on reuse (e.g, data in tables)
Authenticity and integrity
How can we trust that an object is what it claims to be?
Digital information can easily be changed by accident or design

Some general principles (1)
Most of the technical problems associated with long-term digital preservation can be solved if a life-cycle management approach is adopted
i.e. a continual programme of active management
Ideally, combines both managerial and technical processes, e.g., as in the OAIS Reference Model
Many current preservation systems are attempting to support this approach
Digital preservation strategies need to be seen in this wider context

Preservation needs to be considered at a very early stage in an object's life-cycle
There is a need to identify 'significant properties'
Recognises that preservation is context dependent, even user specific (concept of 'designated community')
Helps with choosing an acceptable preservation strategy
Encapsulation
Surrounding the digital object - at least conceptually - with all of the information needed to decode and understand it (including software)
Produces autonomous 'self-describing' objects, reduces external dependencies (linked to the Information Package concept in the OAIS Reference Model)

Metadata and documentation is vitally important
Relates to OAIS concepts like Representation Information and Preservation Description Information
Functions
Records scientific meaning
Records the research context
Enables the development of finding aids
Standards are being developed that support digital preservation activities (e.g., the PREMIS Data Dictionary)
Wherever possible, retain also the original byte-stream

Digital preservation strategies
Three main families:
Technology preservation / digital archaeology
Emulation
Migration

Technology preservation
The preservation of an information object together with all of the hardware and software needed to interpret it
Successfully preserves the look, feel and behaviour of the whole system (at least while the hardware and software still functions)
Severe problems with storage and ongoing maintenance, missing documentation
May have a role for historically important hardware
May have a shorter-term role for supporting the rescue of digital objects (digital archaeology)

Digital archaeology
Not so much a preservation strategy, but the default situation if there isn't one
Using various techniques to recover digital content from obsolete or damaged physical objects (media, hardware, etc.)
A time consuming process, needs specialised equipment and (in most cases) adequate documentation
Considered to be expensive (and risky)
Remains an option for content deemed to be of value that has not been dealt with in any other way

Emulation (1)
Preserving the original bit-streams and application software; running this on emulator programs that mimic the behaviour of obsolete hardware
Emulators evolve over time
Chaining, rehosting
Emulation Virtual Machines
Running emulators on simplified 'virtual machines' that can be run on a range of different platforms

Emulation (2)
Benefits:
Technique already widely used, e.g. for emulating different hardware, computer games
Preserves (and uses) the original bits
Reduces the need for regular object transformations (but emulators and virtual machines may themselves need to be migrated)
Retains ‘look-and-feel’
May be the only approach possible where objects are complex or dependent on executable code
Less 'understanding' of formats is needed; little incremental cost in keeping additional formats

Emulation (3)
Challenges:
Do organisations have the technical skills necessary to implement the strategy?
Preserving 'look and feel' may not be needed for all objects
It will be difficult to know definitively whether user experience has been accurately preserved
Uses:
Promising family of approaches
Needs further practical application and research, e.g. Dioscuri software (National Library of the Netherlands)

Migration (1)
Based on the managed transformation of content:
A set of organised tasks designed to achieve the periodic transfer of digital information from one hardware and software configuration to another, or from one generation of computer technology to a subsequent one - CPA/RLG report (1996)
Abandons attempts to keep old technology (or substitutes for it) working
A 'known' solution used by data archives and software vendors
Focuses on the perceived content (or significant properties) of objects

Migration (2)
Challenges:
Can be labour intensive (batch process, monitoring, QA)
There can be problems with ensuring the ongoing 'integrity and authenticity' of objects
Transformations need to be documented (typically as part of the preservation metadata)
Uses:
Seems to be most suitable for dealing with large collections of similar objects (e-print repositories?)
Migration can often be combined with some form of standardisation process, e.g., as part of ingest
A role for repository managers?

Preservation support on ingest
Formats can be identified and validated on ingest or deposit into a repository
JHOVE (JSTOR/Harvard Object Validation Environment)
PRONOM, DROID (The National Archives)
Metadata
Some tools exist for the automatic capture of metadata
Standardisation on ingest
Perceived wisdom suggests the adoption of open or non-proprietary standards, e.g. databases structured in XML, uncompressed images, 'preservation friendly' standards like PDF/A

Choosing a strategy (1)
Preservation strategies are not in competition
Different strategies will work together, may be value in diversification
Migration strategies mean difficult choices need to be made about target formats
But the strategy chosen has implications for:
The technical infrastructure required (and metadata)
Collection management priorities
Rights management
Owning the rights to re-engineer software
Costs

Choosing a strategy (2)
Plato preservation planning tool (EU Planets project)
A decision support tool that helps users explore the evaluation of potential preservation solutions against specific requirements and for building a plan for preserving a given set of objects
Integrates file format identification (using DROID); some migration services; XML-based generic format characterisation using XCL (eXtensible Characterisation Languages)
http://www.ifs.tuwien.ac.at/dp/plato/intro.html

Repository audit frameworks (1)
Repository audit frameworks first developed out of the OAIS Reference Model
OAIS Mandatory Responsibilities (only six of them):
The main focus was on technical and organisational aspects, e.g.:
That repositories ensure that preserved information (content) can be understood (independently understandable)
That documented policies and procedures are being followed
No clear concept of OAIS compliance (although often claimed by system developers)

Trusted Repositories Audit and Certification (TRAC): Criteria and Checklist
RLG-NARA Digital Repository Certification Task Force checklist, revised (following pilot audits) by the Center for Research Libraries and OCLC
Criteria cover three main aspects:
Organisational Infrastructure
Governance and viability, structure and staffing, financial sustainability, contracts, etc.
Digital Object Management
Ingest, preservation planning, archival storage, etc.
Technologies, Technical Infrastructure, & Security
Systems and infrastructure, etc.

TRAC Checklist example page

DRAMBORA (Digital Repository Audit Method Based on Risk Assessment)
Digital Curation Centre / Digital Preservation Europe
“ Presents a methodology for self-assessment, encouraging organisations to establish a comprehensive self-awareness of their objectives, activities and assets before identifying, assessing and managing the risks implicit within their organisation“
Identifying risks and scoring each one on likelihood and impact
Covers: organisational context, policies, assets, risks, etc.
Online tool (http://www.repositoryaudit.eu/about/)

A means of "asking the right questions" about your repository and documenting appropriate procedures and risks
Both TRAC and DRAMBORA are under consideration by (different) ISO technical committees
External badge of quality (a "certified preservation repository")
vs.
Management tool for self assessment

Web links:
PRESERV project: http://preservation.eprints.org/
KeepIt project: http://preservation.eprints.org/keepit/
Plato Preservation Planning tool: http://www.ifs.tuwien.ac.at/dp/plato/intro.html
DRAMBORA: http://www.repositoryaudit.eu/about/
RSP briefing paper on preservation and storage formats: http://www.rsp.ac.uk/pubs/briefingpapers-docs/technical-preservformats.pdf

Questions? “ Pigabyte” King Bladud’s Pigs in Bath (public art project), Summer 2008 http://www.kingbladudspigs.org/

Repositories and the curation of research data

Dealing with research data
An extremely broad category of material:
“... any information that can be stored in digital form, including text, numbers, images, video or movies, audio, software, algorithms, equations, animations, models, simulations, etc.” (National Science Board, Long-lived digital data collections, 2005)
In practice, it can mean almost anything

Why curate research data? (1)
Part of the normal research process:
The need for others to validate and replicate research
In some disciplines, supporting data is routinely made available to reviewers and linked from journal papers
Principles of sharing and openness are firmly embedded in some disciplines

Extrinsic and intrinsic value;
High investment in research
Data can be very expensive to capture and analyse
Data is impossible to recreate once lost
Observational data (by definition) is irreplaceable
Current generations of instruments can gather more data than can be analysed

The potential for creating 'new' knowledge from existing data:
Re-use, re-analysis, data mining
Annotation, e.g. in molecular biology astronomy
Combining datasets in innovative ways, e.g. mapping biodiversity data onto ecological GIS
“Science 2.0”

It is increasingly a requirement of some research funding bodies
Some have quite mature data retention policies (not necessarily for permanent retention)
Increasing expectation of access to data from publicly-funded research
OECD Principles and guidelines for access to research data from public funding (2007)

Institutional asset management:
Universities and other research organisations invest very large sums of money into research activities
Research data is a key output of this activity
It is, therefore, an institutional asset that needs stewardship

Promoting the institution, research group or individual:
Re-use helps promote visibility and 'impact'
Institutions become acknowledged 'centres of competence'

Who undertakes preservation?
Researchers
Indirectly - they have most direct contact with creation stage, and understand how data can be used
Directly - sometimes responsible for maintaining community data collections
Information professionals
Sometimes, but it depends on the context
IT professionals
Primarily informaticians working with scientists

Roles and responsibilities (1)
Long-lived data collections (NSB)
Data authors
Data managers
Data scientists
Data users
Funding agencies
Dealing with data (JISC)
Scientist
Institution
Data centre
User
Funder
Publisher

Scientists
Initial creation and use of data
Expectation of first use and in gaining appropriate credit and recognition
Responsible for:
Managing data for life of project
For using standards (where possible)
For complying with data policies
For making the data available in a form that can (easily?) be used by others

Institutions:
Role less clear
Institutional policies may require short-term management of data
Advocacy and training
Some institutions are developing repository services
Are rarely currently used for research data
Federated approaches maintain disciplinary involvement

Data centres
Undertakes curation and provides access
Responsible for:
Selection and ingest
Participating in the development of standards
Protecting the rights of data creators
Supporting ingest and metadata capture
Supporting re-use (tools and services)
Training

Users:
Users of third-party data
Responsible for:
Adhering to any licenses and restrictions on use
Acknowledging data creators and curators
Managing any derived data
Provide feedback to scientists and data centres

Funding bodies:
Acting at policy level
Responsible for:
Considering wider policy perspectives
Developing policies in co-operation with other stakeholders
Monitoring and enforcing data policies
Support for long-term data management
Support for data curation

Research data collections (1)
A typology (1):
From National Science Board report Long-lived digital data collections (2005)
Research data collections – the products of one or more focused research projects
Resource or community data collections – collections that emerge to serve particular subject sub-disciplines
Reference data collections – serve a broader and more diverse set of user communities

Research data collections (2)
Data in “research data collections” is most at risk
A modern version of the “file-drawer problem”
Data stored on personal hard-drives or on media; largely undocumented
Particular challenge when the data creator has retired or moved to another institution
Data creators not always aware of its potential value
The reward structure of science is not always helpful

Curation infrastructures (1)
Focus on the generic:
Need for a balance between:
The 'bottom-up' discipline-based drivers that promote the generation of research data
The policy level, looking to make cost effective investment in curation
When building Infrastructures, focus on the generic
Storage systems and middleware
Preservation services
Identifying the needs of the wider community

Curation infrastructures (2)
The need for collaboration:
Need for 'deep-infrastructure' recognised as far back as 1996 by the Task Force on Archiving of Digital Information
Digital preservation involves the "grander problem of organizing ourselves over time and as a society ... [to manoeuvre] effectively in a digital landscape" (p. 7)

Curation challenges: Costs
NSF Task Force looking at this subject
JISC-funded LIFE (Life Cycle Information for E-Literature) project is developing a predictive costing tool (http://www.life.ac.uk/)
JISC-funded study ( Keeping research data safe , 2008) focused on research data curation at the institution level
The complex service requirements for curating research data means that institutions are setting-up federated approaches to repository development
Currently ingest costs are much higher than long-term storage and preservation costs

Curation challenges: Scale (1)
The “digital deluge” in e-Science
New generations of instruments
Computer simulations
Many terabytes generated per day, petabyte scale computing (and growing)
Cory Doctorow, “Welcome to the petacentre.” Nature, 455, pp 17-21, 4 Sep 2008
Are Institutional Repositories ready for this?
Digitised content:
Google Book Search (~7 million items)
A role for research libraries?

Curation challenges: Scale (2)
Problems of scale are particularly acute in traditional 'big-science' disciplines:
Particle physics (e.g., the Large Hadron Collider)
Astronomy (sky surveys, etc)
But “smaller experiments will grow the fastest” (Szalay & Gray, Nature , 440, 413-4, 23 Mar 2006)
Bioinformatics, crystallography, engineering design, and many others
In some cases it may be cheaper just to generate the data again, e.g. for computer simulations

Curation challenges: Complexity (2)
Research data is extremely diverse - not really a single category of material
tabular data, images, GIS, etc.
raw machine output vs, derived data
varying levels of structure (XML, legacy formats, etc.)
many different standards
Research data is not homogeneous
No one-size-fits-all approach possible

Curation challenges: Cultures
Diverse research cultures
Data practices vary widely, even within a single discipline
Gene sequence data is typically deposited in public databases
In proteomics, sharing is not so widespread; partly driven by lack of standards, but there is also concern about who have exploitation rights
Role of commercial interests
Pharmaceuticals, architecture and engineering, geological prospecting

The Future ...
“It is always a mistake for a historian to try and predict the future. Life, unlike science, is simply too full of surprises” - Richard J. Evans, In defence of history (1997, p. 62)

Further reading
National Science Board, Long-lived digital data collections: enabling research and education in the 21st century (NSF, 2005) http//www.nsf.gov/pubs/2005/nsb0540/
Liz Lyon, Dealing with data; roles, rights, responsibilities and relationships (JISC, 2007) http://www.jisc.ac.uk/whatwedo/programmes/digitalrepositories2005/dealingwithdata.aspx
Neil Beagrie, Jullia Chruszcz, and Brian Lavoie, Keeping research data safe: a cost model and guidance for UK universities (JISC, 2008) http://www.jisc.ac.uk/publications/publications/keepingresearchdatasafe.aspx

Acknowledgments
UKOLN is funded by the Joint Information Systems Committee (JISC) of the UK higher and further education funding councils, the Museums, Libraries and Archives Council (MLA), as well as by project funding from the JISC, the European Union, and other sources. UKOLN also receives support from the University of Bath, where it is based.
More information: http://www.ukoln.ac.uk/

Thank You!

Repositories and digital preservation

by Michael Day on Sep 16, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 10

Statistics

Repositories and digital preservation — Presentation Transcript

http://www.slideshare.net	7
http://bogdanpopovici2008.wordpress.com	3