Ldp Executive Slides

Engineering What’s the Problem?
100-year Digital Preservation† Digital infrastructure in its infancy; paper infrastructure has
been slowly refined for over 3000 years
Most content starts in digital form. Society has demonstrated little commitment to digital
preservation
Some content is fundamentally digital.
After a dozen years of archives ’ efforts, no existing digital
People want to save some of it for the long term. preservation program can match Internet scales

“[No] concise and well-developed strategy that represents the views
What's needed to preserve bit-streams?
of a broad community has yet emerged. Since 1989 at least twelv e
What will be needed to interpret saved bit-streams? have been published. … we need to re-think how we are
approaching research … [and] need to engage … researchers in
What can be done to make the content trustworthy? this process, and especially those with a strong computing science
and engineering background.” [Ross, ECDL 2007 keynote address]
How can convenient software support be implemented?
We sketch a solution to every technical challenge
H.M. Gladney , Economics and Engineering for Preserving Digital Content submitted for publication, December
,
2007. Draft available at http://home.pacbell.net/hgladney/LDPeng.pdf
.
†
Adapted from a March 2006 U.S. NIST workshop presentation

© 2008, H.M. Gladney, HMG Consulting http://home.pacbell.net/hgladney/ 1 © 2008, H.M. Gladney, HMG Consulting http://home.pacbell.net/hgladney/ 2

Digital Archiving and
Scope Limitations and Status
Digital Preservation
Digital Preservation h mitigation of the deleterious effects
of technology obsolescence, media degradation, and
“Archiving” consists of
fading human memory —effects that might impair the
value of digital object copies long after originals were
• Exploiting digital library technology
created.
• Tailoring client services
Technical components only (e.g., not addressing professional
education needed).
• Curating collections and catalogs
Increment to available document and digital repository
technology • Providing search services
• Sharing with other institutions
Preservation is a different topic than • Long-term preservation
repository management
These can be treated independently.

Information Producers’
End Users’ Requirements
Requirements
What might a descendant want of content stored
today? For any document preserved somewhere Citizens will want to preserve information without
in the world, help asking anybody’ s permission, with convenient tools
anybody’
and infrastructure:
• To find and retrieve a copy of its representing bit-string;
• To use the content as its producers intended; • To preserve any content, doing so that descendants
• To decide whether the content is trustworthy; can use this content as specified in the prior slide;
• To submit such content to repositories that reliably
• To exploit embedded references to reliably identify and
promise to save it.
retrieve contextual objects recursively; and
• To achieve all this without technical difficulties.

L. Graham and P.T. Metaxas , quot;Of Course It's True; I Saw It on the Internetquot; Critical Thinki g in the Internet Era,
n
Comm. ACM 46(5), 70 -75, May 2003.


© 2006, H.M. Gladney, HMG Consulting 1

Scale of the Digital
Repository Institutions’
Preservation Challenge
Requirements
# of archivists ~ 5000 and stable
What technology will repository institutions want? In # of software engineers ~ 500,000 and stable
addition to perfect world digital library technology, they
>106 and growing
# of information producers
will want support for:
>101 0 and growing
# of digital objects
• Continuing their day -to-day content management without
What fraction is worth saving? Nobody knows!
disruption;
• Sharing content and metadata semi-automatically;
Implication: existing cultural repositories cannot keep up
• Scaling to immense numbers of large documents; and
without radical change
• Sharing preservation effort to avoid costs & burdens.
Plausible increases in repository funding don’t change this
Even if the above estimates are incorrect by a factor of 10

Implications of the Scaling TDO† Approach
Challenge Divide and conquer
Assume the worst and mitigate its potential effects
Ø
Preservation of a significant fraction of what’s
what’ Repositories are lost. Institutions disappear.
Ø
created can be achieved only by: Clever rogues alter information feloniously.
Ø Disgruntled or careless employees defeat procedures.
• Automating everything possible;
Ø The last guy with critical information is dead.
• Shifting preservation work from archivists to
Starting with a “worst case” solution, find least
information creators;
expensive means for simple, common cases.
• Embedding preservation preparation into
Focus on digital objects.æ
existing document processing; and
†
TDO h Trustworthy Digital Object
• Effective partnership of archivists with SW æ
End users don’t care how archives work, just that they deliver what was stored!
engineers and information producers.

Digital Archiving Models, Representation Preservation: a special case of Communication
Languages and Standards
Information consumer cannot obtain producer’s explanations.
Canonical representation for archiving? TDO (Trustworthy Digital Object) structure

How to compress data? Not a digital preservation topic.

How to manage interoperability? Current EDP standards.

How to prepare content for archival (1) Add metadata that archivists want.
submission. (2) Encode bit -strings to be durable.
(3) Sign and seal content packages.

Trustworthiness of archived information? Durable Evidence (this talk)?

Role of standards in information packages? Starting point for Durable Encoding.
OAIS packages and submission interfaces? TDO format (this talk)?

Taxonomies, Thesauri and Ontologies Not a preservation topic.

Talk about the users, not about repositories!
Achieving Semantics Interoperability “the Holy Grail of knowledge management ”

Distinguishing current needs is politically pragmatic.


OAIS Repository
An Information Communication Model
(the conventional focus)?

Can describe transformations objectively, except for 0è1 and 9è10.
What the information consumer receives should not depend on the
transmission path from the information producer. (Implies an OAIS
DIP to OAIS SIP relationship.) ?

Digital Object (or a Collection of Anything)?
Part of the Context: Nested Digital Repositories
Tightly bound
package.
Including
metadata.
Linked schema
& standards.
Nested
versions.
Using ternary
relationships.
Saving every
important
version.
Recursive and
grounded.

Durable Evidence for Info. Packages
Bit-string Durability: Replication (as for books)?
UUIDs for digital objects, for material objects, and for object collections
LOCKSS* successful, but embeds too much policy
Avatar implementing rules for replication Relationships whose elements are identifiers and scalars
Certificates with recursion grounded in a few institutions
è Generalize
Design rules for Share keys by personal connections
replication
All packaged with XML
Design rules for
access control Signed and sealed just as
in the age-old practice
(Japanese ukiyo-e)?

*Reich & Rosenthal, LOCKSS: A Permanent Web Publishing and Access System , D-Lib Magazine 7(6), 2001.
Gladney , Trustworthy … Objects: Evidence After Every Witness is Dead, ACM Trans. Info. Sys. 22(3), 406 -436, 2004.



Durable Encoding for Content Blobs
Durable Encoding for Content Bit-strings
Starting point is a few widely understood standards
Ø Unicode/UTF, small XML core, BNF, ASN.1, identifier syntax
Universal Virtual Machine (enhanced Turing Machine)?
Ø Bit-addressed, any number of registers, condition codes
Ø Segmented memory —some segments private, others shared
Ø Input/output streams,
Ø Still needs multiprogramming and real -time instructions
Need one UVC emulator for each HW architecture Objective: render for human intelligibility and/or
Need one UVC application for each file type (~1000 types)? Make available to be program input
One UVC program for every file type (~1000 world- wide)?
Don’t save anything irrelevant, e.g., O/S portions
One Restore Application per future computer architecture
Save the original file together with UVC application
Problem with proprietary formats, e.g., MS Word files
Lorie & vanDiessen, A Universal Virtual Computer for long-term preservation,IBM Research Report, RJ 10338, 2005.
Gladney & Lorie Trustworthy … Digital Objects: Durable Encoding… , ACM Trans. Info. Sys. 23(3), 299-324, 2005.


Properties of TDO Methodology Summary and Status
Digital Preservation h mitigation of the deleterious effects
Format migration (which is error-prone) is not required
of technology obsolescence, media degradation, and
Consumers can evaluate authenticity without human fading human memory .
administrative help
Metadata-to-object dissociation rare and detectable Every conceptual challenge answered; no criticism received.
Engineering design is available.
Insensitive to Internet security risks
Ø Plug-in compatible with any digital library software package
Links have authentication codes for reliability
Ø Compatible with existing & emerging standards
Implementable without disrupting existing repositories
Conformance with regulatory requirements will be easy Next steps: building a prototype and running a pilot
Implementation easy (based on XML and text editors)?
Scalable


References Recommended Epistemological Texts
Ludwig Wittgenstein, Tractatus Logico-Philosophicus ,
HMG, Preserving Digital Information, Springer Verlag, 2007.
1921; also Philosophical Investigations, 1953.
HMG, Digital Preservation in a National Context: Questions
Rudolf Carnap, Logical Structure of the World, 1926.
and Views of an NDIIPP Outsider, D-Lib Mag. 13(1), 2007.
Gilbert Ryle, The Concept of Mind, 1949.
http://www.dlib.org/dlib/january07/gladney/01gladney.html
S. Ross, Digital Preservation, Archival Science and Ernst Cassirer, The Problem of Knowledge: Philosophy,
Methodological Foundations for Digital Libraries , 11th Science, and History Since Hegel, 1950.
European Conference on Digital Libraries, 2007. Michael Polanyi, Personal Knowledge: Towards a Post-
http://www.ecdl2007.org/Keynote_ECDL2007_SROSS.pdf
Critical Philosophy, 1958.
HMG, Economics and Engineering for Preserving Digital
Willard Van Orman Quine, Word and Object, 1960.
Content, to appear 2008.
J. Alberto Coffa, The Semantic Tradition from Kant to
http://home.pacbell.net/hgladney/LDPeng.pdf
Carnap to the Vienna Station, 1991.



Backup Slides A Practical Definition for “Authentic”
• Given a derivation statement R, “V is a copy of Y ( V=C(Y) )”,
• a provenance statement S, “X said or created Y as part of event
Z”, and
• a copy function, “C(y) = Tn (… (T2( T1(y) ))), ”
• we say that V is a derivative of Y if V is related to Y according to R.
• We say that “by X as part of event Z” is a true provenance of V if R
and S are true.
• We say that V is sufficiently faithful to Y if C conforms to social
conventions for the genre and for the circumstances at hand.
• We say that V is an authentic copy of Y if it is a sufficiently faithful
derivative with true provenance.


Trustworthy Digital Object (TDO)? Essential & Accidental Information
What ’s essential depends on what somebody specific
wants to accomplish (information producer or consumer).
Accidental content cannot be avoided.

Imperfect knowledge capture is a fact of life; many
examples in engineering specifications.

Software Layering and Rules Administration
Schema for a Digital Collection



Some Conclusions
Preservation of manufacturing data does not present
unique technical challenges è look at what members of
other disciplines are writing and saying.
But manufacturing does have specialized file formats and standar ds.
Divide and conquer è treat preservation as content mgmt.
increment.
Focus on data structure first; later on data management.
Be sensitive to the teachings of scientific philosophy.
Age-old procedures for paper have digital counterparts.
Big challenge: persuading information providers to write
metadata
We have an architecture, and are ready to start
software design.

© 2008, H.M. Gladney, HMG Consulting http://home.pacbell.net/hgladney/ 31


Ldp Executive Slides

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Ldp Executive Slides

Similar to Ldp Executive Slides (20)

Recently uploaded

Recently uploaded (20)

Ldp Executive Slides