Digital preservation and institutional repositories

  • 7,044 views
Uploaded on

Delivered at the Summer Institute for Data Curation at the University of Illinois, 21 May 2009.

Delivered at the Summer Institute for Data Curation at the University of Illinois, 21 May 2009.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • You are fantastic. This slideshow and your Roach Motel article have been super helpful for the paper I'm currently slogging my way through (Yay, comps!!) And, have provided me with something new and interesting to think about for a career path. (1.5 semesters left and I still don't have a clue as to what sort of librarian I want to be...)

    Thanks!
    Are you sure you want to
    Your message goes here
  • thank you!
    Are you sure you want to
    Your message goes here
  • Quick comments (from someone who actually does digital video preservation!):

    - BWF (Broadcast Wave Format) is the standard for audio preservation, and has been for some time now. It is lossless, and specified by the European Broadcasting Union.

    - JPEG2000 provides for both lossless and lossy compression.
    Are you sure you want to
    Your message goes here
  • A quick note: for lossless audio, FLAC is the open-source, non-proprietary counterpart to Ogg Vorbis. Ogg is the open-source version of the MP3, and as you mention OGG is certainly less supported than MP3 for things like web playback. FLAC, however, is very well supported by a number of playback softwares, but more importantly can be used to quickly create access derivatives of any bitrate and format - and certainly much smaller than a WAV.
    Are you sure you want to
    Your message goes here
  • Few comments from the EPrints perspective:
    Slide 61: EPrints has an internal default XML schema which is not dublin core, thus is very flexible and designed for you to edit. You can export this data in dc, mets, mods etc by writing a plugin which matches the internal schema data to that you wish to export - lots already written. The same can be applied for importing data.
    Slide 69: EPrints doesn't set what the user interface looks like and we have worked hard to ensure this is the case. Some examples: http://www.edshare.soton.ac.uk/, http://eprints.ucl.ac.uk/, http://strathprints.strath.ac.uk/. http://eprints.iisc.ernet.in/, http://languagebox.eprints.org/. I've tried to avoid Southampton ones here, feel free to browse around them and see how people have customised them.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
7,044
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
166
Comments
8
Likes
10

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide














































































































































Transcript

  • 1. Institutional repositories for the digital arts and humanities Dorothea Salo University of Wisconsin dsalo@library.wisc.edu
  • 2. Preservation for the digital arts and humanities Dorothea Salo University of Wisconsin dsalo@library.wisc.edu
  • 3. Preservation and institutional repositories for the digital arts and humanities Dorothea Salo University of Wisconsin dsalo@library.wisc.edu
  • 4. And I said... ... you’re giving me how much time for this?
  • 5. Threat model • “Preservation” means nothing unmodified. • This is why it becomes such a bogeyman! • Two things you need to know first: • why you’re preserving what you’re preserving, and • what you’re preserving it against. • Your collection-development policy should inform the first question. • Your coll-dev policy doesn’t include local born-digital or digitized materials? This is a problem. Fix it. • The second question is your “threat model.”
  • 6. What is your threat model for print?
  • 7. Homelessness
  • 8. Water
  • 9. Bad materials
  • 10. Flora and fauna
  • 11. Physical damage
  • 12. Loss or destruction
  • 13. Armageddon
  • 14. Why did I just make you do that? • I’m weird. • I’m trying to destroy the myth that any given medium “preserves itself.” • Media do not preserve themselves. People preserve media —or media get bizarrely lucky. • We need not panic over digital preservation any more than we panic about print. • Approach digital preservation the same way you approach print preservation. • Strategically: this approach helps your colleagues get a grip, too. Your colleagues may well be the biggest barrier to digital preservation in your library!
  • 15. In your groups... List important threats to digital data.
  • 16. Physical medium failure
  • 17. “Bitrot”
  • 18. File format obsolescence
  • 19. Forgetting what you have
  • 20. Forgetting what the stuff you have means
  • 21. Rights and DRM
  • 22. Lack (or disappearance) of organizational commitment
  • 23. One word: Geocities.
  • 24. Ignorance ? • “It’s in Google, so it’s preserved.” (Not even “Google Books!”) • “I make backups, so I’m fine.” • “I have a graduate student who takes care of these things.” • “Metadata? What’s that? I have to have it?” • “Digital preservation is an unsolvable problem, so why even try?” (I’ve heard this one from librarians. I bet you have too.)
  • 25. Apathy
  • 26. Armageddon
  • 27. Salo’s needs pyramid Less Less immediate Fidelity tractable to original Usability Format viability Bitrot Physical medium issues More More immediate Acquisition issues tractable
  • 28. Mitigating the risks
  • 29. But first, a word about failure • “We can’t save everything digital!” • Well, no, we can’t. • We can’t save everything printed either. • That’s no excuse, in either medium. Why do we let it be one for digital materials? • Yes, we will lose some stuff. That’s life in the big city. Dive in anyway.
  • 30. And a word about scale • Many of those currently panicking about digital preservation are thinking about huge scales. • At some repository size, bitrot happens faster than you can detect and fix it. • Last I heard, this was somewhere in the exabyte range. • We’re not. So let’s relax about some of this stuff. At our scale, many problems are solvable. • Unless your problem is digital video. Good luck with that. • Our scale problems happen on the front end, as we’ve been learning this week.
  • 31. Physical medium failure • Gold CDs are not the panacea we thought. • They’re not bad; they’re just hard to audit, so they fail (when they fail) silently. Silent failure is DEADLY. • How long will hardware be able to read them? • ALL such physical media are risky, for the same reasons! • Current state of the art: get it on spinning disk. • Back up often. Distribute your backups geographically. Test them now and then. • Consider a LOCKSS cooperative agreement. Others have. • Any physical medium WILL FAIL. Have a plan for when it does.
  • 32. Bitrot • Sometimes used for “file format obsolescence.” • I use it for “the bits flipped unexpectedly.” • Checking a file bit-by-bit against a backup copy is computationally impractical for every day. • Though on ingest it’s a good idea to verify bit-by-bit! • Checksums • A file is, fundamentally, a great big number. • Do math on the number file. Store the result as metadata. • To check for bitrot, redo the math and check the answer against the stored result. If they’re different, scream. • Several checksum algorithms; for our purposes, which one you use doesn’t matter much.
  • 33. File format obsolescence • When possible, prefer file formats that are: • Open/non-proprietary. (If a software vendor goes out of business, does their format?) • Documented • Standardized, non-patent-encumbered • In widespread use. (If the format dies, lots of people have incentive to solve the problem.) • For text, non-binary • For everything else, lossless rather than lossy • For compound objects, compound documents rather than embedded • Realistically? We often have to take what we’re given.
  • 34. Lossless? Lossy? What? • Essential tradeoff: quality and fidelity vs. file size • Clipping information out makes the file size smaller! But once it’s gone, it’s gone. • Tremendous problem with video. Lossless video formats are HUGE. • Lossy image formats: JPEG, JPEG2000 (much less so) • (more or less) Lossless: TIFF, PNG, GIF • Compression may be lossless or lossy. Find out!
  • 35. Example: JPG
  • 36. Audio formats • I am NOT going to talk about codecs vs. container formats. Consider it homework. • No ideal choice here; lossless formats are patent-encumbered and/or proprietary • WAV and AIFF are okay. Ogg Vorbis is ideal, but nobody supports it. • mp3: if you must, it’s lossy.
  • 37. Migration vs. emulation • Migration: move the file to a new format • Don’t throw away your original! You may have made the wrong migration decision. • Not necessarily a lossless process. (Fonts!) • Emulation: create a modern hardware/software environment that can deal with the old format • For some cultural artifacts such as games, this is the only reasonable option. • Emulation advocates make big claims that I’m not sure they can back up. Proceed with caution.
  • 38. Normalization • Migration of a dataset toward a well-defined target. • “Treat the same thing the same way.” • E.g. census data... define a set of data tables, move all data into them. • Great for interoperability and preservation! • Pitfall: “the same thing”? • Humanities: TEI is a de facto normalizer for humanities textual data. • (Other XML formats in other fields: e.g. ChemML, NLM DTD.)
  • 39. Problem: BEHAVIOR. • Migration can preserve information content and (often but not always) appearance. • Preserving interaction patterns is much harder! • E.g. a web page containing Javascript • Or a database with a query engine • Or an applet or Flash object • Or a collection whose interactions are based on an obsolete software system. (DynaText anyone?) • Hard problem. No obvious solutions; certainly no easy ones.
  • 40. When is a PDF not a PDF? • When it’s a .doc with the wrong file extension • When there’s no file extension on it at all • When it’s so old it doesn’t follow the standardized PDF conventions • When it’s otherwise malformed, made by a bad piece of software. • How do you know whether you have a good PDF? (Or .doc, or .jpg, or .xml, or anything else.)
  • 41. File format registries and testing tools • JHOVE: JSTOR/Harvard Object Validation Environment • Java software intended to be pluggable into other software environments • Answers “What format is this thing?” and “Is this thing a good example of the format?” • Limited repertoire of formats • PRONOM/DROID + GDFR = Unified Digital Formats Registry
  • 42. Forgetting what you have • Absolutely pernicious problem. We don’t know what we have to begin with! • Do you know how much Faculty Stuff is scattered throughout your institution’s .edu domain? Me neither. But I know it’s a lot. How much of that is irreplaceable? • We’re also bad at labelling and tracking what we have. • No easy answer to this one; the solution lies in a complete praxis reinvention. • Yeah. Good luck with that.
  • 43. ... but I thought you meant in libraries, Dorothea! • Come on, we’ve solved that one: Metadata! • Once it’s in the library, it’s probably fine. The real problem is all that Other Stuff Out There. • This is a collection-development problem and should be treated as one. • Don’t dump it on some poor “digital preservation librarian!” That flat out doesn’t scale. • Don’t make the mistake of drawing thick lines around “our stuff” and “their stuff.” Like it or not, our coll-dev universe has moved beyond what’s published and what’s canonically “library.”
  • 44. What the stuff you have means • Collect whatever it takes to answer this question: • If the owner of this material were hit by a bus tomorrow, what would be needed for others to use it? • Nasty discipline-specific problem. • This is what the NARA/RLG Trusted Digital Repository checklist is aiming at with “designated community.” • Where NARA/RLG goes off the rails is assuming you have to go through this exercise with EVERYTHING YOU HAVE. • Data-dictionaries, algorithms, specifications, tech metadata, whatever it takes. Use common sense!
  • 45. Rights and DRM • Not having IP rights to something may mean you can’t preserve it. • Brian Lavoie writes well about this problem. • Copyright law and its exceptions haven’t caught up to the digital age! • Third-party services (e.g. blogs, ITunesU, Slideshare) are a headache here. • DRM means that no matter the rights situation, you’re stuck. • PDFs: Users turn on “security” features. This is DRM. Tell them not to do that! • Huge headache with third-party services, again.
  • 46. ... and other hassles • Privacy, confidentiality, and human-subject research issues • Think “we’re the humanities; IRBs don’t happen to us”? Think again. One word: FOLKLORE. • Third-party copyright • Campus musical or dramatic performances • Issues of cultural sensitivity, heritage, repatriation • You need a dark (or at least dim) archive if you’re serious about digital preservation. There is no way around this. Sorry.
  • 47. Organizational commitment • There is only one answer: POLICY. • Unfortunately, it’s not a quick, easy, or uncomplicated answer. • Digital preservation costs money. • People in high places are scared of it. • It requires process and staff change. • You have to make the case. And then make it again. And again. Until they get it! • Where I am, Somebody Else’s Problem fields are everywhere around this issue.
  • 48. You are probably the preservation option of last resort. Be prepared for anything excluded from your policy to disappear.
  • 49. When organizations fail • Remember Geocities? We’re worse. • Mellon: Can’t make a list of its funded on-the-web projects, because most of them are GONE. G-O-N-E. • We do not, as a profession, have a safety net for each others’ projects and materials. • This is, frankly, unconscionable. • I don’t know how to fix it; I am just warning you that project rescues are and will continue to be necessary. • Institutional boundaries are a barrier here.
  • 50. Great policy guidance • Policy-making for research data in repositories: a guide • http://www.disc-uk.org/docs/guide.pdf • Practical data management: a legal and policy guide • http://eprints.qut.edu.au/archive/00014923/01/ Microsoft_Word_-_Practical_Data_Management_- _A_Legal_and_Policy_Guide_doc.pdf • Australian, so take “legal” with a grain of salt • Guide to social science data preparation and archiving • http://www.icpsr.umich.edu/ICPSR/access/dataprep.pdf
  • 51. Summary: the OAIS model • “Reference model” for archival systems • All theory, no praxis, by design. (Because praxis changes!) • Four parts • Vocabulary • Data (and interaction) model • Required responsibilities of an archive • Recommended functions (in the computer-programming sense) for carrying out those responsibilities • My favorite distillation: Ockerbloom • http://everybodyslibraries.com/2008/10/13/what- repositories-do-the-oais-model/
  • 52. Institutional repositories
  • 53. For our purposes... • We’re talking about the software. • I’m not going to rant (much) about what IRs are for or how they’re run. • If you want that, read Roach Motel. Better yet, read Palmer et al. 2009. • We’re interested in the application (or lack thereof ) of IRs to data curation in the arts and humanities. Right? Right. • I’m not afraid of the technical, and neither should you be.
  • 54. IR software • Open source • Fedora Commons: http://fedora-commons.info/ • DSpace: http://dspace.org/ • EPrints: http://eprints.org/ • Commercial • ContentDM: http://contentdm.com/ • VTLS/Vital: http://www.vtls.com/products/vital • Hosted • ContentDM: http://contentdm.com/ • BePress: http://bepress.com/ • Open Repository (based on DSpace): http:// www.openrepository.com/ • Digitool: http://www.exlibrisgroup.com/category/ DigiToolOverview
  • 55. In your groups... Please brainstorm common examples of A&H digital content requiring preservation.
  • 56. Common A&H use-cases • Image collections • Page-scanned books (with or without OCR) • Marked-up books • Theses and dissertations • Website preservation • Audio and video • Complex multimedia • Database (linguistic, geographic...) • Software
  • 57. In your groups... Please brainstorm how you and your patrons expect to use and interact with these genres of data. Make a list of verbs.
  • 58. What they’ll tell you on al ituti st . in y ere!” an itor ve os thing th ha ep “W e r every ut ca np You
  • 59. How you must not respond
  • 60. The IR content use-case • A research paper • In a single file; possibly more than one format available • Is not related to any other item in the history of ever • The user can download it, and... um... just download it, really.
  • 61. How much of our stuff does that work for? • Image collections • Page-scanned books (with or without OCR) • Marked-up books • Theses and dissertations • Website preservation • Audio and video • Complex multimedia • Database (linguistic, geographic...) • Software
  • 62. One user interface does not fit all
  • 63. One metadata standard does not fit all • EAD • METS • The simple fact is that • VRA Core EPrints and DSpace do • MODS Dublin Core, METS, and • TEI Header nothing else natively. • Dublin Core This is purely inadequate for humanities data • MARC curation. • ... the beat goes on.
  • 64. One file format does not fit all • Yes, we have to take what we get. • With discrete files, most IR software is fine. • Forget about streaming audio/video. • DSpace is good with static websites. • For other composite objects, you’re in trouble. • For anything like a database, you’re in trouble.
  • 65. The DSpace/EPrints view of the universe • Communities and collections • “EPeople” • must be given explicit permission to add or edit materials • Metadata entry forms • DSpace: fields configurable by collection • EPrints: auto-configures fields based on content type • Files/bitstreams • Many permitted per item; must upload one by one in DSpace! • Get friendly with the DSpace batch importer. You’ll need it.
  • 66. The Fedora view of the universe • You can do anything at all with anything at all as long as you’re willing to tell Fedora how to do it. Infinite flexibility! But also infinite responsibility. • “Content model:” what’s in this thing? • “Service:” what should the user-interface do with what’s in this thing? • Metadata, relationships, stuff
  • 67. Can you use Fedora for an IR? • Yes, but not alone; you need all the Content Models and Services bolted on top. • Try Islandora or Muradora. Fez is a last resort; it acts like DSpace, and this is not a good thing. • Even if you can’t build a real Fedora digital library now, you may not be able to do so in future if you stick with DSpace... • ... but the Fedora/DSpace merger may change things.
  • 68. What is this FOXML stuff anyway? • Think of it as the Fedora batch-import format. • It’s complex! But it can absorb any amount or type of XML metadata or data, which is really quite nice.
  • 69. Summing up • Out-of-the-box IR software will handle some A&H data-curation jobs adequately... • ... but by no means all of them. • If you need sophisticated UI, bite the bullet and go with Fedora. Islandora and Muradora make Fedora simpler for simple things than it once was. • If you don’t need sophisticated user-facing UI, go with EPrints. • DSpace is a loser choice.
  • 70. Credits • Watch: http://www.flickr.com/photos/fdecomite/406635986/ • Wet book: http://www.flickr.com/photos/dno1967/2979040762/ • “Bookworm and Bug Juice”: http://www.flickr.com/photos/modestospeed/576659116/ • Moldy books: http://www.flickr.com/photos/umjanedoan/496656416/ • Damaged book: http://www.flickr.com/photos/donabelandewen/3375108358/ • Carnegie library: http://www.flickr.com/photos/jhoweaa/436923541/ • Floppy box: http://www.flickr.com/photos/rintakumpu/2684989757/ • Floppy art: http://www.flickr.com/photos/bludgeoner86/2507833950/ • Bitrot: http://www.flickr.com/photos/raver_mikey/2865543940/ • Escape the ring: http://www.flickr.com/photos/hydropeek/2611071166/ • Obsolete grownups: http://www.flickr.com/photos/nietsdoener/1091201075/ • Confusion: http://www.flickr.com/photos/flavinsky/3411791256/ • Confusion II: http://www.flickr.com/photos/demibrooke/2550349404/ • Axeman: http://www.flickr.com/photos/27888428@N00/3163030403/ • Lazy dazy: http://www.flickr.com/photos/hmk/2742398590/ • DRM/Orwell: http://www.flickr.com/photos/jbonnain/523672080/ • Mushroom cloud: http://www.flickr.com/photos/nicholas_t/543334336/ • Pollock: http://www.flickr.com/photos/redneck/215447253/
  • 71. Thank you! • This presentation is available under a Creative Commons Attribution 3.0 United States license. • Please remember to credit images if you reuse individual slides. Thank you!