Taming the Monster Digital Preservation Planning and Implementation Tools Dorothea SaloPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/ One System, One LibraryWorldIslandInfo.com / CC-BY 2.0 2 June 2011
Why is this so scary?Photo: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
Isn’t this just as scary?Photo: “News Paper Origami Dragon Monster”http://www.ﬂickr.com/photos/epsos/3777343342/epSos.de / CC-BY 2.0
Yet we persevere.Photo: “News Paper Origami Dragon Monster”http://www.ﬂickr.com/photos/epsos/3777343342/epSos.de / CC-BY 2.0
DIGITAL IS NO DIFFERENT.Photo: “559 - The Matrix - Seamless Texture”http://www.ﬂickr.com/photos/zooboing/4335531915/Patrick Hoesly / CC-BY 2.0
Many of the same ideas apply... • Planning and policy • Risk assessment • Risk management • (knowing that we can’t save everything) • Materials quality matters! • Problem discovery and remediation • Crisis management • Chief problems: staﬀ, $$$, organizational commitmentPhoto: “Where I Teach”http://www.ﬂickr.com/photos/eklektikos/2541408630/Todd Ehlers / CC-BY 2.0
Planning and assessment toolsPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
Scene-setting • Rosenthal, David. “Requirements for Digital Preservation: a Bottom-Up Approach.” • http://www.dlib.org/dlib/november05/rosenthal/ 11rosenthal.html • If you’re new to this, or trying to ﬁnd your feet, this is the best short introduction I know. • The list of threats is outstanding.Photo: “Bottoms Up! - Duck; San Anton Gardens, Malta”http://www.ﬂickr.com/photos/foxypar4/3123113762/John Haslam / CC-BY 2.0
TRAC• “Trusted Repository Audit Checklist”• Despite the name, covers a LOT more than the technology! ! • Budget • Staﬃng • “designated communities”• CRL will audit you, if you like • (don’t, unless you’re really serious!)• http://catalog.crl.edu/record=b2212602~S1
DRAMBORA• Digital Repository Audit Method Based on Risk Assessment• A “self-test,” if you will. • DRAMBORA is equally good as a pre- or post-test.• Personally, I prefer DRAMBORA to TRAC, ! especially for those just starting out.• http://www.repositoryaudit.eu/ • (registration required for toolkit access)
Coping with ﬁle formatsPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
The one acronym you need to know: FITS• “File Information Tool Set” • (you need to know this; otherwise it’s hard to Google)• Wrapper for several ﬁle-format detector software packages• Intended to be baked into other software• It’s early days yet! • (This means you can’t always trust what the tools tell you, especially when they’re telling you about errors.)
What’s this ﬁle?• wotsit.org “The Programmer’s File and Data Resource”• Directory of ﬁle extensions• When in doubt: open in a browser or text editor and see what you get. • N.b.: Microsoft Word is NOT a text editor!
Solving the geographic distribution problemPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
What problem, now? • The “all your eggs in one basket” problem. • If all your bits are on one server, and the server room is ﬂooded, or your town is nuked—oops. • Not the same as backups! • Don’t get me wrong, backups are important! • Backups are SHORT-TERM, and usually LOCAL. Geographic distribution (plus associated auditing) is intended for the long term. • Don’t forget auditing!Photo: “Nido”http://www.ﬂickr.com/photos/italintheheart/3679974298/Jorge Elías / CC-BY 2.0
LOCKSS• Lots of Copies Keeps Stuﬀ Safe! • (There is also Portico, but Portico only works with e‑journal content.) • Open-source software that handles replication and (some) auditing.• “Private LOCKSS network” • A group of institutions agrees to build a LOCKSS network just for the stuﬀ they’re interested in. • ASERL does this for ETDs. Many institutions (including UW-Madison) participate in a PLN for govdocs.
“The cloud” • Typical cloud-based storage services make NO promises they won’t lose your stuﬀ. • And for large quantities of data, bandwidth can become an issue. • And can they look at your stuﬀ? Should they be able to? • Some early movers in this market fading • Iron Mountain had to kill their service. • DuraCloud • trying to ﬁnesse this issue by negotiating tougher SLAs with cloud-storage providersPhoto: “Sky View From Humboldt Park”http://www.ﬂickr.com/photos/purpleslog/2589612577/Purple Slog / CC-BY 2.0
Repository and digital-library platformsPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
Another friendly word of advice: DON’T CHASE THE SHINY.Photo: “Sparkle Texture”http://www.ﬂickr.com/photos/abbylanes/3214921616/Abby Lane / CC-BY 2.0
Digital-library software • Is almost always VERY BAD at digital preservation! • (most packages don’t even try!) • So if a ﬁle gets corrupted on the server, or whatever... no warnings, no restore, nothing. Also, provenance? Who needs provenance? Event tracking? What’s that? • I’m not saying don’t use it. I’m saying that it doesn’t solve this problem. • In fact, if you’re using this software, you need to solve this problem FOR IT.Photo: “National DIGITAL Library”http://www.ﬂickr.com/photos/schex/193912573/Jesse Schexnayder / CC-BY 2.0
Institutional-repository software • Is SHOCKINGLY bad at digital preservation! • (Though sometimes better than most DL software.) • Examples • Hosted/commercial: Digital Commons (BePress), ContentDM, DigiTool • If you go hosted, you’d better ask about their digital- preservation practices! • Open-source: EPrints, DSpace, FedoraPhoto: “IMG_0668”http://www.ﬂickr.com/photos/12967790@N00/66531124Robert / CC-BY 2.0
A new approach: curation microservicesPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
Do we really needPhoto: “giant crystal blob”http://www.ﬂickr.com/photos/a_of_doom/527905701/A of DooM / CC-BY 2.0 THE BLOB?
How about a jigsaw puzzle instead? • Break the digital-preservation problem down into parts. • Code up each part, making sure that it plays nicely with other parts. • lots of nice APIs! • which means other software can adopt/adapt microservices as well! • Put parts together as you need them.Photo: “Lapsana Apogonoides Puzzle”http://www.ﬂickr.com/photos/gdesigneralex/2313092112/gdesigneralex / CC-BY 2.0
California Digital Library• Pioneering this approach• Has open-sourced code for microservices• Has added microservices together to build its “Merritt” storage/repository service
Escaping the silos: Fedora CommonsPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
What is Fedora Commons?• Blueprints and foundation, not the whole house (analogy credit to Peter Gorman)• You build the house you want!• Or you build condominiums on the same foundation. • Need diﬀerent user interfaces for diﬀerent materials? • Need diﬀerent structures and behaviors? • No problem! Fedora can handle that.• (have I run this analogy into the ground yet?)
We had this... Diagram courtesy of Peter Gorman.
We are building this. Diagram courtesy of Peter Gorman.
E-records managementPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
Axioms• Records management is about policy and procedures. • If your policy doesn’t ﬁt with their procedures, guess what wins? Choose battles wisely.• There is never enough storage space.• Nobody cares until there’s a crisis.• Software will not save you... but it might help! Photo: “The Never Ending Math Problem” http://www.ﬂickr.com/photos/acidwashphotography/2967752733/ d3 Dan / CC-BY 2.0
Duke Data Accessioner• Accessioning tool for digital data • use case: J. Important Scholar dumps her hard drive on your desk, expects you to cope• File migrator, metadata manager, GUI, plugins (e.g. for ﬁle-format detection)• Bit rough, but in production use. • http://library.duke.edu/uarchives/about/tools/data- accessioner.html
Archivematica• Soup-to-nuts records management and digital preservation tool. • Evaluation and accessioning all the way through preservation actions. (Oddly, they seem to be missing disposal... but they’re in alpha, so...)• Open source • Runs on a Linux server; RMs and archivists log in to GUI application remotely.• Normally I hate and fear silos, but this one is smartly built on microservices.
Practical E-Records• Weblog by Chris Prom and protegés• Tool evaluations, conference-session writeups, essays on praxis• Best reading out there for the do-it- yourselfer• If you’re not reading it, why not?• http://e-records.chrisprom.com/
Last thoughtsPhoto: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0
If you can’t do everything... Image: “Confused” http://www.ﬂickr.com/photos/kristiand/3223044657/ Kristian D. / CC-BY 2.0 that’s okay. Who can?
DO SOMETHING.Photo: “Came hame háááá!”http://www.ﬂickr.com/photos/kristiand/3223044657/Guirí R. Reyes / CC-BY 2.0
The worst threat?INACTION. Photo: “Fatty’s role model” http://www.ﬂickr.com/photos/cloudzilla/4910616774/ cloudzilla / CC-BY 2.0
Thank you! This presentation is available under a Creative Commons 3.0 United States license.Photo: “Happy Easter, to my Peeps”http://www.ﬂickr.com/photos/76074333@N00/449028423/WorldIslandInfo.com / CC-BY 2.0