Migrating from OCLC's Digital Archive to DuraCloud


Published on

Presented by Lisa Gregory at Best Practices Exchange, December 2012. This presentation compared OCLC's Digital Archive to DuraCloud, and the process of migrating storage from one to the other.

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Migrating from OCLC's Digital Archive to DuraCloud

  1. 1. State Library • Part of the North Carolina Departmentof North of Cultural ResourcesCarolina • Work closely/pool resources with the State Archives • Digital Information Management Program
  2. 2. CONTENT STAFF State Publications Genealogy Research ~ 4.75 FTE North Caroliniana Local server CONTENTdm (state-supported)Connexion Digital Import Offsite storage (vendor)SYSTEMS STORAGE
  3. 3. Born-Digital 3.25 Digitized .75CONTENTdm CONTENTdm Connexion Project Client Local Storage .75 Remote Storage
  4. 4. CONTENT• We preserve access and master copies• 1.27 TB, 162,000+ files• Mostly .tif, .pdf, .jpg, .txt
  5. 5. CONTENTFile structure by “project” admindocs fulltext images_access images_master images_processed metadataNaming convention pubs_serial_annualreportclean2005.pdf gen_statefair_lifecharacterthomasruffin1871_0001.tif
  6. 6. Local storage • managed by department-wide IT • includes working & preservation content • server is shared, but our directory is restricted • daily incremental backups STORAGE
  7. 7. OCLC’s • Began using in 2008Digital • Web interface for accessArchive • FTP or automatic uploads • Integrated with CONTENTdm • Detailed reporting, broken out by CONTENTdm collection • Fixity checks, virus checks
  8. 8. +• Integration with • Integration with CONTENTdm CONTENTdm• Fixity checks and • Finding and retrieving items virus scans • Manifest/batch• Responsive upload requirement support • Vendor-side error• Extensive reports reporting • Verifying storage contents
  9. 9. DuraSpace’s • Began using in 2012 • Web interface for access • Web interface or client-side tools for upload • Content Management System-agnostic • Fixity checks
  10. 10. +• Presentation is like a traditional gui file manager • Searching• Can designate spaces, • Sorting permissions• Can make a space public • Verifying storage• Powerful upload tools contents• Fixity scans • Overwriting isn’t• Robust reporting• Easy to get content out hard to do• Choice of storage services • Batch delete• VERY collaborative support• Non-profit • MD5
  11. 11. PREPARATION CONTENTdm ? Local OCLC DA server
  12. 12. CONTENTdm Local server1. Exported metadata from CONTENTdm2. Exported file names from local server3. Bashed preservation file names, checksums4. Identified and recovered missing files
  13. 13. 1. Exported metadata Onerous to from CONTENTdm impossible2. Exported file names Easy from local server3. Bashed preservation Easy but time file names, checksums consuming4. Identified and Easy-ish recovered missing files
  14. 14. 1. Exported metadata Onerous to from CONTENTdm impossible• OCLC had to provide export for largest & most critical collection• 363 MB tar file -> 18 x 100+ MB csv files• Added frustration: metadata for compound objects v. multi-page pdfs
  15. 15. 2. Exported file names Easy from local server1. Bashed preservation Easy but time file names, checksums consuming• Spreadsheet gymnastics• Manual review for filename/checksum inconsistencies
  16. 16. 4. Identified and Easy-ish recovered missing files• Missing from CONTENTdm? Added by librarians• Missing from local server? Request to OCLC or re-download from CONTENTdm
  17. 17. THE MOVE Local server DuraCloud1. Tested sync and upload tools2. Discussed spaces3. Ran sync tool on local preservation storage4. Ongoing maintenance: upload tool
  18. 18. 1. Tested sync and upload tools Easy• Helped determine flags to manage computer resources during sync• Verified logging output, permissions• Helped flesh out local workflow
  19. 19. 2. Discussed spaces Easy, and Interesting• Many spaces or few, to accommodate different workflows?• Assignment of permissions
  20. 20. 3. Ran sync tool on local Easy preservation storage• Ran continuously for 5 2/3 days• 94,177 items
  21. 21. 4. Ongoing maintenance: upload tool Easy-ish• Uploads done weekly and monthly• Upload tool used to avoid accidental overwriting• Have to create “mock” file structure
  22. 22. Working Staging – Limited Accessdirectory Local server StagingWorkingdirectory DuraCloudWorkingdirectory
  23. 23. Insights • Room for preservation metadata improvement • Working with full metadata dumps is problematic • Need for more automated monitoring for local storage • Integration with CMS not helpful unless FULL integration in other words: • Streamlined ingest = streamlined preservation
  24. 24. Still more • No, really: manual management and auditing is getting less feasiblethoughts • What is acceptable content loss? • What is acceptable preservation metadata error rate? • Responsiveness to enhancement requests should be figured into vendor choice • At 5 years out, PREMIS lite is just fine