Digital Presentation Best Practices: Lessons Learned From Across the Pond

3,338 views
3,235 views

Published on

Digital Presentation Best Practices: Lessons Learned From Across the Pond. Slavko Manojlovich (Associate University Librarian (IT) / Manager, Digital Archives Initiative Memorial University St Johns Canada) and Benoit Pauwels (Head, Library Automation Team, Université libre de Bruxelles Belgium)

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,338
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
46
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Digital Presentation Best Practices: Lessons Learned From Across the Pond

  1. 1. Digital Preservation Best PracticesLessons Learned From Across the Pond<br />Slavko ManojlovichAssociate University Librarian (IT) / Manager, Digital Archives Initiative<br />and<br />Benoit Pauwels<br />Head, Library Automation Team<br />Université Libre de Bruxelles<br />[with input from Michael J. Bennett, Digital Projects Librarian and Institutional Repository Coordinator, University of Connecticut]<br />
  2. 2. Outline<br />What is digital preservation?<br />Best practices information resources<br />Open Archives Information System (OAIS) <br />Preservation Planning<br />Digital Preservation in Action(Archivematica)<br />Digital preservation @ ULB<br />Our issues<br />
  3. 3. What is digital preservation?<br />Digital preservation is NOT digitization!!!!!!!!<br />
  4. 4. What is digital preservation?<br />Digital preservation is the series of actions and interventions required to ensure continued and reliable access to authentic digital objects for as long as they are deemed to be of value. This encompasses not just technical activities, but also all of the strategic and organisational considerations that relate to the survival and management of digital material.Source<br />
  5. 5. What is digital preservation?<br />Digital preservation is the series of actions and interventions required to ensure continued and reliable access to authenticdigital objects for as long as they are deemed to be of value. This encompasses not just technical activities, but also all of the strategic and organisational considerations that relate to the survival and management of digital material.Source<br />
  6. 6. What is digital preservation?<br />Disaster recovery strategies and backup systems are not sufficient to ensure survival and access to authentic digital resources over time.Source<br />
  7. 7. Digital preservation includes:<br />Digitized analogue content (easy)<br />Born–digital content (more difficult)<br />What is digital preservation?<br />
  8. 8. Recent example from Memorial University<br />Preserve faculty member’s research outputs from 1977 – present stored in a variety of formats. <br />“All of the above represents a vast resource which cannot be lost from the University”.<br />What is digital preservation?<br />
  9. 9. Best practices may not always be the best option for your organization:<br />British Library Microsoft Live Book Data Project<br />The DPT [Digital Preservation Team] have taken the view that since the budget for hard drive storage for this project has already been allocated, it would be impractical to recommend a change in the specifics as far as file format is concerned for this project...... JPEG 2000 files compressed to 70 dB PSNR for the preservation copy.<br />Source<br />Digital Preservation Best Practices<br />
  10. 10. Best practices may not always be the best option for your organization:<br />British Library Microsoft Live Book Data Project<br />The DPT [Digital Preservation Team] have taken the view that since the budget for hard drive storage for this project has already been allocated, it would be impractical to recommend a change in the specifics as far as file format is concerned for this project...... JPEG 2000 files compressed to 70 dB PSNR for the preservation copy.<br />Source<br />Digital Preservation Best Practices<br />
  11. 11. The National Gallery (UK) Preservation of Digital Photographs of the Collection<br /> The National Gallery has photographed their entire collection using a high-end digital MARC camera capable of capturing and rendering colour accuracy which is at least 5 times better than traditional photography. They have selected the proprietary raw camera output format for long-term preservation because it supports an advanced level of colour management. The company supporting the camera and associated software is very smalland is not a market leader.Source: Site Visit to National Gallery Photography Department, April, 2010.<br />Digital Preservation Best Practices<br />
  12. 12. The National Gallery (UK) Preservation of Digital Photographs of the CollectionThe National Gallery has photographed their entire collection using a high-end digital MARC camera capable of capturing and rendering colour accuracy which is at least 5 times better than traditional photography. They have selected the proprietary raw camera output format for long-term preservation because it supports an advanced level of colour management. The company supporting the camera and associated software is very smalland is not a market leader.Source: Site Visit to National Gallery Photography Department, April, 2010.<br />Digital Preservation Best Practices<br />
  13. 13. Eighth European Conference on Digital ArchivingGeneva, Switzerland / April 28 -30, 2010Source<br />Archiving 2010The Hague, Netherlands / June 1-4, 2010Note: Archiving 2011 – Salt Lake City (May, 16-19, 2011)Source<br />Best Practices Information SourcesConferences<br />
  14. 14. OR2010: The 5th International Conference on Open RepositoriesMadrid, Spain / July 6-9, 2010Note: Or2011 – Austin, Texas (June 7-11, 2011) Source<br />iPRES2010: 7th International Conference on Preservation of Digital ObjectsVienna, AustriaSeptember 19-24, 2010Source<br />Best Practices Information SourcesConferences<br />
  15. 15. Digital Preservation – The Planets WayLondon, UK / February 9, 2010Source<br />Digital Futures London 2010: From digitization to delivery King’s Digital Consultancy Services (KDCS)King’s College, London, UK April 19 – 23, 2010Source<br />Best Practices Information SourcesWorkshops<br />
  16. 16. Digital Preservation Management: Implementing Short-term Solutions for Long-term ProblemsCambridge, MA, USA / June 13-18, 2010Note: Albany, New York / June 5-10, 2011Source<br />Short digital preservation workshops are typically offered in conjunction withmost digital preservationconferences.<br />Best Practices Information SourcesWorkshops<br />
  17. 17. Open Planets FoundationSource<br />Digital Curation CentreSource<br />Library of Congress National Digital Information Infrastructure and Preservation ProgramSource<br />Best Practices Information SourcesWeb Sites/Listservs/Blogs<br />
  18. 18. JISC Digital Preservation and Records Management ProgrammeSource<br />PrestoPRIME Keeping Audiovisual Contents AliveSource<br />International Internet Preservation ConsortiumSource<br />Best Practices Information SourcesWeb Sites/Listservs/Blogs<br />
  19. 19. Best Practices Information SourcesWeb Sites/Listservs/Blogs<br />Source<br />
  20. 20. International Journal of Digital CurationSource<br />ARIADNESource<br />D-Lib MagazineSource<br />Best Practices Information SourcesJournals<br />
  21. 21. International Journal of Digital Curation<br />Source<br />
  22. 22. International Journal of Digital Curation<br />Source<br />
  23. 23. International Journal of Digital Curation<br />Source<br />
  24. 24. Best Practices Information SourcesEducation<br />Source<br />
  25. 25. Best Practices Information SourcesEducation<br />Source<br />
  26. 26. Best Practices Information SourcesEducation<br />Source<br />
  27. 27. Best Practices Information SourcesEducation<br />Source<br />
  28. 28. Best Practices Information SourcesEmployment<br />Source<br />
  29. 29. Open Archives Information System(OAIS)<br />Developed by the Consultative Committee for Space Data Systems in 2002 and became an ISO standard in 2003 (ISO 14721:2003).148 pages of heavy reading<br />“Those who will implement OAIS archives or administer them on a daily basisshould read the entire document.”Source<br />
  30. 30. Open Archives Information System <br />OCLC claims OAIS compliance for their “Digital Archive”.Source<br />Library and Archives Canada’s Trusted Digital Repository is based on OAIS.Source<br />National Library of the Netherlands’ e-Depot is an exemplar world classOAIS based digital repository.Source<br />
  31. 31. Open Archives Information System <br />“GPO’s world-class preservation repository [Fdsys] went live in March 2009. The repository was built upon the Open Archival Information System (OAIS) model and provides sufficient control to ensure long-term preservation and access.”<br />Source<br />
  32. 32. Open Archives Information System <br />“The use of this reference model as the basis of any archive implementation is recommended as it allows practitioners to use common language and potentially common tools to address common problems.”<br /> Tessella Technology & Consulting White PaperSource<br />
  33. 33. OAIS Reference Model<br />“The use of <br />Source<br />
  34. 34. OAIS Reference Model<br />Source<br />
  35. 35. OAIS Reference Model<br />Source<br />
  36. 36. OAIS Reference Model - Actors<br />Source<br />
  37. 37. OAIS Reference Model - Objects<br />Source<br />
  38. 38. OAIS Reference Model - Actions<br />Source<br />
  39. 39. Monitor designated community (consumer needs and expectations)<br />Monitor technology<br />Develop preservation strategies and standards<br />Develop packaging designs and migration plans<br />Preservation Planning<br />Source<br />
  40. 40. Monitor TechnologyInternet Archive Wayback Machine<br />Wayback for www.unb.ca<br />
  41. 41. Monitor TechnologyCross-Platform Access Video Format<br />2005: wmv (Windows Media Video) format using Windows Media Player (or other players) for Windows and Flip4MAC Quicktime extension for Macintosh.<br />2005 – 2009: swf (Adobe Flash) format with Adobe flash plug-ins available for Windows and Macintosh browsers becomes the flavour of the day for web delivery of video content.<br />
  42. 42. Monitor TechnologyCross-Platform Access Video Format<br />Fast forward to April, 2010: mp4 (H.264) format with players/support for Windows, Macintosh and IPAD.<br />IPAD does not support wmv or swf video formats.<br />Video conversion history: wmvswfmp4 from original DVD vobs.<br />DVD vob files are being preserved with agoal of converting them to MXF MotionJPEG 2000 for long-term preservation.<br />
  43. 43. Monitor TechnologyGoogle Drops H.264 Support (Jan 11, 2011)<br />Source<br />
  44. 44. Monitor TechnologyMicrosoft Adds H.264 Support (Feb 2, 2011)<br />Source<br />
  45. 45. Plato: The PLANETS Preservation Planning Tool<br />Source<br />
  46. 46. Plato: The PLANETS Preservation Planning Tool<br />Developed by the PLANETS Consortium<br />Source<br />
  47. 47. Plato: The PLANETS Preservation Planning Tool<br />A preservation plan defines a series of preservation actions to be taken by a responsible institution due to an identified risk for a given set of digital objects or records (called collection).<br />The preservation plan takes into account the preservation policies, legal obligations, organisational and technical constraints, user requirements and preservation goals and describes the preservation context, the evaluated preservation strategies and the resulting decision for one strategy, including the reasoning for the decision.<br />It also specifies a series of steps or actions (called preservation action plan) along with responsibilities and rules and conditions for execution on the collection. <br />Provided that the actions and their deployment as well as the technical environment allow it, this action plan is an executable workflow definition.<br />Access to a library of preservation plans.<br />Source<br />
  48. 48. Plato: The PLANETS Preservation Planning Tool<br />Source<br />
  49. 49. Plato: TIFF to JPEG 2000 Case Study<br />Source<br />YouTube Video<br />
  50. 50. Plato: TIFF to JPEG 2000 Case Study<br />British Library’s 2 million newspaper pages in TIFF-5 uncompressed and high quality. File size is 40 MB/ page.<br />PLATO experiment compares image quality and size of TIFF-5 images converted to JPEG 2000 lossless.<br />Experiment results: JPEG 2000 lossless image quality is as good as TIFF-5 uncompressed and image file size is reduced by 25-30 percent. JPEG derivatives from TIFF-5 are as good as JPEG derivativesfrom JPEG 2000 lossless.<br />Source<br />
  51. 51. Planets Time Capsule<br />Source<br />
  52. 52. E-Prints: Integration of Bit-Level and Logical Preservation (New)<br />Source<br />
  53. 53. E-Prints: Integration of Bit-Level and Logical Preservation (New)<br />Source<br />
  54. 54. E-Prints: Integration of Bit-Level and Logical Preservation (New)<br />GIF files will be migrated to PNG with the ImageMagick utility<br />Source<br />
  55. 55. E-Prints: Integration of Bit-Level and Logical Preservation (New)<br />Upload Plato preservation plan to E-Prints<br />Prescribed preservation plan action applied to each set of identified “at risk” classified files<br />E-Prints creates provenance metadata for all preservation actions (i.e. File was migrated from “file format A” to “file format B” on this date according to preservationplan NNN).<br />Source<br />
  56. 56. Sample Media Type Preservation Plan<br />Source<br />
  57. 57. Trustworthy Repositories Audit & Certification (TRAC) Checklist<br />Source<br />
  58. 58. Trustworthy Repositories Audit & Certification (TRAC) Checklist<br />Source<br />
  59. 59. Trustworthy Repositories Audit & Certification (TRAC) Checklist<br />Source<br />
  60. 60. Trustworthy Repositories Audit & Certification (TRAC) Checklist<br />Source<br />The repository commits to continuing maintenance of digital objects for identified community/communities.<br />Demonstrates organizational fitness (including financial, staffing structure, and processes) to fulfill its commitment. <br />
  61. 61. Trustworthy Repositories Audit & Certification (TRAC) Checklist<br />Source<br />Acquires and maintains requisite contractual and legal rights and fulfills responsibilities.<br />Has an effective and efficient policy framework.<br />Acquires and ingests digital objects based upon stated criteria that correspond to its commitments and capabilities. <br />
  62. 62. Trustworthy Repositories Audit & Certification (TRAC) Checklist<br />Maintains/ensures the integrity, authenticity and usability of digital objects it holds over time. <br />Creates and maintains requisite metadata about actions taken on digital objects during preservation as well as about the relevant production, access support, and usage process contexts beforepreservation. <br />Source<br />
  63. 63. Trustworthy Repositories Audit & Certification (TRAC) Checklist<br />Fulfills requisite dissemination requirements.<br />Has a strategic program for preservation planning and action.<br />Has technical infrastructure adequate to continuing maintenance and security of its digital objects.<br />Complete TRAC Document<br />Source<br />
  64. 64. Digital Curation Micro-Services<br />“Micro-services are an approach to digital curation based on<br />devolving curation function into a set of independent, but<br />interoperable, services that embody curation values and strategies.<br />Since each of the services is small and self-contained, they are<br />collectively easier to develop, deploy, maintain, and enhance.<br />Equally as important, they are more easily replaced when they have<br />outlived their usefulness. Although the individual services are<br />narrowly scoped, the complex function needed for effective<br />curation emerges from the strategic combination of<br />individual services.”<br />Source<br />
  65. 65. Archivematica http://archivematica.org is an open source software toolkit that takes the OAIS model and turns its various conceptual entities into actionable functionalities.<br />Take SIPs and turn them into AIPs and DIPs.<br />In v. 0.7 alpha this is accomplished through a Unix pipeline design which makes use of various open-source utilities toperform designated actions.<br />Digital Preservation in Action Archivematica (version 0.7 alpha)<br />
  66. 66. Open source software developed by Artefactual Systems (Vancouver, Canada)<br />Development partners include:<br />UNESCO Memory of the World Programme<br />International Monetary Fund<br />Vancouver City Archives<br />University of British Columbia<br />University of Virginia (Rubymatica)<br />Many alpha installations<br />Digital Preservation in Action Archivematica (version 0.7 alpha)<br />
  67. 67. Archivematica & OAISSIP > AIP > DIP<br />Source<br />
  68. 68. Archivematica & OAIS<br />SIP>AIP>DIP<br />Micro-services<br />Open source tools employed<br />Source<br />
  69. 69. Archivematica & OAISCuration Micro-services<br />Receive SIP<br />verifyChecksum<br />Review SIP<br />extractPackage<br />assignIdentifier<br />parseManifest<br />cleanFilename<br />Source<br />
  70. 70. Archivematica & OAISCuration Micro-services<br />Quarantine SIP<br />lockAccess<br />virusCheck<br />Appraise SIP<br />identifyFormat<br />validateFormat<br />extractMetadata<br />decidePreservationAction<br />Source<br />
  71. 71. Archivematica & OAISCuration Micro-services<br />Prepare AIP<br />gatherMetadata<br />normalizeFiles<br />createPackage<br />Review AIP<br />decideStorageAction<br />Source<br />
  72. 72. Archivematica & OAISCuration Micro-services<br />Store AIP<br />writePackage<br />replicatePackage<br />auditfixity<br />readPackage<br />updatePackage<br />Provide DIP<br />uploadPackage<br />updateMetadata<br />Source<br />
  73. 73. Archivematica & OAISCuration Micro-services<br />Monitor Preservation<br />checkFormatRegistry<br />updatePreservationPlanPolicies<br />migrateFormat<br />synchronizeAIPsandDIPs<br />Source<br />
  74. 74. Digital Curation Software Tools<br />Pronom File Format RegistryPRONOM is a resource for anyone requiring impartial and definitive information about the 320+ file formats, software products and other technical components required to support long-term access to electronic records and other digital objects of cultural, historical or business value. It is maintained by The National Archive(UK). <br />Source<br />Source<br />
  75. 75. Digital Curation Software Tools<br />Pronom File Format Registry (Excel 2.1)<br />Source<br />
  76. 76. Digital Curation Software Tools<br />Pronom File Format Registry (Excel 2.1)<br />Source<br />
  77. 77. Digital Curation Software Tools<br />FITS (Developed by Harvard University)<br />The File Information Tool Set (FITS) identifies, validates, and extracts technical metadata for various file formats. It wraps several third-party open source tools, normalizes and consolidates their output, and reports any errors.<br />Current tools are: Jhove, Exiftool, National Library of New Zealand Metadata Extractor, DROID, FFIdent, File Utility, Fileinfo andXMLMetadata.<br />Source<br />
  78. 78. Digital Curation Software Tools<br />FITS (Developed by Harvard University)<br />File identification using DROID<br />File validation using Jhove<br />Metadata extraction using NZ Metadata Extractor<br />Metadata normalization and consolidation using XMLMetadata<br />Source<br />
  79. 79. Digital Curation Software Tools<br />FITS (Developed by Harvard University)<br />All digital file formats are not supported by every tool as illustrated in the latest FITS release notes:<br />Improved support for audio formats<br />Better identification of JP2 and JPx images<br />Improved identification of EXIF and JFIF JPEGs<br />Fixed DROID format output for SVG files<br />Source<br />
  80. 80. Digital Curation Software Tools<br />FITS (DROID Tool – file identification)<br />DROID (Digital Record Object Identification) uses internal and external signatures, maintained in the PRONOM technical registry, to identify and report the specific file format versions of digital files. <br />Source<br />
  81. 81. Digital Curation Software Tools<br />FITS (JHOVE Tool – file identification, validation and characterization)<br />File identification as per DROID<br />File validation<br />A file is well-formed if it meets the purely syntactic requirements for a format.For example, a TIFF object is well-formed if it starts with an 8 byte header followed by a sequence of Image File Directories (IFDs), each composed of a 2 byteentry count and a series of 8 byte taggedentries.<br />Source<br />
  82. 82. Digital Curation Software Tools<br />FITS (JHOVE Tool – file identification, validation and characterization)<br />File validation (continued)<br />A well-formed file is also valid if it meets additional semantic level requirements.For example, an RGB file must have at least three sample values per pixel.<br />Source<br />
  83. 83. Digital Curation Software Tools<br />FITS (JHOVE Tool – file identification, validation and characterization)<br />File characterization<br />The process of determining the format-specific significant properties of an object of a given format.<br />JHOVE can report the file pathname or URI, last modification date, byte size, format, format version, MIME type, format profiles and, optionally, a checksum.<br />Source<br />
  84. 84. Digital Curation Software Tools<br />FITS (JHOVE Tool – sample output)<br />Source<br />
  85. 85. Digital Curation Software Tools<br />FITS (JHOVE Tool – supported file formats)<br />Source<br />AIFF <br />ASCII <br />BYTESTREAM <br />GIF <br />HTML <br />JPEG <br />JPEG 2000 <br />PDF <br />TIFF <br />UTF-8 <br />WAVE <br />XML <br />
  86. 86. Digital Curation Software Tools<br />FITS (New Zealand Metadata Extraction Tool)<br />Automatically extracts preservation-related metadata from digital files.<br />Supported file formats: <br />Images: BMP, GIF, JPEG and TIFF. <br />Office documents: MS Word (version 2, 6), Word Perfect, Open Office (version 1), MS Works, MS Excel,<br /> MS PowerPoint, and PDF. <br />Audio and Video: WAV and MP3. <br />Markup languages: HTML and XML. <br />Source<br />
  87. 87. Digital Curation Software Tools<br />FITS (New Zealand Metadata Extraction Tool)<br />Potential metadata elements which can be extracted from an audio file header include:<br />Resolution <br />Duration <br />Bitrate <br />Compression <br />Encapsulation <br />Channels <br />Source<br />
  88. 88. Digital Curation Software Tools<br />BagItA specification for the packaging of digital content for transfer. Content is packaged (the bag) along with a small amount of machine-readable text (the tag) to help automate the content's receipt, storage and retrieval. There is no software to install. A bag consists of a base directory containing the tag and a subdirectory that holds the content files. The tag is a simple text-file manifest, like a packing slip, that consists of two elements:<br />An inventory of the content files in the bag<br />A checksum for each file<br />Source<br />Source<br />
  89. 89. Digital Curation Software Tools<br />BagIt: bag directory contents<br />/6‐1999‐06‐07bagit.txtbag‐info.txtmanifest‐md5.txt/data6‐1999‐06‐07.tif6‐1999‐06‐07_general_metadata.xml6‐1999‐06‐07_technical_metadata.xml<br />Source<br />Source<br />
  90. 90. Digital Curation Software Tools<br />BagIt: bagit.txt<br />BagIt‐Version: 0.96<br /> Tag‐File‐Character‐Encoding: UTF‐8<br />Source<br />
  91. 91. Digital Curation Software Tools<br />BagIt: bag‐info.txt<br />Source‐organization: Simon Fraser University LibraryOrganization‐URL: http://www.lib.sfu.caBagging‐Date: 2009‐06‐26External‐Description: TIFF master files and associated metadata for item 6‐1999‐06‐07 in the SFU Editorial Cartoons Collection.<br />Source<br />
  92. 92. Digital Curation Software Tools<br />BagIt: manifest‐md5.txt<br />91a6ce58ad2628b81c46c034d434816f data/6‐ 1999‐06‐07.tif8c2712026f0f54c4ad156674e87f573b data/6‐1999‐06‐07_general_metadata.xml28fa197bbfd61e4da0f6119ed7420bff data/6‐ 1999‐06‐07_technical_metadata.xml<br />Source<br />
  93. 93. Digital Curation Software Tools<br />BagIt: 1999‐06‐07.tif<br />Ingrid Rice, June 7, 1999<br />Source<br />
  94. 94. Digital Curation Software Tools<br />BagIt: General metadata file<br />Source<br />
  95. 95. Digital Curation Software Tools<br />BagIt: Technical metadata file<br />/6‐1999‐06‐07<br />bagit.txt <br />bag‐info.txt <br />manifest‐md5.txt<br />/data<br />6‐1999‐06‐07.tif<br />6‐1999‐06‐07_general_metadata.xml<br />6‐1999‐06‐07_technical_metadata.xml<br />Source<br />
  96. 96. DSpace 1.7 (New Features)<br />AIP Backup and Restore<br />Outputs metadata and bitstreams into zipped self-contained Archival Information Packages which can be loaded into another instance of DSpace or another institutional respository platform (Fedora, CONTENTdm, etc.)<br />DSpace AIPs can function as SIPs or DIPs.<br />Possible to load Archivematica AIPs intoDSpace.<br />Source<br />
  97. 97. DSpace 1.7 (New Features)<br />Curation System<br />Infrastructure to support the implementation of digital curation micro-services for the long-term preservation of your DSpace content.<br />Initial Services include:<br />Bitstream format profiler: examines all the bitstreams and generates a count and support level for each type of bitstream format. Useful tool for format migration. Note: this is not identifying and validating bitstreams.<br />Required metadata: checks to see if requiredmetadata is present in all records.<br />Virus scan: Virus check using ClamAV tool.<br />Source<br />
  98. 98. Archivematica 0.7 Alpha Demo<br />Objectives<br />Show complete process of ingest/archival/dissemination chain for one SIP<br />Our demo SIP contains object files of various image formats: TIFF, BMP, SVG, PNG, JP2, EPS, GIF, JPG, TGA<br />Check contents of ArchiveMatica SIP, throughout the process, as it transforms into a self-contained AIP and DIP<br />
  99. 99. Archivematica 0.7 Alpha Demo<br />Normalization paths used in this demo<br />(*) PNG and JPEG2000 are not normalized to a preservation format<br />
  100. 100. Archivematica 0.7 Alpha Demo<br />Archivematica Release 0.7 Alpha<br />YouTube Video 1 and 2, along with <br />step by step instructions.<br />
  101. 101. Archivematica 0.7 Alpha Demo<br />Boot your PC with the bootable Archivematica DVD.<br />Login as: demo Password: demo<br />You see the File Manager<br />Shortcuts<br />Directories used through the archiving process<br />Imagine you’re an archivist and you have a set of object files sitting in demo/testFiles<br />structured into a number of directories<br />each directory corresponds to a logical unit of resources, be it a distinctive item or a complete fonds<br />each directory in testFiles = one SIP<br />You couldalso drag/drop, copy/pastefromUSB stick<br />
  102. 102. Archivematica 0.7 Alpha Demo<br />Launch dashboard and resize so that it can be viewed as you navigate through the Archivematica processes.<br />FireFox: uncheck File/Work Offline<br />Web-based administration for the archivist<br />Tracks various stages of the archival process<br />(In this demo setup of ) ArchiveMatica manual approval is required from archivist at various stages in the process:<br />we’ll have a look at contents of SIP, AIP and DIP at each of these stages<br />
  103. 103. Archivematica 0.7 Alpha Demo<br />ArchiveMatica-SIP<br />Folder structure, containing metadata, checksums, object files<br />logs<br />logs/fileMeta<br />metadata: checksum and descriptive metadata<br />objects: digital objects to be preserved<br />Content changes as SIP is moved through the different stages of the archiving process<br />Demo SIP = ImagesSIP directory<br />
  104. 104. Archivematica 0.7 Alpha Demo<br />Start the archival process <br />Drap and drop the ImagesSIP directory into the receiveSIP watched directory<br />Rename the SIP<br />The SIP appears in the DashBoard<br />
  105. 105. Archivematica 0.7 Alpha Demo<br />First approval: appraise SIP for submission<br />click on Micro-Services to look at actions performed by ArchiveMatica so far<br />SIP backup, SIP compliant, assign UUIDs (package and object files), check delivered checksums (if any delivered)<br />click on Browse to see contents of SIP at this stage<br />logs/fileUUIDs.log<br />logs/fileMeta/*.xml for each object file: PREMIS-formatted metadata<br /> file name, uuid, sha256 hash<br />events that occurred on the object file<br />
  106. 106. Archivematica 0.7 Alpha Demo<br />First approval: appraise SIP for submission<br />submitted SIP should be in accordance with institution’s submission agreements<br />delete any unwanted files or directories <br />File Manager/appraiseSIPForSubmission<br />add descriptive metadata about the SIP in metadata/dublincore.xml<br />click on Approve<br />
  107. 107. Archivematica 0.7 Alpha Demo<br />SIP quarantined <br />SIP is placed in quarantine for virus checking<br />Why quarantine? <br />Give ClamAV a chance to pick up the latest version of its virus database<br />How long?<br />demo: preset to one minute<br />National Archives of Australia: 1 month<br />archivist can manually remove SIP from quarantine<br />
  108. 108. Archivematica 0.7 Alpha Demo<br />Second approval: appraise SIP for preservation<br />zipped/tarred/… files are extracted<br />check directory and file names<br />scan for viruses<br />using FITS:<br />identify and validate format of object files<br />extract technical metadata – PREMIS<br />
  109. 109. Archivematica 0.7 Alpha Demo<br />Second approval: appraise SIP for preservation<br />logs/clamAVScan.txt: report on virus checking<br />logs/extraction.log: report on extracted zip<br />logs/fileMeta/*.xml: augmentedPREMIS-formatted metadata<br />format designation(PRONOM PUID identifier)<br />events<br />technical metadata<br />
  110. 110. Archivematica 0.7 Alpha Demo<br />Second approval: appraise SIP for preservation<br />technical metadata: object characteristics<br /><fits_output> XML formatted metadata <br /><fits/identification><br /><fits/fileinfo><br /><fits/filestatus>: well-formed / valid<br /><fits/metadata>: technical metadata of object<br /><fits/toolOutput>: output results of used tools Jhove, File Utility, Exiftool, Droid, NLNZ Metadata Extractor, ffident File Information, XML Metadata<br />
  111. 111. Archivematica 0.7 Alpha Demo<br />Second approval: appraise SIP for preservation<br />delete any unwanted files or directories from the SIP <br />FileManager/appraiseSIPForPreservation<br />click on Approve<br />ArchiveMatica now creates an AIP and a DIP for this SIP<br />normalization based on format identified <br />
  112. 112. Archivematica 0.7 Alpha Demo<br />Third approval: push AIP to archival storage<br />storeAIP contains one zip file for the AIP<br />containing a bag (according BagIt specs)<br />Click on Browse next to Store AIP micro-service<br />Look in the bag<br />
  113. 113. Archivematica 0.7 Alpha Demo<br />ArchiveMatica-AIP<br />data/<br />logs/normalizationLog.txt<br />metadata: the dublincore.xml<br />checksum.sha256 for the AIP<br />objects: all original formats + preservation formats<br />METS.xml: METS XML container with structural, descriptive, administrative metadata of AIP<br />
  114. 114. Archivematica 0.7 Alpha Demo<br />Source<br />
  115. 115. Archivematica 0.7 Alpha Demo<br />Source<br />
  116. 116. Archivematica 0.7 Alpha Demo<br />ArchiveMatica-AIP / METS.xml<br /><structMap>: structure of the AIP<br /><fileSec>: list of files included in the AIP<br /><dmdSec>: descriptive metadata for the AIP (the dublincore.xml)<br /><amdSec>: administrative metadata<br /><digiprovMD>: PREMIS-formatted digital provenance metadata<br />most of it is grabbed from the logs/fileMetafiles<br />object identification and characteristics<br />events <br />agents<br />relation between original and preservation copies<br />
  117. 117. Archivematica 0.7 Alpha Demo<br />Third approval: push AIP to archival storage<br />If wanted, check contents of the AIP : you are not able to make any changes though in an AIP<br />click on Approve<br />AIP is pushed into archival storage<br />our demo setup: the AIPsStore directory<br />real life: cloud storage, Amazon S3, your own network storage device, CLOCKSS, …<br />
  118. 118. Archivematica 0.7 Alpha Demo<br />Fourth approval: upload DIP to public access system<br />directory created for this DIP under uploadDIP<br />objects: normalized access copies of the object files <br />objectsBackup: idem<br />METS.xml: identical as in the AIP<br />If wanted, check and change contents of the DIP <br />File Manager / uploadDIP<br />click on Approve<br />removed from SIPbackups<br />copied to DIPbackups<br />our demo setup: DIP is pushed towards an ICA-Atom public access system<br />
  119. 119. Archivematica 0.7 Alpha Demo<br />ICA-AtoM public access system<br />Fully web-basedarchival description application based on International Council on Archives standards<br />AtoM = Access to Memory<br />Point Firefox to http://localhost/ica-atom<br />UploadedDIPs are by default in draft. Change status to ‘published’ for these to become visible in public access<br />Log in: demo@example.com / demo<br />Choosefromarchival descriptions<br />Edit: change publication status to ‘published’<br />Log out<br />Selected archive isnowpublicly visible<br />
  120. 120. Digital preservation @ ULB<br />Context: multiple digital archives<br />DI-pot<br />All academic output (except PhD theses)<br />Most digital born / some digitized by library staff<br />Self-submission by academic staff <br />Extensively modified DSpace 1.4.2 <br />Metadata granularity <br />Semi-automated metadata ingest from PubMed, Scopus, Web of Science, BibTex and RIS files<br />Integrated with central administration databases (staff, departments, controlled vocabulary, ...)<br />55K descriptions<br />8K full-text [ PDF ]<br />
  121. 121. Digital preservation @ ULB<br />Context: multiple digital archives<br />Bictel<br />PhD theses (since 2004)<br />Most digital born / some digitized by library staff<br />Self-submission, with some support from faculty staff<br />ETD software from Virginia Tech<br />Metadata per object file: access restrictions, deposit dates, mime type, location<br />1300 descriptions<br />Typically multiple object files per thesis [ PDF ]<br />
  122. 122. Digital preservation @ ULB<br />Context: multiple digital archives<br />Iconothèque<br />Audiovisual material as support for courses<br />Most digital born / some digitized by faculty staff<br />Self-submission by faculty staff<br />ContentDM 5.4<br />12K descriptions<br />[ JPEG ]<br />
  123. 123. Digital preservation @ ULB<br />Context: multiple digital archives<br />Digithèque<br />Out of print / public domain books and journals<br />Digitized by library staff<br />Submission by library staff<br />Symphony + file system (available over SMB, HTTP)<br />100K pages / 344 publications<br />[ TIFF + PDF ]<br />
  124. 124. Digital preservation @ ULB<br />Context: multiple digital archives<br />Near future: archives of ULB<br />(our ISADG enabled) DSpace<br />
  125. 125. Digital preservation @ ULB<br />All our digital archives :<br />Talk OAI-PMH<br />Expose identical exchange format <br />Based on MPEG21-DIDL<br />Compound object of item and associated object files<br />“Globally unique persistent identifier” (GUPI) for item and each object file<br />Descriptive metadata for item expressed in MODS<br />Metadata for object files: descriptive, version, access restrictions, deposit /embargo dates, mime type, location<br />
  126. 126. DIDL[1]<br />Item[1]<br />Descriptor/Identifier (persistent identifier)<br />Descriptor/modified<br />Item[1..∞] (of type descriptiveMetadata)<br />Descriptor/type (« descriptiveMetadata »)<br />Descriptor/Identifier (persistent identifier)<br />Descriptor/modified<br />Component/Resource -- representation by value (XML)<br />Item[0..∞] (of type objectFile)<br />Descriptor/type (« objectFile »)<br />Descriptor/Identifier (persistent identifier)<br />Descriptor/modified<br />Component/Resource -- representation by ref. (URL)<br />Item[0..1] (of type humanStartPage)<br />Descriptor/type (« humanStartPage »)<br />Component/Resource -- representation by ref. (URL)<br />Digital preservation @ ULB<br />
  127. 127. Digital preservation @ ULB<br />One dissemination platform<br />SAMBURU: harvest and index<br />DIDL records are harvested from the digital archives <br />DIDL record is stored as-is in MySQL database<br />DIDL record is transformed into SOLR document and stored in Lucene indexes<br />DI-fusion: web portal<br />Based on VuFind<br />Search/retrieve records through SOLR<br />Use XSLT to transform DIDL into HTML<br />Additional 2.0 functionality with AJAX technology<br />
  128. 128. Digital preservation @ ULB<br />Samburu<br />SOLR<br />DI-fusion<br />web <br />portal<br />OAI-PMH<br />DI-pot<br />Indexer<br />BicTel<br />Harvester<br />Lucene indexes<br />Icono<br />MySQL<br />Metadata Store<br />Digi<br />UMons<br />OAI-PMH<br />OAI-PMH<br />Metadata<br />Enrichment<br />
  129. 129. Digital preservation @ ULB<br />
  130. 130. Digital preservation @ ULB<br />
  131. 131. Digital preservation @ ULB<br />
  132. 132. Digital preservation @ ULB<br />Enrichment process<br />Fetch DIDL records from SAMBURU md store+ Fetch object files (in function of enrichment type) <br />Calculate enrichment and create DIDL formatted enrichment record<br />Make enrichment record available over OAI-PMH<br />SAMBURU harvests and merges original DIDL record with enrichment DIDL record, before re-indexing into Lucene<br />End user sees enrichment through DI-fusion<br />
  133. 133. Digital preservation @ ULB<br />Enrichment: 3 prototype setups<br />Enrichment service at Erasmus University in Rotterdam fetches publications in economics from md store, and determines JEL classification codes based on text analysis<br />Enrichment service @ ULB extracts texts from PDFs and indexes on all words. DI-fusion permits end user to do a full-text search<br />Enrichment service @ ULB enriches with JCR impact factors (based on ISSN and publication year)<br />
  134. 134. Digital preservation @ ULB<br />Back to digital preservation<br />SUBMISSION<br />metadata and object files (through 4 submission interfaces) <br />DISSEMINATION<br />through DI-fusion<br />ARCHIVAL<br />we need a PAS: “Perpetual Archiving System”<br />based on the idea of enrichment<br />
  135. 135. Digital preservation @ ULB<br />Samburu<br />SOLR<br />DI-fusion<br />web <br />portal<br />OAI-PMH<br />DI-pot<br />Indexer<br />BicTel<br />Harvester<br />Lucene indexes<br />Icono<br />MySQL<br />Metadata Store<br />Digi<br />UMons<br />OAI-PMH<br />OAI-PMH<br />PAS<br />SIPs<br />AIPs<br />DIPs<br />LOCKSS<br />Admin<br />
  136. 136. Digital preservation @ ULB<br />PAS-SIP<br />Retrieve DIDL records over OAI-PMH from SAMBURU metadata store<br />Fetch object files, based on references included in the DIDL record<br />Make and store ArchiveMatica-SIP<br />Alternative to OAI-PMH + web grabbing:<br />Prepare ArchiveMatica-SIPs on a network-attached filesystem<br />More practical for bulk ingest into AM: less network traffic<br />We would probably try a combined approach: bulk + incremental<br />Specific package information registered in PAS-Admin<br />
  137. 137. Digital preservation @ ULB<br />PAS-AIP<br />Use ArchiveMatica micro-services to create and store ArchiveMatica-AIP, according to media type preservation plan<br />Fully automated, at least for certain media types (PDF, JPEG, TIFF)<br />Update package information in PAS-Admin<br />
  138. 138. Digital preservation @ ULB<br />PAS-DIP<br />Use ArchiveMatica micro-services to create and store ArchiveMatica-DIP, according to media type preservation plan<br />DIPped object files made available through web service<br />Update package information in PAS-Admin<br />
  139. 139. Digital preservation @ ULB<br />PAS-Admin<br />Digital preservation status of packages information accessible over a web service:<br />Original digital archive wants to find out archival status of its items, based on gupi of item or object file<br />End user accesses DIPped object files through web service: not publicly available since dependent on accessibility restrictions set by IPR owner in original digital archive<br />AIPs are pushed into outer preservation space, e.g. LOCKSS + registered as suchin PAS-Admin<br />
  140. 140. Digital preservation @ ULB<br />PAS-Admin<br />Throughout SIP/AIP/DIP processing, relevant information should be registered about the packages in a db<br />For each SIP, AIP, DIP:<br />(I) gupi of item and all object files<br />uuid of package<br />(I) identifier of original digital archive<br />(I) date of creation/modification<br />
  141. 141. Digital preservation @ ULB<br />PAS-Admin<br />relevant metadata of DIPs are made available as DIDL-structured (enrichment) records over OAI-PMH for SAMBURU to pick up<br />Parse/extract from METS.xml:<br />Essentially mime type and location<br />sum of original metadata and PAS-created metadata is available to DI-fusion<br />DI-fusion could for example decide to only show DIP version of an object file, and inform end user of the existence of the original object file format<br />
  142. 142. Open DiscussionAlternative options for integrating Archivematica or a subset of digital curation micro-services into your digitization workflow.<br />
  143. 143. Issues<br />Institutional repositories are also used to maintain an institution’s bibliography, with frequent updates of descriptive metadata and object files. <br /> When should digital objects from an IR be preserved?<br />
  144. 144. Issues<br />Dappert, A. & Enders M. Using METS, PREMISand MODS for archiving eJournalsD-Lib Magazine Volume 14 Number (9/10)http://www.dlib.org/dlib/september08/dappert/09dappert.html<br />“AIP per generation” generation: change in md and/or object file<br />
  145. 145. Issues<br />Both ArchiveMatica and LOCKSS are looking into solutions for the normalization of objects and packaging. Both systems seem redundant at first. <br /> How does ArchiveMatica interact with LOCKSS?<br />
  146. 146. Issues<br />ArchiveMatica-AIPs, DSpace-AIPs, exchange of packages between digital archives, nationwide preservation solution.<br /> Need for interoperability standards?<br /> TIPR: Towards Interoperable Preservation <br />Respositories<br /> RXP: Repository eXchange Package<br />
  147. 147. AIP Repository Interoperability<br /> “For reasons of redundancy, succession planning and software migration, repositories must be able to exchange copies of archival information packages with each other. Every different repository application, however, describes and structures its archival packages differently. Therefore each system produces dissemination packages that are rarely understandable or usable as submission packages by other repositories. “<br />Source<br />
  148. 148. AIP Repository Interoperability<br />One possible solution: RXP (Repository eXchange Package), developed by the Towards Interoperable Preservation Repositories (TIPR) project which has defined a standards-based package of metadata files that can act as an intermediary information package, the RXP, a lingua franca all repositories can read and write.<br />Another option: create AIPS followingthe HathiTrust specification for digital objects.<br />Source<br />Source<br />
  149. 149. Issues<br />AIPs are intended for perpetual access and therefore only contain objects that comply to an open documented format. Any human being within 50 years should be able to re-read the contents of the object files, given a textual documentation.<br /> So, why migrate AIPs into a new(er) format?<br />
  150. 150. Issues<br />Archivematica normalizes moving pictures into MPEG2 = loss of quality<br />Lossless conversion would be Motion JPEG2000<br />However: no open-source CLI-based tool for conversion into Motion JPEG2000 format available<br />
  151. 151. Issues<br />The more copies of a digital object are stored all over the place, the less trivial becomes control of copyright. <br /> Is geo-independent perpetual archiving in contradiction with IPR issues?<br />
  152. 152. Issues<br />Packages are self-contained: if you find an AIP, you know what it is about, and you can read, look, hear it. But how do you find the AIP in a see of billions of AIPs?<br /> Don’t forget to preserve finding aids! How?<br />
  153. 153. Contact<br />SlavkoManojlovich<br />Associate University Librarian (IT)<br />Manager, Digital Archives Initiative<br />Memorial University of Newfoundland, St. John’s<br />slavko@mun.ca<br />& <br />Benoit Pauwels<br />Head, Library Automation Team<br />Université Libre de Bruxelles<br />Benoit.Pauwels@ulb.ac.be<br />*This presentation may be downloaded at:<br />http://dl.dropbox.com/u/18652253/phoenix%20presentation.pptx<br />

×