Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

July 2019 Monthly Meetup Australasia Preserves LTO data migration

133 views

Published on

For this month's community meetup, Andrew Martin from DAMsmart shared thoughts and experiences from an audiovisual LTO data migration project for a government agency. He was joined by Tom Eccles from AIATSIS. For more info, checkout the blog post here: https://www.australasiapreserves.org/2019/08/july-2019-monthly-meetup-on-lto-data.html

Published in: Technology
  • Be the first to comment

  • Be the first to like this

July 2019 Monthly Meetup Australasia Preserves LTO data migration

  1. 1. LTO DATA MIGRATION
  2. 2. WHAT IS LTO? • Linear Tape-Open developed in the 1990’s by the LTO Consortium: HP, IBM, and Quantum • An open standard • Magnetic tape single reel storage format • Single Desktop Drives, or autoloader libraries • Backwards compatible drives by 2 generations
  3. 3. CURRENT ISSUES AROUND LTO 8 Changes from GMR (Giant Magneto-Resistance) heads to TMR (Tunnel Magnetoresistance Recording) heads. LTO 8 drives can only read LTO 8 and LTO7 Cartridges LTO 8 cartridges currently unavailable due to manufacturer lawsuits regarding patents
  4. 4. CASE STUDY: AIATSIS ATSIC DIGITISED VIDEO COLLECTION • 50 LTO 4 cartridges to be migrated to LTO 6 (as delivery medium) • 2,900 audiovisual preservation files • 36 TB of data • Written using the POSIX TAR format • No tape contents information
  5. 5. • LTO drives configured and controlled with Cygwin. Correct block size was set on the drive using –mt Correct blocking factor for reading was selected when restoring using -tar • Approx. 120MB/s per data stream • 3 weeks to transfer all data onto 5 x 12 TB Western Digital Surveillance Drives 6 Gb/s SAS 6 Gb/s SAS USB 3 USB 3
  6. 6. DATA RESTORE AND VERIFICATION Video Files Lossless JPEG2000.mxf Accompanying XML Metadata files that contained checksum for the video file Data verified using custom software Look for the same filename Extract checksum Create checksum of video file and compare
  7. 7. COMPARE REPORT • Filename compare • SHA1 from XML • SHA1 calculated • Does it match!
  8. 8. DATA PREPARATION AND LTO6 WRITING • Before writing data, files were renamed. ATSIC0001.mxf-0-0-1 > ATSIC0001.mxf • Data migrated to 17 LTO6 using LTFS • Verification of data written to tape • Verification reports and LTO drive contents created.
  9. 9. WHAT TO WATCH OUT FOR WHEN PREPARING FOR AN LTO DATA MIGRATION PROJECT What format / vendor has the source LTO been written under? Is there a database of the source data? Does the data need specific proprietary software to be restored? Are there existing checksums that can be utilised to ensure correct restore?
  10. 10. WHAT TO WATCH OUT FOR WHEN PREPARING FOR A DATA MIGRATION PROJECT • Is there any additional processing required while files are online in the staging process? (e.g. metadata creation, QC checks, transcoding) • Would you write tape to tape, or tape to disk to tape? • Is LTO the right format to be migrating to? What are the Preservation and business requirements? • Would migrating to the cloud be appropriate?
  11. 11. THANKYOU!! Andrew@damsmart.com.au @audiovisualopen www.damsmart.com.au

×