More Related Content
Viewers also liked
Similar to APTrust CNI Seattle April 2015
Similar to APTrust CNI Seattle April 2015 (20)
Recently uploaded
Recently uploaded (20)
APTrust CNI Seattle April 2015
- 1. Update at CNI, April 14, 2015 Chip German, APTrust Andrew Diamond, APTrust Nathan Tallman, University of Cincinnati Linda Newman, University of Cincinnati Jamie Little, University of Miami APTrust Web Site: aptrust.org
- 2. Sustaining Members: Columbia University Indiana University Johns Hopkins University North Carolina State University Penn State Syracuse University University of Chicago University of Cincinnati University of Connecticut University of Maryland University of Miami University of Michigan University of North Carolina University of Notre Dame University of Virginia Virginia Tech
- 3. • Quick description of APTrust: The Academic Preservation Trust (APTrust) is a consortium of higher education institutions committed to providing both a preservation repository for digital content and collaboratively developed services related to that content. The APTrust repository accepts digital materials in all formats from member institutions, and provides redundant storage in the cloud. It is managed and operated by the University of Virginia and is both a deposit location and a replicating node for the Digital Preservation Network (DPN). • 39 of the registrants for CNI in Seattle are from APTrust sustaining-member institutions
- 4. APTrust How it works Andrew Diamond Senior Preservation & Systems Engineer
- 9. Behind the Scenes, Step by Step
- 16. What You See, Step by Step
- 20. Storage Pending
- 21. Record Pending
- 22. Record Succeeded
- 23. List of Ingested Objects
- 24. Object Detail
- 25. List of Files
- 26. File Detail
- 30. Pending Tasks: Restore & Delete
- 31. Completed Tasks: Restore & Delete
- 32. Detail of Completed Restore
- 33. Detail of Completed Delete
- 34. A Note on Bag Storage & Restoration
- 35. Bag Storage
- 36. Bag Restoration
- 37. Where’s my data?
- 38. On multiple disks in Northern Virginia. S3 Glacier On hard disk, tape or optical disk in Oregon.
- 39. S3 = Simple Storage Service
- 40. S3 = Simple Storage Service 99.999999999% Durability (9 nines)
- 41. S3 = Simple Storage Service 99.999999999% Durability (9 nines) Each file stored on multiple devices in multiple data centers
- 42. S3 = Simple Storage Service 99.999999999% Durability (9 nines) Each file stored on multiple devices in multiple data centers Fixity check on transfer
- 43. S3 = Simple Storage Service 99.999999999% Durability (9 nines) Each file stored on multiple devices in multiple data centers Fixity check on transfer Ongoing data integrity checks & self- healing
- 44. Glacier Long-term storage with 99.999999999% durability (11 nines) Not intended for frequent access Takes five hours or so to restore a file
- 45. Chances of losing an archived object?
- 46. Chances of losing an archived object? 1 in 100,000,000,000 S3
- 47. Chances of losing an archived object? 1 in 100,000,000,000 S3 Glacier 1 in 10,000,000,000,000
- 48. Chances of losing an archived object? 1 in 100,000,000,000 S3 Glacier 1 in 10,000,000,000,000 1 in 1,000,000,000,000,000,000,000,000 chance of losing both to separate events
- 49. Chances of losing an archived object? 1 in 100,000,000,000 S3 Glacier 1 in 10,000,000,000,000 1 in 1,000,000,000,000,000,000,000,000 chance of losing both to separate events Very Slim
- 50. Protecting Against Failure During Ingest
- 51. Protecting Against Failure During Ingest Microservices with state Resume intelligently after failure
- 57. Protecting Against Failure After Ingest
- 58. Protecting Against Failure After Ingest Objects & Files Metadata & Events
- 63. What if...
- 64. What if ingest services stop? They restart themselves and pick up where they left off.
- 65. What if ingest server dies? We spin up a new one in a few hours.
- 66. What if Fedora service stops? We restart it manually.
- 67. What if Fedora server dies? We can build a new one from the data on the hot backup in Oregon.
- 68. What if we lose all of Glacier in Oregon? We have all files in S3 in Virginia. Ingest services and Fedora are not affected.
- 69. What if we lose the whole Virginia data center? Fedora Ingest Services Metadata and Events Files and Objects S3
- 70. What if we lose the whole Virginia data center? Ingest Services We can rebuild services from code and config files in hours.
- 71. What if we lose the whole Virginia data center? We can restore all files from Glacier in Oregon. Files and Objects S3
- 72. What if we lose the whole Virginia data center? We can restore all metadata and events from the Fedora backup in Oregon.Fedora Metadata and Events
- 73. Metadata stored with files
- 74. Attached metadata for last-ditch recovery
- 76. To lose everything... A major catastrophic event in Virginia and Oregon, or...
- 77. To lose everything... Don’t pay the AWS bill A major catastrophic event in Virginia and Oregon, or...
- 78. Partner Tools
- 80. Partner Tools Bag Management Upload Download List Delete ...or use open-source AWS libraries for Java, Python, Ruby, etc.
- 82. Partner Tools Bag Validation Validate a bag before you upload it. Get specific details about what to fix.
- 84. Partner Tools Bag Management & Validation Download from the APTrust wiki https://sites.google.com/a/aptrust.org/aptrust-wiki/home/partner-tools
- 86. Learning to Run University of Cincinnati Libraries path to contributing content to APTrust
- 87. Digital Collection Environment ● 3 digital repositories o DSpace -- UC Digital Resource Commons o LUNA -- UC Libraries Image and Media Collections o Hydra -- Scholar@UC Institutional Repository ● 1 website ● 1 Storage Area Network file system
- 88. Workflow Considerations ● Preservation vs. Recovery o platform-specific bags or platform-agnostic bags? o derivative files ● Bagging level o record level or collection level?
- 89. UC Libraries Workflow Choices ● Recovery is as important as preservation o Include platform specific files and derivatives ● Track platform provenance o use bag naming convention or additional fields in bag-info.text ● Track collection membership o use bag naming convention or additional fields in bag-info.text
- 90. Bagging (crawling) 1st iteration ● Start with DSpace, Cincinnati Subway and Street Improvements collection ● Replication Task Suite to produce Bags o one bag per record ● Bash scripts to transform to APTrust Bags ● Bash scripts to upload to test repository ● Worked well, but… Scripts available at https://github.com/ntallman/APTrust_tools.
- 91. Bagging (walking) 2nd iteration ● Same starting place ● Export collection as Simple Archive Format o ~ 8,800 records, ~ 375 GB ● Create bags using bagit-python o 250 GB is bag-size upper limit
- 92. Bagging (walking) 2nd iteration (continued) ● Collection had three series o two series under 250 GB ▪ each series was put into a single bag o one series over 250 GB ▪ used a multi-part bag to send to APTrust, APTrust merges split bags into one ● Bash script to upload bags to production repository
- 93. Bagging (running) 3rd iteration ● Testing of tools created by Jamie Little of University of Miami ● Jamie can tell you more but…
- 94. Long-Term Strategy (marathons) ● Bag for preservation and recovery ● For most content o one bag per collection ▪ except for Scholar@UC o manual curation and bagging ▪ except for Scholar@UC ● Use bag-info.txt to capture platform provenance and collection membership
- 95. at
- 96. UM Digital Collections Environment ● File system based long-term storage ● CONTENTdm for public access
- 97. Creating a workflow ● Began at the command line working with LOC’s BagIt Java Library ● Bash scripts ● Our initial testing used technical criteria (size, structural complexity, file types)
- 98. What to put in the bag? ● CONTENTdm metadata ● FITS technical metadata ● We maintained our file system structure in the bags
- 99. Beginning APTrust Testing ● Began uploading whole collections as bags ● Early attempts at a user interface using Zenity
- 100. UM’s APTrust Automation ● Sinatra based server application ● HTML & JavaScript interface
- 101. Starting Screen
- 102. Choosing a Repository ID
- 103. Choosing Files
- 105. Final Upload
- 106. Bagging Strategies Moved from bag per collection strategy to bag per file Changed interface to allow selection of individual files
- 107. APTrust Benefits for Developers ● Wide range of open source tools available ● Uploading to APTrust is no different than uploading to S3
- 108. Where to find us github.com/UMiamiLibraries http://merrick.library.miami.edu/
- 109. Why are we doing this again? ● Libraries’ and Archives’ historic commitment to long- term preservation is the raison d'être, a core justification, for our support of repositories. It’s why our stakeholders trust us and will place their content in our repositories. ● We expect APTrust to be our long-term preservation repository to meet our responsibilities to our stakeholders.
- 110. Why are we doing this again? ● We know from some experience, and reports from our peers, that backup procedures, while still essential, are fraught with holes. ● We expect APTrust to be our failover if any near-term restore from backup proves incomplete.
- 111. Linda Newman Head, Digital Collections and Repositories linda.newman@uc.edu UC Libraries, Digital Collections and Repositories Nathan Tallman Digital Content Strategist nathan.tallman@uc.edu http://digital.libraries.uc.edu https://scholar.uc.edu https://github.com/uclibs
Editor's Notes
- You have files to archive.
- You send files to a receiving bucket at AWS US-East in Northern Virginia.
- Files go to S3 preservation bucket. Metadata and PREMIS events go to Fedora. We delete the file from the receiving bucket.
- APTrust copies them to Glacier in Oregon.
- Ingest and fixity generation.
- First ID is ID identifier in Fedora. Use this to look up your file in the web UI. Second ID is URL in S3 preservation bucket.
- The white entries are pending restore and delete operations.
- Use the apt_download tool or one of your own tools to retrieve the restored file. You should delete the file after restoration because we have to pay for that storage.
- AWS modular Oregon data centers are in Umatilla, Boardman, and likely other areas as well. Umatilla: http://www.pcmag.com/article2/0,2817,2396153,00.asp Boardman: http://www.stackdriver.com/public-cloud-data-centers/ Glacier on BDXL optical disk: http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ Glacier on LDO6 tape: https://storageservers.wordpress.com/2012/10/22/amazon-glacier-storage-media-is-a-tape-driven-solution/ Facebooks uses Blu-Ray: http://arstechnica.com/information-technology/2014/01/facebook-uses-10000-blu-ray-discs-to-create-petabytes-of-cold-storage/ and http://money.cnn.com/2014/08/21/technology/facebook-blu-ray/
- Store and retrieve files through HTTPS Restrict access to authorized users Virtually unlimited storage space Pay only for what you use
- 99.99% percent availability over 1 year. That means the service is guaranteed by SLA to be up and running 99.99% of the time in any given year. Source: http://aws.amazon.com/s3/details/#durability
- S3 does ongoing fixity checks, replacing bad files with good ones. Amazon does not specify the schedule for these ongoing checks, but they use MD5. We do our own SHA-256 checks approximately every 30 days.
- Source for Glacier eleven nines: http://media.amazonwebservices.com/AWS_Storage_Options.pdf
- Nine 9’s = 1 in one hundred million
- Eleven nines = one in ten trillion
- This makes it sound like your data will still be there, even when the rest of the universe is gone.
- I don’t believe anything is this certain.
- If we can’t check the receiving buckets for new items, we try again in a few minutes.
- If download fails, we try again in a few minutes. If validate fails, we stop trying to process this bag.
- Storing files to S3 may fail on item #5000 of 10,000. System is smart enough to resume where it left off.
- A large bag may have tens of thousands of metadata records and events to record in Fedora. If Fedora cannot receive all metadata & events, system will try again later, resuming where it left off.
- If copy to Glacier fails after #5000 of 10,000 items, system is smart enough to resume where it left off.
- AWS has four zones in their Virginia data center, each with its own separate lines for power, backup power, and internet. If one zone goes down, others are not affected.
- S3 files are on separate devices in at least three zones.
- Our Fedora server is in one zone. We mirror its data to a second zone, so if the server or disk dies, we can bring up a new server with all the data. Ingest services don’t keep any data other than the bags they are working on. If those services die, we just restart them and they pick up where they left off. If the server dies, it can reconstruct its workload by checking with Fedora and the receiving buckets. It may duplicate work on the files it was processing when the server died, but it won’t lose work.
- Hoping to change this.
- That may take a couple of days.
- This may take a couple of days!
- These are scriptable, command-line tools. They are pre-compiled and require only a 5-line configuration file.
- These are scriptable, command-line tools. They are pre-compiled and require only a 5-line configuration file.
- These are scriptable, command-line tools. They are pre-compiled and require only a 5-line configuration file.
- First bag is not valid because it’s missing the md5 manifest. Second bag is valid.
- These are scriptable, command-line tools. They are pre-compiled and require only a 5-line configuration file.
- AVPreserve info about Glacier: http://www.avpreserve.com/wp-content/uploads/2014/04/AVPreserve_Glacier_profile.pdf
- A handful of digital collections are only available through our website Our SAN has archival masters for nearly (if not all) all of your digital collections, sometimes the only archival masters
- Starting with DSpace, need to work out strategy for other content sources 1st iteration RTS (outputs every community, collection, and record as an individual item) one bag per record Developed scripts to process RTS produced bags into APTrust compliant bags and upload to APTrust receiving bucket (using AWS CLI) <https://github.com/ntallman/APTrust_tools> abandoned because realized late in game that RTS wiped out dates in the file-system metadata
- 2nd iteration Tested bagit-Java and bagit-python, decided to use bagit-python Exported single collection from DSpace in Simple Archive Format 8,822 items, ~377 GB 250 GB bag-size limitation Collection had three series two series under 250 GB each series was put into a single bag one series over 250 GB used a multi-part bag to send to APTrust, APTrust merges split bags into one Should we have done one big multi-part bag? I.e. one bag per collection? Did not worry about tracking platform or collection in this bag Expected to one day wipe this content out and replace with something that matches our final strategy
- 2nd iteration Tested bagit-Java and bagit-python, decided to use bagit-python Exported single collection from DSpace in Simple Archive Format 8,822 items, ~377 GB 250 GB bag-size limitation Collection had three series two series under 250 GB each series was put into a single bag one series over 250 GB used a multi-part bag to send to APTrust, APTrust merges split bags into one Should we have done one big multi-part bag? I.e. one bag per collection? Did not worry about tracking platform or collection in this bag Expected to one day wipe this content out and replace with something that matches our final strategy
- 3rd iteration (really just testing) I'll let Jamie tell you more about this and I'll just say that I wish I had discovered this before creating my artisanal, hand-crafted bags. Basically a GUI wrapper for my same process except, added file characterization
- To Do Develop local preservation database or tool to track which content has been sent to APTrust Scholar@UC is a self-submit institutional repository. No hierarchy of collections (as we traditionally think about them) one record (work), many files Public commitment to long-term preservation of all Scholar content using APTrust for all content The flexible architecture of APTrust Preservation Repository (Are we allowed to say Fluctis?) allows us to make local decisions on how we package our content for preservation