Puglia marac-file formats-20111020


Published on

Published in: Technology, Art & Photos
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Puglia marac-file formats-20111020

  1. 1. Revisiting File Formats for Digitization Steven T. Puglia Digital Conversion Services Manager Office of Strategic Initiatives Library of Congress 101 Independence Ave, SE Washington, DC 20540, USA Phone: 202-707-5726 Email: spug@loc.gov
  2. 2. <ul><li>Federal Agencies Digitization Guidelines Initiative </li></ul><ul><li>www.digitizationguidelines.gov </li></ul><ul><ul><li>Working Groups- </li></ul></ul><ul><ul><ul><li>Still Image </li></ul></ul></ul><ul><ul><ul><li>Audio-Visual </li></ul></ul></ul><ul><ul><ul><li>Authentication </li></ul></ul></ul>
  3. 3. In general, within the digital library community, format and compression recommendations for master and derivative image files remain based on older perspectives regarding digitization, digital preservation, and IT/network/web technologies.
  4. 4. <ul><li>In the current, and likely future, constrained fiscal climate, file format and image compression recommendations should be reviewed and revised to- </li></ul><ul><li>Take into account cost-effectiveness and efficiency in: </li></ul><ul><ul><li>Mass digitization projects </li></ul></ul><ul><ul><li>Short-term digital storage </li></ul></ul><ul><ul><li>Derivative production </li></ul></ul><ul><ul><li>Moving files into and within an IT, network, and web delivery infrastructure </li></ul></ul><ul><ul><li>Digital preservation and long-term storage </li></ul></ul><ul><li>Reflect current perspectives on digital preservation; take advantage of new computer, IT, and web technologies; and respond to the changing expectations of researchers and users. </li></ul>
  5. 5. <ul><li>For still or raster images, considering: </li></ul><ul><ul><li>TIFF </li></ul></ul><ul><ul><li>JPEG 2000 </li></ul></ul><ul><ul><li>JPEG </li></ul></ul><ul><ul><li>PNG </li></ul></ul><ul><ul><li>PDF and PDF/A </li></ul></ul>
  6. 6. <ul><li>Not just sustainability, considering: </li></ul><ul><ul><li>Authenticity </li></ul></ul><ul><ul><li>Data resilience and error recovery </li></ul></ul><ul><ul><li>Embedded metadata support </li></ul></ul><ul><ul><li>Fiscal implications </li></ul></ul><ul><ul><li>Sustainability Confidence and Pros/Cons </li></ul></ul><ul><ul><li>Technical Pros/Cons </li></ul></ul><ul><ul><li>Data resilience and error recovery </li></ul></ul><ul><ul><li>Ease and accuracy of validation </li></ul></ul>
  7. 7. <ul><li>David Rosenthal, Stanford University- </li></ul><ul><li>http://blog.dshr.org/2011/09/whats-wrong-with-research-communication.html </li></ul><ul><li>“ If we were starting with a blank sheet of paper to design a mechanism for communicating about research, what would be our requirements? </li></ul><ul><ul><li>Repeatability </li></ul></ul><ul><ul><li>Reusability </li></ul></ul><ul><ul><li>Immediacy </li></ul></ul><ul><ul><li>Transparency </li></ul></ul><ul><ul><li>Openness </li></ul></ul><ul><ul><li>Sustainability </li></ul></ul><ul><ul><li>Permanence </li></ul></ul><ul><ul><li>Authenticity ” </li></ul></ul>
  8. 8. Recommended Data Formats for Preservation Purposes in the Florida Digital Archive http://fclaweb.fcla.edu/uploads/Lydia%20Motyka/FDA_documentation/recFormats.pdf
  9. 9. <ul><li>Sustainability Confidence: </li></ul><ul><ul><li>High </li></ul></ul><ul><ul><ul><li>TIFF - uncompressed </li></ul></ul></ul><ul><ul><ul><li>JPEG 2000 - JP2 and lossless compression </li></ul></ul></ul><ul><ul><ul><li>PNG </li></ul></ul></ul><ul><ul><ul><li>PDF/A – FCLA recommends for text </li></ul></ul></ul><ul><ul><li>Medium </li></ul></ul><ul><ul><ul><li>TIFF – lossless compressed </li></ul></ul></ul><ul><ul><ul><li>JPEG </li></ul></ul></ul><ul><ul><ul><li>JPEG 2000 – JP2 and lossy compression </li></ul></ul></ul><ul><ul><ul><li>PDF – FCLA recommends for text </li></ul></ul></ul>
  10. 10. <ul><li>For audio: </li></ul><ul><ul><li>Broadcast WAV </li></ul></ul><ul><li>For video: </li></ul><ul><ul><li>MXF (Media eXchange Format) </li></ul></ul><ul><ul><ul><li>Working on application specification ( http://www.digitizationguidelines.gov/guidelines/MXF_app_spec.html) </li></ul></ul></ul><ul><ul><li>Library of Congress planning on lossless JPEG2000 encoding </li></ul></ul><ul><ul><li>( http://www.digitizationguidelines.gov/still-image/documents/Snyder.pdf </li></ul></ul><ul><ul><li>and http://www.digitalpreservation.gov/news/events/other_meetings/storage11/docs/04_snyder.pdf ) </li></ul></ul><ul><ul><li>SMPTE working on AXF or Archive eXchange Format </li></ul></ul>
  11. 11. <ul><li>Image and File Compression? </li></ul><ul><ul><li>Yes! No! Maybe! </li></ul></ul><ul><ul><li>Circling back to yes – when appropriate and using a reasoned approach </li></ul></ul>
  12. 12. <ul><li>Howard Besser, NYU </li></ul><ul><li>File Compression Strategies, ALA Mid-Winter Panel, 1999 (RLG DigiNews) </li></ul><ul><li>The Short Life of Digital Information – The Scrambling Problem </li></ul><ul><ul><li>In order to solve short-term problems resulting from the use of digital technology, we've engaged in practices that may result in long-term peril. </li></ul></ul><ul><ul><li>In the past, because large-scale storage was costly and bandwidth was fairly narrow, many repositories responded to these constraints by compressing their master images or multimedia. </li></ul></ul>
  13. 13. <ul><li>According to the reasoning that dominated until recently, compressed master files take up less storage, are easier to deliver to users with slow network connections, and are more convenient to handle internally. </li></ul><ul><li>In recent years, a number of institutions have come to question this tenet as storage costs have plummeted and network speeds have dramatically increased. </li></ul><ul><li>Compression creates a number of problems. </li></ul><ul><li>Another very important issue is that both lossy and lossless compression add still another level of complexity to the encoding of a file, making it even more difficult for future archeologists trying to decipher its contents. </li></ul>
  14. 14. <ul><li>Lou Sharpe, Picture Elements </li></ul><ul><li>File Compression Strategies, ALA Mid-Winter Panel, 1999 (RLG DigiNews) </li></ul><ul><ul><li>Advocated the use of visually lossless (but lossy) compression with certain types of originals. Because &quot;practical people make practical decisions,&quot; </li></ul></ul><ul><ul><li>Time, conversion costs, staffing, etc., are all factors in the decision-making process for digital reformatting. </li></ul></ul><ul><ul><li>He called attention to a handout that framed his position as a debate proposition: </li></ul></ul><ul><ul><ul><li>1. Resolved, that visually lossless (yet lossy) compression of tonal images of illustrated book pages can be used to create high-quality digital masters if…conditions are met. </li></ul></ul></ul>
  15. 15. 2. Further resolved, that such images are of sufficient quality to serve as preservation images for books which are: Found in the stacks, not in the rare book room. Likely to remain available somewhere in physical form. 3. Further resolved, that such images are of comparable or superior quality to accepted preservation approaches such as microfilm. 4. Further resolved, that cost matters in digital library image conversion projects, even though it is other people's money. With a final nod to the improvements proposed for JPEG 2000, Sharpe argued that at a minimum, the library and archival community should not close the door on the use of visually lossless compression.
  16. 16. <ul><li>Shift in thinking began in the mid-2000s. </li></ul><ul><li>A lot of consideration- </li></ul><ul><ul><li>In Europe </li></ul></ul><ul><ul><li>By large university libraries </li></ul></ul><ul><ul><li>By those dealing with mass digitization </li></ul></ul><ul><li>It is a matter of- </li></ul><ul><ul><li>Scale and economics </li></ul></ul><ul><ul><li>A better understanding of the realities and risks of digital preservation </li></ul></ul>
  17. 17. <ul><li>“ Exabytes: Documenting the ‘digital age’ and huge growth in computing capacity,” by Brian Vastag, Washington Post, Feb. 10, 2011 </li></ul><ul><li>http ://www.washingtonpost.com/wp-dyn/content/article/2011/02/10/AR2011021004916.html </li></ul><ul><ul><li>Megabytes are dead. </li></ul></ul><ul><ul><li>Gigabytes are passe. </li></ul></ul><ul><ul><li>So much digital data now moves around the globe that those who endeavor to measure it employ a new - or new to non-nerds - term. </li></ul></ul><ul><ul><li>Meet the exabyte. </li></ul></ul>
  18. 18. Rise of Information in the Digital Age http://www.washingtonpost.com/wp-dyn/content/graphic/2011/02/11/GR2011021100614.html?sid=ST2011021100514
  19. 19. Really big data: The challenges of managing mountains of information, by John Brandon, October 18, 2011 http://www.computerworld.com/s/article/9220504/Really_big_data_The_challenges_of_managing_mountains_of_information ? The Library of Congress processes 2.5 petabytes of data each year, which amounts to 40TB per week. Thomas Youkel, group chief of enterprise systems engineering at the Library, estimates the data load will quadruple in the next few years as the Library continues to carry out its dual mandates to serve up data for historians and preserve information in all its forms.
  20. 20. <ul><li>David Rosenthal, Stanford University </li></ul><ul><li>http://blog.dshr.org/2010/12/rob-sharpes-case-for-format-migration.html </li></ul><ul><li>My case is that, as we see from the last few years focus on &quot;sustainability of digital preservation&quot;, the major problems in digital preservation are economic. </li></ul><ul><li>(Blue Ribbon Task Force on Sustainable Digital Preservation and Access - http://brtf.sdsc.edu/) </li></ul>
  21. 21. Andy Jackson, The British Library http://www.openplanetsfoundation.org/blogs/2011-01-12-format-obsolescence-and-sustainable-access This means that the long-term cost of preserving our collection scales not only with the size of the files, but also rises as the number of formats we are required to support is increased.
  22. 22. <ul><li>We can not avoid dealing with compressed file formats, including lossy compressed- </li></ul><ul><ul><li>Currently, up to 375 billion digital photos are taken each year and the number continues to increase - “ How many photos have ever been taken?” by Jonathan Good, Sept. 15, 2011 </li></ul></ul><ul><ul><li>This is orders of magnitude more raster image files than are being produced by digitization efforts </li></ul></ul><ul><ul><li>We can expect pretty much all of these 375 billion digital photos per year are JPG files </li></ul></ul><ul><ul><li>The answer for digital preservation is not going to be insisting all image files be saved as uncompressed formats </li></ul></ul>
  23. 23. David Rosenthal, Stanford University http://blog.dshr.org/2011/03/how-few-copies.html Compression reduces the redundancy within a single copy and increases the risk of damage. There are also techniques that increase the redundancy within a single copy and reduce the risk.
  24. 24. <ul><li>David Rosenthal, Stanford University </li></ul><ul><li>http://blog.dshr.org/2011/03/how-few-copies.html </li></ul><ul><li>If you ask the people who run large data centers what are the most important causes of data loss, you get a list like this: </li></ul><ul><ul><li>Operator error </li></ul></ul><ul><ul><li>External Attack </li></ul></ul><ul><ul><li>Insider Attack </li></ul></ul><ul><ul><li>Economic Failure </li></ul></ul><ul><ul><li>Organization Failure </li></ul></ul>
  25. 25. Erik Hetzner, California Digital Library http://groups.google.com/group/digital-curation/msg/b487a1b0188f9c0c I see no reason to store, as a matter of policy, uncompressed files on our disks. In fact, I think we should be more aggressive about compressing files. (Hetzner focuses on lossless compression.)
  26. 26. Erik Hetzner, California Digital Library http://groups.google.com/group/digital-curation/msg/b487a1b0188f9c0c Even without error correcting codes, I don’t think the arguments for storing uncompressed data only as a matter of policy are strong at all. When we take error correcting codes into account, not compressing your data as a policy in order to keep a higher level of redundancy seems like the worst way to increase the redundancy of the data. Smart people have figured out how to make codes which can reliably correct limited errors in bytestreams. Why not use them?
  27. 27. Data corruption is and will remain a problem. An active part of digital preservation will be to overcome this problem. The LOCKSS concept includes one approach for dealing with the problem – “…the bits and bytes are continually audited and repaired…to protect fragile digital content for the very long time.” http://www.eecs.harvard.edu/~mema/publications/SOSP2003.pdf LOCKSS now has a 12 year track record.
  28. 28. <ul><li>David Rosenthal, Stanford University </li></ul><ul><li>http://blog.dshr.org/2011/03/how-few-copies.html </li></ul><ul><li>Thus we can say that some digital content is going to get lost or damaged. This shouldn't be news; the same is true of analog content. </li></ul><ul><li>We have rules of thumb to guide us in trying to reduce the amount of loss and damage: </li></ul><ul><ul><li>The more copies the safer </li></ul></ul><ul><ul><li>The less correlated the copies the safer </li></ul></ul><ul><ul><li>The more reliable each copy the safer </li></ul></ul><ul><ul><li>The faster failures are detected and repaired the safer </li></ul></ul><ul><ul><li>The less aggressive the compression the smaller the effect of damage </li></ul></ul>
  29. 29. If image files are being brought into a managed environment, compression, particularly lossless compression, is much less of a concern. Conversely, if images are being stored on DVDs on a shelf, then compression raises the risks significantly.
  30. 30. One option for file format and compression (lossless and lossy) - JPEG 2000
  31. 31. There remain barriers for many organizations to adoption of JPEG 2000 (limited open source tools), and concerns and related potential risks (corruption and potential legal issues). These issues have been acknowledged within the broader cultural heritage digitization community.
  32. 32. A number of research studies have been conducted on the robustness of JPEG 2000. Studies have seen similar results in terms of susceptibility to corruption. Nevertheless, organizations have concluded that JPEG 2000 is an appropriate file format choice from a robustness perspective – “conclude that JPEG 2000 is a good current solution for our digital repositories.” A Format for Digital Preservation of Images by Buonora and Liberati http://www.dlib.org/dlib/july08/buonora/07buonora.html
  33. 33. It is worth noting the format includes some “resiliency” elements that add robustness and thereby counteract some effects of data loss. These resiliency elements are described in the notes at the bottom of the Sustainability of Digital Formats – Planning for Library of Congress web page ( http://www.digitalpreservation.gov/formats/fdd/fdd000138.shtml) .
  34. 34. Wellcome Library http://jpeg2000wellcomelibrary.blogspot.com/2010/06/we-need-how-much-storage.html In 2009, the Wellcome Library set out an ambitious vision to digitise a large proportion of its historic collections. This would take the annual digitisation activities of the Library from hundreds, or at most, thousands of images per year to several million images per year. … we realised this could see the generation of up to 30m images over 5 years. Exciting, but perhaps slightly daunting, considering we didn't yet have an infrastructure to fully support such a large collection of digital assets.
  35. 35. Wellcome Library- Anyone reading this blog will understand why the scale of the programme is key to the blog topic. When we asked our IT department to tell us how much it would cost to store 30m TIFF files - our de facto standard for the couple hundred thousand images in our existing picture library - we were stunned. Two petabytes of online, spinning disk storage with a top-of-the-line enterprise management system and remote backup would cost how much? We learned that the cost would be something like a fifth of our total budget for the entire digitisation programme.
  36. 36. Wellcome Library- Should we consider a lower-cost storage solution? Even tape back-up was quite expensive for that scale, and you can't serve images up online from tape anyway. We revised our image sizes, factoring in smaller and smaller resolutions and/or bit depths for material like the printed books, which didn't need full colour, high resolution images. We still couldn't afford the storage costs. Finally, we saw the light and started looking into a relatively new image format called JPEG 2000 .
  37. 37. <ul><li>JPEG 2000 Summit </li></ul><ul><ul><li>The thought process and rationale for a decision to adopt and implement JPEG 2000 </li></ul></ul><ul><ul><li>What are the advantages and disadvantages? When does JPEG 2000 make sense? </li></ul></ul><ul><ul><li>What are the barriers to adoption and implementation for organizations interested in JPEG 2000? </li></ul></ul><ul><ul><li>How do we as a community overcome the barriers? </li></ul></ul><ul><ul><li>For organizations that decide JPEG 2000 is a good match for their needs and goals, what is needed to make adoption and implementation practical and feasible? </li></ul></ul><ul><ul><li>Who is willing to work on these efforts and how do we move forward? </li></ul></ul>
  38. 38. <ul><li>Organizations using or accepting JPEG 2000: </li></ul><ul><ul><li>British Library </li></ul></ul><ul><ul><li>National Library of the Czech Republic </li></ul></ul><ul><ul><li>Google </li></ul></ul><ul><ul><li>Harvard University </li></ul></ul><ul><ul><li>Internet Archive </li></ul></ul><ul><ul><li>Library of Congress </li></ul></ul><ul><ul><ul><li>National Audio Visual Conservation Center </li></ul></ul></ul><ul><ul><ul><li>National Digital Newspaper Program </li></ul></ul></ul><ul><ul><li>National Library of the Netherlands </li></ul></ul><ul><ul><li>National Library of Norway </li></ul></ul><ul><ul><li>Wellcome Library </li></ul></ul>
  39. 39. It is very possible, more digital images are produced by mass digitization efforts and saved as JPEG 2000 files than other file formats. Despite concerns and a clear need for organizational support relating to implementing JPEG 2000, far more cultural heritage organizations are using JPEG 2000 for digitization than most people realize.
  40. 40. <ul><li>Also, JPEG 2000 is widely implemented in other communities as well- </li></ul><ul><ul><li>Digital cinema </li></ul></ul><ul><ul><li>Medical imaging </li></ul></ul><ul><ul><li>Geospatial/remote sensing </li></ul></ul><ul><ul><li>Law enforcement (facial image compression for biometrics). </li></ul></ul><ul><li>The Department of Defense and Intelligence Community have adopted the ISO standard for JPEG 2000 for the National Imagery Transmission Format standard as well. </li></ul>
  41. 41. Conclusions: There is not a single answer to the question of file format for raster image files produced by digitization projects. There are a number of file formats worthy of consideration – suitable from technical, sustainability, fiscal, and other perspectives. Compression can represent a reasonable risk for appropriate efforts, and is likely a practical reality as digitization and digital preservation efforts scale. Not using compression likely represents a real risk, particularly given the dramatic and continued growth in digital data.