PRESERVING
                   DIGITAL PUBLIC
                   TELEVISION
Part of the NDIIPP Program of the Library of Co...
NDIIPP =
National Digital Information
Infrastructure and Preservation
Program of the Library of Congress
www.digitalpreser...
PRESERVING
DIGITAL
PUBLIC
TELEVISION
PARTNERS         WNET
                           WGBH
                        NYU
   ...
by massdistraction via Flickr http://www.flickr.com/photos/sharynmorrow/3718174646/in/set-72157621271414097/




DIGITAL AR...
Design and build an
PDPTV   OAIS-compliant
        preservation repository
GOALS   for born digital public
        televis...
Implement and

PDPTV   recommend standards
        for metadata, wrapper

GOALS   and encoding formats,
        production...
PDPTV   Recommend selection
        criteria for long-term
GOALS   retention
PDPTV   Examine and recommend
        strategies for long term
GOALS   sustainability
“    An OAIS is an archive,
 consisting of an organization of
  people and systems, that has
 accepted the responsibility ...
PRESERVATION PLANNING
          DESCRIPTIVE                             DESCRIPTIVE

P            INFO                    ...
TECHNOLOGIES
            {
  some of the
 REPOSITORY

                PROJECT
                SPECIFIC
                 CO...
}
METADATA
 MODEL
by bredgur via Flickr: http://www.flickr.com/photos/bredgur/1323025528/




PHASE   2006-2008
SUBMISSION

wnet


          NYU
wgbh
       repository

pbs
SUBMISSION

wnet


          NYU
wgbh
       repository

pbs



             SIP Class A
SUBMISSION
       SD
wnet      ESS
              EN
                CE
                     A

                           ...
SUBMISSION
       SD
wnet      ES
       ME SEN
          TA     CE
             DA
               TA A
                  ...
SUBMISSION
       SD
wnet      ES
       ME SEN
          TA     CE
             DA
               TA A
                  ...
SUBMISSION
       SD
wnet      ES
       ME SEN
          TA     CE
             DA
               TA A
                  ...
SUBMISSION
       SD
wnet      ES
       ME SEN
          TA     CE
             DA
               TA B A
                ...
SUBMISSION
       SD
wnet      ES
       ME SEN
          TA     CE
             DA
               TA A
                  ...
SUBMISSION

wnet


                        NYU
wgbh
              ATA2   repository
           AD CE C
         ET EN
    ...
SUBMISSION

wnet

       SD ESSENCE E
       METADATA 3        NYU
wgbh
              ATA2    repository
           AD CE ...
1. Aggregate content
2. Normalize filenames *
3. Aggregate & map descriptive metadata to PBCore *
4. Extract Technical Meta...
by kapoue via Flickr http://www.flickr.com/photos/kapoue/2563697039/




PHASE   2008-2009
SUBMISSION



          NYU
wnet
       repository




            SIP Class A
SUBMISSION


       HD ESSENCE
                       NYU
wnet
                    repository




                        ...
SUBMISSION


         HD ESSENCE
       PBCORE METADATA      NYU
wnet
                         repository




            ...
SUBMISSION



          NYU
wnet
       repository




            SIP Class B
SUBMISSION



                         NYU
wnet
       SOURCE FILES   repository




                           SIP Class B
SUBMISSION



       PBCORE METADATA      NYU
wnet
        SOURCE FILES     repository




                              S...
1. Normalize filenames
2. Generate PREMIS
3. Generate METS
4. Validate checksum
5. INGEST
                     REPOSITORY
 ...
PHASE 1      PHASE 2


12 5
processing
   steps
             processing
                steps
?
What changed
[preservation-ready]
FILE-BASED WORKFLOW




      by Brian Daniel Eisenberg via Flickr http://www.flickr.com/photos/pplpwr...
File & Folder
   naming
conventions
      for
 production
 and post-
 production
                by drpritch via Flickr ht...
Standard
                        MXF

                       DV100     Recording
 formats &
settings for
                 ...
Technical
 metadata
extraction in
   house
mediainfo.sourceforge.net/en
PBCore
  records
  created
  in-house
pbcore.vermicel.li
Archiving can integrate
seamlessly into file-based
broadcast workflows if the right
practices are introduced early on
a few more
LESSONS
LEARNED
3
DIGITAL PRESERVATION
      REQUIREMENTS:
1. Bit Preservation
2. Accessibility of Content
3. Organizational Commitment
1
BIT PRESERVATION
ONE COPY IS NO COPY


by NightRPStar via Flickr http://www.flickr.com/photos/ninjanoodles/153893226/
“       (rules define how many copies
         to make, and which locations
          to put these in, with a typical
     ...
Photo by quapan via Flickr http://www.flickr.com/photos/hinkelstone/2435823037/




Consider federated storage models
    f...
2
CONTENT ACCESSIBILITY
by Shira Golding via Flickr http://www.flickr.com/photos/boojee/3743753784/
Define minimum metadata
          creation and collection
requirements & rules, throughout
     the production & broadcast
...
by ScrapyGraphics via Flickr http://www.flickr.com/photos/scrapygraphics/2515645664/




including file and folder naming
co...
Metadata should be
standards-based
by DG Jones via Flickr http://www.flickr.com/photos/dgjones/1225183400/




a few words about
file formats...
“ Businesses may use different
  encoding formats for different
 business processes, but should
  strive to avoid transcod...
3   ORGANIZATIONAL
      COMMITMENT
“
Benign neglect is the default
stewardship, collection policy.
Physical world, even more so
                             ...
TRUSTWORTHY
REPOSITORIES AUDIT
AND CERTIFICATION:
   CRITERIA AND
    CHECKLIST
coming soon...

THE AMERICAN
    ARCHIVE
  of public broadcasting
THANK
 YOU
   www.thirteen.org/ptvdigitalarchive
          kvm211@nyu.edu
http://www.slideshare.net/kvanmalssen
Preserving Digital Public Television: A Status Report
Preserving Digital Public Television: A Status Report
Upcoming SlideShare
Loading in...5
×

Preserving Digital Public Television: A Status Report

1,250

Published on

Some findings from the Preserving Digital Public Television Project as it enters its final month. The project was a collaboration between WNET, WGBH, PBS, and NYU as part of the Library of Congress's National Digital Information Infrastructure and Preservation Program (NDIIPP). This talk was prepared for the Society of Motion Picture and Television Engineers NY Section Meeting, February 24, 2010.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,250
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
40
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

  • Early context for the project was that public television was supposed to depositing one copy of all programs at LC. That hasn’t been happening.
  • WNET and WGBH combined produce about 60% of nationally broadcast programming in the US. But they also produce and distribute local programming. PBS is only a distributor - not a content creator. NYU had expertise in building digital libraries (which public broadcasting did not) and an existing PR. LC is the funder, but also a repository.
  • Why are we doing this?

    - Jacques Cousteau story.


  • We’re not dealing with digitizing things.



  • An archive in this sense is not a server. It is processes, procedures, people with a mission of preservation, for which it is responsible into the indefinite future.

  • “The PR is designed as a set of loosely-coupled components communicating over stable interfaces.”
    Storage Resource Broker = supports shared collections that can be distributed across multiple organizations and heterogeneous storage systems. Can be used as a Data Grid Management System.
    Dspace = DSpace open source software, used for Archival Storage, Data Management, Dissemenation functions of OAIS


































  • These steps need just to process any one of those SIPs. These were the basic steps, but they were slightly different for each SIP class because of different metadata, different file formats, different PREMIS.















  • Processing is the same for both SIP classes (production masters and source files).


  • The “p” word was never used. It actually made sense to make changes during transition, and no practices were too entrenched yet, but they didn’t want to say it was for archival reasons, just that this was the way it was going to be done.


  • Settings: Frame Size (1080i), Aspect Ratio (16/9), Frame Rate (29.97), Data Rate (117 mbps)

  • Including technical & descriptive metadata





  • Consider a grid or distributed but federated system
  • Combined with local storage for most frequently accessed materials with Grid solutions. Make sure your approached is managed. Take a look at the AVATARm project in the UK for more info.

  • One word. You are going to need a lot of it. Video files are not self describing. Filenames are not good for search and retrieval, file-level metadata is not searchable.
  • If there are no cataloging rules for descriptive metadata, everyone will input differently. Combine cataloging rules with controlled vocabularies.

  • Because: you won’t have to reinvent the wheel (elements, definitions, vocab), facilitates exchange, there is a support community.
  • A few things about file formats: There is still no standard format for video preservation, especially for born digital, because it is born compressed. The most important thing to do is choose an open, widely support encoding format, that can be used in all systems in your core business processes without transcoding. MXF or QT (FCP) containers.


  • Preservation does not happen in a vacuum. There must be ongoing commitment, funds, staffing, reviewed and updated policies and procedures, etc.



  • Preserving Digital Public Television: A Status Report

    1. 1. PRESERVING DIGITAL PUBLIC TELEVISION Part of the NDIIPP Program of the Library of Congress A STATUS REPORT Kara Van Malssen Senior Research Scholar & Metadata Specialist New York University February 24, 2010 SYMPTE-NY Section Meeting
    2. 2. NDIIPP = National Digital Information Infrastructure and Preservation Program of the Library of Congress www.digitalpreservation.gov Image by Smiley Man with a Hat via Flickr http://www.flickr.com/photos/smileymanwithahat/2477365291/
    3. 3. PRESERVING DIGITAL PUBLIC TELEVISION PARTNERS WNET WGBH NYU PBS Library of Congress
    4. 4. by massdistraction via Flickr http://www.flickr.com/photos/sharynmorrow/3718174646/in/set-72157621271414097/ DIGITAL ARCHAEOLOGY?
    5. 5. Design and build an PDPTV OAIS-compliant preservation repository GOALS for born digital public television
    6. 6. Implement and PDPTV recommend standards for metadata, wrapper GOALS and encoding formats, production workflow practices
    7. 7. PDPTV Recommend selection criteria for long-term GOALS retention
    8. 8. PDPTV Examine and recommend strategies for long term GOALS sustainability
    9. 9. “ An OAIS is an archive, consisting of an organization of people and systems, that has accepted the responsibility to preserve information and make it available for a designated community.... this distinguishes it “ from other uses of the term ‘archive.’ - Reference Model for an Open Archival Information System, ISO 14721:2003
    10. 10. PRESERVATION PLANNING DESCRIPTIVE DESCRIPTIVE P INFO INFO C R DATA O O MANAGEMENT queries N D S result sets U SIP INGEST ACCESS orders U C M ARCHIVAL E E STORAGE DIP R AIP R AIP ADMINISTRATION MANAGEMENT OAIS Functional Model
    11. 11. TECHNOLOGIES { some of the REPOSITORY PROJECT SPECIFIC CODE
    12. 12. } METADATA MODEL
    13. 13. by bredgur via Flickr: http://www.flickr.com/photos/bredgur/1323025528/ PHASE 2006-2008
    14. 14. SUBMISSION wnet NYU wgbh repository pbs
    15. 15. SUBMISSION wnet NYU wgbh repository pbs SIP Class A
    16. 16. SUBMISSION SD wnet ESS EN CE A NYU wgbh repository pbs SIP Class A
    17. 17. SUBMISSION SD wnet ES ME SEN TA CE DA TA A 1 NYU wgbh repository pbs SIP Class A
    18. 18. SUBMISSION SD wnet ES ME SEN TA CE DA TA A 1 NYU wgbh repository C EC EN E SS SD pbs SIP Class A
    19. 19. SUBMISSION SD wnet ES ME SEN TA CE DA TA A 1 NYU wgbh ATA2 repository AD CE C ET EN M E SS SD pbs SIP Class A
    20. 20. SUBMISSION SD wnet ES ME SEN TA CE DA TA B A 1 NYU wgbh ATA2 repository AD CE C ET EN M E SS SD pbs SIP Class B
    21. 21. SUBMISSION SD wnet ES ME SEN TA CE DA TA A 1 NYU wgbh AT A2 repository AD CE C ET EN M E SS ED D NC S SE ES pbs HD SIP Class C
    22. 22. SUBMISSION wnet NYU wgbh ATA2 repository AD CE C ET EN M E SS SD pbs SIP Class D
    23. 23. SUBMISSION wnet SD ESSENCE E METADATA 3 NYU wgbh ATA2 repository AD CE C ET EN M E SS SD pbs SIP Class D
    24. 24. 1. Aggregate content 2. Normalize filenames * 3. Aggregate & map descriptive metadata to PBCore * 4. Extract Technical Metadata 5. Map Technical Metadata to PBCore * 6. Generate PBCore * 7. Hunt down creating app info 8. Determine playback reqs * 9. Generate PREMIS * 10. Generate checksums 11. Generate METS * REPOSITORY 12. INGEST INGEST TASKS
    25. 25. by kapoue via Flickr http://www.flickr.com/photos/kapoue/2563697039/ PHASE 2008-2009
    26. 26. SUBMISSION NYU wnet repository SIP Class A
    27. 27. SUBMISSION HD ESSENCE NYU wnet repository SIP Class A
    28. 28. SUBMISSION HD ESSENCE PBCORE METADATA NYU wnet repository SIP Class A
    29. 29. SUBMISSION NYU wnet repository SIP Class B
    30. 30. SUBMISSION NYU wnet SOURCE FILES repository SIP Class B
    31. 31. SUBMISSION PBCORE METADATA NYU wnet SOURCE FILES repository SIP Class B
    32. 32. 1. Normalize filenames 2. Generate PREMIS 3. Generate METS 4. Validate checksum 5. INGEST REPOSITORY INGEST TASKS
    33. 33. PHASE 1 PHASE 2 12 5 processing steps processing steps
    34. 34. ? What changed
    35. 35. [preservation-ready] FILE-BASED WORKFLOW by Brian Daniel Eisenberg via Flickr http://www.flickr.com/photos/pplpwrd/2673102206/
    36. 36. File & Folder naming conventions for production and post- production by drpritch via Flickr http://www.flickr.com/photos/drpritch/305053820/
    37. 37. Standard MXF DV100 Recording formats & settings for MXF DV100 Transfer to HDD recording, QT Ingest & Re- editing, DV100 wrap broadcast, QT Edit archiving DV100 (i.e. no transcoding QT during workflow) DV100 Playout
    38. 38. Technical metadata extraction in house mediainfo.sourceforge.net/en
    39. 39. PBCore records created in-house pbcore.vermicel.li
    40. 40. Archiving can integrate seamlessly into file-based broadcast workflows if the right practices are introduced early on
    41. 41. a few more LESSONS LEARNED
    42. 42. 3 DIGITAL PRESERVATION REQUIREMENTS: 1. Bit Preservation 2. Accessibility of Content 3. Organizational Commitment
    43. 43. 1 BIT PRESERVATION
    44. 44. ONE COPY IS NO COPY by NightRPStar via Flickr http://www.flickr.com/photos/ninjanoodles/153893226/
    45. 45. “ (rules define how many copies to make, and which locations to put these in, with a typical strategy being 3 copies in 3 geographically separate “ locations) - M. Addis, et al “Sustainable Archiving and Storage Management of Audiovisual Digital Assets” SMPTE Motion Imaging Journal, Nov/Dec 2009
    46. 46. Photo by quapan via Flickr http://www.flickr.com/photos/hinkelstone/2435823037/ Consider federated storage models for cost and sustainability reasons
    47. 47. 2 CONTENT ACCESSIBILITY
    48. 48. by Shira Golding via Flickr http://www.flickr.com/photos/boojee/3743753784/
    49. 49. Define minimum metadata creation and collection requirements & rules, throughout the production & broadcast workflow
    50. 50. by ScrapyGraphics via Flickr http://www.flickr.com/photos/scrapygraphics/2515645664/ including file and folder naming conventions
    51. 51. Metadata should be standards-based
    52. 52. by DG Jones via Flickr http://www.flickr.com/photos/dgjones/1225183400/ a few words about file formats...
    53. 53. “ Businesses may use different encoding formats for different business processes, but should strive to avoid transcoding wherever possible, because it “ introduces a generation and thus reduces quality. - Peter Thomas “File Formats in Television Archiving and Content Exchange” SMPTE Motion Imaging Journal, Nov/Dec 2009
    54. 54. 3 ORGANIZATIONAL COMMITMENT
    55. 55. “ Benign neglect is the default stewardship, collection policy. Physical world, even more so “ in the digital world. - Cathy Marshall, Senior Researcher, Microsoft Research Keynote at Code4Lib, February 23, 2010. via Twitter @jschneider
    56. 56. TRUSTWORTHY REPOSITORIES AUDIT AND CERTIFICATION: CRITERIA AND CHECKLIST
    57. 57. coming soon... THE AMERICAN ARCHIVE of public broadcasting
    58. 58. THANK YOU www.thirteen.org/ptvdigitalarchive kvm211@nyu.edu http://www.slideshare.net/kvanmalssen
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×