A Library Preservation Challenge Managing the collective collection over time Janifer Gatenby, OCLC Western European Studies Section (ACRL WESS) June 30 2008

Fundamental Questions What are the challenges? What are the responses to these challenges / problems? Are they adequate in themselves? Does the ensemble make coherent sense?

What are the challenges?

What are the responses to these challenges / problems?

Are they adequate in themselves?

Does the ensemble make coherent sense?

Challenge: We lose valuables Natural disaster, Deterioration, War, Lack of care or will We also lose valuable digital things Broken links, broken code, buried data, lack of description (context) Only 20% of silent films survive Theda Bara: 3.5 of 40 films Fire Fire

Natural disaster, Deterioration, War,

Lack of care or will

We also lose valuable digital things

Broken links, broken code, buried data, lack of description (context)

“When scores of books packed into shelves or in piles are set ablaze, the pages fuse and carbonize, turning into clinkers in the intense heat due to the lack of oxygen.” Bosnia's National Library in Sarajevo, shelled and burned August 25-26, 1992

“When scores of books packed into shelves or in piles are set ablaze, the pages fuse and carbonize, turning into clinkers in the intense heat due to the lack of oxygen.”

Bosnia's National Library in Sarajevo, shelled and burned August 25-26, 1992

Looking at the Challenges That which cannot be found…. is lost Increasing part of library collection is outside the building – digital and physical Broken digital links Obscurity – failure to rank highly That which is not regularly accessed…. may be lost E.g. PDFs that will not open because they contain outdated fonts Openers may get lost Little used resources may be weeded, in ignorance of their rare value

That which cannot be found…. is lost

Increasing part of library collection is outside the building – digital and physical

Broken digital links

Obscurity – failure to rank highly

That which is not regularly accessed…. may be lost

E.g. PDFs that will not open because they contain outdated fonts

Openers may get lost

Little used resources may be weeded, in ignorance of their rare value

Looking at the Challenges Limited space in library buildings Yet physical collection continues to grow British Library storage increases by 12 kilometres a year “ in 2006 291,920 new titles were published in the U.S., and the number of new books has increased nearly every year for the last decade, despite the spread of electronic publishing” Robert Darnton Increase in grey material Self publishing; no peer review Increase in material of dubious merit More difficult to detect resources of worth How much should we collect and preserve?

Limited space in library buildings

Yet physical collection continues to grow

British Library storage increases by 12 kilometres a year

“ in 2006 291,920 new titles were published in the U.S., and the number of new books has increased nearly every year for the last decade, despite the spread of electronic publishing” Robert Darnton

Increase in grey material

Self publishing; no peer review

Increase in material of dubious merit

More difficult to detect resources of worth

How much should we collect and preserve?

Preservation Level Regional & National Individually not unique but if lost would destroy regional landscape Streetscape Leiden, Netherlands Global Works of international significance in their own right Rijnland Gemeenschaft, Leiden Ship’s Carpenter’s building, Leiden

Regional & National

Individually not unique but if lost would destroy regional landscape

Streetscape Leiden, Netherlands

Global

Works of international significance in their own right

Rijnland Gemeenschaft, Leiden

Ship’s Carpenter’s building, Leiden

Responses Collective physical stores and services Digitisation Collective digital stores Beyond simple storage to true preservation Exposure

Collective physical stores and services

Digitisation

Collective digital stores

Beyond simple storage to true preservation

Exposure

Collective physical stores Central independent physical stores National Repository Library of Finland, Kuopio (400kms NE of Helsinki) 1989- Norwegian Repository Library CTLes (Centre Technique du Livres de L’Enseignement Supérieur) CASS (Cooperative Academic Store of Scotland) One library nominated for region University of Regensburg, Bavaria (following failures in North Rhine Westphalia and Baden-Württemberg) Central / Distributed hybrid UK Research Reserve Project (based on BL but retaining 2 to 3 other copies among the cooperative)

Central independent physical stores

National Repository Library of Finland, Kuopio (400kms NE of Helsinki) 1989-

Norwegian Repository Library

CTLes (Centre Technique du Livres de L’Enseignement Supérieur)

CASS (Cooperative Academic Store of Scotland)

One library nominated for region

University of Regensburg, Bavaria (following failures in North Rhine Westphalia and Baden-Württemberg)

Central / Distributed hybrid

UK Research Reserve Project (based on BL but retaining 2 to 3 other copies among the cooperative)

Physical Stores – an adequate response? Rely on union catalogues – for collection management and exposure Finnish receive uncatalogued material and catalogue it UK Research Reserve: known problems Serial holdings in union catalogues are “patchy” And not updated regularly enough And don’t include retention policy (new ISO standard 20775 – Schema for Holdings – includes dynamic and policy information for query responses) Rely on effective delivery systems & D igitisation o n D emand For DOD need copyright evidence Lack comprehensive, simple international request and delivery

Rely on union catalogues – for collection management and exposure

Finnish receive uncatalogued material and catalogue it

UK Research Reserve: known problems

Serial holdings in union catalogues are “patchy”

And not updated regularly enough

And don’t include retention policy (new ISO standard 20775 – Schema for Holdings – includes dynamic and policy information for query responses)

Rely on effective delivery systems & D igitisation o n D emand

For DOD need copyright evidence

Lack comprehensive, simple international request and delivery

Digitisation We cannot digitise everything in the forseeable future CENL 2007 est. – only 1% digitised 2005 – Google books 10 yr plan is only covering 35% of WorldCat Jilovsky 2008 – 12% Digitising for indexing versus digitizing for preservation Parts need re-doing Jean Noël Jeanneney & Google Books Cultural, commercial bias Questionable quality Access and re-use restrictions Preservation not guaranteed “ Too many – often small – institutions or institutions with relevant collections of too modest a volume want to digitise too little at too high a price without being able to justify distributed costs of investment and management…” Erland Kolding Nielsen RLG Programs comparison of digitisation contracts

We cannot digitise everything

in the forseeable future

CENL 2007 est. – only 1% digitised

2005 – Google books 10 yr plan is

only covering 35% of WorldCat

Jilovsky 2008 – 12%

Digitising for indexing versus digitizing for preservation

Parts need re-doing

Jean Noël Jeanneney & Google Books

Cultural, commercial bias

Questionable quality

Access and re-use restrictions

Preservation not guaranteed

Digitisation and Copyright OCLC Registry of Copyright Evidence Need more author information – nationality Need definition of “due diligence” “ The extension of the copyright limit from 50 to 70 years… was simply a catastrophe and an enormous obstacle to developing a relevant, adequate and comprehensive EDL with 20th century material of sufficient importance….The legal demands of investigating and finding the heirs..prohibitive” Erland Kolding Nielsen

OCLC Registry of Copyright Evidence

Need more author information – nationality

Need definition of “due diligence”

Centralised and Collective Digital Stores The assets of the libraries Gallica (90,000 works, 80,000 images) KB, Netherlands – 7 projects; 41 million pages EU Digital Library OCLC Digital Archive Harvestable Institutional Repositories Saving the community’s assets CLOCKSS (licensed content on publisher’s sites) Web archiving – saving the community’s web presence

The assets of the libraries

Gallica (90,000 works, 80,000 images)

KB, Netherlands – 7 projects; 41 million pages

EU Digital Library

OCLC Digital Archive

Harvestable Institutional Repositories

Saving the community’s assets

CLOCKSS (licensed content on publisher’s sites)

Web archiving – saving the community’s web presence

European Digital Library EU has funded infrastructure Now warming to funding digitisation itself Needs serious funding “ ..nothing wrong with the vision (EDL with > 12 million by 2012) but [financially] unrealistic “ Erland Kolding Nielsen “ ..A better balance between quality, quantity and costs has to be struck if libraries wish to digitise on a large scale…” Astrid Verheusen

EU has funded infrastructure

Now warming to funding digitisation itself

Needs serious funding

Archiving licensed and research content Licensed digital content – e.g. CLOCKSS Research data EU 7 th Framework FP7 – taking research data to the forefront JISC study: Keeping Research Data Safe: a cost model and guidance for UK Universities Neil Beagrie, Julia Chriszcz and Brian Lavoie RLG Programs, led from Scotland “New modes of scholarship”

Licensed digital content – e.g. CLOCKSS

Research data

EU 7 th Framework FP7 – taking research data to the forefront

JISC study: Keeping Research Data Safe: a cost model and guidance for UK Universities Neil Beagrie, Julia Chriszcz and Brian Lavoie

RLG Programs, led from Scotland “New modes of scholarship”

L ots o f C opies K eep S tuff S afe “ ...let us save what remains: not by vaults and locks which fence them from the public eye and use in consigning them to the waste of time, but by such a multiplication of copies, as shall place them beyond the reach of accident.” — Thomas Jefferson, February 18, 1791 (CLOCKSS web page) CLOCKSS – US/UK cooperative Monitoring, testing, emulating Indiana U, NYPL, Rice U, Stanford U, U. Virginia, OCLC, U. Edinburgh Am.Chem.Assoc, AMA, Am. Physiol.Assoc, Nature, Sage, Taylor & Francis, Elsevier, Oxford UP, Springer, Wiley Blackwell

“ ...let us save what remains: not by vaults and locks which fence them from the public eye and use in consigning them to the waste of time, but by such a multiplication of copies, as shall place them beyond the reach of accident.” — Thomas Jefferson, February 18, 1791 (CLOCKSS web page)

CLOCKSS – US/UK cooperative

Monitoring, testing, emulating

Indiana U, NYPL, Rice U, Stanford U, U. Virginia, OCLC, U. Edinburgh

Am.Chem.Assoc, AMA, Am. Physiol.Assoc, Nature, Sage, Taylor & Francis, Elsevier, Oxford UP, Springer, Wiley Blackwell

Long Term Archiving Systems e-Depot (KB Dutch National Library) kopal (DNB German National Library) LOCKSS (Stanford University) Portico (Ithaka / JSTOR) Outdated data formats Migration or emulation “ ..migration during ingest adds significantly to costs with no guarantee that the chosen formats will solve future problems “ Michael Seadle quoting LOCKSS

e-Depot (KB Dutch National Library)

kopal (DNB German National Library)

LOCKSS (Stanford University)

Portico (Ithaka / JSTOR)

Outdated data formats

Migration or emulation

WARC – Web Archiving ISO 28500 Allows archiving of all control information from the harvesting protocol (e.g., request headers), not just response information. payload content Arbitrary metadata linked to other stored data (e.g., subject classifier, discovered language, encoding) Data transformations linked to other stored data Overly long records by truncation or segmentation Allows duplicate removal (to reduce storage) Support for data compression and maintenance of data record integrity. Standardisation led by Christian Lupovici of the Bibliothèque Nationale de France

Allows archiving of

all control information from the harvesting protocol (e.g., request headers), not just response information.

payload content

Arbitrary metadata linked to other stored data (e.g., subject classifier, discovered language, encoding)

Data transformations linked to other stored data

Overly long records by truncation or segmentation

Allows duplicate removal (to reduce storage)

Support for data compression and maintenance of data record integrity.

Standardisation led by Christian Lupovici of the Bibliothèque Nationale de France

WARC Implementations Bibliothèque Nationale de France Coverage – covered by legal deposit – everything that would be consumed by the French public (not just the .fr domain) 120 terabytes; > 10 billion files Norway, Sweden Everything in the national domain International Internet Preservation Consortium (IIPC) Australia, Canada, Denmark, Finland, France, Iceland, Italy, Norway, Sweden, The British Library (UK), The Library of Congress (USA) and the Internet Archive (USA)

Bibliothèque Nationale de France

Coverage – covered by legal deposit – everything that would be consumed by the French public (not just the .fr domain)

120 terabytes; > 10 billion files

Norway, Sweden

Everything in the national domain

International Internet Preservation Consortium (IIPC)

Australia, Canada, Denmark, Finland, France, Iceland, Italy, Norway, Sweden, The British Library (UK), The Library of Congress (USA) and the Internet Archive (USA)

Exposure Importance of maximum exposure Users expectations – users are playing internationally Need web presence to achieve page rank in search engines Growth of harvesting OAIPMH, sitemaps Need to act on two fronts for maximum exposure Feeding external search engines and sites Enriching the metadata at home “ Visibility in a way can be seen to be indispensable in the survival of the item”…Werner Schwartz

Importance of maximum exposure

Users expectations – users are playing internationally

Need web presence to achieve page rank in search engines

Growth of harvesting OAIPMH, sitemaps

Need to act on two fronts for maximum exposure

Feeding external search engines and sites

Enriching the metadata at home

Catalogue and Union catalogues Cataloguing backlogs still real NUKAT in Poland – just celebrated 1 million records University of Warsaw Bavarian State Library finished retrospective cataloguing 2004 “ Millions of digital documents and resources are uncatalogued and even unmapped and difficult to retrieve” Michael Gorman

Cataloguing backlogs still real

NUKAT in Poland – just celebrated 1 million records

University of Warsaw

Bavarian State Library finished retrospective cataloguing 2004

“ Millions of digital documents and resources are uncatalogued and even unmapped and difficult to retrieve” Michael Gorman

Catalogues becoming richer Supporting improved discovery and exposure Replacing the browsing experience Supporting selection Learning from online shopping experience (Amazon) Importance of user contribution Need large sites to attract users Importance of data mining rather than data crafting Scalability

Supporting improved discovery and exposure

Replacing the browsing experience

Supporting selection

Learning from online shopping experience (Amazon)

Importance of user contribution

Need large sites to attract users

Importance of data mining rather than data crafting

Scalability

VIAF links

Evaluative information Reading lists Ester (Estonia) BibSys Bibliotek.dk Statistics Bibliotek.dk COBISS – Slovenian “Best Read Books” Over 70,000 lists

Reading lists

Ester (Estonia)

BibSys

Bibliotek.dk

Statistics

Bibliotek.dk

COBISS – Slovenian “Best Read Books”

Exposure of Library Resources End User interfaces to Union catalogues - examples NUKAT SUDOC GBV Bibliotek.dk BibSys Libris COBISS Ester

End User interfaces to Union catalogues - examples

NUKAT

SUDOC

GBV

Bibliotek.dk

BibSys

Libris

COBISS

Ester

The same copy can be retrieved from: The Royal Library of the Netherlands web site: http://opc4.kb.nl The Dutch National Union Catalogue web site: http:// www.picarta.nl The European Library web site: http://www.theeuropeanlibrary.org WorldCat: http:// www.worldcat.org Google Books: http:// books.google.com /

WorldCat - Statistics http:// www.oclc.org/worldcat/statistics/default.htm Bibliographic records: 106.938.448 Holdings: 1.284.762.384 Articles: 54+ million (Medline, ERIC, GPO, British Library, ArticleFirst) New – agreements with H. W. Wilson et MLA Libraries: 60.000+ National catalogues loaded: > 40 Australia, Czech Republic, Denmark, Finland, Germany, Poland, Sweden, United Kingdom public libraries ++ http://www.worldcat.org

http:// www.oclc.org/worldcat/statistics/default.htm

Bibliographic records: 106.938.448

Holdings: 1.284.762.384

Articles: 54+ million (Medline, ERIC, GPO, British Library, ArticleFirst)

New – agreements with H. W. Wilson et MLA

Libraries: 60.000+

National catalogues loaded: > 40

Australia, Czech Republic, Denmark, Finland, Germany, Poland, Sweden, United Kingdom public libraries ++

http://www.worldcat.org

WorldCat: Visits per month

Provenance and destination 2007: 110 millions 1 million referrals per month from Google Book Search

Regional & National Management of the collective collection Selection, weeding, claiming Provision of delivery services Global physical delivery architecture Global Exposure of collections bibliographic metadata, holdings, issue level holdings statistics reference query and answer pairs Exposure of services Management of exposure data Links, mining, user contribution Curation of rare & priceless Electronic delivery architecture

Regional & National

Management of the collective collection

Selection, weeding, claiming

Provision of delivery services

Global physical delivery architecture

Global

Exposure of collections

bibliographic metadata,

holdings, issue level holdings

statistics

reference query and answer pairs

Exposure of services

Management of exposure data

Links, mining, user contribution

Curation of rare & priceless

Electronic delivery architecture

Fundamental Questions What are the responses to these challenges / problems? Are they adequate in themselves? Does the ensemble make coherent sense? ? ? ? ? Collective physical stores and services  Digitisation  Collective digital stores Beyond simple storage to true preservation  Exposure 

What are the responses to these challenges / problems?

Are they adequate in themselves?

Does the ensemble make coherent sense?

? ? ? ?

Collective physical stores and services 

Digitisation 

Collective digital stores

Beyond simple storage to true preservation 

Exposure 

“ ..Electronic enterprises come and go. Research libraries last for centuries. Better to fortify them than to declare them obsolete, because obsolescence is built into the electronic media..” Robert Darnton New York review of books 55, no. 10 June 12 2008

“ ..Electronic enterprises come and go. Research libraries last for centuries. Better to fortify them than to declare them obsolete, because obsolescence is built into the electronic media..” Robert Darnton

New York review of books 55, no. 10 June 12 2008

Thank you

Collection Analysis Compare collection – globally or with selected libraries By subject, classification, format, publication date range Supports re-location, purchase, digitisation decisions

Compare collection – globally or with selected libraries

By subject, classification, format, publication date range

Supports re-location, purchase, digitisation decisions

Registries of digital content “ Catalogue information will increase the use of digitised works and their chance to be preserved”…Werner Schwartz OCLC / DLF Registry of digital Masters Agreement with LIBER / CERL – Consortium of European Research Libraries Scope of harvesting in Europe is still insufficient Data not in compliance with DLF guidelines – being converted

“ Catalogue information will increase the use of digitised works and their chance to be preserved”…Werner Schwartz

OCLC / DLF Registry of digital Masters

Agreement with LIBER / CERL – Consortium of European Research Libraries

Scope of harvesting in Europe is still insufficient

Data not in compliance with DLF guidelines – being converted