Metadata for your Digital Collections

Metadata for your
Digital Collections
Jenn Riley
Metadata Librarian
IU Digital Library Program

3/6/07 INCOLSA Workshop 2
Many definitions of metadata
 “Data about data”
 “Structured information about an information
resource of any media type or format.”
(Caplan)
 “Any data used to aid the identification,
description and location of networked
electronic resources.” (IFLA)
 …

Refining a definition
 Other characteristics
 Structure
 Control
 Origin
 Machine-generated
 Human-generated
 In practice, the term often covers data and
meta-metadata

Some uses of metadata
 By information specialists
 Describing non-traditional materials
 Cataloging Web sites
 Navigating digital objects
 Managing digital objects over the long term
 Managing corporate assets
 By novices
 Preparing Web sites for search engines
 Describing Eprints
 Managing personal CD collections

Metadata and cataloging
 Depends on what you mean by:
 metadata, and
 cataloging!
 But, in general:
 Metadata is broader in scope than cataloging
 Much metadata creation takes place outside of libraries
 Good metadata practitioners use fundamental
cataloging principles in non-MARC environments
 Metadata created for many different types of materials
 Metadata is NOT only for Internet resources!

Metadata in digital library projects
 Searching
 Browsing
 Display for users
 Interoperability
 Management of digital objects
 Preservation
 Navigation

Some types of metadata
Type Use
Descriptive metadata Searching
Browsing
Display
Interoperability
Technical metadata Interoperability
Digital object management
Preservation
Preservation metadata Interoperability
Preservation
Rights metadata Interoperability
Digital object management
Structural metadata Navigation

How metadata is used

Creating descriptive metadata
 Digital library content management systems
 ContentDM
 ExLibris Digitool
 Greenstone
 Library catalogs
 Spreadsheets & databases
 XML

Creating other types of metadata
 Technical
 Stored in content management system
 Stored in separate Excel spreadsheet
 Structural
 Created and stored in content management system
 METS XML
 GIS
 Using specialized software
 Content markup
 In XML

Descriptive metadata
 Purpose
 Description
 Discovery
 Some common general schemas
 Dublin Core (unqualified and qualified)
 MARC
 MARCXML
 MODS
 LOTS of domain-specific schemas

Simple Dublin Core (DC)
 15-element set
 National and international standard
 2001: Released as ANSI/NISO Z39.85
 2003: Released as ISO 15836
 Maintained by the Dublin Core Metadata
Initiative (DCMI)
 Other players
 DC Usage Board
 DCMI Communities
 DCMI Task Groups

DCMI mission
 The mission of DCMI is to make it easier to
find resources using the Internet through the
following activities:
 Developing metadata standards for discovery
across domains,
 Defining frameworks for the interoperation of
metadata sets, and,
 Facilitating the development of community- or
disciplinary-specific metadata sets that are
consistent with items 1 and 2

DC Principles
 Original principles
 “Core” across all knowledge domains
 No element required
 All elements repeatable
 1:1 principle
 DC Abstract Model
 “A reference against which particular DC encoding
guidelines can be compared” model
 Two schools of thought on its development
 Clarifies model underlying the metadata standard
 Overly complicates a standard intended to be simple

Content/value standards for DC
 None required
 Some elements recommend a content
or value standard as a best practice
 Relation
 Source
 Subject
 Type
 Coverage
 Date
 Format
 Language
 Identifier

Some limitations of DC
 Can’t indicate a main title vs. other
subordinate titles
 No method for specifying creator roles
 W3CDTF format can’t indicate date ranges or
uncertainty
 Can’t by itself provide robust record
relationships

Good times to use DC
 Cross-collection searching
 Cross-domain discovery
 Metadata sharing
 Describing some types of simple resources
 Metadata creation by novices

DC
[record]
QDC
[record]
[collection]
MARC
[record]
[collection]
MARCXML
[record]
MODS
[record]
[collection]
Record
format
XML
RDF
(X)HTML
Field labels Text
Reliance on
AACR
None
Common
method of
creation
By novices, by
specialists,
and by
derivation

Qualified Dublin Core (QDC)
 Adds some increased specificity to
Unqualified Dublin Core
 Same governance structure as DC
 Same encodings as DC
 Same content/value standards as DC
 Listed in DMCI Terms
 Additional principles
 Extensibility
 Dumb-down principle

Types of DC qualifiers
 Additional elements
 Element refinements
 Encoding schemes
 Vocabulary encoding schemes
 Syntax encoding schemes

DC qualifier status
 Recommended
 Conforming
 Obsolete
 Registered

Limitations of QDC
 Widely misunderstood
 No method for specifying creator roles
 W3CDTF format can’t indicate date ranges or
uncertainty
 Split across 3 XML schemas
 No encoding in XML (yet) officially endorsed
by DCMI

Best times to use QDC
 More specificity needed than simple DC, but
not a fundamentally different approach to
description
 Want to share DC with others, but need a few
extensions for your local environment
 Describing some types of simple resources
 Metadata creation by novices

DC
[record]
QDC
[record]
[collection]
MARC
[record]
[collection]
MARCXML
[record]
MODS
[record]
[collection]
Record
format
XML
RDF
(X)HTML
XML
RDF
(X)HTML
Field labels Text Text
Reliance on
AACR
None None
Common
method of
creation
By novices, by
specialists,
and by
derivation
By novices, by
specialists,
and by
derivation

MAchine Readable Cataloging
(MARC)
 Format for the records in library catalogs
 Used for library metadata since 1960s
 Adopted as national standard in 1971
 Adopted as international standard in 1973
 Maintained by:
 Network Development and MARC Standards
Office at the Library of Congress
 Standards and the Support Office at the
National Library of Canada

More about MARC
 Actually a family of MARC standards
throughout the world
 U.S. & Canada use MARC21
 Structured as a binary interchange format
 ANSI/NISO Z39.2
 ISO 2709
 Field names
 Numeric fields
 Alphabetic subfields

Content/value standards for MARC
 None required by the format itself
 But US record creation practice relies
heavily on:
 AACR2r
 ISBD
 LCNAF
 LCSH

Limitations of MARC
 Use of all its potential is time-consuming
 OPACs don’t make full use of all possible
data
 OPACs virtually the only systems to use
MARC data
 Requires highly-trained staff to create
 Local practice differs greatly

Good times to use MARC
 Integration with other records in OPAC
 Resources are like those traditionally found in
library catalogs
 Maximum compatibility with other libraries is
needed
 Have expert catalogers for metadata creation

DC
[record]
QDC
[record]
[collection]
MARC
[record]
[collection]
MARCXML
[record]
MODS
[record]
[collection]
Record
format
XML
RDF
(X)HTML
XML
RDF
(X)HTML
ISO 2709
[ANSI Z39.2]
Field labels Text Text Numeric
Reliance on
AACR
None None Strong
Common
method of
creation
By novices, by
specialists,
and by
derivation
By novices, by
specialists,
and by
derivation
By
specialists

MARC in XML (MARCXML)
 Copies the exact structure of MARC21 in an
XML syntax
 Numeric fields
 Alphabetic subfields
 Implicit assumption that content/value
standards are the same as in MARC

Limitations of MARCXML
 Not appropriate for direct data entry
 Extremely verbose syntax
 Full content validation requires tools external
to XML Schema conformance

Best times to use MARCXML
 As a transition format between a MARC
record and another XML-encoded metadata
format
 Materials lend themselves to library-type
description
 Need more robustness than DC offers
 Want XML representation to store within
larger digital object but need lossless
conversion to MARC

DC
[record]
QDC
[record]
[collection]
MARC
[record]
[collection]
MARCXML
[record]
MODS
[record]
[collection]
Record
format
XML
RDF
(X)HTML
XML
RDF
(X)HTML
ISO 2709
[ANSI Z39.2]
XML
Field labels Text Text Numeric Numeric
Reliance on
AACR
None None Strong Strong
Common
method of
creation
By novices, by
specialists,
and by
derivation
By novices, by
specialists,
and by
derivation
By
specialists
By derivation

Metadata Object Description
Schema (MODS)
 Developed and managed by the Library of
Congress Network Development and MARC
Standards Office
 For encoding bibliographic information
 Influenced by MARC, but not equivalent
 Usable for any format of materials
 First released for trial use June 2002
 MODS 3.2 released late 2006

MODS differences from MARC
 MODS is “MARC-like” but intended to be
simpler
 Textual tag names
 Encoded in XML
 Some specific changes
 Some regrouping of elements
 Removes some elements
 Adds some elements

Content/value standards for MODS
 Many elements indicate a given content/value
standard should be used
 Generally follows MARC/AACR2/ISBD
conventions
 But not all enforced by the MODS XML
schema
 Authority attribute available on many
elements

Limitations of MODS
 No lossless round-trip conversion from and to
MARC
 Still largely implemented by library community
only
 Some semantics of MARC lost

Good times to use MODS
 Materials lend themselves to library-type
description
 Want to reach both library and non-library
audiences
 Need more robustness than DC offers
 Want XML representation to store within
larger digital object

DC
[record]
QDC
[record]
[collection]
MARC
[record]
[collection]
MARCXML
[record]
MODS
[record]
[collection]
Record
format
XML
RDF
(X)HTML
XML
RDF
(X)HTML
ISO 2709
[ANSI Z39.2]
XML XML
Field labels Text Text Numeric Numeric Text
Reliance on
AACR
None None Strong Strong Implied
Common
method of
creation
By novices, by
specialists,
and by
derivation
By novices, by
specialists,
and by
derivation
By
specialists
By derivation
By specialists
and by
derivation

Visual Resources Association
(VRA) Core
 From Visual Resources Association
 Separates Work from Image
 Library focus
 Inspiration from Dublin Core
 Version 3.0 released on 2002
 Version 4.0 currently in Beta

Categories for the Description of
Works of Art (CDWA) Lite
 Reduced version of the Categories for the
Description of Works of Art (512 categories)
 From J. Paul Getty Trust
 Museum focus
 Conceived for record sharing

Structure standards for learning
materials
 Gateway to Educational Materials (GEM)
 From the U.S. Department of Education
 Based on Qualified Dublin Core
 Adds elements for instructional level, instructional method,
etc.
 “GEM's goal is to improve the organization and accessibility
of the substantial collections of materials that are already
available on various federal, state, university, non-profit, and
commercial Internet sites.”*
 IEEE Learning Object Metadata (LOM)
 Elements for technical and descriptive metadata about
learning resources
* From <http://www.thegateway.org/about/documentation/schemas>

Text Encoding Initiative (TEI)
 TEI in Libraries
 For encoding full texts of documents
 Literary texts
 Letters
 …etc.
 Requires specialized search engine
 Delivery requires specialized software or
offline conversion to HTML

Encoded Archival Description
(EAD)
 Maintained by the Society for American
Archivists EAD Working Group
 Markup language for archival finding aids
 Designed to accommodate multi-level
description
 Requires specialized search engine
 Delivery requires specialized software or
offline conversion to HTML
 EAD 1.0 released in 1998
 EAD2002 finalized in December 2002

Levels of control
 Data structure standards (e.g., MARC)
 Data content standards (e.g., AACR2r)
 Encoding schemes
 Vocabulary
 Syntax
 High-level models (e.g., FRBR)
 Very few metadata standards include a
counterpart to the AACR “chief source of
information”

Some data content standards
 Anglo-American Cataloging Rules, 2nd
edition (AACR2)
 Scheduled to be replaced by RDA in 2009
 Describing Archives: A Content Standard
(DACS)
 Replaces APPM
 Cataloging Cultural Objects (CCO)
 First content standard explicitly designed for
these materials

When there’s no data content
standard…

Vocabulary encoding schemes
 TGM I
 TGM II
 TGN
 GeoNet
 AAT
 LCSH
 LCNAF
 DCMI Type
 MIME Types
 …etc.

Syntax encoding schemes
 ISO8601
 W3CDTF
 URI
 AACR2r
 …etc.

Functional Requirements of
Bibliographic Records (FRBR) model
WORK
EXPRESSION
MANIFESTATION
ITEM
is realized through
is embodied in
is exemplified by

Using FRBR principles in
metadata creation
 Don’t need to take the model literally
 For unique materials, much simplification is
possible
 Make sure you know how your practices
conform to the high-level model
 Be consistent in these practices

How do I pick standards? (1)
 Institution
 Nature of holding institution
 Resources available for metadata creation
 What others in the community are doing
 Capabilities of your delivery software
 The standard
 Purpose
 Structure
 Context
 History

How do I pick standards? (2)
 Materials
 Genre
 Format
 Likely audiences
 What metadata already exists for these materials
 Project goals
 Robustness needed for the given materials and users
 Describing multiple versions
 Mechanisms for providing relationships between records
 Plan for interoperability, including repeatability of elements
 More information on handout

Assessing materials for ease of
metadata creation
 Number of items?
 Homogeneity of items?
 Foreign language?
 Published or unpublished?
 Specialist needed?
 How much information is known?
 Any existing metadata?

Assessing currently existing metadata
 Machine-readable?
 Divided into fields?
 What format?
 What content standards?
 Complete?

Assessing software capabilities
 Are there templates for standard metadata
formats?
 Can you add/remove fields to a template?
 Can you create new templates?
 Can you add additional clarifying information
without creating a separate field?
 Personal vs. corporate names
 Subject vocabulary used
 Is there an XML export? Does it produce valid
records?

Case studies in choosing standards
 Describe your institution
 Describe one collection you’d like to digitize
 Describe your technical infrastructure

Beyond descriptive metadata
 Technical metadata
 Preservation metadata
 Rights metadata
 Structural metadata

Technical metadata
 For recording technical aspects of digital objects
 For long-term maintenance of data
 Migration
 Emulation
 Much can be generate automatically, but not all
 Some examples:
 NISO Z39.87: Data Dictionary – Technical Metadata for Digi
& MIX
 Schema for Technical Metadata for Text
 Forthcoming standard for audio from the Audio
Engineering Society
 LC VMD draft schema for technical metadata for video
files

Image technical metadata
 Might include:
 Color space
 Bit depth
 Byte order
 Compression scheme
 Camera settings
 Operator name

Text technical metadata
 Might include:
 Character set
 Byte order
 Font/script
 Language

Audio technical metadata
 Might include:
 Byte order
 Checksum
 Sample rate
 Duration
 Number of channels

Video technical metadata
 Might include:
 Bits per sample
 Calibration information
 Sample format
 Signal format

Preservation metadata
 The set of everything you need to know to
preserve digital objects over the long term
 Information that supports and documents the
digital preservation process
 Includes technical metadata but also other
elements
 Covers elements such as checksums,
creation environment, and change history
 PREMIS is the prevailing model

Rights metadata
 Machine- or human-readable indications of
rights information for a resource
 Can be used to determine if a user can
access a resource
 Can indicate rights holder of a resource for
payment purposes
 Some current schemas
 METS rights
 XrML
 ODRL

Structural metadata
 For creating a logical structure between
digital objects
 Multiple copies/versions of same item
 Multiple pages within item
 Multiple sizes of each page
 Meaningful groups of content
 Often handled transparently by a delivery
system
 METS is the current primary standard

Why you should care about these
standards
 You will migrate from your current system to
another, probably in the next few years
 File formats become obsolete
 We have too many interesting collections to
have to re-do work we’ve already done
 Standards promote interoperability

Building “Good digital
collections”*
 Interoperable – with the important goal
of cross-collection searching
 Persistent – reliably accessible
 Re-usable – repositories of digital
objects that can be used for multiple
purposes
*Institute for Museum and Library Services. A Framework of Guidance for Building Good Digital
Collections. Washington, D.C.: Institute for Museum and Library Services, November 2001.
http://www.niso.org/framework/Framework2.html

Building “Good digital
collections”
 Interoperable – with the important goal of
cross-collection searching
 Persistent – reliably accessible
 Re-usable – repositories of digital objects that
can be used for multiple purposes
Good metadata promotes good digital
collections.

Sharing your metadata
 Harvesting
 Collects metadata, processes it, and stores it locally to
respond to user queries
 Open Archives Initiative Protocol for Metadata
Harvesting
 Federated searching
 Transmits user queries to multiple destinations in real
time
 ILS vendors currently offering these products
 Protocols used
 Z39.50
 SRU

OAI Protocol Structure
 Intentionally designed to be simple
 Data providers
 Have metadata they want to share
 “Expose” their metadata to be harvested
 Service providers
 Harvest metadata from data providers
 Provide searching of harvested metadata
from multiple sources
 Can also provide other value-added services

Data Providers
 Set up a server that responds to harvesting
requests
 Required to expose metadata in simple
Dublin Core (DC) format
 Can also expose metadata in any other
format expressible with an XML schema

Service Providers
 Harvest and store metadata
 Generally provide search/browse access to
this metadata
 Can be general or domain-specific
 Can choose to collect metadata in formats
other than DC
 Generally link out to holding institutions for
access to digital content

Advantages for Libraries
 Any existing rules for description can be
used
 Can share metadata without sacrificing local
granularity
 Location of unique materials by many users
 Domain-specific service providers
 Middle ground between Google and OCLC
 One of a suite of tools to provide users with
access to all of your materials

Why share metadata?
 Benefits to users
 One-stop searching
 Aggregation of subject-specific resources
 Benefits to institutions
 Increased exposure for collections
 Broader user base
 Bringing together of distributed collections
Don’t expect users will know about your
collection and remember to visit it.

Why share metadata with OAI?
 “Low barrier” protocol
 Shares metadata only, not content,
simplifying rights issues
 Same effort on your part to share with one or
a hundred service providers (basically)
 Wide adoption in the cultural heritage sector
 Quickly eclipsed older methods such as
Z39.50

Three possible architectures
OAIHarvester
Digital asset management system
Metadata
creation
module
OAI data
provider
module
Transformation
Metadata
creation
system
Stand-alone
OAI data
provider
Transformation
DC
QDC MODS
MARCXML
DC MARCXML
QDC MODS
Metadata
creation
module
Static
Repository
Gateway
Transformation

Basic metadata sharing workflow
 Create metadata, thinking about shareability
 Determine format(s) you wish to share your metadata
in
 Transform records into versions appropriate for
sharing via OAI
 Validate transformed metadata
 Load transformed metadata into OAI data provider
 Test with OAI Repository Explorer
 Communicate with service providers
 See what your metadata looks like once a service
provider harvests it

Preparing your metadata for sharing
 Map to common formats; also called
“crosswalking”
 To create “views” of metadata for specific
purposes
 Mapping from robust format to more general
format is common
 Mapping from general format to more robust
format is ineffective

Crosswalks (1)
 For transforming between metadata formats
 Usually refers to transforming between
content standards rather than structure
standards, but not always
 Mapping from more robust format to less
robust format effective; mapping from simpler
format to more robust format less so
 Good practice to create and store most
robust metadata format possible, then create
other views for specific needs

Crosswalks (2)
 Can be in many formats
 Logical sets of rules [example]
 Actual code [example]
 Often need to tweak a generic crosswalk for a
specific implementation
 Accommodating local practice
 Adding institution-specific information
 Adding context not available locally

Types of mapping logic
 Mapping the complete contents of one field to
another
 Splitting multiple values in a single local field
into multiple fields in the target schema
 Translating anomalous local practices into a
more generally useful value
 Splitting data in one field into two or more
fields
 Transforming data values
 Boilerplate values to include in output
schema

Metadata as a view of the resource
 There is no monolithic, one-size-fits-all
metadata record
 Metadata for the same thing is different
depending on use and audience
 Harry Potter as represented by…
 a public library
 an online bookstore
 a fan site

Choice of vocabularies as a
view
 Names
 LCNAF: Michelangelo Buonarroti, 1475-
1564
 ULAN: Buonarroti, Michelangelo
 Places
 LCSH: Jakarta (Indonesia)
 TGN: Jakarta
 Subjects
 LCSH: Neo-impressionism (Art)
 AAT: Pointillism

Finding the right balance
 Metadata providers know the materials
 Document encoding schemes and controlled
vocabularies
 Document practices
 Ensure record validity
 Aggregators have the processing power
 Format conversion
 Reconcile known vocabularies
 Normalize data
 Batch metadata enhancement

What does this record describe?
identifier: http://name.university.edu/IC-FISH3IC-X0802]1004_112
publisher: Museum of Zoology, Fish Field Notes
format: jpeg
rights: These pages may be freely searched and displayed.
Permission must be received for subsequent distribution in
print or electronically.
type: image
subject: 1926-05-18; 1926; 0812; 18; Trib. to Sixteen Cr. Trib. Pine
River, Manistee R.; JAM26-460; 05; 1926/05/18; R10W;
S26; S27; T21N
language: UND
source: Michigan 1926 Metzelaar, 1926--1926;
description: Flora and Fauna of the Great Lakes Region
Example courtesy of Sarah Shreeves, University of Illinois at Urbana-Champaign

Shareable metadata defined
 Metadata for aggregation with records from other
institutions
 Promotes search interoperability - “the ability to
perform a search over diverse sets of metadata
records and obtain meaningful results” (Priscilla
Caplan)
 Is human understandable outside of its local
context
 Is useful outside of its local context
 Preferably is machine processable

6 Cs and lots of Ss of shareable
metadata
Content
Consistency
Coherence
Context
Communication
Conformance
Metadata standards
Vocabulary and encoding standards
Descriptive content standards
Technical standards

Content
 Choose appropriate vocabularies
 Choose appropriate granularity
 Make it obvious what to display
 Make it obvious what to index
 Exclude unnecessary “filler”
 Make it clear what links point to

Consistency
 Records in a set should all reflect the same
practice
 Fields used
 Vocabularies
 Syntax encoding schemes
 Allows aggregators to apply same
enhancement logic to an entire group of
records

Coherence
 Record should be self-explanatory
 Values must appear in appropriate elements
 Repeat fields instead of “packing” to explicitly
indicate where one value ends and another
begins

Context
 Include information not used locally
 Exclude information only used locally
 Current safe assumptions
 Users discover material through shared
record
 User then delivered to your environment for
full context
 Context driven by intended use

Communication
 Method for creating shared records
 Vocabularies and content standards used in
shared records
 Record updating practices and schedules
 Accrual practices and schedules
 Existence of analytical or supplementary
materials
 Provenance of materials

Conformance to Standards
 Metadata standards (and not just DC)
 Vocabulary and encoding standards
 Descriptive content standards (AACR2, CCO,
DACS)
 Technical standards (XML, Character
encoding, etc)

Before you share…
 Check your metadata
 Appropriate view?
 Consistent?
 Context provided?
 Does the aggregator have what they need?
 Documented?
Can a stranger tell you what the record
describes?

The reality of sharing metadata
 We can no longer afford to only think about our local
users
 Creating shareable metadata will require more work
on your part
 Creating shareable metadata will require our vendors
to support (more) standards
 Creating shareable metadata is no longer an option,
it’s a requirement
 Indiana is moving toward a portal of Indiana-related
digital content – you should be planning for this now

Putting it all into practice
 Develop written documentation
 Develop a quality control workflow for
metadata creation
 Share your findings with others
 Get better with every new online collection

Further information
 jenlrile@indiana.edu
 These presentation slides: <
http://www.dlib.indiana.edu/~jenlrile/presentations/incolsa2007/incolsa.ppt>
 Metadata librarians listserv: <http:
//metadatalibrarians.monarchos.com>
 Priscilla Caplan: Metadata Fundamentals for
all Librarians, 2003

Metadata for your Digital Collections

In this document