DSpace 4.2 Transmission: Import/Export

DSpace 4.2
Advanced Training –
Content Transmission
DSpace 4.2 Advanced Training by James Creel is licensed under a
Creative Commons Attribution 4.0 International License. Special
thanks to the DuraSpace Foundation and the Texas Digital Library
for making this course possible.

Module Outline
• Harvesting and Disseminating with OAI/PMH
• Reading content with REST
• Export and Import with SIPs
• Depositing content with SWORD
• Importing content with the Simple Archive Format (SAF)

Introduction to Harvesting
• Open Archives Initiative
• Protocol for Metadata Harvesting
• Object Reuse and Exchange
• Harvesting with DSpace XMLUI
• Choice of collection source
• Replicate metadata (OAI-PMH) or metadata + data (OAI-PMH +
OAI-ORE)
• What an excellent way to rapidly populate one’s repository!

Introduction to Harvesting
• Go ahead and create a new collection wherever you please.
• We will be harvesting content from remote DSpace
repositories.
• Having created the collection, one is taken to the edit view.
Click the tab for Content Source

How do we learn about the harvest
source?
• Point your browser to
http://repository.tamu.edu/dspace-
oai/request?verb=ListSets to see a list of
collections at TAMU.
• There are several interesting verbs for which an OAI server will
grant requests-
• Point your browser to
http://www.openarchives.org/OAI/openarchives
protocol.html for details
• In the 1.8.x days, one would need to keep that page open when
trying to craft queries to OAI. Under 3.x and higher, there is a
lovely stylesheet courtesy of Lyncode that makes typical queries
easy and automatic.

Configuring the Content Source
• A sample OAI Provider – OAK Trust: The Texas A&M Digital
Repository: http://repository.tamu.edu/dspace-
oai/request
• OAI Set spec: com_1969.1_5670
• Test the settings to make sure things are copasetic, then save.

Save your pointers to the
external OAI service. Select
Import Now…

Your oai webapp provides a
machine-readable dissemination
service.
• Try some requests:
• http://localhost:8080/oai/request?verb=Ident
ify
• http://localhost:8080/oai/request?verb=ListM
etadataFormats
• http://localhost:8080/oai/request?verb=ListS
ets
• http://localhost:8080/oai/request?verb=ListR
ecords&metadataPrefix=ore

We can experiment with
harvesting from each other’s
repositories
• From your command line, run ipconfig
• Your ip address will be listed as the IPv4 address
• You can craft a OAI request URL for your server using
the ip address as the host name.
• If you like, invite a neighbor to harvest one of your
collections.

Automating Harvesting (1/3)
• Requests to harvest large collections can easily time
out.
• This calls for a scheduler that runs independently of
the browser.
• Find it in the XMLUI under
the control panel.

• When automated, the harvester will conduct its
activity on all collections that are configured to
harvest.
• Once started, the harvester will operate at regular
intervals as specified by
harvester.harvestFrequency in
modules/oai.cfg.

• Start – initiate the periodic process
• Pause – wait for the current operation to complete,
then suspend further operations
• Stop – wait for the currently harvested item to
complete, then suspend further operations (which will
likely break further harvests of the containing
collection)
• Reset Harvest Status – clears the status of each
harvested collection so that they may be initiated
anew

Which formats are available to
your harvester?
• This is configurable in [dspace-install-
dir]modulesoai.cfg under the
harvester.oai.metadataformats.[declar
ed-metadata-format-name] values
• Where [declared-metadata-format-name] is declared in
your xoai.xml
• Let’s add “rdf” to that list and try harvesting with it.

Dissemination – Metadata
Crosswalks
• Metadata in DSpace exist in key-value pairs with field names
given by the metadata registry.
• Fields may be exported in the formats that oai indicates from
the ListMetadataFormats verb.
• Dissemination crosswalks are encoded as XSL files inside the
[dspace-install-dir/config/crosswalks]
directory
• The .properties seem to have stopped being used for OAI
dissemination since DSpace went to version 3.x
• The crosswalks are active in specific contexts that can be
configured.

Configuring Metadata Crosswalks –
XOAI Configuration Entities
• Open up the
C:dspaceconfigcrosswalksoaixoai.xml
file with jEdit.
• The top level Configuration element contains <Contexts>,
<Formats>, <Transformers>, <Filters>, and
<Sets>.
• Each of these contain, in turn, what you would expect -
<Context> elements, <Format> elements,
<Transformer> elements, <Filter> elements, and
<Set> elements.
• Each of these does its own thing.

XOAI Configuration – Setting up
Contexts
• The <Context> element refers to instances of all the other
elements.
• The baseurl attribute determines how to address the context
in your url path
• The <Format> elements name the crosswalks to be available
• The <Transformer> element names a stylesheet to apply to
the final XML output
• The <Filter> elements name Java classes that will eliminate
results unacceptable to the context
• The <Set> element appears simply to alias the set of all records
in the context.

Formats
• The <Format> elements have an id attribute which allows
them to be referenced in the <Context>
• They also contain, minimally, a
• <Prefix> by which they are addressed in OAI requests
• <XSLT> designating the xsl file doing the crosswalk
• And should include
• <Namespace> designating the namespace of XML output
• <SchemaLocation> designating the schema specification of
that XML

Transformers
• The <Transformer> element contains an id attribute by
which it is referenced in the <Context> and an <XSLT>
element designating its XSL file.

Filters
• The <Filter> elements contain an id attribute by which
they are referenced in the <Context> and
• <Class> which names the java class doing the filtering
• <Parameter> with a key attribute and one or more <Value>
elements that are used to parameterize the filtering method.

Sets
• The <Set> element has the usual id attribute and
• <Pattern> which renders as the set spec in the OAI response
• <Name> which renders as the set’s name

Exercise – A Custom Context
• Let’s imagine a use case where there is a requirement to be
harvested by a vendor or partner.
• Only items with certain fields are suitable for their index (for
example, those with a title, author, and type)
• Create a new context with an appropriate filter.

Styling for Human Readability
• The webappsoaistaticstyle.xsl stylesheet is used to render
the OAI responses in a nice readable format with the links of
interest also provided.
• One may also change the stylesheet being used by OAI by
changing the stylesheet attribute of the
<Configuration> root element of xoai.xml.
• Let’s experiment with some changes to the style –
• New branding
• Links to each of the contexts

The REST Webapp (1/4)
• Representational State Transfer – A scaleable, simple approach
to web services.
• Stateless on the server side – client maintains any session data
• Cacheable – responses should indicate whether the client can
save them in a web cache
• Layerable – Client need not know or care whether the server is
behind a proxy
• Simple, Uniform Requests – resources identifiable by URI,
responses report their format and their cacheability

• Read Only in 4.x
• JSON or XML depending on your HTTP Header: Accept
• Possible values are application/xml and application/json
• Your browser may default to one or the other, but your
application code (or developer’s browser) can specify.
• Communities, Collections, Items and Bitstreams are queryable
resources
• The ?expand query parameter followed by a comma
delimited list will provide more detail than the default queries

• Communities
• /rest/communities lists all
• /rest/communities/:id gets one
• ?expand possibilities: parentCommunity,
collections, subCommunities, logo, all
• Collections
• /rest/collections lists all
• /rest/collections/:id gets one
• ?expend possibilities: parentCommunityList,
parentCommunity, items, license, logo, all

• Items
• /items/:id lists one
• ?expand possibilities: metadata, parentCollection,
parentCollectionList, parentCommunityList,
bitstreams, all
• Bitstreams
• /bitstreams/:bitstreamID lists one
• /bitstreams/:bitstreamID/retrieve to download
• ?expend possibilities: parent, all

The DSpace Packager
• Utilized with the dspace packager command-line script
• Submission Information Packages
• Dissemination Information Packages

Submission Packages (SIPs)
• Four package formats supported by default:
• DSpace Archival Information Package (AIP) – used for backing up
and restoring DSpace repository content
• DSPACE-ROLES – used for backing up and restoring DSpace groups
and epersons
• METS – A zipfile containing MODS descriptive metadata and
designating content bitstreams and their disposition
• PDF – A single PDF file can be considered a package (supposing its
embedded metadata are suitable

• An example – importing a PDF as a package
• Track down a pdf on the interwebs – here’s one!
• http://hdl.handle.net/1969.1/2313
• Copy it to [dspace-install-dir] i.e. C:dspace
• Learn about the packager with the
C:dspacebindspace packager --help --
type PDF command
• Can you craft the command to make the submission?

Submission Packages (SIPs) –
PDF example
• We need a –t for type, -p for parent collection, -e for eperson
email, and finally the name of the “package”
• Once this succeeds, however, the quality of the metadata is
likely to be very poor indeed! Embedded metadata are
seldom well populated.

• An example – importing a METS package
• Of interest as this is also the package used by default for SWORD
deposits
• Find the file mets-sip-example.zip in the
W:Developmentresources directory.
• Copy it to [dspace-install-dir] i.e. C:dspace
• Learn about the packager with the C:dspacebindspace
packager --help --type METS command
• Can you craft the command to make the submission?

Submission Packages (SIPs) –
METS example
• We need at least the –t flag for type, -p for parent collection, -
e for eperson, and finally the filename of the package.
• C:dspacebindspace packager –t METS –p
[collection-handle] –e admin@admin.com
mets-sip-example.zip

Dissemination Packages (DIPs)
• DSpace Archival Information Package
• DSPACE-ROLES
• METS
• No need to export PDFs, we might suppose.
• As a final packaging exercise, use the packager to disseminate
an item. This will require the additional –i (identifier, i.e.
handle of the object) and –d (disseminate instead of the
default, submit)
• Can you craft the command?

Dissemination Packages (DIPs)
• A successful dissemination:
• Let’s complete the circle by submitting this package to another
(or even the same) collection.

SWORD
• Simple Web Service Offering Repository Deposit
• DSpace comes with servers for v1 and v2
• Big innovation of v2 is ability to update items, but client
support is currently limited
• Accessible via a client or (e.g.) a cURL command.
• Accepts deposits via METS packages by default
• Requires an administrative eperson account

SWORD – accessing via cURL
command
• A cURL executable is provided at W:Developmentcurl-
7.37.0-win32
• Copy that directory to your own C:Development.
• This command is an extremely robust tool that enables
communication of data over protocols with and without
encryption – we here are interested just in HTTP today.

command – getting the
servicedocument
• Clues to the meaning may be found at
http://curl.haxx.se/docs/manpage.html

command – Making a deposit
• A long, long command indeed…
• curl
• -i
• --data-binary "@mets-sip-example.zip"
• -H "Content-Disposition: filename=mets-sip-example.zip"
• -H "Content-Type: application/zip"
• -H "X-Packaging: http://purl.org/net/sword-
types/METSDSpaceSIP"
• -H "X-No-Op: false“
• -H "X-Verbose: true“
• --user "admin@admin.com:admin"
http://localhost:8080/sword/deposit/123456789/26

command – Making a deposit
• Find that text in the W:Developmentresourcescurl-deposit-
notes file.
• In an amusing turn of events, this deposit will fail from most of
our localhost machines, as behind the scenes the SWORD
server will attempt to write a temporary file named after your
IP address which contains colon characters which are illegal in
Windows filenames.
• This can be gleaned from the
C:Developmenttomcatlogslocalhost.[today].log
• Instead, let’s experiment with deposits to other servers in the
room.

SWORD – Bringing up the
DSpace Client
• Activate the aspect in xmlui.xconf
• Target repositories are configured in the [dspace-
install-dir]configmodulessword-
client.cfg file

SWORD – Utilizing the DSpace
SWORD Client
• Serves at this time only to copy existing items to another
SWORD-enabled repository.
• To utilize, navigate to the item’s page while logged in as an
administrator.
• Let’s try some
deposits to
localhost and
our neighbors.

SWORD – Looking Forward to
Sword v2 in Practice
• Sword v2 offers the capability to change the content and
metadata of previously deposited items
• Java libraries for the client are available, but I have not seen an
implemented GUI.
• cURL usage is also theoretically quite possible, but also looks
like a little bit of heavy lifting.

Batch Imports
• DSpace Simple Archive Format (SAF)
• The DSpace import script
• Adding items
• Replacing items
• Deleting items
• Importing from real sources
• Example: CSV
• Example: MARC XML

DSpace SAF (1/3) - Overview
• The top level directory contains one directory for each item in
the batch.
• Each item directory must contain:
• The bitstream files
• A contents manifest contents
• A metadata file dublin_core.xml
• Optionally, other metadata files with names like
metadata_[schema].xml where [schema] is the schema’s
name.
Scott Phillips provides a fine guide at
http://www.scottphillips.com/2009/05/howto-dspace-batch-ingest/

DSpace SAF (2/3) – Contents
Manifest
• The contents manifest contents names each bitstream
that will be in the item as well as it’s disposition:
• Bundle
• Permissions
• Primacy

DSpace SAF (3/3) – Metadata
• The SAF uses a specific XML format for the encoding of Dublin
Core style metadata.
• dublin_core.xml
• metadata_[schema].xml where [schema] is another
metadata schema in your repository’s registry
• The containing element is dublin_core with a schema
attribute.
• The field elements are dcvalue with schema, element, and
qualifier attributes.

Example imports…
• Provided are some rough code examples that will parse a CSV
metadata file (and associated content files) or a MARC XML
file (and associated content files).
• The code examples are in Java and best comprehended in a
nicely configured development environment, but we can work
with them using jEdit and the command line.
• We will conduct these imports into the repository and
consider the advantages and disadvantages of the approach.

An example import: CSV
• Create the import processor application in your
C:DevelopmentSAFCreator directory
• mvn clean package
• Run it with java –jar targetSAFCreator-0.0.1-
SNAPSHOT.one-jar.jar
• You will be presented with a Java Swing interface where you can
specify a csv metadat a file, a directory for source files, and
directory for SAF output, and other details for the batch.

An example import: CSV
• Import the SAF as follows:
• c:dspacebindspace import -a -e
admin@admin.com -s c:DevelopmentSAFtest-
output -c 123456789/2 -m
c:DevelopmentSAFtest-outputmap.map

An example import: MARC XML
• This example may be found in the import/marc directory
• Create the program with
• javac –sourcepath . *.java
• jar cfm xslimporter.jar manifest.mf *.class
• Run with
• java –jar xslimporter.jar
To see a common import difficulty, attempt an import as we did for
the CSV example.
-This will result in some schema-related errors, a very common
problem when doing imports.

An example import: MARC XML
• Add the following to a new thesis metadata schema and
re-attempt the import.
• degree.name
• degree.level
• degree.discipline
• degree.department

Consider the Import Results
• Idiosyncrasies of certain field values are more apparent in
different syntactic contexts.
• Different metadata origins entail different complexities in the
processing.
• Importation into a digital repository is a crucial step in the life
of a digital resource, as it is a chance to refine metadata, after
which it can be easily transmitted via crosswalks.
• However, it is a time when metadata are at risk of loss for lack
of care.

Final Thoughts on Content
Transmission
• Along with preservation, one of the greatest services provided
by digital repositories
• Yet, like preservation, good transmission requires constant work
• Crosswalks must be maintained to standards as well as local
practices
• Our means of importing content are constantly improving but
face a moving target
• New collection types inevitably require new development work if
their ingestion is to be automated

DSpace 4.2 Transmission: Import/Export

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to DSpace 4.2 Transmission: Import/Export

Similar to DSpace 4.2 Transmission: Import/Export (20)

More from DuraSpace

More from DuraSpace (20)

Recently uploaded

Recently uploaded (20)

DSpace 4.2 Transmission: Import/Export