Biblio to Fedora Commons REST API Chris Moyers March 30, 2009 CiteBank

CiteBank, Biblio, and Fedora Commons CiteBank is using Biblio for presentation layer. Biblio and Drupal provide much in the way of content management. CiteBank is using Fedora Commons is Preservation Layer. In the event of disaster, Fedora will help make sense of massive body of content.

CiteBank is using Biblio for presentation layer.

Biblio and Drupal provide much in the way of content management.

CiteBank is using Fedora Commons is Preservation Layer.

In the event of disaster, Fedora will help make sense of massive body of content.

Biblio Stores records, (Bibliographic citations) in MySQL. Allows users to attach files to Bibliographic citations. These could be PDFs, images, OCR text, etc. Uploads reside in filesystem. Upload metadata is stored in MySQL.

Stores records, (Bibliographic citations) in MySQL.

Allows users to attach files to Bibliographic citations. These could be PDFs, images, OCR text, etc.

Uploads reside in filesystem. Upload metadata is stored in MySQL.

Fedora Commons A Digital Object is a unit of storage in Fedora. A journal article could be a Digital Object, for example. Digital Objects contain Datastreams. Digital Object (stored as FOXML files in filesystem) Datastream 1 Datastream 2 Xml metadata URI Datastream n URI …

A Digital Object is a unit of storage in Fedora. A journal article could be a Digital Object, for example. Digital Objects contain Datastreams.

Fedora Commons Stores Digital Object metadata on filesystem as FOXML files. Allows Externally Referenced datastreams, which simply point to files external to Fedora. FOXML provides context for external files referenced within a datastream. Has SOAP API (API-M) which allows programmatic ingestion, modification, and purging of digital objects and datastreams.

Stores Digital Object metadata on filesystem as FOXML files.

Allows Externally Referenced datastreams, which simply point to files external to Fedora.

FOXML provides context for external files referenced within a datastream.

Has SOAP API (API-M) which allows programmatic ingestion, modification, and purging of digital objects and datastreams.

Distributed Filesystem (DFS) DFS will create redundant data across nodes. Files are not stored on the DFS in a way that relates them. Due to Fedora and Drupal’s different ways of handling files. Drupal files are in one place, and metadata used to bind them together are in another place (MySQL). FOXML files are in yet another place. Consolidate filesystem paths? In event of disaster, change of technology, we need to be able to retain and make sense of data.

DFS will create redundant data across nodes.

Files are not stored on the DFS in a way that relates them. Due to Fedora and Drupal’s different ways of handling files.

Drupal files are in one place, and metadata used to bind them together are in another place (MySQL). FOXML files are in yet another place.

Consolidate filesystem paths?

In event of disaster, change of technology, we need to be able to retain and make sense of data.

Why Yet Another REST API? Biblio and Fedora are not integrated. Islandora is a module for Drupal that interfaces with Fedora Commons. Requires user to explicitly add content to Fedora No direct tie-in with Biblio. Adds complexity to Fedora Commons. We need some way to bridge the gap.

Biblio and Fedora are not integrated.

Islandora is a module for Drupal that interfaces with Fedora Commons.

Requires user to explicitly add content to Fedora

No direct tie-in with Biblio.

Adds complexity to Fedora Commons.

We need some way to bridge the gap.

Goals of REST API Get data from Biblio to Fedora in a fashion that is transparent to the end user. Digital Objects must be self-describing. Keep it as implementation-neutral as possible. Provide some basic logging and recovery from failure. Make something that can be contributed back to the community. Keep it as simple as possible, both to maintain and to implement.

Get data from Biblio to Fedora in a fashion that is transparent to the end user.

Digital Objects must be self-describing.

Keep it as implementation-neutral as possible.

Provide some basic logging and recovery from failure.

Make something that can be contributed back to the community.

Keep it as simple as possible, both to maintain and to implement.

Process Overview

The REST API Biblio provides the node ID for the citation. The API then performs several steps: Generate Fedora’s PID from the node ID provided by Biblio Determine whether item exists in Fedora, uses SOAP API-M, API-A Create Bibtex file for the citation. This is essentially a duplicate of the record Biblio creates within MySQL. Add/Chg/Del digital object with its datastreams (including the Bibtex file) Perform logging

Biblio provides the node ID for the citation.

The API then performs several steps:

Generate Fedora’s PID from the node ID provided by Biblio

Determine whether item exists in Fedora, uses SOAP API-M, API-A

Create Bibtex file for the citation. This is essentially a duplicate of the record Biblio creates within MySQL.

Add/Chg/Del digital object with its datastreams (including the Bibtex file)

Perform logging

Recovery, Redundancy DFS creates redundant copies of files. Biblio records are replicated as Bibtex records in filesystem, and tied to PDFs, images, etc, with a Fedora Digital Object. FOXML files from Fedora are stored on DFS. Loosely coupled, technology independent, and redundant.

DFS creates redundant copies of files.

Biblio records are replicated as Bibtex records in filesystem, and tied to PDFs, images, etc, with a Fedora Digital Object.

FOXML files from Fedora are stored on DFS.

Loosely coupled, technology independent, and redundant.

Recovery, Redundancy (cont’d) DFS allows for hardware failure. Uses commodity hardware. Data kept in open formats. Not “hostage” to Drupal, Biblio, Fedora. FOXML provides context for resources within datastreams Bibtex datastreams provide comprehensive metadata for items in CiteBank. Images, PDFs, etc, are bound to Bibtex records via FOXML. Other key elements of a DR plan, such as off-site backups, also need to be considered.

DFS allows for hardware failure. Uses commodity hardware.

Data kept in open formats. Not “hostage” to Drupal, Biblio, Fedora.

FOXML provides context for resources within datastreams

Bibtex datastreams provide comprehensive metadata for items in CiteBank.

Images, PDFs, etc, are bound to Bibtex records via FOXML.

Other key elements of a DR plan, such as off-site backups, also need to be considered.

Future Steps for REST API Contribute code… To Islandora? To Biblio? As a separate Drupal module?

Contribute code…

To Islandora?

To Biblio?

As a separate Drupal module?