Surviving the Deluge: Lessons from DOIs and Electronic Publishing at John Wiley & Sons

Surviving the Deluge: Lessons from DOIs and Electronic Publishing at John Wiley & Sons By Matthew Larson

Who I Am
Developer, tech lead, and system architect at Wiley for the last 9 years
Responsible for DOI registration and other systems that support electronic publishing activities

The Challenge
A lot of content from a lot of places
“ A lot of content”: > 1000 new DOIs per day from journals, books and major reference works
“ A lot of places”: Offices in New Jersey, Boston, San Francisco, Oxford, Germany, Singapore, and elsewhere
How to handle DOI registration for all this content?

Surviving the Deluge: Some Principles
Here are five system design principles we’ve used for handling the constant stream of content
We’ll look at how we employ them in our CrossRef registration application, XIRS (eXternal Identifier Registration System)
XIRS was built and launched in 2008 to help handle the combined Wiley/Blackwell content load
XIRS receives CrossRef XML, submits it to CrossRef, and tracks the CrossRef responses
Written in Java (1.6) and deployed on Tomcat 6

XIRS Data Flows

Principle 1: Centralization
Content from multiple sources must go through a single system to be manageable.
Might be tempting to offer a client library instead but that invites trouble
XIRS handles all Wiley DOI registrations
All error handling, reporting and support in one place
Any system that needs to store or process content must be on the critical path for publication
Otherwise you will always be chasing synchronization

Principle 2: Transparency
Make the system as transparent as possible
It’s easy to miss errors when batch processing this much content
How?
Always make real-time system views available
Provides quick assurance that everything is alright and shows when it isn’t
Email daily error reports with the most critical data in the subject line
Show as much information as possible to make support easier
Provide easy-to-use reporting so anyone can query the system

Real-time System Views

Notifications

As Much Info As Possible

Simple Reporting

Principle 3: Heal Thyself
Systems that batch process this much content must be self healing
Lowers support load and increases accuracy
One good method: queue everything
Assume that nothing will work on the first try
Assume that networks, disks, and external services will fail.
The system that sends data to XIRS queues that data--if XIRS is down the data flow will start as soon as XIRS is available.
XIRS itself queues the data it receives. If CrossRef is down, it will try until the data submission is successful.

Principle 3: Heal Thyself
Another good technique: belt and braces (have backup methods)
We process the CrossRef emails coming back
But if we get no CrossRef response (for whatever reason), we call CrossRef directly to get the response
Lets us avoid extra support work if there are emails problems—the system will find those responses itself

Principle 4: Trust No One
Assume that bad data will show up sooner rather than later
QA everything
We parse the CrossRef XML before it is sent to XIRS
And then XIRS parses it again and rejects it if it doesn’t parse
Proactively check and fix data
We insert the timestamp in XIRS so it’s always the latest
We shorten fields to fit within the size limits
We check every file for multiple prefixes in the same file and split the file up automatically
All XSLT must check for string-length(.) > 0 before inserting any elements to avoid empty elements

Principle 4: Trust No One
Assume files will be too big
Some reference articles can have 500+ citations
We limit by DOI count and size on disk
Assume files will be too small
Files can come very quickly with the same container-level (journal) DOI
Causes database contention
We have to remove container-level DOIs that have already been registered to avoid this

Principle 5: Synchronicity
All system communications should be synchronous wherever possible
Delayed/asynchronous responses will be lost when batch processing large amounts of data and support will be much tougher
Long waits and timeout values are acceptable to ensure that the calling system knows whether the call was successful or not

http://crossref.org	19
http://www.crossref.org	6
http://haleinej.blogspot.com	1

Surviving the Deluge: Lessons from DOIs and Electronic Publishing at John Wiley & Sons

by CrossRef on Nov 16, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 26

Statistics

Surviving the Deluge: Lessons from DOIs and Electronic Publishing at John Wiley & Sons — Presentation Transcript