BHL Tech Overview for BHL-Europe

BHL Technology Overview Chris Freeland Technical Director, BHL Director of Bioinformatics, Missouri Botanical Garden

About BHL: Usage, History

Goals of BHL

Scan public domain biodiversity literature.

Negotiate rights to digitize copyrighted materials.

Ingest content digitized by others.

Provide interfaces & APIs for repository.

GUIs

Services for data mining & citation resolution

http://www.biodiversitylibrary.org

More than:

33,000 volumes

13.3 million pages

Avg. monthly growth rate

1,500 volumes

600,000 pages

Now Online

Monthly Usage Stats

45,000 unique users

250,000 pageviews

History

Preliminary work: MOBOT’s Botanicus

http://www.botanicus.org

Funded by Keck Foundation & IMLS

Working demonstration of how nomenclators/databases can link into digitized scientific literature

Architecture

Distributed

Digitized content on Internet Archive servers in California

Metadata index on MOBOT servers in Missouri

Image server on MBL servers in Massachusetts

Nice, but not global

MOBOT Petabox cluster Internet Archive Image Server MBL

Scanning Workflow

Scanning Operations

BHL uses scanning centers established by Internet Archive for mass scanning.

Some partner libraries also scan in-house.

Want to expand international footprint:

mirrored content

ingest from global data providers

Locations of BHL/IA Scanning Centers

Workflow Selection Preparation Post Production (Re)publication Digitization Conservation

Open Access Data

Flora medica , oder, Abbildung der wichtigsten officinellen Pflanzen…[Heft 1-18]

Publisher: Jena,August Schmid,1831 [i.e. 1829-1831].

PDF OCR XML JP2

Complexities of distributed, mass scanning from NYBG from Smithsonian

Post Processing & Derivatives

Derivatives

JPEG2000 (JP2) images

OCR: ABBY FineReader

PDF: LuraTech PDF Compressor

XML metadata

Name Finding via TaxonFinder

Raw Image Converted to text via OCR Name finding via TaxonFinder Extract names Submit to NameBank SOAP response Name Finding in action with Taxonomic Intelligence…

Name Finding Stats to date *

Have mined more than 42 million name string occurrences

More than 30 million name strings verified by NameBank

1.5 million unique

*12 May 2009

Content Delivery

OCR error rate for names only Top OCR errors Study in 2008 found that for sample population of 3,003 names, 1,056 were incorrectly transcribed by OCR. http://biodiversitylibrary.blogspot.com/2008/10/evaluation-of-taxonomic-name-finding.html 1 Insert Space 8 n->v 2 Omit Space 9 l->i 3 e->c 10 r->i 4 u->I 11 u->ii 5 u->n 12 h->l 6 i->l 13 h->ii 7 c->e 14 e->o 35.16%

Current image delivery: djatoka

Images stored as JPEG2000 (.jp2)

Decoded & delivered to browser via djatoka

Open source JP2 image server

Developed by digital librarians

Scalable

Rapid development cycle (v1.1)

Growing community of users

djatoka Browser IIPViewer www.biodiversitylibrary.org .jp2 .jpg IA /page/1274907 pageid: 1274907 BHLdb http://www.archive.org/download/mushroomsofameri00palm/.../mushroomsofameri00palm_0010.jp2 images.biodivlibrary.org A user requests Mushrooms of America, edible and poisonous , Plate X: http://www.biodiversitylibrary.org/page/1274907 locate: BHL/IA architecture St. Louis San Francisco Woods Hole

New delivery option: IA Bookreader

Open source

Example: Flora medica

http://www.us.archive.org/GnuBook/?id=floramedicaodera118diet#229

IA Book Viewer http://www.us.archive.org/GnuBook/?id=floramedicaodera118diet#229

APIs & Data Sharing

Name Service ( Documentation )

REST: XML or JSON

Data Export ( Documentation )

Monthly export of BHL titles, volumes, pages, names, other metadata in delimited files

*Soon: Citation resolver via OpenURL

Beetle, A. A. 1977. Noteworthy grasses from Mexico V. Phytologia 37(4): 317–407.

http://example.edu/cgi?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:article &rft.jtitle=Phytologia &rft.atitle=Noteworthy+grasses+from+Mexico &rft.aulast=Beetle &rft.aufirst=A &rft.date=1977 &rft.volume=37&rft.issue=4&rft.spage=317&rft.epage=407

Articles

Article repository