BHL Tech Overview for BHL-Europe

BHL Technology Overview Chris Freeland Technical Director, BHL Director of Bioinformatics, Missouri Botanical Garden

About BHL: Usage, History

Goals of BHL
Scan public domain biodiversity literature.
Negotiate rights to digitize copyrighted materials.
Ingest content digitized by others.
Provide interfaces & APIs for repository.
GUIs
Services for data mining & citation resolution
http://www.biodiversitylibrary.org

More than:
33,000 volumes
13.3 million pages
Avg. monthly growth rate
1,500 volumes
600,000 pages
Now Online

Monthly Usage Stats
45,000 unique users
250,000 pageviews

History
Preliminary work: MOBOT’s Botanicus
http://www.botanicus.org
Funded by Keck Foundation & IMLS
Working demonstration of how nomenclators/databases can link into digitized scientific literature

Architecture

Distributed
Digitized content on Internet Archive servers in California
Metadata index on MOBOT servers in Missouri
Image server on MBL servers in Massachusetts
Nice, but not global

MOBOT Petabox cluster Internet Archive Image Server MBL

Scanning Workflow

Scanning Operations
BHL uses scanning centers established by Internet Archive for mass scanning.
Some partner libraries also scan in-house.
Want to expand international footprint:
mirrored content
ingest from global data providers
Locations of BHL/IA Scanning Centers

Workflow Selection Preparation Post Production (Re)publication Digitization Conservation

Open Access Data
Flora medica , oder, Abbildung der wichtigsten officinellen Pflanzen…[Heft 1-18]
Publisher: Jena,August Schmid,1831 [i.e. 1829-1831].
PDF OCR XML JP2

Complexities of distributed, mass scanning from NYBG from Smithsonian

Post Processing & Derivatives

Derivatives
JPEG2000 (JP2) images
OCR: ABBY FineReader
PDF: LuraTech PDF Compressor
XML metadata

Name Finding via TaxonFinder

Raw Image Converted to text via OCR Name finding via TaxonFinder Extract names Submit to NameBank SOAP response Name Finding in action with Taxonomic Intelligence…

Name Finding Stats to date *
Have mined more than 42 million name string occurrences
More than 30 million name strings verified by NameBank
1.5 million unique
*12 May 2009

Content Delivery

OCR error rate for names only Top OCR errors Study in 2008 found that for sample population of 3,003 names, 1,056 were incorrectly transcribed by OCR. http://biodiversitylibrary.blogspot.com/2008/10/evaluation-of-taxonomic-name-finding.html 1 Insert Space 8 n->v 2 Omit Space 9 l->i 3 e->c 10 r->i 4 u->I 11 u->ii 5 u->n 12 h->l 6 i->l 13 h->ii 7 c->e 14 e->o 35.16%

Current image delivery: djatoka
Images stored as JPEG2000 (.jp2)
Decoded & delivered to browser via djatoka
Open source JP2 image server
Developed by digital librarians
Scalable
Rapid development cycle (v1.1)
Growing community of users

djatoka Browser IIPViewer www.biodiversitylibrary.org .jp2 .jpg IA /page/1274907 pageid: 1274907 BHLdb http://www.archive.org/download/mushroomsofameri00palm/.../mushroomsofameri00palm_0010.jp2 images.biodivlibrary.org A user requests Mushrooms of America, edible and poisonous , Plate X: http://www.biodiversitylibrary.org/page/1274907 locate: BHL/IA architecture St. Louis San Francisco Woods Hole

New delivery option: IA Bookreader
Open source
Example: Flora medica
http://www.us.archive.org/GnuBook/?id=floramedicaodera118diet#229

IA Book Viewer http://www.us.archive.org/GnuBook/?id=floramedicaodera118diet#229

APIs & Data Sharing
Name Service ( Documentation )
REST: XML or JSON
Data Export ( Documentation )
Monthly export of BHL titles, volumes, pages, names, other metadata in delimited files

*Soon: Citation resolver via OpenURL
Beetle, A. A. 1977. Noteworthy grasses from Mexico V. Phytologia 37(4): 317–407.
http://example.edu/cgi?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:article &rft.jtitle=Phytologia &rft.atitle=Noteworthy+grasses+from+Mexico &rft.aulast=Beetle &rft.aufirst=A &rft.date=1977 &rft.volume=37&rft.issue=4&rft.spage=317&rft.epage=407

Articles

Article repository
Needed a way to display these PDFs
Wanted to extend contribution functionality to users
“ Safe harbor” model
BHL provides platform
Community provides content
Scientists, students, libraries

http:// cite.biodiversitylibrary.org
Drupal with Biblio module
Multi-lingual interface
Customizable display, layout
Solr search/faceting
OAI & other services for discovery/sharing

Outreach

BHL Blog
Updates
Announcements
1,500 users / month

Twitter
twitter.com/BioDivLibrary
Communication tool
Connecting with LinkedData community, other users
Receiving assistance, guidance
FAST turnaround

If BHL-E is not a Research Project…

Technologies in hand:
TaxonFinder
djatoka
IA Bookreader
Drupal/Biblio
OAI-PMH
OpenURL
Fedora Commons

Needed:
Deduplication Tools
Storage
OCR
Markup/rekeying
UI/UX
Interface translation
Data synchronization

Thank you
Chris Freeland
4344 Shaw Blvd.
St. Louis, MO 63110
[email_address]
http://www.biodiversitylibrary.org

http://blog.chrisfreeland.com	42
http://advancephysicsatgeorge.wikispaces.com	21
http://www.slideshare.net	1

BHL Tech Overview for BHL-Europe

by Chris Freeland on May 12, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 64

Statistics

BHL Tech Overview for BHL-Europe — Presentation Transcript