Forward in Reverse

Forward in Reverse
A Gentle Overview Of Forward System Architecture
Eric, Mike & Steve – WiLSWorld 2010

Outline
Intro to Forward with Demo
Batch Processing (Backend)
Web Application (Frontend)
Challenges
Q&As throughout

Intro & Demo

http://forward.library.wisconsin.edu

Batch Processing

We have gobs & gobs of data.

1) Extract it

1a) ILS Data

Sort, Deduplicate, Merge

Antique Style Key
By Stars*Go*Blue
http://www.flickr.com/photos/artbydebora/1406682449/

Common Identifier = OCLC Number

Catalog Extract Processing Details
14 Voyager Instances
13M MARC bibliographic records extracted
Approximately 14 hours
Local C code
Sorted, deduplicated and merged output:
8M records
10GB Raw MARC data

Why Merge?
URLs
Formats
Holdings

1b) Digital Collection Data

Fedora Extract Processing Details
1 Fedora Repository
13K “First Class” XML Objects extracted
Approximately 4 hours
Repository query language
XML output:
METS XML package
Structural XML
MODS Bibliographic XML
41MB XML data

2) Index it

We take raw library data and process it with MARC/XML parsing tools and local parsing rulesin order to build a Solr search index.

Raw data (MARC & METS XML)
Parsing libraries (Java code: marc4j, SAXParser)
Local code that defines parsing rules
Solr index

1. Raw data

LEADER 02000cam a22003734a 45 0
001 6939454
005 20051208125417.0
008 051104s2004 enka $b 001 0 eng
010 $a 2003045349
035 $a (OCoLC)ocm52165958
040 $aDLC $c DLC $d XMA $d BAKER $d UKM
015 $a GBA430162 $2bnb
016 7$a 012906573 $2Uk
020 $a 0754605175 (alk. paper)
024 $a 99811375970
042 $apcc
049 $a GZMA
050 00$a B3376.W564 $b W55355 2004
082 00 $a 111/.85/092 $2 21
245 00 $a Wittgenstein, aesthetics, and philosophy / $c edited by Peter B. Lewis.
260 $aAldershot, Hants, England ; $a Burlington, VT : $bAshgate, $c c2004.
300 $a xii, 255 p. : $b ill. ; $c 24 cm.
440 0 $aAshgateWittgensteinian studies
505 0$a Wittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg
504 $a Includes bibliographical references (p. 235-247) and index.

02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enkab 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. HagbergaIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM

2. MARC/XML parsing libraries

02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enkab 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. HagbergaIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM

3. Local code

4.
http://lucene.apache.org/solr/

What is Solr?
An XML API over a Lucene search index.

Access to Raw Formats
Raw MARC stored for Merged record
Live calls made to Fedora
web services

Data Refresh
Bibliographic: weekly
Circulation status: nightly

For more information, see
http://sdg.library.wisc.edu/blog/2010/03/03/solr-marc-indexing-based-on-diffs/

Web Application

Frontend?
(X)HTML
JavaScript
Cascading Style Sheets
Design
Information Architecture
User experience
Chrome (images, icons, pretty)

Forward Colophon
ActiveRecordBaseWithoutTable (Rails plugin)
Apache
Blacklight (Rails plugin)
Blueprint CSS
Bookreader (jQuery)
Capistrano
Crontab
Engines (Rails plugin)
Fedora
Freebase API
GeoIP (Ruby gem)
Google Books API
Haml (Rails plugin)
Happymapper (Ruby gem)
HathiTrust API
jQuery
Ken (Ruby gem)
LowPro (Prototype JS)
MARC4J
Passenger (modrails)
Prototype JS
PostgreSQL
Raphael
Ruby on Rails
Shibboleth
Subversion
Solr / Lucene
Summon (Ruby gem)
UW-Madison Libraries Staff Directory API
UWDC (Rails plugin)
Voyager API
Tender love and attention

Campus Affiliation
Users localize to a school, allows us scope many features to their campus.
GeoIPRubyGem
Match IP addresses with physical locations.
Raphaël—JavaScript Library
“Small JavaScript library that should simplify your work with vector graphics on the web”.

Raphaël
SVG elements, like the circles and squares in the Forward splash page, can be treated as XHTML elements allowing us to manipulate them with JavaScript and CSS.
http://raphaeljs.com/

Campus Homepage
Forward application stack:
Apache+Passenger (modrails)
Ruby on Rails
PostgreSQL
Apache Solr

Apache+Passenger
Phusion Passenger is an Apache module, which makes deploying Ruby and Ruby on Rails applications on Apache a breeze.
http://www.modrails.com/

Ruby on Rails
“Ruby on Rails is an open-source web framework that’s optimized for programmer happiness and sustainable productivity.”
http://rubyonrails.org/

PostgreSQL
“PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness.”
http://www.postgresql.org/

Apache Solr
“Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling.”
http://lucene.apache.org/solr/

Results

Results – Three columns

Results – Data sources

Results – Facets – Solr

Results – Solr + PostgreSQL + APIs

Results – Context – APIs

Results – Three main columns

Results – CSS grid

Blueprint
“Blueprint is a CSS framework, which aims to cut down on your development time. It gives you a solid foundation to build your project on top of, with an easy-to-use grid, sensible typography, useful plugins, and even a stylesheet for printing.”
http://blueprintcss.org/

Show – Book

Show – Image

Show – Full Text Book

Show – View Full Text Book

BookReader
“The Internet Archive BookReader is used to view books from the Internet Archive online and can also be used to view other books. ”
http://github.com/openlibrary/bookreader

Challenges

Challenges
Merging MARC, METS extracts
Batch processing time (Time/CPU constraints)
Page level indexing (Bookviewer - memory/disk constraints)
Voyager API
Organization challenges
big project, small shop
dealing with vendor silos
multiple cataloging standards
quality of services challenges

Thanks!

Forward in Reverse

by Eric Larson on Jul 22, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 99

Statistics

Forward in Reverse — Presentation Transcript