Open Library

Hello.

http://ﬂic.kr/p/4Pg28f
Wednesday, February 3, 2010

My name is George Oates, and I’m leading the Open Library project.

By way of
Introduction


http://www.ﬂickr.com/photos/dork/4040497259/

The Open Library project is produced out of the Internet Archive. We recently moved in to this
church in The Richmond in San Francisco. We’re all dreaming about ways we might be able to
transform it into a library.

Universal Access to
All Knowledge


Since 1996, the non-proﬁt Internet Archive has been building a digital library of Internet sites
and other things in digital form. archive.org has a ton of texts, video, software, live music...
all sorts of things.

http://www.ﬂickr.com/photos/george/3481337237/

We have many computers.

web.archive.org

Browse through over 150 billion web pages archived from 1996 to a few months ago.

archive-it.org

Archive-It, a subscription service from the Internet Archive, allows institutions to build and preserve
collections of born digital content. Collections are hosted at the Internet Archive data center and are
accessible to the public with full-text search. If you’re interested, you can see live demos online - the
times are listed on the main page there.

nasaimages.org

NASA Images was created through a partnership with NASA to bring public access to NASA's
image, video, and audio collections in a single, searchable resource.

archive.org/details/texts

1.8MM books

http://www.ﬂickr.com/photos/internetarchive/330205088/

- Many of our scanned books are delivered on the One Laptop Per Child machines...

openlibrary.org

The Open Library project is about 3 years old, and its overarching goal for is to have a page on the web
for every book ever published.

We have gathered about 30 million edition records (23 million are available through the site now) from
people like the Library of Congress, University of Toronto, TALIS, San Francisco Public Library and
others, with more on the way. We have built a database infrastructure and the wiki interface, and you
can search millions of book records, narrow results by facet, and search across the full text of about 1.8
million scanned books.

Wandered into The Library.
What did I find?


I started about 8 months ago. I am a web designer by trade, mostly ignorant of the practice
of librarianship and cataloging. In fact, I’d just come from the world of folksonomy at Flickr.
No classiﬁcation systems, full of humans and gorgeous photography.

Hmm...
• Dense library metadata
• Designed for classic institutional
search/retrieve practice
• Data is very “dry”, often of poor
quality Only title and author, for example
• No insight into the community


Coming in cold... so... what am I dealing with here?

Good!
• Loads of data > 23 million edition records

• Small user base < 20,000

• Small team 4 people + 2 advisors

• Small architecture 12 servers

• Flexible framework infogami, web.py


- But, there were also some good things!!

Understand relationships

http://ﬂic.kr/p/6xCJQS

So, what have we got, and how does it all inter-relate?

Any relationship can be made into a hyperlink.


Functional Requirement for Bibliographic Records. FRBR.

This is my attempt to understand it. By renaming things :)


Sketches, sketches, sketches

What does it mean to be an
Open Library?


Open to
Exploration


At a guess, ~95% of ins2tu2onal online collec2ons begin with a search UI

‐ There’s a presump2on of knowledge, not encouragement of explora2on
‐ How do I know what to search for if I don’t know what you’ve got?

OBJECTS - MEANS - REASONS 24
Rules for a Dictionary Catalogue, Charles A. Cutter. 1904, Page 12.

Charles CuJer: Rules for a Dic2onary Catalog ‐ par2cularly interested to hear that a librarian should righNully expect their patrons to have
a 2tle, author or subject in mind.

25


The trouble was, with many Open Library records, there just wasn’t much there... virtually useless?

26

Records in isola2on have no story to tell, nothing to engage with, nothing to explore. Nothing to navigate.

27


28

Wait a minute... what are these things?

29


30


Catalog as
Landscape?
http://www.ﬂickr.com/photos/nov03/3639455345/

Deconstruction

http://ﬂickr.com/photos/tupwanders/3356077817/

I’ve learned a wee bit about the history of library metadata... And museum metadata for that
matter.... It seems like the 1960s are a bit of a blight for human understanding, since that’s
the time when we got all excited about computers and their processing power, and seemingly
overwrote a lot of the crafty, poetic description and allusion...

What happens if you blow everything up?

LEADER: 01378cam 2200373I 4500
001 ocmocm01143845
003 OCoLC
005 19951211171151.0
008 750117r19531945nyu 000 1 eng u
019    $a4338553
040    $cSLC$dOCL$dTXA$dSFR$dOCoLC
049    $aSFRA
092    $aF$bSaLinger 1953
100 1 $aSalinger, J. D.$q(Jerome David),$d1919-
245 14 $aThe catcher in the rye.
260    $a[New York] :$bNew American Library,$c[1953, c1951]
300    $a192 p.$c18 cm.
490 0 $aSignet book,$vD1667
500    $aReprint of the 1945 ed. published by Little, Brown, Boston.
590    $aBarbara Grier and Donna McBride collection.
650 0 $aTeenage boys$vFiction.
650 0 $aBrothers and sisters$vFiction.
650 0 $aPreparatory schools$vFiction.
650 4 $aAlienation in teenagers$vFiction.
650 4 $aTeenage boys$xInterpersonal relations$vFiction.
650 4 $aEmotionally disturbed teenage boys$vFiction.
690    $aBarbara Grier and Donna McBride collection.
655 4 $aQueer pulps.
907    $a.b15331775$b10-24-07$c07-20-03
998    $axsf$b07-01-03$cm$da$e-$feng$gnyu$h4$i1
935    $aADM-9576
907    $a.b15331775$b02-23-04$c07-20-03
998    $axsf$b07-01-03$cm$da$e-$feng$gnyu$h4$i1
945    $aF SaLinger 1953$g1$i31223037153153$lxsfgl$o-$p$0.00$q-$rc$so
$t1$u0$v0$w0$x0$y.i25499191$z08-05-03


- I don’t want Open Library to jettison librarianship, or neglect to acknowledge the brilliant
hard work of librarians over the years...
- You could argue that this sort of computer-y librarianship (or any type of “educated
classiﬁcation”) was (perhaps unintentionally) designed to obscure the personal... the
practical... the human

- How might we adapt or extend (or revert?) this librarians’ work to appeal to a broader
audience?
- Let’s see what happens when you explode Library of Congress Subject Headings. This data
isn’t even in Open Library - we borrowed it from loc.gov then pulled out the dynamite...

650 0 $aPreparatory schools$vFiction.
650 4 $aAlienation in teenagers$vFiction.
650 4 $aTeenage boys$xInterpersonal relations$vFiction.
650 4 $aEmotionally disturbed teenage boys$vFiction.



audience?

650 0 $aPreparatory schools vFiction.
650 0 $aAlienation in teenagers vFiction.
650 0 $aTeenage boys$xInterpersonal relations vFiction.
650 0 $aEmotionally disturbed teenage boys vFiction.



audience?

Teenage boys, Fiction, Brothers and sisters,
Preparatory schools, Alienation in teenagers,
Teenage boys, Interpersonal relations,
Emotionally disturbed teenage boys



audience?


So, we’ve exploded all the subject headings into constituent parts, retaining their types
(subject, person, place, time, work, org etc), and made them searchable. You can see here a
search for any subjects that mention Brothers and Sisters


Looking at the subject page, you can see the Works with the most editions in the top panel,
with a handy indicator to tell you if you can read an electronic version....


If I scroll down...we’ve collated all the publish dates of all the editions with that subject


And, we can also display subjects that are used most often in conjunction with “Brothers and
Sisters”, as well as the authors who write most about them, and publishers who publish
books about them


We can also collect subjects together at the author level. Here you can see what sorts of
subjects Salinger writes about...


Subjects related to J. D. Salinger - note that we’ve retained the Place/Person/Time categories,
but it’s likely we’ll fold Orgs, Works etc into the more general subject bucket.


Incidentally, my colleague Lance Arthur popped in and updated the Salinger record with a
note of his death.

“Books, even after they have been
given a shelf and a number, retain a
mobility of their own. Left to their own
devices, they assemble in unexpected
formations; they follow secret rules of
similarity, unchronicled genealogies,
common interests and themes.”
Alberto Manguel, The Library at Night
Page 163, “The Library as Chance”


“Books, even after they have been
given a shelf and a number, retain a
mobility of their own. Left to their own
devices, they assemble in unexpected
formations; they follow secret rules of
similarity, unchronicled genealogies,
common interests and themes.”
Alberto Manguel, The Library at Night
Page 163, “The Library as Chance”


Here are some other interesting examples...

Civil War, 1861-1865


Wok Cookery


I’m not sure about you, but I certainly remember the Oates family’s ﬁrst wok. We tried. Oh,
how we tried.

Streets



Any guesses as to this subject?

La Comète de Halley!



Every 75 or 76 years, people write about it again :)


Place and subject?
Wondering about whether or not you could actually stand on the surface of Halley’s Comet...
Is that a helpful classiﬁcation of a place?

Which leads me on to the next chapter - a sort of radical exposure...

Open to
Exposure


“Library metadata is
diabolically rational.”
Karen Coyle, kcoyle.net


http://ﬂic.kr/p/v5uNz

The act of adding a book to a library catalog is a bit like playing tetris.


- incidentally, there were only 2 matches for the singular “gold mine”
- usage volume of any particular heading is indicated by the order of the list. most books ==
most used heading
- i wonder if these lists could help to normalise heading usage somehow... if that’s what we
want to do, of course...

http://www.ﬂickr.com/photos/blackbeltjones/4294354526/

We’ve noticed a TON of minor variations in the way cataloguers enter data... Trivial to us, but
very hard for computers to differentiate

U.S. Dept. of Agriculture, Forest Service, Forest Products Laboratory, 127 books
Forest Products Laboratory, 50 books
Dept. of Agriculture, Forest Service, Forest Products Laboratory, 38 books
U.S. Dept. of Agriculture, 37 books
U.S. Dept. of Agriculture, Forest Service, 30 books
Forest Products Laboratory, Forest Service, U.S. Dept. of Agriculture, 13 books
U.S. Forest Products Laboratory, 10 books
Dept. of Agriculture, 7 books


Proliﬁc authors on the subject of Wood
http://upstream.openlibrary.org/subjects/wood

So, diabolically rational, and yet, pretty inconsistent, when you can see it in the aggregate
like this.

Norton 5,072 books
W.W. Norton 2,371 books
W. W. Norton & Company 2,320 books
W. W. Norton 1,577 books
W W Norton & Co Inc 933 books
W W Norton & Co Ltd 824 books
W.W. Norton & Co. 490 books
W.W. Norton & Company 281 books
Distributed by W.W. Norton 269 books
W W Norton & Co Inc (Np) 207 books
Jeffrey Norton Pub 151 books
Norton*(ww Norton Co 144 books
W. W. Norton & company, inc. 124 books
W. W. Norton & Co. 120 books
W.W.Norton 112 books
W.W. Norton & Company, inc. 85 books
Distributed by W.W. Norton & Co. 65 books
W W Norton & Co (Sd) 63 books
W.W. Norton & company, inc. 54 books
Distributed to the book trade by W.W. Norton 52 books
W.W. Norton & Company, Inc. 51 books
W.W. Norton & Company Ltd 48 books
W. W. Norton & Company, inc. 46 books


Variants on a publisher’s name


There’s also the issue of using old headings, for example, the sort of heading about a person
that contains birth/death dates. Useful disambiguation information. But, what happens when
they die?


For those of you who may be unaware, Mr. Bacon is no longer with us.


And it looks like 32 books know that.

History


Publish dates... I’m wondering if a cataloguing system had a required ﬁeld somewhere...

?
History


Published in the future? Let alone nine thousand nine hundred and ninety nine?

“Build it so anyone can
contribute any amount.”
Clay Shirky


So, the idea is, someone sees an error like those publishing dates, and can go into the record
to correct them.


Admittedly, this is the editing history of a personal friend, Dinah Sander - who some of you
may know - but, after seeing the history graphs with incorrect info - Dinah couldn’t stop
herself from jumping in and removing the dodgy data.

Open to
Elaboration


Substrate:
any surface on which a plant or animal lives or on
which a material sticks

http://ﬂic.kr/p/4itJcB

There’s also an alternate deﬁnition which suggests a substrate is catalytic; something that
facilitates a reaction.

What if we consider the library
records like that?

http://flic.kr/p/4itJcB

- The trick is, these records are like a new language. To use them and operate within them
requires specific training. While this is not necessarily a bad thing, and experts are wonderful,
it means that people like me (reasonably clever, been to Uni) can’t make use of them.
- What we’ve tried to do is reveal the substrate; to show the landscape of librarianship and
the beautiful work of classification that has been happening for centuries.
- Using the data we already had in our catalog, and without colliding with the taxonomy/vs
folksonomy issue.

http://ﬂic.kr/p/6zyU3U Tension?

The Taxonomy vs Folksonomy debate may be represented thusly.

1) Books are for use.
2) Every reader his [or her] book.
3) Every book its reader.
4) Save the time of the User.
5) The library is a growing organism.


So, on the basis of the idea of our current catalog being a substrate, as Ranganathan
suggests in his ﬁve laws of library science...

1) Books are for use.
2) Every reader his [or her] book.
3) Every book its reader.
4) Save the time of the User.
5) The library is a growing organism.


Some suggest that this last law was intended more as a pointer to a library’s physical space:
its staff, its buildings, its shelves. I think it’s also useful to consider the law in the
classiﬁcation space, particularly if you imagine that the Open Library catalog today is
effectively a substrate for further connections, elaboration and even corrections.

http://www.ﬂickr.com/photos/curiousexpeditions/2406513532/

Release the shackles!

http://ﬂic.kr/p/38TZ

What if a catalog looks like this? Is crystalline? What if it is unconstrained by the need to sort,
say, alphabetically?

From the artist of this image, Jared Tarbell: “Lines like crystals form at perpendicular angles
to existing lines. A complex form emerges.
1000 classic computational substrate, color palette stolen from Jackson Pollock: A simple
perpendicular growth rule creates intricate city-like structures. The simple rule, the complex
results, the enormous potential for modiﬁcation; this has got to be one of my all time favorite
self-discovered algorithms. Lines likes crystals grow on a computational substrate.”

Activity/History


One of the key components to any happy social system is the visibility of other people, and a
sense of activity. This is one of the key elements we’re focussed on in the redesign. This
particular list shows all edits by humans on Open Library, and actually, turns out to be a
handy way to spot check what’s happening. You’ll notice too, there’s a special tab for the
variety of edits that we run across the system using bots. Often pretty mechanical and
repetitive, we found that the bots obscure the humans if you just mush everything up in a big
list, so we separated them.


So, here’s an example of a record I happened to spot one day as I glanced through the Recent
Activity list...


If you look closely, you’ll notice that apparently, this person believes the Collected Poems to
be part of the “pooop” series... and that they enjoy bacon.


Open Library is a Wiki. That means that you can see the entire editing history of any one
record, and easily undo any errors (or mischief).

http://www.ﬂickr.com/photos/eyeliam/2562666943/

Open to
Exchange


Shelf

http://www.ﬂickr.com/photos/emdot/400280705/

Network

http://www.ﬂickr.com/photos/arenamontanus/352130655/

The question is: Now that these records are online...how many things can we connect them
to?


There are lots and lots of sites on the web that deal with bookish information. Goodreads,
Librarything etc. Why not connect Open Library records to these sites too?


The Guardian, for example, also broadcasts !mely bookish information. We’re wondering how
we might fold that in to providing jump points for people into Open Library...

“We shouldn't waste
people's time making fixes
that would be better done
by machine.”
Edward Betts, Chief Data Munger, Open Library


Canonical ID?
Collect them.


Canonical ID?
Exchange them.


http://openlibrary.org/books/olid/OL7440033M
http://openlibrary.org/books/isbn/0385472579
http://openlibrary.org/books/lccn/93005405
http://openlibrary.org/books/oclc/28419896
http://openlibrary.org/books/id/240727
http://openlibrary.org/books/amazon/...
http://openlibrary.org/books/bookmooch/...
http://openlibrary.org/books/goodreads/...
http://openlibrary.org/books/ocaid/...
http://openlibrary.org/books/librarything/...
http://openlibrary.org/books/paperback_swap/...
http://openlibrary.org/books/Your ID Here/...


You can already ping Open Library with an ID other than the Open Library identiﬁer to see if
we have any matches.

http://openlibrary.org/books/olid/OL7440033M
http://openlibrary.org/books/lccn/93005405
http://openlibrary.org/books/oclc/28419896
http://openlibrary.org/books/id/240727
http://openlibrary.org/books/amazon/...
http://openlibrary.org/books/bookmooch/...
http://openlibrary.org/books/goodreads/...
http://openlibrary.org/books/librarything/...
http://openlibrary.org/books/ocaid/...
http://openlibrary.org/books/paperback_swap/...
http://openlibrary.org/books/Your ID Here/...


Looking
Forward


Tools

http://www.ﬂickr.com/photos/genkigecko/3371739666/

Tools to help people help the data.


This is part of the list of Works by J. D. Salinger, which as you can see is far from perfect.
Humans can spot in a moment that some of these should be blended. Computers? Not so
much.


Similarly, with authors.

Lists

http://www.ﬂickr.com/photos/cibi/3149659494/

- keep track of edits on things you’re interested in, or have edited
- export a small subset of records (bibliography, MARC, XML etc)
- provide another pivot for navigation in the networked catalog - “George has this book on
her “Famous Cheeses” list”

http://www.ﬂickr.com/search/?q=landscape&l=cc

Over time, the hope is that we’ll see a proliferation of different landscapes; different paths
through the catalog...

Lending

http://www.ﬂickr.com/photos/readinginpublic/3999260222/

Starting to investigate what it might mean to lend of out-of-print ebooks.
If anyone would like to join in on that, please let me know.

upstream.openlibrary.org

http://www.ﬂickr.com/photos/nationallibrarynz_commons/3326203787/ http://ﬂickr.com/photos/daveynin/560170975/

We still have a lot to do. I’ve shown you some of the muddiness in the catalog, probably
mainly because our data is aggregated from a variety of systems. But, we think it’s a new way
to look for a book to read, and that’s exciting! I hope you’ll take some time to poke around
the new site. And please, do let me know what you think!

Thank You!
George Oates, glo@archive.org
http://ﬂickr.com/photos/roadsidepictures/244926428/

Open Library

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Similar to Open Library

Similar to Open Library (20)

More from George Oates

More from George Oates (20)

Open Library