A comparison of two digital libraries based on pre-established criteria
Stephen J. Stose June 19, 2010 1IST 676: Assignment 2In what follows is a general outline for a method of evaluating digital libraries. This paper hastwo goals: 1) to refine and continue to develop effective criteria for the evaluation of digitallibraries, and 2) the use of these criteria in the practice of evaluating two specific digital libraries.Two federated digital libraries are evaluated across five criteria of evaluation. The criteria wereadapted from multiple sources, most principally from that of Saracevic (2005) and Choudhury(2002). While the criteria presented here do not necessarily follow the structure of these previousauthors, they do attempt to incorporate and re-organize the same multiple dimensions generallyaccepted as essential. These dimensions are listed in bold, each followed by a series of questionsthat attempt to manifest each concept. They do not constitute orthogonal dimensions. Yet they doseek to represent psychologically distinct experiences of a typical user’s episode with the digitallibrary resource. For this reason the criteria attempts to model this user flow. That is to say thatupon entry, the user has 1) a first impression, followed by 2) a judgment of its aesthetics andoverall design, which incorporates signposts (links etc) that allow/disallow its 3) usability; entryto the digital objects themselves is facilitated or constrained by the exposure and effectiveness ofthe sites 4) taxonomic system, which lead to the 5) digital content itself a user hopefully cancontextualize and learn from. Such a description of an episode is of course a user-centric ideal,and exposes the author’s own preferences towards good digital library design (i.e., how hebelieves a site should foster navigation and resource discovery). However, it also serves as amodel for further research, given that these propositions regarding user-centric episodes and thecriteria by which they should be judged are themselves testable and hence falsifiable. University of Wyoming Digital Initiative The Arizona Memory Project http://digital.uwyo.edu/ http://azmemory.lib.az.us/The overall quality The front page immediately invites you A very brief time here assures you thatof organization: Is into its various “recently digitized” one interface is the rule. The front pagethe content collections. Navigating from the main quickly divulges a lot, in a friendlyimmediately interface into separate collection manner. It has teasers of some of itsaccessible and interfaces is realistic, but there is seldom a content (usually collectionsinviting with clarity clear way back to the collection options. highlighted), to attract the user inside. Itin its presentation? Many collections are federated within one boasts of teacher resources that, withinDoes its overall space (LUNA), which provides coherency seconds, show you the availability ofstructure serve its if the user is lucky to land there. Others lesson plans for teachers. You seestated goals? What lead one quite astray into university quickly that this is some kind ofimpression does the programs with uncertain connection to federated system with many institutions,first 15 seconds of this digital initiative. This fragmentation and a quick click on ‘browse’ orentry leave? is disorienting, a fact the ‘about’ section ‘search’ reveals much of the metadata at failed to allay. The search feature on the one’s fingertips. An extremely attractive front page was both ineffective and start. misleading. The portal’s overall quality is therefore very low.Design: Is the site The main page attractively The design is simple, yet effective.attractive and presented/exposed the various collections There are few items in the menu, andsimple to visually available up front. If lucky, most these are repeated with shortnavigate? Does the collections lead to a separate LUNA explanations within the main content.design assist the workspace (a DAM), which is very More specialized features (‘help’, ‘my
Stephen J. Stose June 19, 2010 2IST 676: Assignment 2user in orienting its attractive, simple and customizable, and favorites’, and ‘my preferences’) arecontents? Is the federates all searching for collections less conspicuous, but availabledesign stable and within its domain. If unlucky, other immediately. The site uses only onepsychologically collections lead to very distinct spaces interface throughout, and a fluid widthcoherent and this can be very disorienting and allows use of your entire screen. Thethroughout, or does organizationally distinct. The front page is site is attractive to navigate, mostlyit require continual simple and visually attractive, but the because it is simple with links thatvisual and menu options usually just offer a series of resolve with the information they boast.conceptual re- links that lead users into a separate web The ‘help’ and ‘about’ are long htmlorientation as one interface, thereby visually and documents, many times unadvised innavigates? psychologically placing them again in a information architecture, but a quick new space. Thus, the design lacks an menu upfront immediately breaks up umbrella space and visual guideposts that this content by scrolling you to each organize collection subspaces and ensure question listed in its menu. Very navigational cohesion. effective, even if a trifle bland.Usability: Is the If lucky to land inside LUNA, where This is an incredibly simple andresource easy to many of the collections reside, the users effective digital library, which speaksuse and effortless to must learn one mere interface in order to well of OCLC’s CONTENTdm, itsnavigate? Do the navigate across its collections. Links underlying architecture. This is alinks resolve in resolve very quickly and the navigational federated digital library, and the linksexpected places, menu is stable, but a user does need to illustrate this effectively. Within oneand is the user experiment a while to learn how they click of the front page you can readcorrectly oriented function. It is not immediately clear when about the contributing institutions, andto retrace this navigation is within or between within two clicks (‘browse>’select aspace? Are collections, but this price of federated collection from the list’) you can viewnavigational cohesion is resolved soon enough. There the items just from that institution’ssignposts stable is very effective user support, but using collection. One click on ‘favorites’ orand informative? Is this resource effectively is not effortless, ‘my preferences’ shows the user howthe user aware of given its power to afford users many the interface can be changed tothe various features organization powers: media groups, accommodate personalization foravailable with presentations, API embedding, and other improved usage. The help section issufficient guidance options given registration. The site may available, but given the ease of thisin their operation? be too powerful at the cost of ordinary interface, almost superfluous, I dare and quick presentational capacity for say. A user has no issue at all finding more ordinary user groups. what s/he needs, or just browsing.System and The main taxonomy is the collection-level A taxonomy of categories ismetadata: Does a categorization. The “Search” feature on immediately available by clicking ontaxonomy of the main page is useless. If within LUNA, ‘browse’. While this list is quite sparsecategories enable the advanced search has excellent (broken into ‘collections’, ‘topics’,effective resource dropdown fields with extensive metadata ‘formats, and ‘time periods’ only), itdiscovery? Is the and Boolian options, and allows federated broadness is indicative of themetadata exposed searching within or amongst all abstraction needed to properly federateto users to enable collections. Simple searching reveals the collections of over 90 institutions.effective search? faceted options in a sidebar, expanding or This is a wonderful way to get usersDo queries return refining the results. It is difficult to find into the content immediately, and listedpresentable results the “browse by category” section in order beside each thumbnail is basic ‘subject’digestible to human to immediately learn about the items and ‘description’ metadata, whichusers? Are queries available, but when found the “what”, allows for quick digestion of its
Stephen J. Stose June 19, 2010 3IST 676: Assignment 2handled efficiently? “who”, and “when” categories do provide relevancy before going further. FacetsAre there faceted a needed conceptual framework for the in the sidebar allow narrowing theoptions for collection’s holdings that might serve the fields. When choosing an image, muchcontinued user better if constantly available in a more metadata unfolds beneath therefinement? sidebar. Queries provide results with image itself, or when doing an advance various grid and object resize options, search. Queries are handled with item-level summaries available immediately, such that you forget you below or upon hovering over each item. are amongst 67,000 digital objects.Digital content: This single-interface federation (in The links ‘my preferences’ or ‘myDoes informative LUNA) is excellent for combining high- favorites’ allows you to change thedescriptive (e.g., quality items across/within the collection, screen defaults (sort order, backgroundhistorical) material but contains little collection-level color, grid/list options), such that thecontextualize the information, leaving little to learn about content itself appears according to eachdigital objects? Are the history and context of each collection. user’s desire. You learn a lot verythe objects Attempts to return to the main portal for quickly regardless of the format, asinformatively collection-level guidance are difficult and beneath each thumbnail are effectivelabeled at the item- otherwise unhelpful, as “about” section descriptors, and beneath each imageand collection- discusses issues more relevant to itself the metadata is co-referenced withlevel, to make it university members, and the “exhibits” links to that reveal all the objects withineasy to learn? Is and other menu items show links that lead that ‘category’. Browsing is fun, whichthe content a high to unfamiliar space. Individual items are implies you are learning and engaged indigital quality, and well documented in the left sidebar with a the content. Checkboxes beside eachcan users complete listing of technical, descriptive, object allows you to collect contentmanipulate (e.g., and administrative metadata. Users can (‘my favorites’). Image size is limited,collate, zoom, print, add/collate items to a workspace or but you can request larger sizes. Theitemize) the presentation space, zoom and compare, if best feature is the item-level toobjects? they care to learn these features (and collection-level coherency. Regardless register). This plethora of multimedia of how you came upon an item, it is co- features overshadows the storytelling and referenced to information regarding its contextualized nature a digital library also collection. Thus, one learns about the needs to represent. item and its context easily and effectively.Is the digital The site has potential, once loose ends are This digital library is fantastic, one oflibrary mission collected. The site fails to serve as an the best federated sources I have used,supported? umbrella portal except that the first page and very worthy of emulation. It allows users to enter specific collections informs on the item- and collection- (with little way back, however). The level, integrates these seamlessly within LUNA federated interface is an excellent one interface, while still recognizing the multimedia tool that exhibits great control collection has a separate institutional over item-level metadata, but it fails to provenance. The mission is completely incorporate collection-level information, supported. Besides a few design issues and probably sacrifices historical regarding attractiveness (menu fonts) storytelling of the items within each and fuzzy header images, it is top-class collection for multimedia options ordinary in terms of usability. users may find difficult.
Stephen J. Stose June 19, 2010 4IST 676: Assignment 2A brief conclusion is warranted. Obviously the Wyoming Digital Initiative is under construction.This I know, as I know one of the directors (iSchool Graduate Ben Goldman). For this reason,the analysis may not be completely fair. However, given that I mostly concentrated on thefederated LUNA system, some words of direct comparison may be granted. While the power andcomplexity of the LUNA system is quickly apparent, these benefits have as costs the joy ofbrowsing I experienced within the Arizona Memory Project. Too much complexity may instillonly confusion, and given that most visitors just wish to browse and perhaps do basic educationalresearch, the power afforded within the LUNA system is perhaps a trifle unnecessary, if not atodds with its educational purpose. In my opinion, if and when a user requires high-levelmultimedia capabilities, these should be secondary to a digital library’s public and educational(i.e., information quality should be first). The LUNA system does raise the bar for digital libraryarchitecture. This power, however, should be secondary to a good experience.Saracevic, Tefko (2005). How were digital libraries evaluated? Presented at Libraries in the Digital Age(LIDA), Dubrovnik and Mljet, Crotia, May 30-June.Choudhury, G.S.; Hobbs, B.; M Lorie, Flores, N.E. (2002). A Framework for Evaluating Digital LibraryService. D-Lib Magazine July/August 2002. Volume 8 Number 7/8