Presentation for Digital Government Research Conference on usability of two multi-database retrieval systems.

1. Evaluating What Really Counts
Expert Users Assess Data Access
Mechanisms and Interfaces
Peter Sommer
Director of Education
Center for New Media Teaching and Learning
Columbia University
Linda Catalano
Doctoral Candidate
Department of Sociology
Columbia University

2. A panel of expert users (social science data librarians
with a minimum of 4 years professional experience in
the field) were interviewed to:
 Identify what elements make for a good information
querying and retrieval system
 Assess the OGIRS and AskCal systems according
to the above-identified criteria.

3. The panel cited the following elements as contributing
to a good information querying and retrieval system:
 An ability to search a number of related terms
simultaneously (as opposed to separately in consecutive
searches)
 An ability to “see where you are” in the system
 Explanations of and/or background information on both
the data sought and the data found; e.g.,
 Definitions of terms
 Sources of data
 How the data are computed

4.  Helpful feedback related to user actions
 Indication of required search parameters; e.g.,
 Geographic area
 Time frame
 Unit of measure
 A consistent user-system interface
 A record of user actions and corresponding system
responses

5. The panel then developed a query* based on the New
York Times Editorial page of that day** and sought to
answer it utilizing two different energy data access
mechanisms:
 OGIRS (Oil and Gas, Extracting Data), an Access
database that presents an interface of “peepholed”
hierarchical categories; and
 AskCal (Philpot et al.), in which data are mediated by a
set of conceptual categories linked into an ontology and
with both a natural language and a structured menu
interface.
*“When did the price of gas go over 75 cents per gallon?”
**5/19/04.

6. The panel was presented with the opening query
screen of each system. They were provided no
additional directions; rather, they were asked to tell
us how they might best proceed.
The OGIRS search was soon abandoned. The
AskCal search was successfully completed, returning
satisfactory answers in two series.

7. The panel attributed several advantages to the
AskCal system, specifically,
 An obvious starting point
 An intuitive transition between natural language and
menu driven query formulation and refinement
 An automated text response to the natural language
query, which provided helpful feedback as to the
required search parameters
 Immediate and continuous visibility of all system
components – i.e., natural language text field, category
menus, user inputs and corresponding search results

8.  The ability to search for related terms simultaneously
from a single screen
 Once results were narrowed down, information on data
sources
 Notation of changes (feedback) in the natural language
query box in response to menu selections, leading to...
 A strong ability to learn how to query the system to the
greatest advantage.

9. Panelists were next introduced to a third, wholly
different yet complementary system – GetGloss (Co
et al.), a meta-glossary of energy terms pooled from
sources across the internet – and asked to imagine
how using it in conjunction with a system like AskCal
would assist information retrieval or not.

10. The response to GetGloss, as an appendix to AskCal,
was overwhelmingly positive. Among the benefits
panelists envisioned it providing were:
 Identification and/or specification of search parameters
 Demonstration of variations in terminology across data
sources, which, together with AskCal’s features,
provided
 A bridge between user/consumers’ “everyday” language
and experts’ generally more specific language

11. Panelists were eager to see a system uniting the
features and functions of AskCal and GetGloss
developed. This evaluation suggests a number of
recommendations in doing so:
 Open with the natural language query string only
 Respond to user inputs with bold changes in format;
e.g., new text marked in red
 Maintain a record of user actions and system responses
(feedback) as a scrollable “chat”
 Hyperlink GetGloss terms to descriptions of information
source agencies and means of computation for
particular statistical results