2. In 1985,blair and mornon published a report
on a large
scale experiment aimed at evaluating the
retrieval effectiveness of a full-text search
and retrieval system.
This is known as STAIRS (storage and
information retrieval system) study.
3. The database examined in the STAIRS study
consisted
of nearly 40,000 documents, representing
rouhgly 3,50,000 pages of hard copy text used
in the defence of a large corporate lawsuit.
The full texts of all the pages where available
online texts and could be retrieved where
specified words appeared either simply or in
Boolean combinations.
the major objective of the STAIRS evaluation
was to asses how well the system could retrieve
all the documents (and only those) relevant to a
given request, and measures of recall and
precisitions were used this purpose.
4. The lawyers generated a total of 51 general
requests.
The lawyers evaluated those documents and
grouped them into ‘vital’, ‘satisfactory’,
marginally relevant’ , or ‘irrelevant’ in relation
to the request.
A sampling techniques was adapted .
Random samples were taken and these were
evaluated by the lawyers .
The total number of relevant of documents
that existed in these subsets was estimated.
5. Out of the 51 requests, values for recall and
Presidion were calculated for 40 and the
remaining 11 were used to check the
sampling techniques and control for possible
bias in the evaluation of retrieved and sample
tests.
The value of precision ranged in percentage
terms from 100%.
6. In attempting to find out why STAIRS could
retrieve only one out of five relevant items in
response to a request.
They point out that a retrieved set of several
thousand documents is impractical to browse
on the part of the user, and quit naturally the
user in such cases wants to reformulate the
query by adding more and more search terms
to bring the out put size to a manageable
limit.
7. Early evaluation experiments have produced a
number of facts and figures that can be
utilized in many ways –in designing a new
system.
The STAIRS study reported retrieval results
that are contradictory to the earlier studies.
In 1986 salton published a paper
commenting on the major points of objection
raised by salton.
8. There is evidence for output overload in large
systems.
The effectiveness of automatic indexing.
Four major methodological problems of the
previous studies.
Small database unreliable techniques for judging
the relevance.
The first concern is with the size and
composition of the collections used for testing in
research .
9. The second concern is with the nature of the
queries used in research collections. for
example: looking at the INSPEC test collection
of 12,684documents, if a typical query maps
to 33 relevant documents, this would
extrapolate to an STN user retrieving over
24,000 documents.
In chemical abstracts database containing 9.5
million documents.
10. The third points raises the issue of whether the
issues the performance of a ranked retrieval
system is a large enough improvement over the
Boolean search model search model to represent
a cost-effective alternative.
The use of research collections with larger
vocabularies and more records.
the investigation of retrieval schemes that
incorporate proximity information
The use of test collection that contain more
specific queries.
