The document discusses the Text REtrieval Conference (TREC) workshops which provide infrastructure for large-scale testing of text retrieval technologies using realistic test collections. TREC aims to advance the state of the art through system evaluations, discussion of research methodology, and an open forum for idea exchange. It describes the typical yearly conference cycle including task definition, document collection, experimentation, and results analysis.

1. TEXT RETRIEVAL CONFERENCES
(TREC)
PRESENTEDTED
BY
S K A B D U L G A F FA R

2. WHAT IS TREC?
 A workshop series that provides the infrastructure for large – scale testing of
(text) retrieval technology.
 Realistic test collections.

Uniform, approprite scoring procedures
 A forum for the exchange of research ideas & for the discussion of research
methodology.

3. TREC PHILOSOPHY

TREC is modern example of the cranfield tradition.
 System evaluation based on test collections
 Emphasis on advancing the state of the art from
evaluation results.
 TREC’s primary purpose is not competitive
benchmark
 Experimental workshop: some times experiments
fail.

4. YEARLY CONFERENCE CYCLE
TASK FOR DEFINITION
DOCUMENT
PROCUREMENT
COLLECTION FOR
PUBLICATION
TOPIC DEVELOMENT
PROCEDING
PUBLICATION
IR EXPERIMENT
TREC
CONFERENCE
RELEVENCE
ASSESSMENT
RESULTS ANALYSIS
RESULTS EVALUATION

5. TREC 2007 PROGRAMME COMMITTEE
Ellen Voorhees
Chair persion:David Lewis
James Allan
Chris Buckley
Gord Cormac
Bill Hersh
Rechard Tong
Ross Wilkinson

6. TREC 2007 TRACK COORDINATORS
 Bloag : Craig Mackdonald, Ladh Ounis, Lan
Soboroff
 Entriprise :Bailey , Cras Well
 Genories : Bill Hersh
 Legal: Barson, Oard, Thompson
 Million query :James allan
 QA: Hon Dang ,Diahe kelly
 Spam: Gord Cormbac

7. HISTOY
(I)1992
First TREC conference started by Donna Harman & Chorles Wayne
Two task s: (a) Ad hoc retrieval , (b) Routing
(ii)1993 (TREC -2) True baseline performance for main tasks
(iii)1995(TREC-4) Official beginning of TREC track structure
(iv)1998(Trec-7) Routing dropped a as main task
(v)2000(TREC 9) Ad – hoc main task dropped ,first all –track TREC
TREC Tracks :
Task that focuses on a particular sub problem of the text retrieval
Tracks invigorate tree & keep tree ahead of the state of art
Specialized collection support search in new areas
First large scale experiments debug what the task really is .
Set of tracks in a particular TREC depend on :
Interest of participants
Appropriateness of task of TREC
Need of sponsors
Resource constraints

8. TREC IMPACT
 Test collection
 Incubator for new research areas
 Common evaluation methodology & improved measures
for text retrieval
 Open forum for exchange of research
 Technology transfer

9. TEST COLLECTIONS
 Paper in general IR literature are TREC collection
 common evaluation methodology & improved
 Measures for text retrieval
 Incubator for new research areas
 Phd theses resulting from CLIR ,DR,QA - Participation .
Open forum for exchange of research
publication of all results prevents unsuccessful research
Technology transfer

10. CRANFIELD TRADITION
 Laboratory testing of system components
 Fine control over variables
 Abstraction from operational setting
COMPARATIVE TESTING :
TEST COLLECTION :
 Set of documents
 Set of questions
 Relevance judgment

11. TREC APPROACH
Assessors create topics at NIST :
 Topics are set to participants ,who return
documents per topics .
ranking of best 1000
 NIST forms pools of unique documents from all submission which
the assessors judge for relevance ;
 System are evaluated using relevance judgment

12. VARIOUS TYPE OF TRECS
TREC -1
1992, TREC-1
was the fast
conference held at NIST.
text retrieval
Brought information
retrieval researchers to
compare the result of their different systems
which used on a large new test collection
(called TIPSTER collection)
The first conference attracted 28 groups of
researchers from academia & industry and
generated wide spared
interest from the
information retrieval community .

13. TREC -2
TREC-2 took place in august 1993 , total participating group was
31st .
The participating group choose three level of participating group
.
Categary “A”- FULL PARTICIPATION
Categary “B” - FULL PARTICAPTION USING ONE –QUARTER OF
THE FULL DOCUMENT
SET.
CATEGARY “C” – FOR EVALUATION ONLY
TWO TYPES OF RETRIVEAL WERE EXAMINED ,RETRIVEAL
USING AN “ad-hoc” query. & routing query.

14. TREC - 3 TO 5
TREC-3 they were made shorter by excluding some keywords &
TREC -4 they were made even shorter to investigate the problems with
very short user statements
TREC-5 included both short & long version of the topic s with the goal
of caring out deeper investigation into witch types of techniques
work well on various length of topics

15. TREC -6
In TREC -6 ,three new tracks –
a. Speech ,cross-language and high –precision information retrieval .
b. The goal of the cross language information retrieval track is to
facilities research on system that are able to retrieve relevant
documents .
c. Speech documents retrieval track – is to stimulate research on
retrieval techniques for spoken documents .(example Radio
broadcasts)
d. The high precisions tracks was designed to deal with task s.

16. TRACK 7
Track -7 contains seven tracks out of which 2 tracks –
query track and very large corpus track - were new.
The goal of the query track was to create a large query
collection . The query step was designed as a means of
creating a large set of different queries for an existing
TREC topic set, topic 1- 50 .
The very large corpus track used a gigabyte document
collection in order to explore how well retrieval
algorithms scale to large document collections

17. TREC 8-12
 RTEC -8 contained seven tracks out of which two –
question –answering (QA) and web tracks are new .The
objectives of QA track is to explore the possibilities of
providing answer to specific natural language queries .
 TREC-9 also introduce seven track s.
A video track was introduced in TREC -10 , and a novelty
track was introduced in TREC -11
 The goal of the novelty tracks is to investigate system’s
abilities to locate relevant and new information with in
the track set of documents returned by a traditional
document retrieval system TREC -12.(2003)

18. C0NCLUSION
 TREC series of experiment s have brought together researchers from
across the world to work on common.
 Goal is build up large text collection .

19. REFERENCES
1. Introduction to modern information retrieval
by
G .G. Chowdhury
2. NIST (National Institute of Standards and Technology)
U.S. Department of commerce

20. THANK YOU