TEXT RETRIEVAL CONFERENCES
(TREC)

PRESENTEDTED
BY
S K A B D U L G A F FA R
WHAT IS TREC?
 A workshop series that provides the infrastructure for large – scale testing of
(text) retrieval technolog...
TREC PHILOSOPHY


TREC is modern example of the cranfield tradition.

 System evaluation based on test collections
 Emp...
YEARLY CONFERENCE CYCLE
TASK FOR DEFINITION

DOCUMENT
PROCUREMENT

COLLECTION FOR
PUBLICATION

TOPIC DEVELOMENT

PROCEDING...
TREC 2007 PROGRAMME COMMITTEE
Ellen Voorhees
Chair persion:David Lewis
James Allan
Chris Buckley
Gord Cormac
Bill Hersh
Re...
TREC 2007 TRACK COORDINATORS
 Bloag : Craig Mackdonald, Ladh Ounis, Lan
Soboroff
 Entriprise :Bailey , Cras Well
 Genor...
HISTOY
(I)1992

First TREC conference started by Donna Harman & Chorles Wayne

Two task s: (a) Ad hoc retrieval , (b) Rout...
TREC IMPACT
 Test collection
 Incubator for new research areas
 Common evaluation methodology & improved measures
for t...
TEST COLLECTIONS
 Paper in general IR literature are TREC collection
 common evaluation methodology & improved
 Measure...
CRANFIELD TRADITION
 Laboratory testing of system components
 Fine control over variables
 Abstraction from operational...
TREC APPROACH
Assessors create topics at NIST :
 Topics are set to participants ,who return
documents per topics .

ranki...
VARIOUS TYPE OF TRECS
TREC -1

1992, TREC-1
was the fast
conference held at NIST.

text retrieval

Brought information
ret...
TREC -2
TREC-2 took place in august 1993 , total participating group was
31st .
The participating group choose three level...
TREC - 3 TO 5
TREC-3 they were made shorter by excluding some keywords &
TREC -4 they were made even shorter to investigat...
TREC -6
In TREC -6 ,three new tracks –
a. Speech ,cross-language and high –precision information retrieval .
b. The goal o...
TRACK 7
Track -7 contains seven tracks out of which 2 tracks –
query track and very large corpus track - were new.
The goa...
TREC 8-12
 RTEC -8 contained seven tracks out of which two –
question –answering (QA) and web tracks are new .The
objecti...
C0NCLUSION
 TREC series of experiment s have brought together researchers from
across the world to work on common.
 Goal...
REFERENCES
1. Introduction to modern information retrieval
by
G .G. Chowdhury
2. NIST (National Institute of Standards and...
THANK YOU
Upcoming SlideShare
Loading in...5
×

Text Retrieval Conferences (TREC)

275

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
275
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Text Retrieval Conferences (TREC)

  1. 1. TEXT RETRIEVAL CONFERENCES (TREC) PRESENTEDTED BY S K A B D U L G A F FA R
  2. 2. WHAT IS TREC?  A workshop series that provides the infrastructure for large – scale testing of (text) retrieval technology.  Realistic test collections.  Uniform, approprite scoring procedures  A forum for the exchange of research ideas & for the discussion of research methodology.
  3. 3. TREC PHILOSOPHY  TREC is modern example of the cranfield tradition.  System evaluation based on test collections  Emphasis on advancing the state of the art from evaluation results.  TREC’s primary purpose is not competitive benchmark  Experimental workshop: some times experiments fail.
  4. 4. YEARLY CONFERENCE CYCLE TASK FOR DEFINITION DOCUMENT PROCUREMENT COLLECTION FOR PUBLICATION TOPIC DEVELOMENT PROCEDING PUBLICATION IR EXPERIMENT TREC CONFERENCE RELEVENCE ASSESSMENT RESULTS ANALYSIS RESULTS EVALUATION
  5. 5. TREC 2007 PROGRAMME COMMITTEE Ellen Voorhees Chair persion:David Lewis James Allan Chris Buckley Gord Cormac Bill Hersh Rechard Tong Ross Wilkinson
  6. 6. TREC 2007 TRACK COORDINATORS  Bloag : Craig Mackdonald, Ladh Ounis, Lan Soboroff  Entriprise :Bailey , Cras Well  Genories : Bill Hersh  Legal: Barson, Oard, Thompson  Million query :James allan  QA: Hon Dang ,Diahe kelly  Spam: Gord Cormbac
  7. 7. HISTOY (I)1992 First TREC conference started by Donna Harman & Chorles Wayne Two task s: (a) Ad hoc retrieval , (b) Routing (ii)1993 (TREC -2) True baseline performance for main tasks (iii)1995(TREC-4) Official beginning of TREC track structure (iv)1998(Trec-7) Routing dropped a as main task (v)2000(TREC 9) Ad – hoc main task dropped ,first all –track TREC TREC Tracks : Task that focuses on a particular sub problem of the text retrieval Tracks invigorate tree & keep tree ahead of the state of art Specialized collection support search in new areas First large scale experiments debug what the task really is . Set of tracks in a particular TREC depend on : Interest of participants Appropriateness of task of TREC Need of sponsors Resource constraints
  8. 8. TREC IMPACT  Test collection  Incubator for new research areas  Common evaluation methodology & improved measures for text retrieval  Open forum for exchange of research  Technology transfer
  9. 9. TEST COLLECTIONS  Paper in general IR literature are TREC collection  common evaluation methodology & improved  Measures for text retrieval  Incubator for new research areas  Phd theses resulting from CLIR ,DR,QA - Participation . Open forum for exchange of research publication of all results prevents unsuccessful research Technology transfer
  10. 10. CRANFIELD TRADITION  Laboratory testing of system components  Fine control over variables  Abstraction from operational setting COMPARATIVE TESTING : TEST COLLECTION :  Set of documents  Set of questions  Relevance judgment
  11. 11. TREC APPROACH Assessors create topics at NIST :  Topics are set to participants ,who return documents per topics . ranking of best 1000  NIST forms pools of unique documents from all submission which the assessors judge for relevance ;  System are evaluated using relevance judgment
  12. 12. VARIOUS TYPE OF TRECS TREC -1 1992, TREC-1 was the fast conference held at NIST. text retrieval Brought information retrieval researchers to compare the result of their different systems which used on a large new test collection (called TIPSTER collection) The first conference attracted 28 groups of researchers from academia & industry and generated wide spared interest from the information retrieval community .
  13. 13. TREC -2 TREC-2 took place in august 1993 , total participating group was 31st . The participating group choose three level of participating group . Categary “A”- FULL PARTICIPATION Categary “B” - FULL PARTICAPTION USING ONE –QUARTER OF THE FULL DOCUMENT SET. CATEGARY “C” – FOR EVALUATION ONLY TWO TYPES OF RETRIVEAL WERE EXAMINED ,RETRIVEAL USING AN “ad-hoc” query. & routing query.
  14. 14. TREC - 3 TO 5 TREC-3 they were made shorter by excluding some keywords & TREC -4 they were made even shorter to investigate the problems with very short user statements TREC-5 included both short & long version of the topic s with the goal of caring out deeper investigation into witch types of techniques work well on various length of topics
  15. 15. TREC -6 In TREC -6 ,three new tracks – a. Speech ,cross-language and high –precision information retrieval . b. The goal of the cross language information retrieval track is to facilities research on system that are able to retrieve relevant documents . c. Speech documents retrieval track – is to stimulate research on retrieval techniques for spoken documents .(example Radio broadcasts) d. The high precisions tracks was designed to deal with task s.
  16. 16. TRACK 7 Track -7 contains seven tracks out of which 2 tracks – query track and very large corpus track - were new. The goal of the query track was to create a large query collection . The query step was designed as a means of creating a large set of different queries for an existing TREC topic set, topic 1- 50 . The very large corpus track used a gigabyte document collection in order to explore how well retrieval algorithms scale to large document collections
  17. 17. TREC 8-12  RTEC -8 contained seven tracks out of which two – question –answering (QA) and web tracks are new .The objectives of QA track is to explore the possibilities of providing answer to specific natural language queries .  TREC-9 also introduce seven track s. A video track was introduced in TREC -10 , and a novelty track was introduced in TREC -11  The goal of the novelty tracks is to investigate system’s abilities to locate relevant and new information with in the track set of documents returned by a traditional document retrieval system TREC -12.(2003)
  18. 18. C0NCLUSION  TREC series of experiment s have brought together researchers from across the world to work on common.  Goal is build up large text collection .
  19. 19. REFERENCES 1. Introduction to modern information retrieval by G .G. Chowdhury 2. NIST (National Institute of Standards and Technology) U.S. Department of commerce
  20. 20. THANK YOU
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×