Chances and Challenges in Comparing Cross-Language Retrieval Tools
Upcoming SlideShare
Loading in...5
×
 

Chances and Challenges in Comparing Cross-Language Retrieval Tools

on

  • 511 views

Presentation at IRF symposium 2010

Presentation at IRF symposium 2010
Vienna June 3, 2010

Statistics

Views

Total Views
511
Views on SlideShare
504
Embed Views
7

Actions

Likes
0
Downloads
7
Comments
0

2 Embeds 7

http://www.linkedin.com 5
https://www.linkedin.com 2

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Chances and Challenges in Comparing Cross-Language Retrieval Tools Chances and Challenges in Comparing Cross-Language Retrieval Tools Presentation Transcript

  • Chances and Challenges in Comparing Cross-Language Retrieval Tools Giovanna Roda Vienna, Austria Irf Symposium 2010 / June 3, 2010
  • CLEF-IP: the Intellectual Property track at CLEF CLEF-IP is an evaluation track within the Cross Language Evaluation Forum (Clef). 1 1 http://www.clef-campaign.org
  • CLEF-IP: the Intellectual Property track at CLEF CLEF-IP is an evaluation track within the Cross Language Evaluation Forum (Clef). 1 organized by the IRF 1 http://www.clef-campaign.org
  • CLEF-IP: the Intellectual Property track at CLEF CLEF-IP is an evaluation track within the Cross Language Evaluation Forum (Clef). 1 organized by the IRF first track ran in 2009 1 http://www.clef-campaign.org
  • CLEF-IP: the Intellectual Property track at CLEF CLEF-IP is an evaluation track within the Cross Language Evaluation Forum (Clef). 1 organized by the IRF first track ran in 2009 running this year for the second time 1 http://www.clef-campaign.org
  • CLEF-IP: the Intellectual Property track at CLEF CLEF-IP is an evaluation track within the Cross Language Evaluation Forum (Clef). 1 organized by the IRF first track ran in 2009 running this year for the second time 1 http://www.clef-campaign.org
  • What is an evaluation track? An evaluation track in Information Retrieval is a cooperative action aimed at comparing different techniques on a common retrieval task.
  • What is an evaluation track? An evaluation track in Information Retrieval is a cooperative action aimed at comparing different techniques on a common retrieval task. produces experimental data that can be analyzed and used to improve existing systems
  • What is an evaluation track? An evaluation track in Information Retrieval is a cooperative action aimed at comparing different techniques on a common retrieval task. produces experimental data that can be analyzed and used to improve existing systems fosters exchange of ideas and cooperation
  • What is an evaluation track? An evaluation track in Information Retrieval is a cooperative action aimed at comparing different techniques on a common retrieval task. produces experimental data that can be analyzed and used to improve existing systems fosters exchange of ideas and cooperation produces a reusable test collection, sets milestones
  • What is an evaluation track? An evaluation track in Information Retrieval is a cooperative action aimed at comparing different techniques on a common retrieval task. produces experimental data that can be analyzed and used to improve existing systems fosters exchange of ideas and cooperation produces a reusable test collection, sets milestones Test collection A test collection consists traditionally of target data, a set of queries, and relevance assessments for each query.
  • Clef–Ip 2009: the task The main task in the Clef–Ip track was to find prior art for a given patent.
  • Clef–Ip 2009: the task The main task in the Clef–Ip track was to find prior art for a given patent. Prior art search Prior art search consists in identifying all information (including non-patent literature) that might be relevant to a patent’s claim of novelty.
  • Participants - 2009 track 1 Tech. Univ. Darmstadt, Dept. of CS, Ubiquitous Knowledge Processing Lab (DE)
  • Participants - 2009 track 1 Tech. Univ. Darmstadt, Dept. of CS, Ubiquitous Knowledge Processing Lab (DE) 2 Univ. Neuchatel - Computer Science (CH)
  • Participants - 2009 track 1 Tech. Univ. Darmstadt, Dept. of CS, Ubiquitous Knowledge Processing Lab (DE) 2 Univ. Neuchatel - Computer Science (CH) 3 Santiago de Compostela Univ. - Dept. Electronica y Computacion (ES)
  • Participants - 2009 track 1 Tech. Univ. Darmstadt, Dept. of CS, Ubiquitous Knowledge Processing Lab (DE) 2 Univ. Neuchatel - Computer Science (CH) 3 Santiago de Compostela Univ. - Dept. Electronica y Computacion (ES) 4 University of Tampere - Info Studies (FI)
  • Participants - 2009 track 1 Tech. Univ. Darmstadt, Dept. of CS, Ubiquitous Knowledge Processing Lab (DE) 2 Univ. Neuchatel - Computer Science (CH) 3 Santiago de Compostela Univ. - Dept. Electronica y Computacion (ES) 4 University of Tampere - Info Studies (FI) 5 Interactive Media and Swedish Institute of Computer Science (SE)
  • Participants - 2009 track 1 Tech. Univ. Darmstadt, Dept. of CS, Ubiquitous Knowledge Processing Lab (DE) 2 Univ. Neuchatel - Computer Science (CH) 3 Santiago de Compostela Univ. - Dept. Electronica y Computacion (ES) 4 University of Tampere - Info Studies (FI) 5 Interactive Media and Swedish Institute of Computer Science (SE) 6 Geneva Univ. - Centre Universitaire d’Informatique (CH)
  • Participants - 2009 track 1 Tech. Univ. Darmstadt, Dept. of CS, Ubiquitous Knowledge Processing Lab (DE) 2 Univ. Neuchatel - Computer Science (CH) 3 Santiago de Compostela Univ. - Dept. Electronica y Computacion (ES) 4 University of Tampere - Info Studies (FI) 5 Interactive Media and Swedish Institute of Computer Science (SE) 6 Geneva Univ. - Centre Universitaire d’Informatique (CH) 7 Glasgow Univ. - IR Group Keith (UK)
  • Participants - 2009 track 1 Tech. Univ. Darmstadt, Dept. of CS, Ubiquitous Knowledge Processing Lab (DE) 2 Univ. Neuchatel - Computer Science (CH) 3 Santiago de Compostela Univ. - Dept. Electronica y Computacion (ES) 4 University of Tampere - Info Studies (FI) 5 Interactive Media and Swedish Institute of Computer Science (SE) 6 Geneva Univ. - Centre Universitaire d’Informatique (CH) 7 Glasgow Univ. - IR Group Keith (UK) 8 Centrum Wiskunde & Informatica - Interactive Information Access (NL)
  • Participants - 2009 track 9 Geneva Univ. Hospitals - Service of Medical Informatics (CH)
  • Participants - 2009 track 9 Geneva Univ. Hospitals - Service of Medical Informatics (CH) 10 Humboldt Univ. - Dept. of German Language and Linguistics (DE)
  • Participants - 2009 track 9 Geneva Univ. Hospitals - Service of Medical Informatics (CH) 10 Humboldt Univ. - Dept. of German Language and Linguistics (DE) 11 Dublin City Univ. - School of Computing (IE)
  • Participants - 2009 track 9 Geneva Univ. Hospitals - Service of Medical Informatics (CH) 10 Humboldt Univ. - Dept. of German Language and Linguistics (DE) 11 Dublin City Univ. - School of Computing (IE) 12 Radboud Univ. Nijmegen - Centre for Language Studies & Speech Technologies (NL)
  • Participants - 2009 track 9 Geneva Univ. Hospitals - Service of Medical Informatics (CH) 10 Humboldt Univ. - Dept. of German Language and Linguistics (DE) 11 Dublin City Univ. - School of Computing (IE) 12 Radboud Univ. Nijmegen - Centre for Language Studies & Speech Technologies (NL) 13 Hildesheim Univ. - Information Systems & Machine Learning Lab (DE)
  • Participants - 2009 track 9 Geneva Univ. Hospitals - Service of Medical Informatics (CH) 10 Humboldt Univ. - Dept. of German Language and Linguistics (DE) 11 Dublin City Univ. - School of Computing (IE) 12 Radboud Univ. Nijmegen - Centre for Language Studies & Speech Technologies (NL) 13 Hildesheim Univ. - Information Systems & Machine Learning Lab (DE) 14 Technical Univ. Valencia - Natural Language Engineering (ES)
  • Participants - 2009 track 9 Geneva Univ. Hospitals - Service of Medical Informatics (CH) 10 Humboldt Univ. - Dept. of German Language and Linguistics (DE) 11 Dublin City Univ. - School of Computing (IE) 12 Radboud Univ. Nijmegen - Centre for Language Studies & Speech Technologies (NL) 13 Hildesheim Univ. - Information Systems & Machine Learning Lab (DE) 14 Technical Univ. Valencia - Natural Language Engineering (ES) 15 Al. I. Cuza University of Iasi - Natural Language Processing (RO)
  • Participants - 2009 track
  • Participants - 2009 track 15 participants
  • Participants - 2009 track 15 participants 48 experiments submitted for the main task
  • Participants - 2009 track 15 participants 48 experiments submitted for the main task 10 experiments submitted for the language tasks
  • 2009-2010: participants
  • 2009-2010: evolution of the CLEF-IP track 2009 1 task: prior art search targeting granted patents 15 participants all from academia families and citations manual assessments standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 1 task: prior art search targeting granted patents 15 participants all from academia families and citations manual assessments standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 2010 1 task: prior art search targeting granted patents 15 participants all from academia families and citations manual assessments standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 2010 1 task: prior art search prior art candidate search and classification task targeting granted patents 15 participants all from academia families and citations manual assessments standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 2010 1 task: prior art search prior art candidate search and classification task targeting granted patents patent applications 15 participants all from academia families and citations manual assessments standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 2010 1 task: prior art search prior art candidate search and classification task targeting granted patents patent applications 15 participants 20 participants all from academia families and citations manual assessments standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 2010 1 task: prior art search prior art candidate search and classification task targeting granted patents patent applications 15 participants 20 participants all from academia 4 industrial participants families and citations manual assessments standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 2010 1 task: prior art search prior art candidate search and classification task targeting granted patents patent applications 15 participants 20 participants all from academia 4 industrial participants families and citations include forward citations manual assessments standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 2010 1 task: prior art search prior art candidate search and classification task targeting granted patents patent applications 15 participants 20 participants all from academia 4 industrial participants families and citations include forward citations manual assessments expanded lists of relevant docs standard evaluation mea- sures
  • 2009-2010: evolution of the CLEF-IP track 2009 2010 1 task: prior art search prior art candidate search and classification task targeting granted patents patent applications 15 participants 20 participants all from academia 4 industrial participants families and citations include forward citations manual assessments expanded lists of relevant docs standard evaluation mea- new measure: pres, more sures recall-oriented
  • What are relevance assessments A test collection (also known as gold standard) consists of a target dataset, a set of queries, and relevance assessments corresponding to each query.
  • What are relevance assessments A test collection (also known as gold standard) consists of a target dataset, a set of queries, and relevance assessments corresponding to each query. The CLEF-IP test collection:
  • What are relevance assessments A test collection (also known as gold standard) consists of a target dataset, a set of queries, and relevance assessments corresponding to each query. The CLEF-IP test collection: target data: 2 million EP patents
  • What are relevance assessments A test collection (also known as gold standard) consists of a target dataset, a set of queries, and relevance assessments corresponding to each query. The CLEF-IP test collection: target data: 2 million EP patents queries: full-text patents (without images)
  • What are relevance assessments A test collection (also known as gold standard) consists of a target dataset, a set of queries, and relevance assessments corresponding to each query. The CLEF-IP test collection: target data: 2 million EP patents queries: full-text patents (without images) relevance assessments: extended citations
  • Relevance assessments We used patents cited as prior art as relevance assessments.
  • Relevance assessments We used patents cited as prior art as relevance assessments. Sources of citations:
  • Relevance assessments We used patents cited as prior art as relevance assessments. Sources of citations: 1 applicant’s disclosure: the Uspto requires applicants to disclose all known relevant publications
  • Relevance assessments We used patents cited as prior art as relevance assessments. Sources of citations: 1 applicant’s disclosure: the Uspto requires applicants to disclose all known relevant publications 2 patent office search report: each patent office will do a search for prior art to judge the novelty of a patent
  • Relevance assessments We used patents cited as prior art as relevance assessments. Sources of citations: 1 applicant’s disclosure: the Uspto requires applicants to disclose all known relevant publications 2 patent office search report: each patent office will do a search for prior art to judge the novelty of a patent 3 opposition procedures: patents cited to prove that a granted patent is not novel
  • Extended citations as relevance assessments direct citations and their families
  • Extended citations as relevance assessments direct citations of family members ...
  • Extended citations as relevance assessments ... and their families
  • Patent families A patent family consists of patents granted by different patent authorities but related to the same invention.
  • Patent families A patent family consists of patents granted by different patent authorities but related to the same invention. simple family all family members share the same priority number
  • Patent families A patent family consists of patents granted by different patent authorities but related to the same invention. simple family all family members share the same priority number extended family there are several definitions, in the INPADOC database all documents which are directly or indirectly linked via a priority number belong to the same family
  • Patent families Patent documents are linked by priorities
  • Patent families Patent documents are linked by INPADOC family. priorities
  • Patent families Patent documents are linked by Clef–Ip uses simple families. priorities
  • Relevance assessments 2010 Expanding the 2009 extended citations:
  • Relevance assessments 2010 Expanding the 2009 extended citations: 1 include citations of forward citations ...
  • Relevance assessments 2010 Expanding the 2009 extended citations: 1 include citations of forward citations ... 2 ... and their families
  • Relevance assessments 2010 Expanding the 2009 extended citations: 1 include citations of forward citations ... 2 ... and their families This is apparently a well-known method among patent searchers.
  • Relevance assessments 2010 Expanding the 2009 extended citations: 1 include citations of forward citations ... 2 ... and their families This is apparently a well-known method among patent searchers. Zig-zag search?
  • How good are the CLEF-IP relevance assessments? CLEF-IP uses families + citations:
  • How good are the CLEF-IP relevance assessments? how complete are extended citations as a relevance assessments?
  • How good are the CLEF-IP relevance assessments? how complete are extended citations as a relevance assessments? will every prior art patent be included in this set?
  • How good are the CLEF-IP relevance assessments? how complete are extended citations as a relevance assessments? will every prior art patent be included in this set? and if not, what percentage of prior art items are captured by extended citations?
  • How good are the CLEF-IP relevance assessments? how complete are extended citations as a relevance assessments? will every prior art patent be included in this set? and if not, what percentage of prior art items are captured by extended citations? when considering forward citations, how good are extended citations as a prior art candidate set?
  • Feedback from patent experts needed Quality of prior art candidate sets has to be assessed
  • Feedback from patent experts needed Know-how of patent search experts is needed
  • Feedback from patent experts needed at Clef–Ip 2009 7 patent search professionals assessed 12 search results
  • Feedback from patent experts needed at Clef–Ip 2009 7 patent search professionals assessed 12 search results the task was not well defined and there were misunderstandings on the concept of relevance
  • Feedback from patent experts needed at Clef–Ip 2009 7 patent search professionals assessed 12 search results the task was not well defined and there were misunderstandings on the concept of relevance amount of data was not sufficient to draw conclusions
  • Feedback from patent experts needed
  • Some initiatives associated with Clef–Ip The results of evaluation tracks are mostly useful for the research community.
  • Some initiatives associated with Clef–Ip The results of evaluation tracks are mostly useful for the research community. This community often produces prototypes that are of little interest to the end-user.
  • Some initiatives associated with Clef–Ip The results of evaluation tracks are mostly useful for the research community. This community often produces prototypes that are of little interest to the end-user. Next I’d like to present two concrete outcomes - not of Clef–Ip directly but arising from work in patent retrieval evaluation
  • Soire
  • Soire developed at Matrixware
  • Soire developed at Matrixware service-oriented architecture - available as a a Web service
  • Soire developed at Matrixware service-oriented architecture - available as a a Web service allows to replicate IR experiments based on classical evaluation model
  • Soire developed at Matrixware service-oriented architecture - available as a a Web service allows to replicate IR experiments based on classical evaluation model tested on the CLEF-IP data
  • Soire developed at Matrixware service-oriented architecture - available as a a Web service allows to replicate IR experiments based on classical evaluation model tested on the CLEF-IP data customized for the evaluation of machine translation
  • Spinque
  • Spinque a spin-off (2010) from CWI (the Dutch National Research Center in Computer Science and Mathematics)
  • Spinque a spin-off (2010) from CWI (the Dutch National Research Center in Computer Science and Mathematics) introduces search-by-strategy
  • Spinque a spin-off (2010) from CWI (the Dutch National Research Center in Computer Science and Mathematics) introduces search-by-strategy provides optimized strategies for patent search - tested on CLEF-IP data
  • Spinque a spin-off (2010) from CWI (the Dutch National Research Center in Computer Science and Mathematics) introduces search-by-strategy provides optimized strategies for patent search - tested on CLEF-IP data transparency: understand your search results to improve strategy
  • Clef–Ip 2009 learnings The Humboldt University implemented a model for patent search that produced the best results.
  • Clef–Ip 2009 learnings The Humboldt University implemented a model for patent search that produced the best results. The model combined several strategies:
  • Clef–Ip 2009 learnings The Humboldt University implemented a model for patent search that produced the best results. The model combined several strategies: using metadata (IPC, ECLA)
  • Clef–Ip 2009 learnings The Humboldt University implemented a model for patent search that produced the best results. The model combined several strategies: using metadata (IPC, ECLA) indexes built at lemma level
  • Clef–Ip 2009 learnings The Humboldt University implemented a model for patent search that produced the best results. The model combined several strategies: using metadata (IPC, ECLA) indexes built at lemma level an additional phrase index for English
  • Clef–Ip 2009 learnings The Humboldt University implemented a model for patent search that produced the best results. The model combined several strategies: using metadata (IPC, ECLA) indexes built at lemma level an additional phrase index for English crosslingual concept index (multilingual terminological database)
  • Some additional investigations Some citations were hard to find
  • Some additional investigations % runs class ≤5 hard 5 < x ≤ 10 very difficult Some citations were hard to find 10 < x ≤ 50 difficult 50 < x ≤ 75 medium 75 < x ≤ 100 easy
  • Some additional investigations We looked at the content of citations and citing patents.
  • Some additional investigations Ongoing investigations.
  • Thank you for your attention.