The CLEF Initiative From 2010 to 2012 and Onwards
Upcoming SlideShare
Loading in...5
×
 

The CLEF Initiative From 2010 to 2012 and Onwards

on

  • 711 views

 

Statistics

Views

Total Views
711
Views on SlideShare
711
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

The CLEF Initiative From 2010 to 2012 and Onwards The CLEF Initiative From 2010 to 2012 and Onwards Presentation Transcript

  • The CLEF Initiative I N I T I AT I V E Conference and Labs of the Evaluation Forum http:/ /www.clef-initiative.eu/ The CLEF Initiative From 2010 to 2012 and Onwards Nicola Ferro Information Management Systems (IMS) Research Group Department of Information Engineering University of Padua, ItalyMediaEval Benchmarking Initiative for Multimedia EvaluationMediaEval 2012 Workshop, 5 October 2012, Santa Croce in Fossabanda, Pisa, Italy
  • I N I T I AT I V E Outline The CLEF Initiative From CLEF 2010 to CLEF 2012 CLEF 2013 and BeyondThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 2MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • The CLEF Initiative I N I T I AT I V E
  • I N I T I AT I V E The CLEF Initiative: Mission The CLEF Initiative is a self-organized body whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure. multilingual and multimodal system testing, tuning and evaluation; investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access; creation of reusable test collections for benchmarking; exploration of new evaluation methodologies and innovative ways of using experimental data; discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge.The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 4MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E The CLEF Initiative: Structure The CLEF Initiative is structured in two main parts: a series of Evaluation Labs, i.e. laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities; a peer-reviewed Conference on a broad range of issues, including investigation continuing the activities of the Evaluation Labs; experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities; research in evaluation methodologies and challenges.The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 5MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E The CLEF Initiative promisehttp://www.clef-initiative.eu/The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 6MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E The CLEF Initiative: Steering Committee Steering Committee Chair Nicola Ferro, University of Padua, Italy Deputy Steering Committee Chair for the Conference Julio Gonzalo, National Distance Education University (UNED), Spain Deputy Steering Committee Chair for the Evaluation Labs Carol Peters, ISTI, National Council of Research (CNR), Italy Members Martin Braschler, Zurich University of Applied Sciences, Switzerland Khalid Choukri, Evaluations and Language resources Distribution Agency (ELDA), France Paul Clough, University of Sheffield, United Kingdom Donna Harman, National Institute for Standards and Technology (NIST), USA Djoerd Hiemstra, University of Twente, The Netherlands Jaana Kekäläinen, University of Tampere, Finland Emanuele Pianta, Centre for the Evaluation of Language and Communication Technologies (CELCT), Italy Maarten de Rijke, University of Amsterdam UvA, The Netherlands Giuseppe Santucci, Sapienza University of Rome, Italy Alan Smeaton, Dublin City University, Ireland Christa Womser-Hacker, University of Hildesheim, GermanyThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 7MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF: the NameThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 8MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF: the NameThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 8MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • CLEF 2010
  • I N I T I AT I V E Approach Scientific and Technological Advancement in Multilingual and Multimedia Information Systems Workshops Workshops Workshops Workshops Workshops Workshops Labs and Labs and Labs and Labs and Labs and Labs and ConferenceThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 10MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2010: 20-23 September 2010 in Padua Ov er 140 es and attende onsors exte rnal sp http://www.clef2010.org/The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 11MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2010 Organization Honorary Chair Carol Peters, ISTI-CNR, Italy General Chairs Maristella Agosti, University of Padua, Italy Maarten de Rijke, University of Amsterdam, The Netherlands Program Chairs Nicola Ferro, University of Padua, Italy Alan Smeaton, Dublin City University, Ireland Lab Chairs Martin Braschler, Zurich University of Applied Sciences, Switzerland Donna Harman, NIST, USAThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 12MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2010 Conference 12 papers (8 full papers and 4 short papers) out of 21 submissions (17 full papers and 4 short papers) Two keynote talks Norbert Fuhr, “IR Between Science and Engineering, and the Role of Experimentation” Ricardo Baeza-Yates, “Retrieval Evaluation in Practice” Other Evaluation Initiatives Ellen Voorhees for TREC; Noriko Kando for NTCIR; Prasenjit Majumder for FIRE; Jaap Kamps for INEX; Pavel Braslavski for ROMIP Two panels Donna Harman, Noriko Kando, Mounia Lalmas, Carol Peters, “The Four Ladies of Experimental Evaluation” “A PROMISE for Experimental Evaluation” by the partners of the projectThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 13MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2010 Labs and Workshops 5 Benchmarking activities CLEF-IP: a benchmarking activity on intellectual property ImageCLEF: a benchmarking activity on image retrieval PAN: a benchmarking activity on plagiarism detection RespubliQA: A benchmarking activity on question answering using multilingual political data WePS: A benchmarking activity on web people search 2 Workshops CriES: a workshop aimed at exploring the evaluation of searching for expertise in social media LogCLEF: a workshop aimed at exploring methodologies for studying search engine log filesThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 14MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • CLEF 2011
  • I N I T I AT I V E CLEF 2011 Approach: Interleaved Sessions 16The CLEF Initiative: From 2010 to 2012 and Onwards Nicola FerroMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2011: 19-22 September 2011 in Amsterdam Over 170 tende es and at nsors e xtern al spohttp://www.clef2011.org/ http://www.clef2011.org/The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 17MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2011 Organization General chairs Julio Gonzalo, National Distance Education University (UNED), Spain Maarten de Rijke, University of Amsterdam, The Netherlands Program chairs Jaana Kekäläinen, University of Tampere, Finland Mounia Lalmas, Yahoo! Research Barcelona, Spain Lab chairs Paul Clough, University of Sheffield, United Kingdom Vivien Petras, Humboldt University, GermanyThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 18MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2011 Conference 14 papers (10 full papers and 4 short papers) out of 23 submissions (19 full papers and 4 short papers) Two keynote talks Elaine Toms , “Would you trust your IR system to choose your date? Re-thinking IR Evaluation in the 21st Century” Omar Alonso, “Crowdsourcing for Information Retrieval Experimentation and Evaluation” Community Sessions Other Evaluation Initiatives: Ellen Voorhees for TREC; Hideo Joho for NTCIR; Jaap Kamps for INEX; Gareth Jones for MediaEval CHORUS+ Session: Information Retrieval from Scientific Multimedia Data Infrastructure Session: the Promise evaluation infrastructure, where Visual Analytics meets Information Retrieval Funding Opportunity Session: Intelligent Information Management - FP7 ICT Workprogramme 2011 - 2012, Strategic Objective 4.4. - General information and conditions of Call 8The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 19MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2011 Labs and Workshops 5 Benchmarking activities CLEF-IP: a benchmarking activity on intellectual property ImageCLEF: a benchmarking activity on image retrieval LogCLEF: a benchmarking activity on Multilingual Log File Analysis: Language identification, query classification, success of a query PAN: a benchmarking activity on plagiarism detection QA4MRE: a benchmarking activity on the evaluation of Machine Reading systems through Question Answering and Reading Comprehension Test 2 Workshops/Pilots CHiC: a workshop aimed at moving towards a systematic and large-scale evaluation of cultural heritage digital libraries and information access systems and helping to shape a possible roadmap for it MusicCLEF: a pilot benchmarking activity on the evaluation of music search engines that are based both on audio content and on multilingual textual descriptionsThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 20MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • CLEF 2012
  • I N I T I AT I V E CLEF 2012: 17-20 September 2012, Rome Ov er 190 es and attende onsors exte rnal sp http://www.clef2012.org/The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 22MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2012 Organization General chairs Tiziana Catarci, Sapienza University of Rome, Italy Djoerd Hiemstra, University of Twente, The Netherlands Program chairs Anselmo Peñas, National Distance Learning University, Spain Giuseppe Santucci, Sapienza University of Rome, Italy Lab chairs Jussi Karlgren, Swedish Institute of Computer Science, Sweden Christa Womser-Hacker, University of Hildesheim, GermanyThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 23MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2012 Conference 17 papers (9 full papers, 5 short papers, and 3 posters) out of 33 submissions Two keynote talks Peter Clark, “From Information Retrieval to Knowledgeable Machines” Tobias Schreck, “Visual Search and Analysis in Textual and Non-Textual Document Repositories - Approaches, Applications, and Research Challenges” Community Sessions Other Evaluation Initiatives: Donna Harman for TREC and TAC; Noriko Kando for NTCIR; Pavel Braslavski for ROMIP; Prasenjit Majumder for FIRE; Gareth Jones for MediaEval (remote) Evaluation Clinic: participants had the possibility of meeting evaluation experts and discussing with them their own evaluation problemsThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 24MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF 2012 Labs and Workshops 7 Benchmarking activities CHiC: a last-year workshop turning into a benchmarking activity for the cultural heritage domain based on Europeana collections CLEF-IP: a benchmarking activity on intellectual property ImageCLEF: a benchmarking activity on image retrieval INEX: the well-known Initiative for Evaluation of XML retrieval joins efforts with CLEF to target new synergies between multilingual, multimodal and semi-structured information access PAN: a benchmarking activity on plagiarism detection QA4MRE: a benchmarking activity on the evaluation of Machine Reading systems through Question Answering and Reading Comprehension Test RepLab: a benchmarking activity on microblog data for online reputation management 1 Workshop eHealth: a workshop on new evaluation issues in the health domain, related to the Louhi series of workshops on NLP in Health InformaticsThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 25MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • CLEF 2013 and Beyond
  • I N I T I AT I V E Next CLEFs CLEF 2013 23 - 26 September 2013 Technical University of Valencia, Spain More information soon on the CLEF 2013 Web site http://www.clef2013.org/ CLEF 2014 15 - 19 September 2014 University of Sheffield, United Kingdom CLEF 2015 Bids are open Send a proposal at chair@clef-initiative.eu by 5 April 2013 Template available at: http://www.clef-initiative.eu/documents/71612/87713/ CLEF-Initiative-Template_for_bids.docxThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 27MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E CLEF Publication We are now missing a single publication that acts as an hub for citation of all the CLEF activities (conference and labs) that happen during the year Therefore, the Springer proceedings delivered at the conference will contain the conference papers selected by the review process carried out by the program committee the lab overview papers, after a review process carried out by the lab chairs To meet the deadlines to have this publication ready by September, we need to reschedule the overall CLEF evaluation campaign periodThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 28MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E Conference - CLEF 2013 Call for Papers coming out soon, late November 2012. Don’t miss it!!! Paper submission deadline around late April 2013 More emphasis on evaluation Focused range of hot topicsThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 29MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E Lab Coordination Deadlines - CLEF 2013 1 October 2012: Call for Lab proposals 15 October 2012: Deadline for presentation of proposals 29 October 2012: Notification of acceptance/rejection 5 November 2012: Release of first lab flyer and call for participation 1 December 2012: Labs registration opens 15 December 2012: Labs begin to release data 15 December 2012 - 1 May 2013: Evaluation Campaign 1 May 2013: Result submission 1 June 2013: Results release 15 June 2013: Working notes papers submitted 30 June 2013: Lab overview papers submitted in Springer format 30 June 2013 - 7 July 2013: Lab Chairs review lab overviews 15 July 2013: Springer proceedings doneThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 30MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E ELIAS Lab Organizers Meeting at ECIR 2013 ELIAS can support a meeting at ECIR 2013 (24-27 March 2013) in Moscow between Lab Organizers to share ideas, plans, and strategies to coordinate http://www.elias-network.eu/The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 31MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E Lab Coordination Deadlines - CLEF 2014 15 June 2013: Call for Lab Proposals 1 July 2013: Deadline for presentation of proposals 20 July 2013: Notification of acceptance/rejection CLEF 2014: Release of first lab flyer and call for participation 1 November 2013: Labs registration opens 15 November 2013: Labs begin to release data 15 November 2013 - 1 May 2014: Evaluation Campaign 1 May 2014: Result submission 1 June 2014: Results release 15 June 2014: Working notes papers submitted 30 June 2014: Lab overview papers submitted in Springer format 30 June 2014 - 7 July 2014: Lab Chairs review lab overviews 15 July 2014: Springer proceedings doneThe CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 32MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • I N I T I AT I V E PROMISE: a supporter of CLEF promise PROMISE Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation FP7 ICT 2009.4.3, Intelligent Information Management PROMISE Retreat Report Prospects and Opportunities for Information Access Evaluation Editors Nicola Ferro, University of Padua, Italy http://www.promise-noe.eu/ Richard Berendsen, University of Amsterdam, The Netherlands Allan Hanbury, Vienna University of Technology, Austria Mihai Lupu, Vienna University of Technology, Austria Vivien Petras, Humboldt University Berlin, Germany Maarten de Rijke, University of Amsterdam, The Netherlands Gianmaria Silvello, University of Padua, Italy Network of Excellence co-funded by the 7th Framework Program of the European Commission, grant agreement no. 258191The CLEF Initiative: From 2010 to 2012 and Onwards Nicola Ferro 33MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
  • Thank Youhttp://www.clef-initiative.eu/