Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TREC 2016: Looking Forward Panel

Opening statement at the "Looking forward" panel at the 25 years of TREC celebration event, Nov 15th, 2016.

Webcast to appear within a week:

  • Login to see the comments

  • Be the first to like this

TREC 2016: Looking Forward Panel

  1. 1. Looking Forward Arjen P. de Vries Gaithersburg MD, November 15th, 2016
  2. 2. Q: “TREC Anniversary”
  3. 3. Top Result:  50 years of Star Trek (Article on the Verge about Facebook Like buttons)
  4. 4. Science Fiction  Defining a TREC task or a track is like time-travel in Back to the Future Note to the audience: that is just 74 characters You could even add the hashtag #TREC #TRECCelebrations and my Twitter handle @arjenpdevries
  5. 5. Better Search – “Deep Personalization”  “Even more broadly than trying to get people the right content based on their context, we as a community need to be thinking about how to support people through the entire search experience.” Jaime Teevan on “Slow Search”  Search as a dialogue My first journal paper: De Vries, Van der Veer and Blanken: Let’s talk about it: dialogues with multimedia databases (1998)
  6. 6. Moving Forward  Elements of the “Slow Search movement” at TREC today: - Sessions - Tasks - Dynamic domains - Total recall - Complex Answer Retrieval (new!)
  7. 7. Missing from TREC!  Access to rich personal data including email, browsing history, documents read and contents of the user’s home directory…
  8. 8. Trade log data! IR-809: (2011) Feild, H., Allan, J. and Glatt, J., "CrowdLogging: Distributed, private, and anonymous search logging," Proceedings of the International Conference on Research and Development in Information Retrieval (SIGIR'11), pp. 375-384. [View bibtex] We describe an approach for distributed search log collection, storage, and mining, with the dual goals of preserving privacy and making the mined information broadly available. [..] The approach works with any search behavior artifact that can be extracted from a search log, including queries, query reformulations, and query- click pairs.
  9. 9. Open challenges  How to select the part of your log data you are willing to trade?  How to estimate the value of this log data?  And a social challenge, not so much scientific: How to get people to participate?
  10. 10. Branding
  11. 11. Branding (NL)
  12. 12. The TREC Brand  A community that creates reusable test collections
  13. 13. Extra Slides
  14. 14. Reproducibility vs. Representativeness  Increasing representativeness of a TREC task should not come at the cost of sacrificing reproducibility (104 characters ) Samar, T., Bellogín, A. & de Vries, A.P. Inf Retrieval J (2016) 19: 230. doi: 10.1007/s10791-015-9276-9
  15. 15. Baltimore
  16. 16. Baltimore  Title query of TREC topic 478 for the information need “Who is the mayor of Baltimore”  “The honest conclusion of this year’s evaluation should be that we underestimated the problem of handling Web data. Surprising is the performance of the title-only queries doing better than queries including description or even narrative. It seems that the web-track topics are really different from the previous TREC topics in the ad- hoc task, for which we never weighted title terms different from description or narrative.” (Quote from the CWI TREC-9 paper)