Wolverhampton Talk
December 2013

Contextual IR:
Participations in TREC Contextual
Suggestion Tracks 2012 and 2013
Gilles ...
Summary
1.

Information Retrieval

2.

Contextual Information Retrieval

3.

TREC

4.

TREC Contextual Track 2012
1. Our a...
Information Retrieval
Usual IR process
Document
Document
Document

Query

Indexing

Query
representation

Indexing

Matchi...
Contextual Information Retrieval
Notion of context in IR
Information

User

Software

Device

How to consider context in I...
Contextual IR
Context integration in Q1

Document
Document
Document

Query

Context
Indexing

Query
representation

Indexi...
TREC
Text Retrieval Conference
Organized by the NIST (USA) since 1992
Based on the Cranfield paradigm of retrieval system ...
TREC
Campaign principles

(Voorhees, 2007)

7
TREC Contextual Suggestion Track 2012

Where to go around here
on this Sunday afternoon?

Great summer !!!

8
TREC Contextual Suggestion Track 2012
Retrieve items corresponding to the context (Q1)
Items = Suggestions
Places to visit...
TREC Contextual Suggestion Track 2012
Two subtasks
S1 : Suggestions corresponding to spatiotemporal data
List of suggestio...
TREC Contextual Suggestion Track 2012: Our approach
Contextual IRS framework 2012

Input data

Internal process

Output da...
TREC Contextual Suggestion Track 2012: Our approach
Spatiotemporal data

User preferences
Coarse-grained approach : iritSp...
TREC Contextual Suggestion Track 2012: Results
Evaluations
For each profile and each context
Different dimensions : W (Web...
TREC Contextual Suggestion Track 2012: Results
P@5

14
TREC Contextual Suggestion Track 2012: Results
MRR

15
TREC Contextual Suggestion Track 2013

Where to go around here?

16
TREC Contextual Suggestion Track 2013
Context =
Spatial only
{
"1": {
"lat": "40.71427", "city": "New York City", "state":...
TREC Contextual Suggestion Track 2013
Two subtasks
Open Web
Same question: Suggest places items corresponding to the conte...
TREC Contextual Suggestion Track 2013: Our approach
Contextual IRS framework 2013
Input data

Intermediate data

L: Lucene...
Example of suggestion in 2013
Title: Celtic Mist Pub
Description:
Place types: bar, establishment.
This place is about .3 ...
Example of suggestion in 2012
Title: Oakley Pub and Grill
Description
Oakley Pub and Grill - Located in Oakley Square, Cin...
Final Results
Open Web

22
Final Results
ClueWeb

23
Result Analysis
First edition (2012)
All the participants discovered the track principles
Worst results: Descriptions of s...
Wolverhampton Talk
December 2013

Thank you for your attention
Questions?
Upcoming SlideShare
Loading in …5
×

Contextual IR: Participations in TREC Contextual Suggestion Tracks 2012 and 2013

1,118 views

Published on

Lecture at the Scientometrics Festival 2013, University of Wolverhampton, UK

Published in: Education, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,118
On SlideShare
0
From Embeds
0
Number of Embeds
40
Actions
Shares
0
Downloads
18
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Contextual IR: Participations in TREC Contextual Suggestion Tracks 2012 and 2013

  1. 1. Wolverhampton Talk December 2013 Contextual IR: Participations in TREC Contextual Suggestion Tracks 2012 and 2013 Gilles Hubert http://bit.ly/wlvHubert2013
  2. 2. Summary 1. Information Retrieval 2. Contextual Information Retrieval 3. TREC 4. TREC Contextual Track 2012 1. Our approach 2. Results 5. TREC Contextual Track 2013 1. Our approach 2. Results 6. Result analysis and Future work 2
  3. 3. Information Retrieval Usual IR process Document Document Document Query Indexing Query representation Indexing Matching Document Document representation Document representation representation List of estimated relevant documents 3
  4. 4. Contextual Information Retrieval Notion of context in IR Information User Software Device How to consider context in IR process ? Q1 : Retrieve items corresponding to the context Q2 : Retrieve the context corresponding to items 4
  5. 5. Contextual IR Context integration in Q1 Document Document Document Query Context Indexing Query representation Indexing Matching Document Document representation Document representation representation List of estimated relevant documents Reranking List of estimated relevant documents 5
  6. 6. TREC Text Retrieval Conference Organized by the NIST (USA) since 1992 Based on the Cranfield paradigm of retrieval system evaluation A set of documents (Collection) A set of information needs (Topics/Queries) A set of relevance judgments (Qrels) Various tracks: AdHoc, Robust, Web… Evaluation measures System output: retrieved documents A relevant, retrieved (True positive) C Information need: relevant documents relevant, not retrieved (False negative) irrelevant, retrieved (False positive) B irrelevant, not retrieved (True negative) Document collection D precision = A A+B A recall = A+C AP (Average Precision), MAP (Mean Average Precision), P@5 (Precision at 5 retrieved documents) … 6
  7. 7. TREC Campaign principles (Voorhees, 2007) 7
  8. 8. TREC Contextual Suggestion Track 2012 Where to go around here on this Sunday afternoon? Great summer !!! 8
  9. 9. TREC Contextual Suggestion Track 2012 Retrieve items corresponding to the context (Q1) Items = Suggestions Places to visit (shops, restaurants, parks…) around the user (5 hours by car max.) Collection = Open Web (Websites) Context = Spatiotemporal data <context number=”1”> <city>Portland</city> <state>Oregon</state> <lat>45.5</lat> <long>-122.7</long> <day>weekday</day> <time>evening</time> <season>fall</season> </context> User preferences <profile number=”1”> <example number=”1” initial=”1” final=”1”/> <example number=”2” initial=”0” final=”-1”/> </profile > <example number=”1”> <title> Dogfish Head Alehouse </title> <description>Craft Brewed Ales and tasty wood grilled food </description> <url>http://www.dogfishalehouse.com/</url> </example> <example number=”2”> <title>The Flaming Pit</title> <description> The Flaming Pit Restaurant and Piano Lounge, home of Tyrone DeMonke. </description> <url>http://www.flamingpitrestaurant.com/</url> </example> 9
  10. 10. TREC Contextual Suggestion Track 2012 Two subtasks S1 : Suggestions corresponding to spatiotemporal data List of suggestions for each context S2 : S1 + user preferences List of suggestions for each profile (user) and each context Suggestion = Title + Description + Url <context2012 groupid=”waterloo” runid=”watcs12a”> <suggestion profile=”1” context=”1” rank=”1”> <title>Deschutes Brewery Portland Public House</title> <description> Deschutes Brewery’s distinct Northwest brew pub in Portland’s Pearl District has become a convivial gathering spot of beer and food lovers since it’s 2008 opening. </description> <url>http://www.deschutesbrewery.com</url> </suggestion> etc. </context2012> 2 “runs” maximum Our participation Team : G. Cabanac & G. Hubert (IRIT – Univ. of Toulouse) 2 runs submitted to S2 subtask 10
  11. 11. TREC Contextual Suggestion Track 2012: Our approach Contextual IRS framework 2012 Input data Internal process Output data Place sets Intermediate data Database External resource Context processing Contexti Preference processing Profilei Place query Place selection Preference definition Contextual list of places Google Places API Positive preferencesi Personalization Negative Contextual list of detailed places Place description enrichment preferencesi Examples Personalized suggestions Bing Google Useri 11
  12. 12. TREC Contextual Suggestion Track 2012: Our approach Spatiotemporal data User preferences Coarse-grained approach : iritSplit3CPv1 Merging of descriptions of examples with initial and final = 1 -> Pref+(P) Merging of descriptions of examples with initial and final = -1 -> Pref-(P) score(P,r) = cosine(Pref+(P),R) − cosine(Pref−(P),R) Fine-grained approach : iritSplit3CPv2 Example description with initial and final = 1 -> Pref+l(P) Example description with initial and final = -1 -> Pref-m(P) score(P, r) = max(cosine(Pref+l (P), r))− max(cosine(Pref−m(P), r)) 12
  13. 13. TREC Contextual Suggestion Track 2012: Results Evaluations For each profile and each context Different dimensions : W (Website), G (Geographical), T (Temporal), and D (Description), and combinations (WGT and GT) Two measures : P@5 and MRR (Mean Reciprocal Rank) iritSplit3CPv1 iritSplit3CPv2 13
  14. 14. TREC Contextual Suggestion Track 2012: Results P@5 14
  15. 15. TREC Contextual Suggestion Track 2012: Results MRR 15
  16. 16. TREC Contextual Suggestion Track 2013 Where to go around here? 16
  17. 17. TREC Contextual Suggestion Track 2013 Context = Spatial only { "1": { "lat": "40.71427", "city": "New York City", "state": "NY", "long": "-74.00597” }, … } { "1": { "url": http://www.freshrestaurants.ca, "description": "Our vegan menu boasts an array of exotic starters, multi-layered salads, filling wraps, high protein burgers and our signature Fresh bowls.”, "title": "Fresh on Bloor” }, “2": { "url": http://www.flamingpitrestaurant.com/, "description": "The Flaming Pit Restaurant and Piano Lounge, home of Tyrone DeMonke.”, "title": "The Flaming Pit” }, … User preferences { "1": [ {"attraction_id": 1, "website": 1, "description": 0}, ... ], "2": [ {"attraction_id": 1, "website": 4, "description": 3}, … ], ”3": [ {"attraction_id": 1, "website": -1, "description": 2}, … ], … } } 17
  18. 18. TREC Contextual Suggestion Track 2013 Two subtasks Open Web Same question: Suggest places items corresponding to the context (Q1) Places to visit (restaurants, museums…) around the user (5 hours by car) Collection = Open Web (Websites) ClueWeb ClueWeb12 (Same question as OpenWeb) ClueWeb12 Contextual suggestion subcollection Sets of ClueWeb12 documents per context Question: Personalization per user profile 2 “runs” maximum Our participation Team: G. Cabanac, G. Hubert & K. Pinel-Sauvagnat (IRIT – Univ. of Toulouse) C. Sallaberry (LIUPPA – Univ. of Pau) D. Palacio (GeoComp – Univ. of Zurich) 1 “run” Open Web 1 “run” ClueWeb (Contextual suggestion subcollection) 18
  19. 19. TREC Contextual Suggestion Track 2013: Our approach Contextual IRS framework 2013 Input data Intermediate data L: Lucene W: WordNet GP: Google Places T: Terrier Useri GN: Geonames Output data Y: Yahoo! BOSS Geo Process P: PostGis GG: Gisgraphy Useri Examples Examples Profilei Contexti B: Bing 1 Profilei Posi ve preferencesi L, T, W Preference processing 1 Contexti Nega ve preferencesi Posi ve preferencesi L, T, W Preference processing Nega ve preferencesi Categories of interesti Categories of interesti Predefined categories 2 2 GP Context processing 3 3 GN, Y, P, GG, B Contextual list of places Place filtering & descrip on enrichment 4 T Retrieval B Place filtering & descrip on enrichment list of places 4 Ranking & refinement Ranking Personalized sugges ons Personalized sugges ons a) Open Web b) ClueWeb 19
  20. 20. Example of suggestion in 2013 Title: Celtic Mist Pub Description: Place types: bar, establishment. This place is about .3 Km West from here (2 min by car with no traffic). Address: 117 South 7th Street, Springfield. There are 11 POIs around: 2 Hotels, 3 Libraries, 3 Parks, 1 PostOffice, 2 Religious. Snippet: Located in Springfield, IL the Celtic Mist is your home away from home with over 16 imported beers on tap and a friendly staff ready to serve you… URL: http://www.celticmistpub.com/ 20
  21. 21. Example of suggestion in 2012 Title: Oakley Pub and Grill Description Oakley Pub and Grill - Located in Oakley Square, Cincinnati, Ohio. Local pub with pleasant atmoshpere and great food. Voted #1 Best Burger in Cincinnati. Outdoor ... PUB and GRILL OAKLEYOAKLEY Oakley Pub and Grill ~ 3924 Isabella Avenue ~ Cincinnati, Ohio 45209 On Oakley Square ~ (513) 531-2500 www.oakleypub.com Used with permission… URL: http://oakleypubandgrill.com/ 21
  22. 22. Final Results Open Web 22
  23. 23. Final Results ClueWeb 23
  24. 24. Result Analysis First edition (2012) All the participants discovered the track principles Worst results: Descriptions of suggestions Second edition (2013) OpenWeb Focus on suggestion descriptions Changes in relevance judgments ClueWeb Misunderstanding of guidelines or insufficient details Next edition: TREC Contextual Suggestion Track 2014 Close to TREC Contextual Suggestion Track 2013 Future work Experiment framework variants on 2013 data Replace limited online tools/services Process larger collection: ClueWeb12 (870 millions pages, ~27TB) 24
  25. 25. Wolverhampton Talk December 2013 Thank you for your attention Questions?

×