SlideShare a Scribd company logo
Wolverhampton Talk
December 2013

Contextual IR:
Participations in TREC Contextual
Suggestion Tracks 2012 and 2013
Gilles Hubert
http://bit.ly/wlvHubert2013
Summary
1.

Information Retrieval

2.

Contextual Information Retrieval

3.

TREC

4.

TREC Contextual Track 2012
1. Our approach
2. Results

5.

TREC Contextual Track 2013
1. Our approach
2. Results

6.

Result analysis and Future work

2
Information Retrieval
Usual IR process
Document
Document
Document

Query

Indexing

Query
representation

Indexing

Matching

Document
Document
representation
Document
representation
representation

List of
estimated
relevant
documents

3
Contextual Information Retrieval
Notion of context in IR
Information

User

Software

Device

How to consider context in IR process ?
Q1 : Retrieve items corresponding to the context
Q2 : Retrieve the context corresponding to items
4
Contextual IR
Context integration in Q1

Document
Document
Document

Query

Context
Indexing

Query
representation

Indexing

Matching

Document
Document
representation
Document
representation
representation

List of estimated
relevant documents

Reranking

List of estimated
relevant documents
5
TREC
Text Retrieval Conference
Organized by the NIST (USA) since 1992
Based on the Cranfield paradigm of retrieval system evaluation
A set of documents (Collection)
A set of information needs (Topics/Queries)
A set of relevance judgments (Qrels)

Various tracks: AdHoc, Robust, Web…
Evaluation measures

System output:
retrieved documents

A
relevant, retrieved
(True positive)

C
Information need:
relevant documents

relevant, not retrieved
(False negative)

irrelevant, retrieved
(False positive)

B
irrelevant, not retrieved
(True negative)

Document collection

D

precision =

A
A+B

A
recall =
A+C
AP (Average Precision),
MAP (Mean Average Precision),
P@5 (Precision at 5 retrieved documents)
…
6
TREC
Campaign principles

(Voorhees, 2007)

7
TREC Contextual Suggestion Track 2012

Where to go around here
on this Sunday afternoon?

Great summer !!!

8
TREC Contextual Suggestion Track 2012
Retrieve items corresponding to the context (Q1)
Items = Suggestions
Places to visit (shops, restaurants, parks…) around the user (5 hours by
car max.)

Collection = Open Web (Websites)
Context =
Spatiotemporal data
<context number=”1”>
<city>Portland</city>
<state>Oregon</state>
<lat>45.5</lat>
<long>-122.7</long>
<day>weekday</day>
<time>evening</time>
<season>fall</season>
</context>

User preferences
<profile number=”1”>
<example number=”1” initial=”1” final=”1”/>
<example number=”2” initial=”0” final=”-1”/>
</profile >

<example number=”1”>
<title> Dogfish Head Alehouse </title>
<description>Craft Brewed Ales and tasty wood
grilled food
</description>
<url>http://www.dogfishalehouse.com/</url>
</example>
<example number=”2”>
<title>The Flaming Pit</title>
<description>
The Flaming Pit Restaurant and Piano Lounge,
home of Tyrone DeMonke.
</description>
<url>http://www.flamingpitrestaurant.com/</url>
</example>
9
TREC Contextual Suggestion Track 2012
Two subtasks
S1 : Suggestions corresponding to spatiotemporal data
List of suggestions for each context
S2 : S1 + user preferences
List of suggestions for each profile (user) and each context
Suggestion = Title + Description + Url
<context2012 groupid=”waterloo” runid=”watcs12a”>
<suggestion profile=”1” context=”1” rank=”1”>
<title>Deschutes Brewery Portland Public House</title>
<description>
Deschutes Brewery’s distinct Northwest brew pub in Portland’s Pearl District has
become a convivial gathering spot of beer and food lovers since it’s 2008 opening.
</description>
<url>http://www.deschutesbrewery.com</url>
</suggestion>
etc.
</context2012>

2 “runs” maximum
Our participation
Team : G. Cabanac & G. Hubert (IRIT – Univ. of Toulouse)
2 runs submitted to S2 subtask

10
TREC Contextual Suggestion Track 2012: Our approach
Contextual IRS framework 2012

Input data

Internal process

Output data

Place sets

Intermediate data
Database

External resource

Context processing

Contexti

Preference processing
Profilei

Place
query

Place selection

Preference
definition

Contextual
list of
places

Google Places
API

Positive
preferencesi

Personalization

Negative

Contextual
list of
detailed
places

Place
description
enrichment

preferencesi

Examples

Personalized
suggestions

Bing

Google

Useri

11
TREC Contextual Suggestion Track 2012: Our approach
Spatiotemporal data

User preferences
Coarse-grained approach : iritSplit3CPv1
Merging of descriptions of examples with initial and final = 1 -> Pref+(P)
Merging of descriptions of examples with initial and final = -1 -> Pref-(P)
score(P,r) = cosine(Pref+(P),R) − cosine(Pref−(P),R)
Fine-grained approach : iritSplit3CPv2
Example description with initial and final = 1 -> Pref+l(P)
Example description with initial and final = -1 -> Pref-m(P)
score(P, r) = max(cosine(Pref+l (P), r))− max(cosine(Pref−m(P), r))
12
TREC Contextual Suggestion Track 2012: Results
Evaluations
For each profile and each context
Different dimensions : W (Website), G (Geographical), T (Temporal),
and D (Description), and combinations (WGT and GT)
Two measures : P@5 and MRR (Mean Reciprocal Rank)
iritSplit3CPv1

iritSplit3CPv2

13
TREC Contextual Suggestion Track 2012: Results
P@5

14
TREC Contextual Suggestion Track 2012: Results
MRR

15
TREC Contextual Suggestion Track 2013

Where to go around here?

16
TREC Contextual Suggestion Track 2013
Context =
Spatial only
{
"1": {
"lat": "40.71427", "city": "New York City", "state": "NY", "long": "-74.00597”
},
…
}
{
"1": {
"url": http://www.freshrestaurants.ca,
"description": "Our vegan menu boasts an array of
exotic starters, multi-layered salads, filling wraps,
high protein burgers and our signature Fresh
bowls.”,
"title": "Fresh on Bloor”
},
“2": {
"url": http://www.flamingpitrestaurant.com/,
"description": "The Flaming Pit Restaurant and
Piano Lounge, home of Tyrone DeMonke.”,
"title": "The Flaming Pit”
},
…

User preferences
{
"1": [

{"attraction_id": 1, "website": 1, "description": 0},
...
],
"2": [
{"attraction_id": 1, "website": 4, "description": 3},
…
],
”3": [
{"attraction_id": 1, "website": -1, "description": 2},
…
],
…

}

}
17
TREC Contextual Suggestion Track 2013
Two subtasks
Open Web
Same question: Suggest places items corresponding to the context (Q1)
Places to visit (restaurants, museums…) around the user (5 hours by car)
Collection = Open Web (Websites)
ClueWeb
ClueWeb12 (Same question as OpenWeb)
ClueWeb12 Contextual suggestion subcollection
Sets of ClueWeb12 documents per context
Question: Personalization per user profile

2 “runs” maximum
Our participation
Team: G. Cabanac, G. Hubert & K. Pinel-Sauvagnat (IRIT – Univ. of Toulouse)
C. Sallaberry (LIUPPA – Univ. of Pau)
D. Palacio (GeoComp – Univ. of Zurich)
1 “run” Open Web
1 “run” ClueWeb (Contextual suggestion subcollection)
18
TREC Contextual Suggestion Track 2013: Our approach
Contextual IRS framework 2013
Input data

Intermediate data

L: Lucene W: WordNet

GP: Google Places

T: Terrier

Useri

GN: Geonames

Output data
Y: Yahoo! BOSS Geo

Process

P: PostGis GG: Gisgraphy

Useri
Examples

Examples

Profilei

Contexti

B: Bing

1

Profilei

Posi ve
preferencesi

L, T, W
Preference
processing

1
Contexti

Nega ve
preferencesi

Posi ve
preferencesi

L, T, W
Preference
processing

Nega ve
preferencesi
Categories
of interesti

Categories
of interesti
Predefined
categories

2

2

GP
Context
processing

3

3

GN, Y, P, GG, B
Contextual
list of places

Place filtering &
descrip on
enrichment
4

T
Retrieval

B
Place
filtering &
descrip on
enrichment

list of places

4

Ranking &
refinement

Ranking

Personalized
sugges ons

Personalized
sugges ons

a) Open Web

b) ClueWeb

19
Example of suggestion in 2013
Title: Celtic Mist Pub
Description:
Place types: bar, establishment.
This place is about .3 Km West from here (2 min by car with no
traffic).
Address: 117 South 7th Street, Springfield.
There are 11 POIs around: 2 Hotels, 3 Libraries, 3 Parks, 1
PostOffice, 2 Religious.
Snippet: Located in Springfield, IL the Celtic Mist is your home
away from home with over 16 imported beers on tap and a
friendly staff ready to serve you…
URL: http://www.celticmistpub.com/
20
Example of suggestion in 2012
Title: Oakley Pub and Grill
Description
Oakley Pub and Grill - Located in Oakley Square, Cincinnati, Ohio.
Local pub with pleasant atmoshpere and great food. Voted #1
Best Burger in Cincinnati. Outdoor ...
PUB and GRILL OAKLEYOAKLEY Oakley Pub and Grill ~ 3924
Isabella Avenue ~ Cincinnati, Ohio 45209 On Oakley Square ~
(513) 531-2500 www.oakleypub.com Used with permission…
URL: http://oakleypubandgrill.com/

21
Final Results
Open Web

22
Final Results
ClueWeb

23
Result Analysis
First edition (2012)
All the participants discovered the track principles
Worst results: Descriptions of suggestions

Second edition (2013)
OpenWeb
Focus on suggestion descriptions
Changes in relevance judgments
ClueWeb
Misunderstanding of guidelines or insufficient details

Next edition: TREC Contextual Suggestion Track 2014
Close to TREC Contextual Suggestion Track 2013

Future work
Experiment framework variants on 2013 data
Replace limited online tools/services
Process larger collection: ClueWeb12 (870 millions pages, ~27TB)

24
Wolverhampton Talk
December 2013

Thank you for your attention
Questions?

More Related Content

Similar to Contextual IR: Participations in TREC Contextual Suggestion Tracks 2012 and 2013

Representing the world: How web users become web thinkers and web makers
Representing the world: How web users become web thinkers and web makersRepresenting the world: How web users become web thinkers and web makers
Representing the world: How web users become web thinkers and web makers
judell
 
Tapping the Data Deluge with R
Tapping the Data Deluge with RTapping the Data Deluge with R
Tapping the Data Deluge with R
Jeffrey Breen
 
Linked data experiments at the National Library of Scotland / Alexandra De Pr...
Linked data experiments at the National Library of Scotland / Alexandra De Pr...Linked data experiments at the National Library of Scotland / Alexandra De Pr...
Linked data experiments at the National Library of Scotland / Alexandra De Pr...
CIGScotland
 
I ii x_slides_albakour_online
I ii x_slides_albakour_onlineI ii x_slides_albakour_online
I ii x_slides_albakour_online
Dyaa AlBakour
 
AnalogIST/ezPAARSE : Analysing Locally Gathered Logfiles to Determine Users’ ...
AnalogIST/ezPAARSE : Analysing Locally Gathered Logfiles to Determine Users’ ...AnalogIST/ezPAARSE : Analysing Locally Gathered Logfiles to Determine Users’ ...
AnalogIST/ezPAARSE : Analysing Locally Gathered Logfiles to Determine Users’ ...
My French Library - IFLA 2014
 
Lifting the Lid on Linked Data
Lifting the Lid on Linked DataLifting the Lid on Linked Data
Lifting the Lid on Linked Data
Jane Stevenson
 
Jumpstart: Introduction to Schema Design
Jumpstart: Introduction to Schema DesignJumpstart: Introduction to Schema Design
Jumpstart: Introduction to Schema Design
MongoDB
 
Preparing for the UK Research Data Registry and Discovery Service
Preparing for the UK Research Data Registry and Discovery ServicePreparing for the UK Research Data Registry and Discovery Service
Preparing for the UK Research Data Registry and Discovery Service
Repository Fringe
 
Linking Open Government Data at Scale
Linking Open Government Data at Scale Linking Open Government Data at Scale
Linking Open Government Data at Scale
Bernadette Hyland-Wood
 
Jarrar: SPARQL - RDF Query Language
Jarrar: SPARQL - RDF Query LanguageJarrar: SPARQL - RDF Query Language
Jarrar: SPARQL - RDF Query Language
Mustafa Jarrar
 
Navigating the Transition from relational to NoSQL - CloudCon Expo 2012
Navigating the Transition from relational to NoSQL - CloudCon Expo 2012Navigating the Transition from relational to NoSQL - CloudCon Expo 2012
Navigating the Transition from relational to NoSQL - CloudCon Expo 2012
exponential-inc
 
Restaurants new york toronto
Restaurants new york torontoRestaurants new york toronto
Restaurants new york toronto
marinaHunt1
 

Similar to Contextual IR: Participations in TREC Contextual Suggestion Tracks 2012 and 2013 (12)

Representing the world: How web users become web thinkers and web makers
Representing the world: How web users become web thinkers and web makersRepresenting the world: How web users become web thinkers and web makers
Representing the world: How web users become web thinkers and web makers
 
Tapping the Data Deluge with R
Tapping the Data Deluge with RTapping the Data Deluge with R
Tapping the Data Deluge with R
 
Linked data experiments at the National Library of Scotland / Alexandra De Pr...
Linked data experiments at the National Library of Scotland / Alexandra De Pr...Linked data experiments at the National Library of Scotland / Alexandra De Pr...
Linked data experiments at the National Library of Scotland / Alexandra De Pr...
 
I ii x_slides_albakour_online
I ii x_slides_albakour_onlineI ii x_slides_albakour_online
I ii x_slides_albakour_online
 
AnalogIST/ezPAARSE : Analysing Locally Gathered Logfiles to Determine Users’ ...
AnalogIST/ezPAARSE : Analysing Locally Gathered Logfiles to Determine Users’ ...AnalogIST/ezPAARSE : Analysing Locally Gathered Logfiles to Determine Users’ ...
AnalogIST/ezPAARSE : Analysing Locally Gathered Logfiles to Determine Users’ ...
 
Lifting the Lid on Linked Data
Lifting the Lid on Linked DataLifting the Lid on Linked Data
Lifting the Lid on Linked Data
 
Jumpstart: Introduction to Schema Design
Jumpstart: Introduction to Schema DesignJumpstart: Introduction to Schema Design
Jumpstart: Introduction to Schema Design
 
Preparing for the UK Research Data Registry and Discovery Service
Preparing for the UK Research Data Registry and Discovery ServicePreparing for the UK Research Data Registry and Discovery Service
Preparing for the UK Research Data Registry and Discovery Service
 
Linking Open Government Data at Scale
Linking Open Government Data at Scale Linking Open Government Data at Scale
Linking Open Government Data at Scale
 
Jarrar: SPARQL - RDF Query Language
Jarrar: SPARQL - RDF Query LanguageJarrar: SPARQL - RDF Query Language
Jarrar: SPARQL - RDF Query Language
 
Navigating the Transition from relational to NoSQL - CloudCon Expo 2012
Navigating the Transition from relational to NoSQL - CloudCon Expo 2012Navigating the Transition from relational to NoSQL - CloudCon Expo 2012
Navigating the Transition from relational to NoSQL - CloudCon Expo 2012
 
Restaurants new york toronto
Restaurants new york torontoRestaurants new york toronto
Restaurants new york toronto
 

Recently uploaded

How to Create Sequence Numbers in Odoo 17
How to Create Sequence Numbers in Odoo 17How to Create Sequence Numbers in Odoo 17
How to Create Sequence Numbers in Odoo 17
Celine George
 
matatag curriculum education for Kindergarten
matatag curriculum education for Kindergartenmatatag curriculum education for Kindergarten
matatag curriculum education for Kindergarten
SarahAlie1
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
heathfieldcps1
 
New Features in Odoo 17 Sign - Odoo 17 Slides
New Features in Odoo 17 Sign - Odoo 17 SlidesNew Features in Odoo 17 Sign - Odoo 17 Slides
New Features in Odoo 17 Sign - Odoo 17 Slides
Celine George
 
(T.L.E.) Agriculture: Essentials of Gardening
(T.L.E.) Agriculture: Essentials of Gardening(T.L.E.) Agriculture: Essentials of Gardening
(T.L.E.) Agriculture: Essentials of Gardening
MJDuyan
 
How to Add a Filter in the Odoo 17 - Odoo 17 Slides
How to Add a Filter in the Odoo 17 - Odoo 17 SlidesHow to Add a Filter in the Odoo 17 - Odoo 17 Slides
How to Add a Filter in the Odoo 17 - Odoo 17 Slides
Celine George
 
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
Celine George
 
How to Manage Access Rights & User Types in Odoo 17
How to Manage Access Rights & User Types in Odoo 17How to Manage Access Rights & User Types in Odoo 17
How to Manage Access Rights & User Types in Odoo 17
Celine George
 
Kesadaran_Berbangsa_dan_Bernegara_Nasion.pptx
Kesadaran_Berbangsa_dan_Bernegara_Nasion.pptxKesadaran_Berbangsa_dan_Bernegara_Nasion.pptx
Kesadaran_Berbangsa_dan_Bernegara_Nasion.pptx
artenzmartenkai
 
Webinar Innovative assessments for SOcial Emotional Skills
Webinar Innovative assessments for SOcial Emotional SkillsWebinar Innovative assessments for SOcial Emotional Skills
Webinar Innovative assessments for SOcial Emotional Skills
EduSkills OECD
 
How to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POSHow to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POS
Celine George
 
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdfThe Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
JackieSparrow3
 
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ..."DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
thanhluan21
 
Cómo crear video-tutoriales con ScreenPal (2 de julio de 2024)
Cómo crear video-tutoriales con ScreenPal (2 de julio de 2024)Cómo crear video-tutoriales con ScreenPal (2 de julio de 2024)
Cómo crear video-tutoriales con ScreenPal (2 de julio de 2024)
Cátedra Banco Santander
 
NC Public Schools Involved in NCDPI, Zipline Partnership
NC Public Schools Involved in NCDPI, Zipline PartnershipNC Public Schools Involved in NCDPI, Zipline Partnership
NC Public Schools Involved in NCDPI, Zipline Partnership
Mebane Rash
 
CTD Punjab Police Past Papers MCQs PPSC PDF
CTD Punjab Police Past Papers MCQs PPSC PDFCTD Punjab Police Past Papers MCQs PPSC PDF
CTD Punjab Police Past Papers MCQs PPSC PDF
hammadmughal76316
 
Edukasyong Pantahanan at Pangkabuhayan 1: Personal Hygiene
Edukasyong Pantahanan at  Pangkabuhayan 1: Personal HygieneEdukasyong Pantahanan at  Pangkabuhayan 1: Personal Hygiene
Edukasyong Pantahanan at Pangkabuhayan 1: Personal Hygiene
MJDuyan
 
ENGLISH-7-CURRICULUM MAP- MATATAG CURRICULUM
ENGLISH-7-CURRICULUM MAP- MATATAG CURRICULUMENGLISH-7-CURRICULUM MAP- MATATAG CURRICULUM
ENGLISH-7-CURRICULUM MAP- MATATAG CURRICULUM
HappieMontevirgenCas
 
How to Empty a One2Many Field in Odoo 17
How to Empty a One2Many Field in Odoo 17How to Empty a One2Many Field in Odoo 17
How to Empty a One2Many Field in Odoo 17
Celine George
 
Year-to-Date Filter in Odoo 17 Dashboard
Year-to-Date Filter in Odoo 17 DashboardYear-to-Date Filter in Odoo 17 Dashboard
Year-to-Date Filter in Odoo 17 Dashboard
Celine George
 

Recently uploaded (20)

How to Create Sequence Numbers in Odoo 17
How to Create Sequence Numbers in Odoo 17How to Create Sequence Numbers in Odoo 17
How to Create Sequence Numbers in Odoo 17
 
matatag curriculum education for Kindergarten
matatag curriculum education for Kindergartenmatatag curriculum education for Kindergarten
matatag curriculum education for Kindergarten
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
New Features in Odoo 17 Sign - Odoo 17 Slides
New Features in Odoo 17 Sign - Odoo 17 SlidesNew Features in Odoo 17 Sign - Odoo 17 Slides
New Features in Odoo 17 Sign - Odoo 17 Slides
 
(T.L.E.) Agriculture: Essentials of Gardening
(T.L.E.) Agriculture: Essentials of Gardening(T.L.E.) Agriculture: Essentials of Gardening
(T.L.E.) Agriculture: Essentials of Gardening
 
How to Add a Filter in the Odoo 17 - Odoo 17 Slides
How to Add a Filter in the Odoo 17 - Odoo 17 SlidesHow to Add a Filter in the Odoo 17 - Odoo 17 Slides
How to Add a Filter in the Odoo 17 - Odoo 17 Slides
 
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
 
How to Manage Access Rights & User Types in Odoo 17
How to Manage Access Rights & User Types in Odoo 17How to Manage Access Rights & User Types in Odoo 17
How to Manage Access Rights & User Types in Odoo 17
 
Kesadaran_Berbangsa_dan_Bernegara_Nasion.pptx
Kesadaran_Berbangsa_dan_Bernegara_Nasion.pptxKesadaran_Berbangsa_dan_Bernegara_Nasion.pptx
Kesadaran_Berbangsa_dan_Bernegara_Nasion.pptx
 
Webinar Innovative assessments for SOcial Emotional Skills
Webinar Innovative assessments for SOcial Emotional SkillsWebinar Innovative assessments for SOcial Emotional Skills
Webinar Innovative assessments for SOcial Emotional Skills
 
How to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POSHow to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POS
 
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdfThe Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
 
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ..."DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
 
Cómo crear video-tutoriales con ScreenPal (2 de julio de 2024)
Cómo crear video-tutoriales con ScreenPal (2 de julio de 2024)Cómo crear video-tutoriales con ScreenPal (2 de julio de 2024)
Cómo crear video-tutoriales con ScreenPal (2 de julio de 2024)
 
NC Public Schools Involved in NCDPI, Zipline Partnership
NC Public Schools Involved in NCDPI, Zipline PartnershipNC Public Schools Involved in NCDPI, Zipline Partnership
NC Public Schools Involved in NCDPI, Zipline Partnership
 
CTD Punjab Police Past Papers MCQs PPSC PDF
CTD Punjab Police Past Papers MCQs PPSC PDFCTD Punjab Police Past Papers MCQs PPSC PDF
CTD Punjab Police Past Papers MCQs PPSC PDF
 
Edukasyong Pantahanan at Pangkabuhayan 1: Personal Hygiene
Edukasyong Pantahanan at  Pangkabuhayan 1: Personal HygieneEdukasyong Pantahanan at  Pangkabuhayan 1: Personal Hygiene
Edukasyong Pantahanan at Pangkabuhayan 1: Personal Hygiene
 
ENGLISH-7-CURRICULUM MAP- MATATAG CURRICULUM
ENGLISH-7-CURRICULUM MAP- MATATAG CURRICULUMENGLISH-7-CURRICULUM MAP- MATATAG CURRICULUM
ENGLISH-7-CURRICULUM MAP- MATATAG CURRICULUM
 
How to Empty a One2Many Field in Odoo 17
How to Empty a One2Many Field in Odoo 17How to Empty a One2Many Field in Odoo 17
How to Empty a One2Many Field in Odoo 17
 
Year-to-Date Filter in Odoo 17 Dashboard
Year-to-Date Filter in Odoo 17 DashboardYear-to-Date Filter in Odoo 17 Dashboard
Year-to-Date Filter in Odoo 17 Dashboard
 

Contextual IR: Participations in TREC Contextual Suggestion Tracks 2012 and 2013

  • 1. Wolverhampton Talk December 2013 Contextual IR: Participations in TREC Contextual Suggestion Tracks 2012 and 2013 Gilles Hubert http://bit.ly/wlvHubert2013
  • 2. Summary 1. Information Retrieval 2. Contextual Information Retrieval 3. TREC 4. TREC Contextual Track 2012 1. Our approach 2. Results 5. TREC Contextual Track 2013 1. Our approach 2. Results 6. Result analysis and Future work 2
  • 3. Information Retrieval Usual IR process Document Document Document Query Indexing Query representation Indexing Matching Document Document representation Document representation representation List of estimated relevant documents 3
  • 4. Contextual Information Retrieval Notion of context in IR Information User Software Device How to consider context in IR process ? Q1 : Retrieve items corresponding to the context Q2 : Retrieve the context corresponding to items 4
  • 5. Contextual IR Context integration in Q1 Document Document Document Query Context Indexing Query representation Indexing Matching Document Document representation Document representation representation List of estimated relevant documents Reranking List of estimated relevant documents 5
  • 6. TREC Text Retrieval Conference Organized by the NIST (USA) since 1992 Based on the Cranfield paradigm of retrieval system evaluation A set of documents (Collection) A set of information needs (Topics/Queries) A set of relevance judgments (Qrels) Various tracks: AdHoc, Robust, Web… Evaluation measures System output: retrieved documents A relevant, retrieved (True positive) C Information need: relevant documents relevant, not retrieved (False negative) irrelevant, retrieved (False positive) B irrelevant, not retrieved (True negative) Document collection D precision = A A+B A recall = A+C AP (Average Precision), MAP (Mean Average Precision), P@5 (Precision at 5 retrieved documents) … 6
  • 8. TREC Contextual Suggestion Track 2012 Where to go around here on this Sunday afternoon? Great summer !!! 8
  • 9. TREC Contextual Suggestion Track 2012 Retrieve items corresponding to the context (Q1) Items = Suggestions Places to visit (shops, restaurants, parks…) around the user (5 hours by car max.) Collection = Open Web (Websites) Context = Spatiotemporal data <context number=”1”> <city>Portland</city> <state>Oregon</state> <lat>45.5</lat> <long>-122.7</long> <day>weekday</day> <time>evening</time> <season>fall</season> </context> User preferences <profile number=”1”> <example number=”1” initial=”1” final=”1”/> <example number=”2” initial=”0” final=”-1”/> </profile > <example number=”1”> <title> Dogfish Head Alehouse </title> <description>Craft Brewed Ales and tasty wood grilled food </description> <url>http://www.dogfishalehouse.com/</url> </example> <example number=”2”> <title>The Flaming Pit</title> <description> The Flaming Pit Restaurant and Piano Lounge, home of Tyrone DeMonke. </description> <url>http://www.flamingpitrestaurant.com/</url> </example> 9
  • 10. TREC Contextual Suggestion Track 2012 Two subtasks S1 : Suggestions corresponding to spatiotemporal data List of suggestions for each context S2 : S1 + user preferences List of suggestions for each profile (user) and each context Suggestion = Title + Description + Url <context2012 groupid=”waterloo” runid=”watcs12a”> <suggestion profile=”1” context=”1” rank=”1”> <title>Deschutes Brewery Portland Public House</title> <description> Deschutes Brewery’s distinct Northwest brew pub in Portland’s Pearl District has become a convivial gathering spot of beer and food lovers since it’s 2008 opening. </description> <url>http://www.deschutesbrewery.com</url> </suggestion> etc. </context2012> 2 “runs” maximum Our participation Team : G. Cabanac & G. Hubert (IRIT – Univ. of Toulouse) 2 runs submitted to S2 subtask 10
  • 11. TREC Contextual Suggestion Track 2012: Our approach Contextual IRS framework 2012 Input data Internal process Output data Place sets Intermediate data Database External resource Context processing Contexti Preference processing Profilei Place query Place selection Preference definition Contextual list of places Google Places API Positive preferencesi Personalization Negative Contextual list of detailed places Place description enrichment preferencesi Examples Personalized suggestions Bing Google Useri 11
  • 12. TREC Contextual Suggestion Track 2012: Our approach Spatiotemporal data User preferences Coarse-grained approach : iritSplit3CPv1 Merging of descriptions of examples with initial and final = 1 -> Pref+(P) Merging of descriptions of examples with initial and final = -1 -> Pref-(P) score(P,r) = cosine(Pref+(P),R) − cosine(Pref−(P),R) Fine-grained approach : iritSplit3CPv2 Example description with initial and final = 1 -> Pref+l(P) Example description with initial and final = -1 -> Pref-m(P) score(P, r) = max(cosine(Pref+l (P), r))− max(cosine(Pref−m(P), r)) 12
  • 13. TREC Contextual Suggestion Track 2012: Results Evaluations For each profile and each context Different dimensions : W (Website), G (Geographical), T (Temporal), and D (Description), and combinations (WGT and GT) Two measures : P@5 and MRR (Mean Reciprocal Rank) iritSplit3CPv1 iritSplit3CPv2 13
  • 14. TREC Contextual Suggestion Track 2012: Results P@5 14
  • 15. TREC Contextual Suggestion Track 2012: Results MRR 15
  • 16. TREC Contextual Suggestion Track 2013 Where to go around here? 16
  • 17. TREC Contextual Suggestion Track 2013 Context = Spatial only { "1": { "lat": "40.71427", "city": "New York City", "state": "NY", "long": "-74.00597” }, … } { "1": { "url": http://www.freshrestaurants.ca, "description": "Our vegan menu boasts an array of exotic starters, multi-layered salads, filling wraps, high protein burgers and our signature Fresh bowls.”, "title": "Fresh on Bloor” }, “2": { "url": http://www.flamingpitrestaurant.com/, "description": "The Flaming Pit Restaurant and Piano Lounge, home of Tyrone DeMonke.”, "title": "The Flaming Pit” }, … User preferences { "1": [ {"attraction_id": 1, "website": 1, "description": 0}, ... ], "2": [ {"attraction_id": 1, "website": 4, "description": 3}, … ], ”3": [ {"attraction_id": 1, "website": -1, "description": 2}, … ], … } } 17
  • 18. TREC Contextual Suggestion Track 2013 Two subtasks Open Web Same question: Suggest places items corresponding to the context (Q1) Places to visit (restaurants, museums…) around the user (5 hours by car) Collection = Open Web (Websites) ClueWeb ClueWeb12 (Same question as OpenWeb) ClueWeb12 Contextual suggestion subcollection Sets of ClueWeb12 documents per context Question: Personalization per user profile 2 “runs” maximum Our participation Team: G. Cabanac, G. Hubert & K. Pinel-Sauvagnat (IRIT – Univ. of Toulouse) C. Sallaberry (LIUPPA – Univ. of Pau) D. Palacio (GeoComp – Univ. of Zurich) 1 “run” Open Web 1 “run” ClueWeb (Contextual suggestion subcollection) 18
  • 19. TREC Contextual Suggestion Track 2013: Our approach Contextual IRS framework 2013 Input data Intermediate data L: Lucene W: WordNet GP: Google Places T: Terrier Useri GN: Geonames Output data Y: Yahoo! BOSS Geo Process P: PostGis GG: Gisgraphy Useri Examples Examples Profilei Contexti B: Bing 1 Profilei Posi ve preferencesi L, T, W Preference processing 1 Contexti Nega ve preferencesi Posi ve preferencesi L, T, W Preference processing Nega ve preferencesi Categories of interesti Categories of interesti Predefined categories 2 2 GP Context processing 3 3 GN, Y, P, GG, B Contextual list of places Place filtering & descrip on enrichment 4 T Retrieval B Place filtering & descrip on enrichment list of places 4 Ranking & refinement Ranking Personalized sugges ons Personalized sugges ons a) Open Web b) ClueWeb 19
  • 20. Example of suggestion in 2013 Title: Celtic Mist Pub Description: Place types: bar, establishment. This place is about .3 Km West from here (2 min by car with no traffic). Address: 117 South 7th Street, Springfield. There are 11 POIs around: 2 Hotels, 3 Libraries, 3 Parks, 1 PostOffice, 2 Religious. Snippet: Located in Springfield, IL the Celtic Mist is your home away from home with over 16 imported beers on tap and a friendly staff ready to serve you… URL: http://www.celticmistpub.com/ 20
  • 21. Example of suggestion in 2012 Title: Oakley Pub and Grill Description Oakley Pub and Grill - Located in Oakley Square, Cincinnati, Ohio. Local pub with pleasant atmoshpere and great food. Voted #1 Best Burger in Cincinnati. Outdoor ... PUB and GRILL OAKLEYOAKLEY Oakley Pub and Grill ~ 3924 Isabella Avenue ~ Cincinnati, Ohio 45209 On Oakley Square ~ (513) 531-2500 www.oakleypub.com Used with permission… URL: http://oakleypubandgrill.com/ 21
  • 24. Result Analysis First edition (2012) All the participants discovered the track principles Worst results: Descriptions of suggestions Second edition (2013) OpenWeb Focus on suggestion descriptions Changes in relevance judgments ClueWeb Misunderstanding of guidelines or insufficient details Next edition: TREC Contextual Suggestion Track 2014 Close to TREC Contextual Suggestion Track 2013 Future work Experiment framework variants on 2013 data Replace limited online tools/services Process larger collection: ClueWeb12 (870 millions pages, ~27TB) 24
  • 25. Wolverhampton Talk December 2013 Thank you for your attention Questions?