Real-Time News Recommendation and
Lifelogging: An Introduction to the
Evaluation Campaigns NEWSREEL (CLEF)
and Lifelog (NT...
A few words about me
Past: Various positions in Berlin
(TUB), Dublin (DCU), Berkeley
(ICSI), and London (QMUL)
Research on...
Overview
Evaluation Campaigns
Overview
NEWSREEL
(CLEF)
LifeLog
(NTCIR-12)
Introduction
Career Collaborations Research
affiliated with UC Berkeley
*now: INSIGHT Centre
Smart Information Systems
Collaborations
Recommender Systems
Aggregated Search
Multimedia Analysis
Gamification
Large-Scale
Evaluation
Semantic Sear...
Example: Video Search Engine
[ACM Multimedia’08, ECIR’08, SIVP’08, ACM TOIS’11]
Adapt retrieval results based on
•User’s s...
Example: Video Search Engine
[ACM/Springer Multimedia Systems (2010), MTAP (2012)]
Recommend news stories based on
user’s ...
Example: Video Browsing
[MMM’12-14, Int. Journal of Multimedia Information Retrieval (2014)]
Video Browser Showdown Compet...
Example: Other Video Search Engines
Example: Image Browser
[IICAI’09]
Ostensive Browsing
Example: Visual Lifelog Browser
[published as book chapter, Springer Verlag (2013)]
Providing easy access to visual lifelo...
Example: Enterprise Search Engine
[AAMAS’14]
Aggregated Search
Example: Health Information Portal
[PervasiveHealth’13, MIS’14, Book chapter (2015)]
Semantic Search
Multilingual Search
Example: Knowledge Management System
[ECIR’14, Book Chapter (2015)]
Targeting intrinsic motivation using
Gamification desi...
Multimedia Analysis
Video
segmentation
Voice activity
detection
Quality
assessment
of user-
generated
video
Affective
cont...
Data Analysis
Activity and Energy expenditure estimation using
accelerometer data
!
Overview
Evaluation Campaigns
Overview
NEWSREEL
(CLEF)
LifeLog
(NCTIR-12)
Introduction
Career Collaborations Research
How do we evaluate information access systems?
Document
collection
Topic
set
Relevance
assessments
Testcollection
Document...
Evaluation Campaigns
TREC CLEF
FIRE
NTCIR
 Common dataset
 Pre-defined tasks
 Ground truth
 Evaluation protocol
 Eval...
Focus on different domains
 Microblogging
 Ad-hoc and Web Search
 Multimedia
 Federated Web Search
 XML Retrieval
 I...
Overview
Evaluation Campaigns
Overview
NEWSREEL
(CLEF)
LifeLog
(NTCIR-12)
Introduction
Career Collaborations Research
CLEF Tracks
Source: http://www.clef-initiative.eu/
 eHealth
 ImageCLEF
 LifeCLEF
 Living Labs for IR (LL4IR)
 News Re...
In CLEF NEWSREEL, participants can
develop news recommendation
algorithms and have them tested by
millions of users over t...
Recommender Systems help users
to find items that they were not
searching for.
What are recommender systems?
Items?
Example: YouTube
Example: Netflix
Example: News Articles
Source (Image): T. Brodt of plista.com
What are living labs?
Rely on feedback from real users to develop convincing demonstrators that
showcase potentials of an ...
In an IR context…
“A living laboratory on the Web that
brings researchers and searchers together
is needed to facilitate I...
A / B testing
Evaluate
submit to
SIGIR
CLEF NEWSREEL
2014
 Frank Hopfgartner
 Andreas Lommatzsch
 Benjamin Kille
 Torben Brodt
 Tobias Heintz
Co-Organisers
...
Who are the users?
Devices
• Given a dataset, predict news articles a user
will click on
Offline Evaluation
• Recommend articles in real-time over
se...
Predict interactions based on an OFFLINE dataset
Task 1: Offline Evaluation
DATASET
EVALUATION
 Traffic and content updat...
Recommend news articles in REAL-TIME
Task 2: Online Evaluation
LIVINGLAB
EVALUATION
 Provide recommendations for
visitors...
Living Lab Scenario
…
Publisher A
Publisher n
Researcher 1
Researcher n
…
plista
ORP
…
Millions of visitors Publishers Tea...
Open Recommender Platform
Number of clicks
[CLEF’14]
Number of requests
[CLEF’14]
Click-Through Rate
[CLEF’14]
Overall results
Advertisement: Join the Living Lab - Tutorial at ECIR’15
Overview
Evaluation Campaigns
Overview
NEWSREEL
(CLEF)
LifeLog
(NTCIR-12)
Introduction
Career Collaborations Research
NTCIR
Source:HideoJoho
NTCIR-12 Tasks
NTCIR-12
 Second round:
 Search-Intent Mining
 Mobile Click
 Temporal Information Access
 Spoken Query...
Encourage research advances in organising
and retrieving from lifelog data.
LifeLog @ NTCIR-12
Lifelogging Challenges
The challenges are how to sense
the person, their actions, their
life and make it accessible using
...
LifeLog @ NTCIR-12
CO-ORGANISERS
TIMELINE
 Cathal Gurrin, Dublin City
University
 Hideo Joho, University of
Tsukuba
 Fr...
Multimodal dataset with information needs
Created by 5-10
individuals over
10+ days
TESTCOLLECTION
 1,500 images, locatio...
Evaluate different methods of
retrieval and access.
Tasks
T1:LIFELOGSEMANTICACCESS(LSAT)
T2:LIFELOGINSIGHT
 Models the re...
Thank you
Come and talk to me
Frank Hopfgartner, PhD
Frank.Hopfgartner@glasgow.ac.uk
@OkapiBM25
www.hopfgartner.co.uk
Real-Time News Recommendation and Lifelogging: An Introduction to the evaluation campaigns NewsREEL (CLEF) and Lifelog (NT...
Upcoming SlideShare
Loading in …5
×

Real-Time News Recommendation and Lifelogging: An Introduction to the evaluation campaigns NewsREEL (CLEF) and Lifelog (NTCIR-12)

470 views

Published on

Slides of my invited talk on 3 Februrary 2015 at the HATII research seminar series at University of Glasgow. In this presentation, I provided an overview of my prior research and introduced two academic evaluation campaigns that I am currently co-organising, namely NEWSREEL and Lifelog. NEWSREEL is a living lab on the real-time evaluation of news recommenders which is organized as part of CLEF 20114 and 2015. Lifelog is a pilot task of NTCIR-12 on Personal Lifelog Access & Retrieval.

Published in: Data & Analytics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
470
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Real-Time News Recommendation and Lifelogging: An Introduction to the evaluation campaigns NewsREEL (CLEF) and Lifelog (NTCIR-12)

  1. 1. Real-Time News Recommendation and Lifelogging: An Introduction to the Evaluation Campaigns NEWSREEL (CLEF) and Lifelog (NTCIR-12) Frank Hopfgartner @OkapiBM25
  2. 2. A few words about me Past: Various positions in Berlin (TUB), Dublin (DCU), Berkeley (ICSI), and London (QMUL) Research on Information Access Systems Lecturer in Information Studies (HATII, Glasgow) PhD in Information Retrieval (University of Glasgow)
  3. 3. Overview Evaluation Campaigns Overview NEWSREEL (CLEF) LifeLog (NTCIR-12) Introduction Career Collaborations Research
  4. 4. affiliated with UC Berkeley
  5. 5. *now: INSIGHT Centre
  6. 6. Smart Information Systems
  7. 7. Collaborations Recommender Systems Aggregated Search Multimedia Analysis Gamification Large-Scale Evaluation Semantic Search Multimedia Access Lifelogging Sensor Analysis Multimedia Retrieval & Recommendation User Modelling User Simulation Co-Author graph of first 100 publications (2006-2015)
  8. 8. Example: Video Search Engine [ACM Multimedia’08, ECIR’08, SIVP’08, ACM TOIS’11] Adapt retrieval results based on •User’s search queries •Relevance feedback
  9. 9. Example: Video Search Engine [ACM/Springer Multimedia Systems (2010), MTAP (2012)] Recommend news stories based on user’s long-term interests
  10. 10. Example: Video Browsing [MMM’12-14, Int. Journal of Multimedia Information Retrieval (2014)] Video Browser Showdown Competition
  11. 11. Example: Other Video Search Engines
  12. 12. Example: Image Browser [IICAI’09] Ostensive Browsing
  13. 13. Example: Visual Lifelog Browser [published as book chapter, Springer Verlag (2013)] Providing easy access to visual lifelogs
  14. 14. Example: Enterprise Search Engine [AAMAS’14] Aggregated Search
  15. 15. Example: Health Information Portal [PervasiveHealth’13, MIS’14, Book chapter (2015)] Semantic Search Multilingual Search
  16. 16. Example: Knowledge Management System [ECIR’14, Book Chapter (2015)] Targeting intrinsic motivation using Gamification design elements
  17. 17. Multimedia Analysis Video segmentation Voice activity detection Quality assessment of user- generated video Affective content analysis [ECIR’09, ACM Multimedia (2014), CMBI (‘12, ‘13), MMM (2010, 2013, 2014) , ICMR (2014)]
  18. 18. Data Analysis Activity and Energy expenditure estimation using accelerometer data !
  19. 19. Overview Evaluation Campaigns Overview NEWSREEL (CLEF) LifeLog (NCTIR-12) Introduction Career Collaborations Research
  20. 20. How do we evaluate information access systems? Document collection Topic set Relevance assessments Testcollection Document collection
  21. 21. Evaluation Campaigns TREC CLEF FIRE NTCIR  Common dataset  Pre-defined tasks  Ground truth  Evaluation protocol  Evaluation metrics
  22. 22. Focus on different domains  Microblogging  Ad-hoc and Web Search  Multimedia  Federated Web Search  XML Retrieval  Information Access in the Legal Domain  Document Similarity  …
  23. 23. Overview Evaluation Campaigns Overview NEWSREEL (CLEF) LifeLog (NTCIR-12) Introduction Career Collaborations Research
  24. 24. CLEF Tracks Source: http://www.clef-initiative.eu/  eHealth  ImageCLEF  LifeCLEF  Living Labs for IR (LL4IR)  News Recommendation Evaluation Lab (NEWREEL)  Uncovering Plagiarism, Authorship and Social Software Misuse (PAN)  Question Answering (QA)  Social Book Search (SBS) CLEF’15
  25. 25. In CLEF NEWSREEL, participants can develop news recommendation algorithms and have them tested by millions of users over the period of a few months in a living lab. NEWSREEL
  26. 26. Recommender Systems help users to find items that they were not searching for. What are recommender systems?
  27. 27. Items?
  28. 28. Example: YouTube
  29. 29. Example: Netflix
  30. 30. Example: News Articles Source (Image): T. Brodt of plista.com
  31. 31. What are living labs? Rely on feedback from real users to develop convincing demonstrators that showcase potentials of an idea or a product. Real-life test and experimentation environment to fill the pre-commercial gap between fundamental research and innovation.
  32. 32. In an IR context… “A living laboratory on the Web that brings researchers and searchers together is needed to facilitate ISSS (Information- Seeking Support System) evaluation.” Kelly et al., 2009
  33. 33. A / B testing Evaluate submit to SIGIR
  34. 34. CLEF NEWSREEL 2014  Frank Hopfgartner  Andreas Lommatzsch  Benjamin Kille  Torben Brodt  Tobias Heintz Co-Organisers 2015  Frank Hopfgartner  Torben Brodt  Benjamin Kille  Jonas Seiler  Balázs Hidasi  Andreas Lommatzsch  Roberto Turrin  Martha Larson Scenario
  35. 35. Who are the users?
  36. 36. Devices
  37. 37. • Given a dataset, predict news articles a user will click on Offline Evaluation • Recommend articles in real-time over several months Online Evaluation CLEF NEWSREEL 2014 TASK1TASK2 @clefnewsreel http://www.clef-newsreel.org/
  38. 38. Predict interactions based on an OFFLINE dataset Task 1: Offline Evaluation DATASET EVALUATION  Traffic and content updates of 9 German-language news content provider websites  Traffic: Reading article, clicking on recommendations  Updates: adding and updating news articles  Recorded in June 2013  65 GB, 84 Million records  [Kille et al., 2013]  Dataset split into different time segments  Participants have to predict interactions of these segments  Quality measured by the ratio of successful predictions by the total number of predictions
  39. 39. Recommend news articles in REAL-TIME Task 2: Online Evaluation LIVINGLAB EVALUATION  Provide recommendations for visitors of the news portals of plista’s customers  Ten portals (local news, sports, business, technology)  Communication via Open Recommender Platform (ORP)  Provide recommendations within <10ms (VM provided if necessary)  Three pre-defined evaluation periods  5-23 February 2014  1-14 April 2014  5-19 May 2014  Evaluation criteria  Number of clicks  Number of requests  Click-through rate
  40. 40. Living Lab Scenario … Publisher A Publisher n Researcher 1 Researcher n … plista ORP … Millions of visitors Publishers Teams
  41. 41. Open Recommender Platform
  42. 42. Number of clicks [CLEF’14]
  43. 43. Number of requests [CLEF’14]
  44. 44. Click-Through Rate [CLEF’14]
  45. 45. Overall results
  46. 46. Advertisement: Join the Living Lab - Tutorial at ECIR’15
  47. 47. Overview Evaluation Campaigns Overview NEWSREEL (CLEF) LifeLog (NTCIR-12) Introduction Career Collaborations Research
  48. 48. NTCIR Source:HideoJoho
  49. 49. NTCIR-12 Tasks NTCIR-12  Second round:  Search-Intent Mining  Mobile Click  Temporal Information Access  Spoken Query & Spoken Document Retrieval  QA Lab for Entrance Exam  First round:  Medical NLP for Clinical Documents  Personal Lifelog Access & Retrieval  Short Text Conversation
  50. 50. Encourage research advances in organising and retrieving from lifelog data. LifeLog @ NTCIR-12
  51. 51. Lifelogging Challenges The challenges are how to sense the person, their actions, their life and make it accessible using appropriate interfaces, search, recommendation engines and visual/aural feedback. Further, exploiting the lifelog to identify context for adaptive information services. Source (Graphic): DAI-Labor, Berlin
  52. 52. LifeLog @ NTCIR-12 CO-ORGANISERS TIMELINE  Cathal Gurrin, Dublin City University  Hideo Joho, University of Tsukuba  Frank Hopfgartner, University of Glasgow  Brian Moynagh, Dublin City University  Rami Albatal, Dublin City University  27 Feb 2015: Kickoff event  30 Jun 2015: Task Registration  01 Jul 2015: Dataset release  Sep 2015 – Feb 2016: Formal run  01 Feb 2016: Evaluation Results  01 Mar 2016: Paper for the Proceedings  7-10 Jun 2016: NCTIR-12 conference
  53. 53. Multimodal dataset with information needs Created by 5-10 individuals over 10+ days TESTCOLLECTION  1,500 images, location, GSR, heart-rate, others… per lifelogger per day  Accompanying output of 1,000 concepts  Data processed pre-release (removal of personal content; face blurring, translation of concepts)  Detailed user queries and judgments generated by the lifelogging data gatherers
  54. 54. Evaluate different methods of retrieval and access. Tasks T1:LIFELOGSEMANTICACCESS(LSAT) T2:LIFELOGINSIGHT  Models the retrieval need from lifelogs (Known-Item Search)  Retrieve N segments that match information need  Interactive or Automatic participation  Interactive: Time limit for fair and comparative evaluation in an interactive system with users  Automatic: Fully-automatic retrieval system. Automated query processing  Models the need for reflection over lifelog data  Exploratory task, the aim is to:  encourage broad participation  novel methods to visualise and explore lifelogs  Same data as LSAT task  Presented via demo/poster.
  55. 55. Thank you Come and talk to me Frank Hopfgartner, PhD Frank.Hopfgartner@glasgow.ac.uk @OkapiBM25 www.hopfgartner.co.uk

×