Recommended
More Related Content
Similar to MediaEval 2018: NewsREEL Multimedia at MediaEval 2018: News Recommendation with Image and Text Content
Similar to MediaEval 2018: NewsREEL Multimedia at MediaEval 2018: News Recommendation with Image and Text Content (20)
More from multimediaeval
More from multimediaeval (20)
Recently uploaded
Recently uploaded (20)
MediaEval 2018: NewsREEL Multimedia at MediaEval 2018: News Recommendation with Image and Text Content
- 1. Sophia Antipolis, France, October 29-31, 2018 NewsREEL Multimedia at MediaEval 2018: News Recommendation with Image and Text Content Andreas Lommatzsch, Benjamin Kille, Frank Hopfgartner, Torben Brodt
- 2. • Recommender systems help users to find relevant items / matching the individual preferences • Can be applied in a growing number of domains • Online shops • Videos and movie portals • Social media • News Portals • Scenarios may differ with respect to • Set of users (explicit profiles, sessions, anonymous users; context data) • Set of items (static / dynamic, collection size) • Available user-item interactions and content (ratings, reviews, impressions, sales) • Metrics Motivation Recommender Systems
- 3. • Images have an strong influence on our perception Living Labs and Stream-based Evaluation are on the rise. A recent inquiry of recommendation experts reveals shockingly low levels of confidence in offline evaluation. 9 out of 10 experts agree that existing protocols can Recommendation #1 Recommended Article Headline Recommendation #2 Recommended Article Headline Recommendation #3 Recommended Article Headline Recommendation #4 Recommended Article Headline The TUB presents NewsREEL at ECIR 2016 Motivation Multi-Media in Recommender Systems
- 4. • Recommender Research focuses on Collaborative Filtering • Text content-based approaches • Not much research how to use multimedia content in recommender systems • NewsREEL Multimedia addresses this challenge • Real-world scenario • Dataset • Baseline • Evaluation Tools Motivation Objectives of NewsREEL Multimedia Living Labs and Stream-based Evaluation are on the rise. A recent inquiry of recommendation experts reveals shockingly low levels of confidence in offline evaluation. 9 out of 10 experts agree that existing protocols can provide spurious results. They suggest to rely on socalled „real“ user feedback. Such feedback can be obtained in living labs such as clef Newsreel. Researchers connect their servers with the Open Recommendation Platform (http://orp.plista.com). They Recommendation #1 Recommended Article Headline Recommendation #2 Recommended Article Headline Recommendation #3 Recommended Article Headline Recommendation #4 Recommended Article Headline The TUB presents NewsREEL at ECIR 2016 Recommendation A: Recommended Article Headline
- 5. NewsREEL scenario • Recommend interesting news on several different German News portals • News items consist of • The headline [text] • A snippet [text] • An image [URL] • Additional meta data could be provided • User-Item interactions • Impressions • Click on recommendations Scenario NewsREEL Living Labs and Stream-based Evaluation are on the rise. A recent inquiry of recommendation experts reveals shockingly low levels of confidence in offline evaluation. 9 out of 10 experts agree that existing protocols can provide spurious results. They suggest to rely on socalled „real“ user feedback. Such feedback can be obtained in living labs such as clef Newsreel. Researchers connect their servers with the Open Recommendation Platform (http://orp.plista.com). They Recommendation #1 Recommended Article Headline Recommendation #2 Recommended Article Headline Recommendation #3 Recommended Article Headline Recommendation #4 Recommended Article Headline The TUB presents NewsREEL at ECIR 2016 Recommendation A: Recommended Article Headline
- 7. The NewsREEL architecture • Recommender teams register for the challenge • Receive requests that must be answered with recommendations • Evaluation based on live user feedback • An offline version is provided • Replay data and bin-based • Long-term statistics • No real time constraints Scenario NewsREEL
- 8. • Motivation • The Dataset and Statistics • Evaluation Metrics • Discussion and Conclusions Outline Structure of the Presentation
- 9. NewsREEL MultiMedia • Offline dataset (image processing is still time consuming) • Aggregated data on a weekly basis • The dataset consists of • Item data (headline, snippets, imageURL) • Image labels for most items (computed with 6 different configurations) • A low level image representation • Impression statistic (training weeks) • Potentially relevant items (test weeks) Dataset NewsREEL Multimedia Image labels itemIDs in 12 bins (weeks) #impressions Text data
- 10. NewsREEL MultiMedia – dataset statistic • Size: 200 MB (txt/csv) ; 8.5G low level features • Number of pairs (items, week): 311k • #tupels (item, config, label, score): 3Mio • 7 image annotator configurations: Dataset NewsREEL Multimedia Image labels itemIDs #impressions Textual data
- 11. Dataset NewsREEL Multimedia • publishers exhibit different engagement patterns • impressions show a power law distributions: • few articles accumulate majority of impressions • many articles get few impressions
- 12. • Motivation • The Dataset and Statistics • Evaluation Metrics • Discussion and Conclusions Outline Structure of the Presentation
- 13. Task: Predict the most popular items for the test weeks • What are appropriate metrics? • Precision@10 • Precision@N • Average Precision • RMSE / MAE NewsREEL MultiMedia Evaluation
- 14. Task: Predict the most popular items for the test weeks NewsREEL MultiMedia Evaluation f(text, image) score; sort(score) Article : #Impressions 438737736 : 342,428 438007519 : 340,840 437351823 : 268,656 437515691 : 253,596 437560377 : 203,943 439089552 : 169,430 439128028 : 160,133 438283775 : 152,881 436755527 : 145,553 437918358 : 127,624 week 0 | publisher 17614 Article : #Impressions 438737736 : - 438007519 : + 437351823 : - 437515691 : - 437560377 : - 439089552 : + 439128028 : + 438283775 : - 436755527 : - 437918358 : - p@10 = 0.3
- 15. Evaluation Scheme NewsREEL MultiMedia Evaluation w0 w2 w4 w6 w8 w10 w12 Train Train Test Test
- 16. • Precision@10 + Fits best with the application scenario Unstable, hard metric, small codomain {0.0, 0.1, .., 1.0} • Precision@N • Generalization of Precision@10 + Overcomes the problem of small codomain, weak recommender can reach a score > 0 • Find an appropriate N; we decided to use N = 0.1 * #items • Average Precision@N • Precision does not consider the order of the elements + Average Precision overcomes this problem More difficult to interpret • RMSE / MAE • Common metric is recommender system Inappropriate due to the power law distribution of the #impressions NewsREEL MultiMedia Evaluation-Metrics
- 17. • Motivation • The Dataset and statistics • Evaluation Metrics • Discussion and Conclusions Outline Structure of the Presentation
- 18. • Problem setting ranking optimisation • Dataset characteristics • impressions • preprocessed images • texts • own pipeline can be used • Evaluation scheme precision@10, precision@10% • 8 registrations • 1 submission ( how to improve?) • To continue with • baseline (presented remotely) • contribution (presented remotely) Conclusion
- 19. ? Questions?
- 20. • The code is available on request (Java/Maven) Contact
Editor's Notes
- Distinct number of items: 52k
- Distinct number of items: 52k