The document describes a workflow for cross media recommendations based on linked data analysis. It discusses two use cases - editor support and chat analysis - where the MICO project can provide recommendations of related articles, videos, or other media based on content analysis and metadata. The workflow involves crawling media items, analyzing and storing them as linked data, focusing on a specific item being edited, processing recommendation queries against the stored data, and returning results to the editing platform. Evaluation of recommender systems is also discussed.

1. © Fraunhofer IDMT
T. Köllmer†, E. Berndl‡, T. Weißgerber‡, P. Aichroth† , H. Kosch‡
A Workflow for Cross Media Recommendations
based on Linked Data Analysis
† Fraunhofer IDMT ‡ University of Passau
Thomas Köllmer
thomas.koellmer@idmt.fraunhofer.de

2. © Fraunhofer IDMT 2
The MICO Project
 http://www.mico-project.eu
 FP7 STREP, GA# 610480
 7 Partners: Salzburg Research, Fraunhofer IDMT, Oxford University,
University of Passau, Umeå University, Zaizi, InsideOut10
 Duration: 36 Months (end: 10/2016)
 Goal of MICO is to develop a platform to analyse “media in context” …
 ...by orchestrating different content extraction tools that can work
simultaneously or in a sequence and …..
 ...provide valuable metadata to third party applications.

3. © Fraunhofer IDMT 3
Cross Media Recommendations in MICO
 The Recommendation Problem
 “Providing suggestions for items to be of use to a user”
 Broad Application domains:
 e-Commerce, Travel, Multimedia, News, …
 Two main data sources:
 User behaviour data (Collaborative Filtering)
 Item Metadata (Content based recommendation)

4. © Fraunhofer IDMT 4
Cross Media Recommendations in MICO
 Media Recommendation
 “Classic” Recommendation problems (Music-Recommendation, Netflix
Challenge)
 Mostly based on collaborative filtering, but also content analysis
 Cross Media Recommendation
 Cross Media ≠ Cross Modal
 Multiple Input
 Media Items + Associated Media context, e.g., commentary
 Multiple Output
 Not necessarily more of the same thing

5. © Fraunhofer IDMT 5
Editor Support Usecase
As an editor, while I create or edit articles using WordPress, I want to
automatically
get related articles and videos that I might link to the article.
 Project Use Case: Recommendations for Greenpeace Magazine Italia
 Articles
 Videos
 User behaviour data

6. © Fraunhofer IDMT 6
Editor Support Usecase
As an editor, while I create or edit articles using WordPress, I want to
automatically
get related articles and videos that I might link to the article.
 How can Mico Help?
 Articles: NER components for content spotting
 Videos
 Text to speech + NER
 Metadata Analysis
 User behaviour data
 Generic collaborative filtering component (prediction.io)

7. © Fraunhofer IDMT 7
Chat Analysis Usecase
As a zooniverse user, I want get more information that helps me during the
current classification task, e.g., by the recommendation of similar subjects
or training data.
 Input:
 Results from animal detection
 Sentiment and competence classification of talks
 NER for animal spotting
 Output:
 Classification Hints
 Hints for moderators / experts

8. © Fraunhofer IDMT 8
Processing Workflow for Editor Support Use Case
1. Crawler feeds media items to platform
e.g., videos from Greenpeace channel, news articles, internal archive….
2. Analysis results are stored as Linked Data
Currently: using Apache Marmotta as triplestore

9. © Fraunhofer IDMT 9
Processing Workflow for Editor Support Use Case
3. Item gets into Focus (someone is writing an article on a specific topic)
4. Focused item is fed to the same platform, using customized pipelines
5. Results are processed by LD matching, i.e Recommendation component..
6. … enriched by stored annotations …
7. And returned to editing platform

10. © Fraunhofer IDMT 10
Recommendation Workflow in MICO

11. © Fraunhofer IDMT 11
Ontology Matching
 Semantic distance of entities important for analysis
 Animal detection: “I see a bird on the tree”, “Yes it’s definitely a
sparrow”
 Topic detection: Wind energy, sustainability, renewables, oil, …
 Non-Mico:
 Genres! (e.g., currently 1400 genres on Spotify…)
 Not less controversial: movie genres

12. © Fraunhofer IDMT 12
Evaluation
 How to evaluate a recommender system?
 Forget about precision and recall!
 Observing user behaviour
 Online experiment
 Offline: extrapolating past user behavior
 Lab Studies / Questionnaire
 Open problem: standardized + comparable testing of recommender
systems

13. © Fraunhofer IDMT 13
Conclusions
 Proposed setting tries to formalize a certain subset of recommendation
problems: Cross media recommendation
 Focused on Media Items
 Profits from Content Analysis
 New items, have a high contextual dependency on existing items, that
will be exploited for recommendation
 Work in Progress:
 Stable release of Mico-platform and extractors: right now
 PoC recommendation code will be integrated in the platform
 Demo Services: available in June
 Incorporate feedback into recommendation

14. © Fraunhofer IDMT
T. Köllmer†, E. Berndl‡, T. Weißgerber‡, P. Aichroth† , H. Kosch‡
A Workflow for Cross Media Recommendations
based on Linked Data Analysis
† Fraunhofer IDMT ‡ University of Passau
Thank you!
Questions?