Combining Multimedia and Semantics (LACNEM2010)
Upcoming SlideShare
Loading in...5
×
 

Combining Multimedia and Semantics (LACNEM2010)

on

  • 1,897 views

Invited talk given at LACNEM2010 (Cali, Colombia), in September 2010, on the combination of semantics and multimedia.

Invited talk given at LACNEM2010 (Cali, Colombia), in September 2010, on the combination of semantics and multimedia.

Statistics

Views

Total Views
1,897
Slideshare-icon Views on SlideShare
1,893
Embed Views
4

Actions

Likes
2
Downloads
41
Comments
0

1 Embed 4

http://www.linkedin.com 4

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Combining Multimedia and Semantics (LACNEM2010) Combining Multimedia and Semantics (LACNEM2010) Presentation Transcript

    • Combining Multimedia and Semantics
      Oscar Corcho (ocorcho@fi.upm.es)
      Universidad Politécnica de Madrid
      http://www.oeg-upm.net/
      LACNEM 2010, Cali, ColombiaSeptember 9th 2010
      Credits: Adrián Siles, Mariano Rico, Víctor Méndez, Hector Andrés García-Silva, María del Carmen Suárez-Figueroa, Ghislain Atemezing, Raphaël Troncy
      WorkdistributedunderthelicenseCreativeCommonsAttribution-Noncommercial-Share Alike3.0
      http://www.slideshare.net/ocorcho
    • 2
      Asunción Gómez Pérez
      OntologyEngineering Group. Whomwe are
      Director: A. Gómez-Pérez
      Research Group (37 people)
      2 Full Professor
      4 AssociateProfessors
      1 AssistantProfessor
      3 Postdocs
      17 PhD Students
      8 MScStudents
      2 Software Engineers
      Management (4 people)
      2 Project Managers
      1 SystemAdministrator
      1 Secretary
      50+ PastCollaborators
      10+ visitors
    • Research Areas
      2004
      2008
      1995
      1997
      2000
    • Beforewestart…
      Howmany of youhaveeverheardabouttheword “Ontology”?
      And howmany of you do actuallyknowwhatitmeans?
      4
    • Comingtotermswithontologies and semantics
      An ontology is an engineering artifact, which provides:
      A vocabulary of terms
      A set of explicit assumptions regarding the intended meaning of the vocabulary.
      Almost always including concepts and their classification
      Almost always including properties between concepts
      Shared understanding of a domain of interest
      Agreement on the meaning of terms
      Formal and machine manipulable model of a domain of interest
      Besides...
      The meaning (semantics) of such terms is formally specified
      New terms can be formed by combining existing ones
      Can also specify relationships between terms in multiple ontologies
      5
    • Example: Anontologyaboutsatellites
      6
    • Outline
      Introduction
      What I willbetalkingabout and what I willnot…
      Therewereseveraloptionsthat I exploredbeforeselectingtheonethatyouwillbehearing in thistalk…
      7
    • Option 1: The Semantic Gap
      The lack of coincidencebetweentheinformationthatone can extractfromthesensory data and theinterpretationthatthesame data has for a user in a givensituation
      8
      A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain: Content-based image retrieval at the end of the early years, IEEE PAMI, 1349–1380, 2000.
      However, I alreadyassumedthatEbroulwouldbetalking a lotaboutit in hisopeningkeynote (as he did).
      Besides, I havenotworked at allonthelow-levelpart, so itmaybedifficultfor me toprovideyouwith a goodinsightonthe (many) open problems in thisarea
    • Option 2: MPEG-7 and the Semantic Web
      ISO standard since December 2001
      Main components:
      Descriptors (Ds) and Description Schemes (DSs)
      DDL (XML Schema + extensions)
      Concern all types of media
      A good number of ontologies developed around it
      9
      Part 5 – MDSMultimedia Description Schemes
    • Option 2: MPEG-7 and the Semantic Web
      However, thetalkmay:
      Be a bit boring and tootechnical
      Maylackthemix of state of the art and visionthataninvitedtalkshouldnormallyhave
      And MPEG-7 isnotusedtoomuch
      …So I willcoveronlysomeaspects of thislater, when I talkabout multimedia ontologies.
      10
    • Option 3: Canonical Processes of Media Production (and semantics, obviously)
      For example….
      http://www.cewe-photobook.com
      Application for authoring digital photo books
      Automatic selection, sorting and ordering of photos
      Context analysis methods: timestamp, annotation, etc.
      Content analysis methods: color histograms, edge detection, etc.
      Customized layout and background
      Print by the European leader photo finisher company
      11
      Credits: Raphaël Troncy, LyndaHardman
    • CeWe Color PhotoBook Processes
      My winter ski holidays with my friends
      Credits: Raphaël Troncy, LyndaHardman
    • CeWe Color PhotoBook Processes
      Credits: Raphaël Troncy, LyndaHardman
    • CeWe Color PhotoBook Processes
      Credits: Raphaël Troncy, LyndaHardman
    • CeWe Color PhotoBook Processes
      Credits: Raphaël Troncy, LyndaHardman
    • CeWe Color PhotoBook Processes
      Credits: Raphaël Troncy, LyndaHardman
    • CeWe Color PhotoBook Processes
      Credits: Raphaël Troncy, LyndaHardman
    • Semantics can be important in the process
      Credits: Raphaël Troncy, LyndaHardman
    • Option 3: Canonical Processes of Media Production
      However, some of youprobablyattended Raphaël Troncy’stalklastyear (available in slideshare)
      19
    • In summary…
      I decidedtotalkaboutsomethingthat I havebeenworking in forthelastcouple of years, and which combines
      Semantics (of course, thisisthekeyexpertise of ourgroup)
      Mainlyannotation, Linked Data and a bit of Multimedia OntologyEngineering
      Social networks, collaboration, sharing and collectiveintelligence
      Exploiting home networks and online multimedia sites
      And, obviously, multimedia
      And hence I stillleaveoutmanyinterestingtopics (e.g., semanticsin user interfaces)
      20
    • Outline
      Introduction
      What I willbetalkingabout and what I willnot
      Sem-UPnP-Grid
      Sharing multimedia contentacrosshomesthroughsemanticannotations
      Credits: Mariano Rico and Adrián Siles (UPM), Víctor Méndez and José Manuel Gómez-Pérez (iSOCO), José Manuel Palacios and Mónica Pérez (TID)
      Sem4Tags
      Tagdisambiguation in Flickr
      M3 Ontology(onlyif time permits)
      A semanticbackboneforour multimedia-relatedwork
      Conclusions and outlook
      21
    • Internet
      Motivation
      Multimedia resources in Web2.0 are stored in centralised servers.
      You lose some of yourrights as anauthorwhenyouuploadtheseresourcestothese servers.
      Privacyproblems.
      Poorannotations and metadata.
      Theseresourcescannotbesharedwithotherresources in your home.
      22
      UpGrid
    • Multimedia Content SharingwithUpGrid
      23
      Annotation:
      “Ángel onthebeach”
      Reasoning:
      • “Ángel is my son”
      • “Pedro is my brother”
      • “Juan is my brother”
      • -------------------------------
      • Ángel is my nephew
      Juan
      P2P
      Semantic-basedquery:
      “multimedia contentrelatedto my nephew”
      Annotation:
      “Ángel playing soccer”
      Pedro
      Additionalsemanticinformation:
      • “Ángel is my son”
      • “Pedro is my brother”
      Additionalsemanticinformation:
      “Juan is my brother”
    • Architecture
    • Architecture (anotherviewonit)
    • Snapshotsfromtheapplication
      Checkhttp://www.youtube.com/results?search_query=UPnPGrid
    • Summary
      Aneffectivemeansforsharing multimedia contentsacrosshomes, avoiding Web2.0 siteswhereyourrightsmaybecompromised
      However, itisstill a prototype, and no serioususabilitytesting has been done
      Muchworkstillneeded in ordertogointo a real system
      And endusersfinditdifficulttoprovideannotations
      Do you imagine yourparents and grandparentsannotatingphotos and videos likethat?
      Let’sseehowthiscouldbeamelioratedwiththenextpart of ourpresentation.
      27
    • Outline
      Introduction
      What I willbetalkingabout and what I willnot
      Sem-UPnP-Grid
      Sharing multimedia contentacrosshomesthroughsemanticannotations
      Sem4Tags
      Tagdisambiguation in Flickr
      Credits: Héctor Andrés García Silva
      M3 Ontology(onlyif time permits)
      A semanticbackboneforour multimedia-relatedwork
      Conclusions and outlook
      28
      Egresado de laUniversidad del Valle
    • Introduction
      Social Tagging Systems
      Web 2.0 applications
      Applications for storing, sharing, and discovering information resources.
      Users assign tagsto identify information resources
      Tags are used to search/discover resources
      29
    • Introduction
      Folksonomy
      Emerging classification scheme from social tagging systems
      Folk: People, Taxonomy: Classification
      Represented by: Users, Tags, Resources
      Taxonomy
      Folksonomy
      • Top-down
      • Controlled Vocabulary
      • Hierarchical structure
      • Exclusive/Restrictive
      • Expensive to maintain
      • Bottom-up (user created)
      • No fixed vocabulary
      • No Hierarchical structure
      • No Exclusive/Flexible
      • Low cost
      30
    • Introduction
      Why is tagging so popular?
      Reduce cognitive burdens
      it’s easy to use
      Users don´t need any special skill or experience
      The benefits of tagging are immediate
      Future retrieval
      Contribution and sharing
      Attract Attention
      Self Presentation
      Opinion Expression
      31
    • Introduction
      However
      Tags can be ambiguous
      Polysemy: partyas a celebration as opposed to partyas a political organization
      Synonym: party and celebration 
      Morphological variations:
      party, parties, partying, partyign
      Plurals
      Acronyms
      Conjugated verbs
      Misspelling
      Compound words
      Political party, PoliticalParty, Political_party,
      Political-Party, etc.
      Detail/granularity level
      A general tag as partyin contrast to a specific tag as banquet.
      32
    • Motivation
      The problem: Morphological variations, synonyms, granularity, and polysemy hamper information retrieval processes based on folksonomies.
      Systems ignore resources tagged with morphological variationsor synonyms of that tag, as well as the resources tagged with more generic or more specific tags
      710.659 results
      8.661.581 Results
      33
    • When searching with polysemous tags, all the resources tagged with that tag are retrieved without taking into account the tag sense the user was looking for.
      (e.g., Query flickr with bank results in photos about financial institutions, river edges, fog banks, and sand banks, etc. )
      34
      Motivation
    • Motivation
      What if we associate tags with semantic entities?
      http://morpheus.cs.umbc.edu/aks1/ontosem.owl 
      #non-work-activity
      We can avoid the aforementioned pitfalls
      #organization
      #special-occasion
      #political-entity
      #party
      #Celebration
      #political-party
      #Coalition
      #federation
      #Birthday
      #Anniversary
      uk, tories, party, conservative, speech
      party, balloons, colors, bar, crowd
      35
    • State of the Art: Semantic Grounding of Cross-Lingual Folksonomies
      Garcia HA, Corcho O, Alani H, Gómez-Pérez A. Review of the state of the art: Discovering and Associating Semantics to Folksonomies. Knowledge Engineering Review (in press)
      None of the analyzed approaches deals with multilingual tags
      36
    • Semantic Grounding of Cross-Lingual Folksonomies
      MSR: a Multilingual Sense Repository based on Wikipedia and enriched with semantic information taken from DBpedia.
      Terms and
      frequency
      Banco
      Bank
      http://dbpedia.org/resource/Bank
      Terms and
      frequency
      Banco
      Cardumen
      Swarm
      http://dbpedia.org/resource/Swarm
      Banco de
      Arena
      http://dbpedia.org/resource/
      SandBank
      Terms and
      frequency
      Sandbank
      37
    • Semantic Grounding of Cross-Lingual Folksonomies
      Sem4Tags: A process for Associating Semantics to Tags.
      38
      Dinero,
      Calle,
      Santander,
      Money,
      Madrid,
      Atm,
      cajero
      Europe
      Euro
      Finance
      Central bank
      awesomePic
      Nikon ..
      Bank
      Banco
      http://dbpedia.org/resource/Bank
    • Semantic Grounding of Cross-Lingual Folksonomies
      Disambiguation activity
      The candidate senses and the tag context are represented as vectors.
      The vector components are the set of most frequent terms in each Wikipedia page representing a sense.
      For each sense the values of the vector are calculated using TF-IDF.
      For the tag context the values in each position are 1 or 0 if the corresponding term appears in the tag context.
      The tag context vector is compared against each sense vector using the cosine of the angle as similarity measure.
      The most similar sense to the tag context is selected as the one representing the meaning of the analyzed tag
      39
      39
    • Semantic Grounding of Cross-Lingual Folksonomies
      Disambiguation activity
      We use the information of the wikipedia default sense for a term.
      Sim(TagContext, Sensei)= λ*Cosine + β*defaultSense
      We experimentally defined β = 0,2 and λ = 0.8
      We attempt to use DBpedia semantic information in the disambiguation activity:
      Sim(TagContext, Sensei)= λ*Cosine + β*defaultSense + δ*SemanticInfo
      Studies have shown that tags in flickr refers mainly to: Locations, Time, Given Names, Potography related subjects among others.
      We use DBpedia and YAGO relations to classify the senses according to this categories.
      However, we found that not all the senses related to a term have the same amount of relations. (e.g. Madrid is not a city)
      40
    • Let’s try it
      http://robinson.dia.fi.upm.es:8080/SemanticTagsWebApp/index.jsp
      Whatdoes “bernabeu” mean ifitscontextis…?
      estadio, madrid, fútbol
      41
    • Experiment
      Baseline: Directly associate tags with DBpedia resources
      Look for spaces and replace them with ' _‘.
      For tags in English:
      Create a URI of the form http://en.wikipedia.org/wiki/tag
      Query DBpedia using the http://xmlns.com/foaf/0.1/page relation
      For tags in Spanish:
      Create a URI of the form http://es.wikipedia.org/wiki/tag
      Query DBpedia using the http://dbpedia.org/property/wikipage-es relation
      42
    • Experiment
      Approaches:
      Baseline: Selection of the sense without a disambiguation activity.
      Sem4Tags: For each sense we use the whole Wikipedia article as source for frequentterms.
      Sem4TagsAC: Same as Sem4Tags including the selection of the Active Context.
      Sem4TagsAbs: For each sense we use the Wikipedia article abstract (extracted from DBpedia) as source for frequent terms.
      Sem4TagsAbsAC: Same as Sem4TagsAbs including the selection of the Active Context.
      43
    • Experiment
      Initial Data Set
      Wide range of Users, photos, and tags.
      764 photos uploaded by 719 users to Flickr that have been tagged with tags describing tourist places in Spain
      12.4 (+/- 7.85) tags per photo
      9484 tagging activities (TAS) : <user,photo,tag>
      4135 distinct tags where used
      Processed Data Set
      From each photo we processed on average 2 tags
      2260 taggingactivities (TAS)
      44
    • Experiment
      Evaluation Campaign
      41 Evaluators
      Evaluate semantic associations produce by each approach: <user; tag; photo; DBpedia resource; language>
      Three different evaluators evaluated each semantic association.
      Questions:
      Able to identify the tag meaning (known or Unknown)
      Tag language (English, Spanish, Both, other)
      The tag correspond to a Named entity
      According to the identified tag language they evaluate the semantic association in terms of
      Highly related, Related, Not Related.
      45
    • Experiment
      Results
      Evaluators identified the semantics of the 87% of TAS (known)
      62.6 % of TAS were considered in English
      87.7% of TAS were considered in Spanish
      Agreement among evaluators (Fleiss’ kappa statistics):
      k=0.76 for highly related
      K=0.71 for the related case/highly related case
      46
    • Experiment
      Precision and RecallforHighlyRelevantresults
      47
      English
      Spanish
    • Experiment
      Conclusions
      Baseline obtained high precision, however it was able to find semantic resources for just a fraction of the analyzed data set:
      Baseline: 27.7% in English and 19.4% in Spanish.
      Sem4Tags: 79.1 % in English and 81.4% in Spanish
      All approaches obtained better precision with named entities than with unnamed entities.
      Sem4Tags and Sem4TagsAC are the approaches that obtained the best results in terms of Precision and Recall.
      Sometimes Sem4TagsAC obtains better P@1 values but the improvements are supported by no or low statistical evidence.
      Sem4TagsAbs and Sem4TagsAbs are clearly the worst approaches.
      48
    • Outline
      Introduction
      What I willbetalkingabout and what I willnot
      Sem-UPnP-Grid
      Sharing multimedia contentacrosshomesthroughsemanticannotations
      Sem4Tags
      Tagdisambiguation in Flickr
      M3 Ontology(onlyif time permits)
      A semanticbackboneforour multimedia-relatedwork
      Conclusions and outlook
      49
    • 50
      Ontología M3
    • There are already multimedia ontologies
      MDS Upper Layer represented in RDFS
      2001: Hunter
      Later on: link to the ABC upper ontology
      MDS fully represented in OWL-DL
      2004: Tsinaraki et al., DS-MIRF model
      MPEG-7 fully represented in OWL-DL
      2005: Garcia and Celma, Rhizomik model
      Fully automatic translation of the whole standard
      MDS and Visual parts represented in OWL-DL
      2007: Arndt et al., COMM model
      Re-engineering MPEG-7 using DOLCE design patterns
      However, their requirements are not always clear nor have they been developed with clear methodological guidelines
    • 52
      Knowledge Resources
      Ontological Resources
      O. Design Patterns
      3
      4
      O. Repositories and Registries
      5
      6
      Flogic
      RDF(S)
      OWL
      Ontological Resource
      Reuse
      O. Aligning
      O. Merging
      5
      6
      2
      Ontology Design
      Pattern Reuse
      Non Ontological Resource
      Reuse
      4
      3
      6
      Non Ontological Resources
      2
      Ontological Resource
      Reengineering
      7
      Glossaries
      Dictionaries
      Lexicons
      5
      Non Ontological Resource
      Reengineering
      4
      6
      Classification
      Schemas
      Thesauri
      Taxonomies
      Alignments
      2
      RDF(S)
      1
      Flogic
      O. Conceptualization
      O. Implementation
      O. Formalization
      O. Specification
      Scheduling
      OWL
      8
      Ontology Restructuring
      (Pruning, Extension,
      Specialization, Modularization)
      9
      O. Localization
      1,2,3,4,5,6,7,8, 9
      Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation;
      Configuration Management; Evaluation (V&V); Assessment
      NeOnMethodology
    • 53
      Ontology Requirements Specification (I)
      • Non-functional ontology requirements:
      • Characteristics not related to the ontology content
    • 54
      Ontology Requirements Specification (II)
      • Functional ontology requirements:
      • Content specific requirements referred to the particular knowledge to be represented by the ontology
      • Requirements in natural language
      • in the form ofCQs
      • in the form of sentences (General Characteristics)
      • Strategies: (1) Top-Down, (2) Bottom-Up, and (3) Middle out
    • 55
      Ontology Requirements Specification (III): Functional Requirements on M3
      Perspectiva Multidominio
      Perspectiva Multimedia
      Perspectiva Multilenguaje
    • 56
      Ontology Requirements Specification (IV): ORSD
      Perspectiva Multidominio
      M3
      Perspectiva Multimedia
      Perspectiva Multilenguaje
    • 57
      Knowledge Resources
      Ontological Resources
      O. Design Patterns
      3
      4
      O. Repositories and Registries
      5
      6
      Flogic
      RDF(S)
      OWL
      Ontological Resource
      Reuse
      O. Aligning
      O. Merging
      5
      6
      2
      Ontology Design
      Pattern Reuse
      Non Ontological Resource
      Reuse
      4
      3
      6
      Non Ontological Resources
      2
      Ontological Resource
      Reengineering
      7
      Glossaries
      Dictionaries
      Lexicons
      5
      Non Ontological Resource
      Reengineering
      4
      6
      Classification
      Schemas
      Thesauri
      Taxonomies
      Alignments
      2
      RDF(S)
      1
      Flogic
      O. Conceptualization
      O. Implementation
      O. Formalization
      O. Specification
      Scheduling
      OWL
      8
      Ontology Restructuring
      (Pruning, Extension,
      Specialization, Modularization)
      9
      O. Localization
      1,2,3,4,5,6,7,8, 9
      Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation;
      Configuration Management; Evaluation (V&V); Assessment
      NeOnMethodology
    • 58
      Scheduling using gOntt
      gOntt
      Life cycle model selection
      I need to schedule the development of the M3 ontolgy network
      Scenarios selection
    • 59
      Scheduling using gOntt (II)
      gOntt
    • 60
      Knowledge Resources
      Ontological Resources
      O. Design Patterns
      3
      4
      O. Repositories and Registries
      5
      6
      Flogic
      RDF(S)
      OWL
      Ontological Resource
      Reuse
      O. Aligning
      O. Merging
      5
      6
      2
      Ontology Design
      Pattern Reuse
      Non Ontological Resource
      Reuse
      4
      3
      6
      Non Ontological Resources
      2
      Ontological Resource
      Reengineering
      7
      Glossaries
      Dictionaries
      Lexicons
      5
      Non Ontological Resource
      Reengineering
      4
      6
      Classification
      Schemas
      Thesauri
      Taxonomies
      Alignments
      2
      RDF(S)
      1
      Flogic
      O. Conceptualization
      O. Implementation
      O. Formalization
      O. Specification
      Scheduling
      OWL
      8
      Ontology Restructuring
      (Pruning, Extension,
      Specialization, Modularization)
      9
      O. Localization
      1,2,3,4,5,6,7,8, 9
      Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation;
      Configuration Management; Evaluation (V&V); Assessment
      NeOnMethodology
    • 61
      Reusing Ontological Resources: Comparative Analysis (I)
    • 62
      Reusing Ontological Resources: Comparative Analysis
    • Outline
      Introduction
      What I willbetalkingabout and what I willnot
      Sem-UPnP-Grid
      Sharing multimedia contentacrosshomesthroughsemanticannotations
      Sem4Tags
      Tagdisambiguation in Flickr
      M3 Ontology(onlyif time permits)
      A semanticbackboneforour multimedia-relatedwork
      Conclusions and outlook
      63
    • Conclusions and outlook
      Weallagreethat…
      Multimedia UGC has beenone of thebasis of Web2.0
      The use of semantics can provide…
      Betterunderstanding of thedomain and of theircontent
      Heavyweight: addressingthesemantic gap automatically
      Ligthweight: allowinguserstoannotate
      Middleweight: from free tagstoknowledge
      Betterexploratorynavigation and serendipity
      Interconnecting multimedia contentwiththeLinked Data cloud
      However, privacyissues are still a majorbarrierfor a largeruptake, especiallyforsomepopulationsegments
      Allowing P2P exchangebetween “known” homes, whileexploitingsemantic-basedsearch
      64
    • Combining Multimedia and Semantics
      Oscar Corcho (ocorcho@fi.upm.es)
      Universidad Politécnica de Madrid
      http://www.oeg-upm.net/
      LACNEM 2010, Cali, ColombiaSeptember 9th 2010
      Credits: Adrián Siles, Mariano Rico, Víctor Méndez, Hector Andrés García-Silva, María del Carmen Suárez-Figueroa, Ghislain Atemezing, Raphaël Troncy
      WorkdistributedunderthelicenseCreativeCommonsAttribution-Noncommercial-Share Alike3.0
      http://www.slideshare.net/ocorcho