Combining Multimedia and Semantics (LACNEM2010)
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Combining Multimedia and Semantics (LACNEM2010)

on

  • 1,967 views

Invited talk given at LACNEM2010 (Cali, Colombia), in September 2010, on the combination of semantics and multimedia.

Invited talk given at LACNEM2010 (Cali, Colombia), in September 2010, on the combination of semantics and multimedia.

Statistics

Views

Total Views
1,967
Views on SlideShare
1,963
Embed Views
4

Actions

Likes
2
Downloads
41
Comments
0

1 Embed 4

http://www.linkedin.com 4

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Combining Multimedia and Semantics (LACNEM2010) Presentation Transcript

  • 1. Combining Multimedia and Semantics
    Oscar Corcho (ocorcho@fi.upm.es)
    Universidad Politécnica de Madrid
    http://www.oeg-upm.net/
    LACNEM 2010, Cali, ColombiaSeptember 9th 2010
    Credits: Adrián Siles, Mariano Rico, Víctor Méndez, Hector Andrés García-Silva, María del Carmen Suárez-Figueroa, Ghislain Atemezing, Raphaël Troncy
    WorkdistributedunderthelicenseCreativeCommonsAttribution-Noncommercial-Share Alike3.0
    http://www.slideshare.net/ocorcho
  • 2. 2
    Asunción Gómez Pérez
    OntologyEngineering Group. Whomwe are
    Director: A. Gómez-Pérez
    Research Group (37 people)
    2 Full Professor
    4 AssociateProfessors
    1 AssistantProfessor
    3 Postdocs
    17 PhD Students
    8 MScStudents
    2 Software Engineers
    Management (4 people)
    2 Project Managers
    1 SystemAdministrator
    1 Secretary
    50+ PastCollaborators
    10+ visitors
  • 3. Research Areas
    2004
    2008
    1995
    1997
    2000
  • 4. Beforewestart…
    Howmany of youhaveeverheardabouttheword “Ontology”?
    And howmany of you do actuallyknowwhatitmeans?
    4
  • 5. Comingtotermswithontologies and semantics
    An ontology is an engineering artifact, which provides:
    A vocabulary of terms
    A set of explicit assumptions regarding the intended meaning of the vocabulary.
    Almost always including concepts and their classification
    Almost always including properties between concepts
    Shared understanding of a domain of interest
    Agreement on the meaning of terms
    Formal and machine manipulable model of a domain of interest
    Besides...
    The meaning (semantics) of such terms is formally specified
    New terms can be formed by combining existing ones
    Can also specify relationships between terms in multiple ontologies
    5
  • 6. Example: Anontologyaboutsatellites
    6
  • 7. Outline
    Introduction
    What I willbetalkingabout and what I willnot…
    Therewereseveraloptionsthat I exploredbeforeselectingtheonethatyouwillbehearing in thistalk…
    7
  • 8. Option 1: The Semantic Gap
    The lack of coincidencebetweentheinformationthatone can extractfromthesensory data and theinterpretationthatthesame data has for a user in a givensituation
    8
    A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain: Content-based image retrieval at the end of the early years, IEEE PAMI, 1349–1380, 2000.
    However, I alreadyassumedthatEbroulwouldbetalking a lotaboutit in hisopeningkeynote (as he did).
    Besides, I havenotworked at allonthelow-levelpart, so itmaybedifficultfor me toprovideyouwith a goodinsightonthe (many) open problems in thisarea
  • 9. Option 2: MPEG-7 and the Semantic Web
    ISO standard since December 2001
    Main components:
    Descriptors (Ds) and Description Schemes (DSs)
    DDL (XML Schema + extensions)
    Concern all types of media
    A good number of ontologies developed around it
    9
    Part 5 – MDSMultimedia Description Schemes
  • 10. Option 2: MPEG-7 and the Semantic Web
    However, thetalkmay:
    Be a bit boring and tootechnical
    Maylackthemix of state of the art and visionthataninvitedtalkshouldnormallyhave
    And MPEG-7 isnotusedtoomuch
    …So I willcoveronlysomeaspects of thislater, when I talkabout multimedia ontologies.
    10
  • 11. Option 3: Canonical Processes of Media Production (and semantics, obviously)
    For example….
    http://www.cewe-photobook.com
    Application for authoring digital photo books
    Automatic selection, sorting and ordering of photos
    Context analysis methods: timestamp, annotation, etc.
    Content analysis methods: color histograms, edge detection, etc.
    Customized layout and background
    Print by the European leader photo finisher company
    11
    Credits: Raphaël Troncy, LyndaHardman
  • 12. CeWe Color PhotoBook Processes
    My winter ski holidays with my friends
    Credits: Raphaël Troncy, LyndaHardman
  • 13. CeWe Color PhotoBook Processes
    Credits: Raphaël Troncy, LyndaHardman
  • 14. CeWe Color PhotoBook Processes
    Credits: Raphaël Troncy, LyndaHardman
  • 15. CeWe Color PhotoBook Processes
    Credits: Raphaël Troncy, LyndaHardman
  • 16. CeWe Color PhotoBook Processes
    Credits: Raphaël Troncy, LyndaHardman
  • 17. CeWe Color PhotoBook Processes
    Credits: Raphaël Troncy, LyndaHardman
  • 18. Semantics can be important in the process
    Credits: Raphaël Troncy, LyndaHardman
  • 19. Option 3: Canonical Processes of Media Production
    However, some of youprobablyattended Raphaël Troncy’stalklastyear (available in slideshare)
    19
  • 20. In summary…
    I decidedtotalkaboutsomethingthat I havebeenworking in forthelastcouple of years, and which combines
    Semantics (of course, thisisthekeyexpertise of ourgroup)
    Mainlyannotation, Linked Data and a bit of Multimedia OntologyEngineering
    Social networks, collaboration, sharing and collectiveintelligence
    Exploiting home networks and online multimedia sites
    And, obviously, multimedia
    And hence I stillleaveoutmanyinterestingtopics (e.g., semanticsin user interfaces)
    20
  • 21. Outline
    Introduction
    What I willbetalkingabout and what I willnot
    Sem-UPnP-Grid
    Sharing multimedia contentacrosshomesthroughsemanticannotations
    Credits: Mariano Rico and Adrián Siles (UPM), Víctor Méndez and José Manuel Gómez-Pérez (iSOCO), José Manuel Palacios and Mónica Pérez (TID)
    Sem4Tags
    Tagdisambiguation in Flickr
    M3 Ontology(onlyif time permits)
    A semanticbackboneforour multimedia-relatedwork
    Conclusions and outlook
    21
  • 22. Internet
    Motivation
    Multimedia resources in Web2.0 are stored in centralised servers.
    You lose some of yourrights as anauthorwhenyouuploadtheseresourcestothese servers.
    Privacyproblems.
    Poorannotations and metadata.
    Theseresourcescannotbesharedwithotherresources in your home.
    22
    UpGrid
  • 23. Multimedia Content SharingwithUpGrid
    23
    Annotation:
    “Ángel onthebeach”
    Reasoning:
    • “Ángel is my son”
    • 24. “Pedro is my brother”
    • 25. “Juan is my brother”
    • 26. -------------------------------
    • 27. Ángel is my nephew
    Juan
    P2P
    Semantic-basedquery:
    “multimedia contentrelatedto my nephew”
    Annotation:
    “Ángel playing soccer”
    Pedro
    Additionalsemanticinformation:
    • “Ángel is my son”
    • 28. “Pedro is my brother”
    Additionalsemanticinformation:
    “Juan is my brother”
  • 29. Architecture
  • 30. Architecture (anotherviewonit)
  • 31. Snapshotsfromtheapplication
    Checkhttp://www.youtube.com/results?search_query=UPnPGrid
  • 32. Summary
    Aneffectivemeansforsharing multimedia contentsacrosshomes, avoiding Web2.0 siteswhereyourrightsmaybecompromised
    However, itisstill a prototype, and no serioususabilitytesting has been done
    Muchworkstillneeded in ordertogointo a real system
    And endusersfinditdifficulttoprovideannotations
    Do you imagine yourparents and grandparentsannotatingphotos and videos likethat?
    Let’sseehowthiscouldbeamelioratedwiththenextpart of ourpresentation.
    27
  • 33. Outline
    Introduction
    What I willbetalkingabout and what I willnot
    Sem-UPnP-Grid
    Sharing multimedia contentacrosshomesthroughsemanticannotations
    Sem4Tags
    Tagdisambiguation in Flickr
    Credits: Héctor Andrés García Silva
    M3 Ontology(onlyif time permits)
    A semanticbackboneforour multimedia-relatedwork
    Conclusions and outlook
    28
    Egresado de laUniversidad del Valle
  • 34. Introduction
    Social Tagging Systems
    Web 2.0 applications
    Applications for storing, sharing, and discovering information resources.
    Users assign tagsto identify information resources
    Tags are used to search/discover resources
    29
  • 35. Introduction
    Folksonomy
    Emerging classification scheme from social tagging systems
    Folk: People, Taxonomy: Classification
    Represented by: Users, Tags, Resources
    Taxonomy
    Folksonomy
    • Top-down
    • 36. Controlled Vocabulary
    • 37. Hierarchical structure
    • 38. Exclusive/Restrictive
    • 39. Expensive to maintain
    • 40. Bottom-up (user created)
    • 41. No fixed vocabulary
    • 42. No Hierarchical structure
    • 43. No Exclusive/Flexible
    • 44. Low cost
    30
  • 45. Introduction
    Why is tagging so popular?
    Reduce cognitive burdens
    it’s easy to use
    Users don´t need any special skill or experience
    The benefits of tagging are immediate
    Future retrieval
    Contribution and sharing
    Attract Attention
    Self Presentation
    Opinion Expression
    31
  • 46. Introduction
    However
    Tags can be ambiguous
    Polysemy: partyas a celebration as opposed to partyas a political organization
    Synonym: party and celebration 
    Morphological variations:
    party, parties, partying, partyign
    Plurals
    Acronyms
    Conjugated verbs
    Misspelling
    Compound words
    Political party, PoliticalParty, Political_party,
    Political-Party, etc.
    Detail/granularity level
    A general tag as partyin contrast to a specific tag as banquet.
    32
  • 47. Motivation
    The problem: Morphological variations, synonyms, granularity, and polysemy hamper information retrieval processes based on folksonomies.
    Systems ignore resources tagged with morphological variationsor synonyms of that tag, as well as the resources tagged with more generic or more specific tags
    710.659 results
    8.661.581 Results
    33
  • 48. When searching with polysemous tags, all the resources tagged with that tag are retrieved without taking into account the tag sense the user was looking for.
    (e.g., Query flickr with bank results in photos about financial institutions, river edges, fog banks, and sand banks, etc. )
    34
    Motivation
  • 49. Motivation
    What if we associate tags with semantic entities?
    http://morpheus.cs.umbc.edu/aks1/ontosem.owl 
    #non-work-activity
    We can avoid the aforementioned pitfalls
    #organization
    #special-occasion
    #political-entity
    #party
    #Celebration
    #political-party
    #Coalition
    #federation
    #Birthday
    #Anniversary
    uk, tories, party, conservative, speech
    party, balloons, colors, bar, crowd
    35
  • 50. State of the Art: Semantic Grounding of Cross-Lingual Folksonomies
    Garcia HA, Corcho O, Alani H, Gómez-Pérez A. Review of the state of the art: Discovering and Associating Semantics to Folksonomies. Knowledge Engineering Review (in press)
    None of the analyzed approaches deals with multilingual tags
    36
  • 51. Semantic Grounding of Cross-Lingual Folksonomies
    MSR: a Multilingual Sense Repository based on Wikipedia and enriched with semantic information taken from DBpedia.
    Terms and
    frequency
    Banco
    Bank
    http://dbpedia.org/resource/Bank
    Terms and
    frequency
    Banco
    Cardumen
    Swarm
    http://dbpedia.org/resource/Swarm
    Banco de
    Arena
    http://dbpedia.org/resource/
    SandBank
    Terms and
    frequency
    Sandbank
    37
  • 52. Semantic Grounding of Cross-Lingual Folksonomies
    Sem4Tags: A process for Associating Semantics to Tags.
    38
    Dinero,
    Calle,
    Santander,
    Money,
    Madrid,
    Atm,
    cajero
    Europe
    Euro
    Finance
    Central bank
    awesomePic
    Nikon ..
    Bank
    Banco
    http://dbpedia.org/resource/Bank
  • 53. Semantic Grounding of Cross-Lingual Folksonomies
    Disambiguation activity
    The candidate senses and the tag context are represented as vectors.
    The vector components are the set of most frequent terms in each Wikipedia page representing a sense.
    For each sense the values of the vector are calculated using TF-IDF.
    For the tag context the values in each position are 1 or 0 if the corresponding term appears in the tag context.
    The tag context vector is compared against each sense vector using the cosine of the angle as similarity measure.
    The most similar sense to the tag context is selected as the one representing the meaning of the analyzed tag
    39
    39
  • 54. Semantic Grounding of Cross-Lingual Folksonomies
    Disambiguation activity
    We use the information of the wikipedia default sense for a term.
    Sim(TagContext, Sensei)= λ*Cosine + β*defaultSense
    We experimentally defined β = 0,2 and λ = 0.8
    We attempt to use DBpedia semantic information in the disambiguation activity:
    Sim(TagContext, Sensei)= λ*Cosine + β*defaultSense + δ*SemanticInfo
    Studies have shown that tags in flickr refers mainly to: Locations, Time, Given Names, Potography related subjects among others.
    We use DBpedia and YAGO relations to classify the senses according to this categories.
    However, we found that not all the senses related to a term have the same amount of relations. (e.g. Madrid is not a city)
    40
  • 55. Let’s try it
    http://robinson.dia.fi.upm.es:8080/SemanticTagsWebApp/index.jsp
    Whatdoes “bernabeu” mean ifitscontextis…?
    estadio, madrid, fútbol
    41
  • 56. Experiment
    Baseline: Directly associate tags with DBpedia resources
    Look for spaces and replace them with ' _‘.
    For tags in English:
    Create a URI of the form http://en.wikipedia.org/wiki/tag
    Query DBpedia using the http://xmlns.com/foaf/0.1/page relation
    For tags in Spanish:
    Create a URI of the form http://es.wikipedia.org/wiki/tag
    Query DBpedia using the http://dbpedia.org/property/wikipage-es relation
    42
  • 57. Experiment
    Approaches:
    Baseline: Selection of the sense without a disambiguation activity.
    Sem4Tags: For each sense we use the whole Wikipedia article as source for frequentterms.
    Sem4TagsAC: Same as Sem4Tags including the selection of the Active Context.
    Sem4TagsAbs: For each sense we use the Wikipedia article abstract (extracted from DBpedia) as source for frequent terms.
    Sem4TagsAbsAC: Same as Sem4TagsAbs including the selection of the Active Context.
    43
  • 58. Experiment
    Initial Data Set
    Wide range of Users, photos, and tags.
    764 photos uploaded by 719 users to Flickr that have been tagged with tags describing tourist places in Spain
    12.4 (+/- 7.85) tags per photo
    9484 tagging activities (TAS) : <user,photo,tag>
    4135 distinct tags where used
    Processed Data Set
    From each photo we processed on average 2 tags
    2260 taggingactivities (TAS)
    44
  • 59. Experiment
    Evaluation Campaign
    41 Evaluators
    Evaluate semantic associations produce by each approach: <user; tag; photo; DBpedia resource; language>
    Three different evaluators evaluated each semantic association.
    Questions:
    Able to identify the tag meaning (known or Unknown)
    Tag language (English, Spanish, Both, other)
    The tag correspond to a Named entity
    According to the identified tag language they evaluate the semantic association in terms of
    Highly related, Related, Not Related.
    45
  • 60. Experiment
    Results
    Evaluators identified the semantics of the 87% of TAS (known)
    62.6 % of TAS were considered in English
    87.7% of TAS were considered in Spanish
    Agreement among evaluators (Fleiss’ kappa statistics):
    k=0.76 for highly related
    K=0.71 for the related case/highly related case
    46
  • 61. Experiment
    Precision and RecallforHighlyRelevantresults
    47
    English
    Spanish
  • 62. Experiment
    Conclusions
    Baseline obtained high precision, however it was able to find semantic resources for just a fraction of the analyzed data set:
    Baseline: 27.7% in English and 19.4% in Spanish.
    Sem4Tags: 79.1 % in English and 81.4% in Spanish
    All approaches obtained better precision with named entities than with unnamed entities.
    Sem4Tags and Sem4TagsAC are the approaches that obtained the best results in terms of Precision and Recall.
    Sometimes Sem4TagsAC obtains better P@1 values but the improvements are supported by no or low statistical evidence.
    Sem4TagsAbs and Sem4TagsAbs are clearly the worst approaches.
    48
  • 63. Outline
    Introduction
    What I willbetalkingabout and what I willnot
    Sem-UPnP-Grid
    Sharing multimedia contentacrosshomesthroughsemanticannotations
    Sem4Tags
    Tagdisambiguation in Flickr
    M3 Ontology(onlyif time permits)
    A semanticbackboneforour multimedia-relatedwork
    Conclusions and outlook
    49
  • 64. 50
    Ontología M3
  • 65. There are already multimedia ontologies
    MDS Upper Layer represented in RDFS
    2001: Hunter
    Later on: link to the ABC upper ontology
    MDS fully represented in OWL-DL
    2004: Tsinaraki et al., DS-MIRF model
    MPEG-7 fully represented in OWL-DL
    2005: Garcia and Celma, Rhizomik model
    Fully automatic translation of the whole standard
    MDS and Visual parts represented in OWL-DL
    2007: Arndt et al., COMM model
    Re-engineering MPEG-7 using DOLCE design patterns
    However, their requirements are not always clear nor have they been developed with clear methodological guidelines
  • 66. 52
    Knowledge Resources
    Ontological Resources
    O. Design Patterns
    3
    4
    O. Repositories and Registries
    5
    6
    Flogic
    RDF(S)
    OWL
    Ontological Resource
    Reuse
    O. Aligning
    O. Merging
    5
    6
    2
    Ontology Design
    Pattern Reuse
    Non Ontological Resource
    Reuse
    4
    3
    6
    Non Ontological Resources
    2
    Ontological Resource
    Reengineering
    7
    Glossaries
    Dictionaries
    Lexicons
    5
    Non Ontological Resource
    Reengineering
    4
    6
    Classification
    Schemas
    Thesauri
    Taxonomies
    Alignments
    2
    RDF(S)
    1
    Flogic
    O. Conceptualization
    O. Implementation
    O. Formalization
    O. Specification
    Scheduling
    OWL
    8
    Ontology Restructuring
    (Pruning, Extension,
    Specialization, Modularization)
    9
    O. Localization
    1,2,3,4,5,6,7,8, 9
    Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation;
    Configuration Management; Evaluation (V&V); Assessment
    NeOnMethodology
  • 67. 53
    Ontology Requirements Specification (I)
    • Non-functional ontology requirements:
    • 68. Characteristics not related to the ontology content
  • 54
    Ontology Requirements Specification (II)
    • Functional ontology requirements:
    • 69. Content specific requirements referred to the particular knowledge to be represented by the ontology
    • 70. Requirements in natural language
    • 71. in the form ofCQs
    • 72. in the form of sentences (General Characteristics)
    • 73. Strategies: (1) Top-Down, (2) Bottom-Up, and (3) Middle out
  • 55
    Ontology Requirements Specification (III): Functional Requirements on M3
    Perspectiva Multidominio
    Perspectiva Multimedia
    Perspectiva Multilenguaje
  • 74. 56
    Ontology Requirements Specification (IV): ORSD
    Perspectiva Multidominio
    M3
    Perspectiva Multimedia
    Perspectiva Multilenguaje
  • 75. 57
    Knowledge Resources
    Ontological Resources
    O. Design Patterns
    3
    4
    O. Repositories and Registries
    5
    6
    Flogic
    RDF(S)
    OWL
    Ontological Resource
    Reuse
    O. Aligning
    O. Merging
    5
    6
    2
    Ontology Design
    Pattern Reuse
    Non Ontological Resource
    Reuse
    4
    3
    6
    Non Ontological Resources
    2
    Ontological Resource
    Reengineering
    7
    Glossaries
    Dictionaries
    Lexicons
    5
    Non Ontological Resource
    Reengineering
    4
    6
    Classification
    Schemas
    Thesauri
    Taxonomies
    Alignments
    2
    RDF(S)
    1
    Flogic
    O. Conceptualization
    O. Implementation
    O. Formalization
    O. Specification
    Scheduling
    OWL
    8
    Ontology Restructuring
    (Pruning, Extension,
    Specialization, Modularization)
    9
    O. Localization
    1,2,3,4,5,6,7,8, 9
    Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation;
    Configuration Management; Evaluation (V&V); Assessment
    NeOnMethodology
  • 76. 58
    Scheduling using gOntt
    gOntt
    Life cycle model selection
    I need to schedule the development of the M3 ontolgy network
    Scenarios selection
  • 77. 59
    Scheduling using gOntt (II)
    gOntt
  • 78. 60
    Knowledge Resources
    Ontological Resources
    O. Design Patterns
    3
    4
    O. Repositories and Registries
    5
    6
    Flogic
    RDF(S)
    OWL
    Ontological Resource
    Reuse
    O. Aligning
    O. Merging
    5
    6
    2
    Ontology Design
    Pattern Reuse
    Non Ontological Resource
    Reuse
    4
    3
    6
    Non Ontological Resources
    2
    Ontological Resource
    Reengineering
    7
    Glossaries
    Dictionaries
    Lexicons
    5
    Non Ontological Resource
    Reengineering
    4
    6
    Classification
    Schemas
    Thesauri
    Taxonomies
    Alignments
    2
    RDF(S)
    1
    Flogic
    O. Conceptualization
    O. Implementation
    O. Formalization
    O. Specification
    Scheduling
    OWL
    8
    Ontology Restructuring
    (Pruning, Extension,
    Specialization, Modularization)
    9
    O. Localization
    1,2,3,4,5,6,7,8, 9
    Ontology Support Activities: Knowledge Acquisition (Elicitation); Documentation;
    Configuration Management; Evaluation (V&V); Assessment
    NeOnMethodology
  • 79. 61
    Reusing Ontological Resources: Comparative Analysis (I)
  • 80. 62
    Reusing Ontological Resources: Comparative Analysis
  • 81. Outline
    Introduction
    What I willbetalkingabout and what I willnot
    Sem-UPnP-Grid
    Sharing multimedia contentacrosshomesthroughsemanticannotations
    Sem4Tags
    Tagdisambiguation in Flickr
    M3 Ontology(onlyif time permits)
    A semanticbackboneforour multimedia-relatedwork
    Conclusions and outlook
    63
  • 82. Conclusions and outlook
    Weallagreethat…
    Multimedia UGC has beenone of thebasis of Web2.0
    The use of semantics can provide…
    Betterunderstanding of thedomain and of theircontent
    Heavyweight: addressingthesemantic gap automatically
    Ligthweight: allowinguserstoannotate
    Middleweight: from free tagstoknowledge
    Betterexploratorynavigation and serendipity
    Interconnecting multimedia contentwiththeLinked Data cloud
    However, privacyissues are still a majorbarrierfor a largeruptake, especiallyforsomepopulationsegments
    Allowing P2P exchangebetween “known” homes, whileexploitingsemantic-basedsearch
    64
  • 83. Combining Multimedia and Semantics
    Oscar Corcho (ocorcho@fi.upm.es)
    Universidad Politécnica de Madrid
    http://www.oeg-upm.net/
    LACNEM 2010, Cali, ColombiaSeptember 9th 2010
    Credits: Adrián Siles, Mariano Rico, Víctor Méndez, Hector Andrés García-Silva, María del Carmen Suárez-Figueroa, Ghislain Atemezing, Raphaël Troncy
    WorkdistributedunderthelicenseCreativeCommonsAttribution-Noncommercial-Share Alike3.0
    http://www.slideshare.net/ocorcho