Keynote at the 4th International Workshop on Linked Media (LIME) http://www.mico-project.eu/pages/lime-workshop-2016/

1. Linking media, data,
and services
Ruben Verborgh
Ghent University – iMinds

5. The problem
The solution
The problems

6. The problem
The solution
The problems

9. How can semantic technologies
assist with multimedia analysis?

13. analysis
algorithms
Linked
Data
planning
knowledge
bases
reasoning

14. goal-driven
data-driven

15. • subrequest 1
…
• subrequest n Engine
EXIFJPEG
metadata
JPEG
inputrequest

16. The problem
The solution
The problems

17. multimedia
data
+
metadata
Supervisor
requested
metadata
description
Blackboard
requested
metadata
description
gathered
metadata
Collaborator 3
semantic
capability
description
semantic
requirements
description
Collaborator 2
semantic
capability
description
semantic
requirements
description
Collaborator 1
semantic
capability
description
semantic
requirements
description
multimedia
data
+
metadata
r
requested
metadata
description
metadata
Blackboard
requested
metadata
description
gathered
metadata
n
1
tic
ity
ion
multimedia
data
+
metadata
requested
metadata
description
metadata
Blackboard
requested
metadata
description
gathered
metadata

18. multimedia
data
+
metadata
Supervisor
requested
metadata
description
Blackboard
requested
metadata
description
gathered
metadata
Collaborator 3
semantic
capability
description
semantic
requirements
description
Collaborator 2
semantic
capability
description
semantic
requirements
description
Collaborator 1
semantic
capability
description
semantic
requirements
description
multimedia
data
+
metadata
r
requested
metadata
description
metadata
Blackboard
requested
metadata
description
gathered
metadata
c
y
on
r 1
ntic
bility
ption
multimedia
data
+
metadata
visor
requested
metadata
description
metadata
Blackboard
requested
metadata
description
gathered
metadata
or3
mantic
ability
cription
tor2
semantic
capability
description
rator1
semantic
capability
description

19. query request
input
Collaborator
output
result
2.
3.
5.
1.
4.

20. Collaboratorinput output
Toolkit

21. Collaborator
Wrapper
input output

23. start

24. <Loft.jpg> a foaf:Image.
<Loft.jpg> foaf:depicts ?person.
input
goal

25. composition

26. {
?image a foaf:Image.
}
=>
{
?regionList a rdf:List.
_:region owl:oneOf ?regionList;
rdfs:subClassOf face:FaceRegion,
[ owl:hasValue ?image;
owl:onProperty imreg:regionOf ].
}
service rule

27. {
?region a face:FaceRegion.
}
=>
{
?region imreg:regionDepicts ?face.
?face a face:Face;
face:isFaceOf ?person.
?person a foaf:Person.
}.
service rule

28. {
?image imreg:region ?region.
?region imreg:regionDepicts ?face.
?face face:isFaceOf ?person.
}
=>
{
?image foaf:depicts ?person.
}.
knowledge rule

29. Example 7.8 Composition from image input to person request
8
>>>>>>><
>>>>>>>:
Iæ := (param1 = ) √ ( )
IFD := (regionList = ) √ (image = )
IFR := (f ace = , per son = )
√ (region = )
I≤ := ( ) √ (per son = )
Iæ and I≤ are the virtual start and end services, respectively.
Iæ IFD IFR I≤
62

30. execution

31. <Loft.jpg#xywh=45,121,51,51> a face:FaceRegion;
imreg:regionOf <Loft.jpg>.
<Loft.jpg#xywh=221,91,56,56> a face:FaceRegion;
imreg:regionOf <Loft.jpg>.
<Loft.jpg#xywh=535,118,43,43> a face:FaceRegion;
imreg:regionOf <Loft.jpg>.
<Loft.jpg#xywh=734,83,69,69> a face:FaceRegion;
imreg:regionOf <Loft.jpg>.

33. Example 7.8 Composition from image input to person request
8
>>>>>>><
>>>>>>>:
Iæ := (param1 = ) √ ( )
IFD := (regionList = ) √ (image = )
IFR := (f ace = , per son = )
√ (region = )
I≤ := ( ) √ (per son = )
Iæ and I≤ are the virtual start and end services, respectively.
Iæ IFD IFR I≤
62

35. _:face owl-s-sparql:bindsVariable "face";
owl-s-sparql:boundTo
[face:isFaceOf dbpedia:Koen_De_Bouw].
_:person owl-s-sparql:bindsVariable "person";
owl-s-sparql:boundTo dbpedia:Koen_De_Bouw.
<Loft.jpg#xywh=221,91,56,56>

36. dbpedia:Koen_De_Bouw ???
??? dbpedia:Bruno_Vanden_Broucke

37. failure

38. alternative
plan

39. CONSTRUCT WHERE {
dbpedia:Koen_De_Bouw ?p ?o.
}
CONSTRUCT WHERE {
?s ?p dbpedia:Koen_De_Bouw.
}

40. dbpedia:Koen_De_Bouw a dbpedia-owl:Actor;
rdfs:label "Koen De Bouw"@nl.
dbpedia:Bruno_Vanden_Broucke a dbpedia-owl:Actor;
rdfs:label "Bruno Vanden Broucke"@nl.
!
dbpedia:Loft_%282008_film%29
dbpprop:director dbpedia:Erik_Van_Looy;
dbpprop:name "Loft"@nl;
dbpprop:producer dbpedia:Woestijnvis;
dbpprop:starring dbpedia:Bruno_Vanden_Broucke,
dbpedia:Filip_Peeters,
dbpedia:Jan_Decleir,
dbpedia:Koen_De_Bouw,
dbpedia:Koen_De_Graeve,
dbpedia:Matthias_Schoenaerts,
dbpedia:Veerle_Baetens.

41. {
?image foaf:depicts ?person
?person foaf:knows ?acquaintance.
}
=>
{
?image face:maybeDepicts ?acquaintance.
}.
knowledge rule

42. recomposition

43. acquaintances to the parameter. Note that the actual process is slightly more
complex, but some details were omitted for brevity.
Example 7.14 Adapted composition, recovering from the face recognition failure
8
>>>>>>><
>>>>>>>:
...
I0
æ := (param1 = ) √ ( )
I0
FR := (f ace = , per son = )
√ (region = ,
candidates = )
Iæ IFD IFR I≤
I0
æ I0
FR
67

44. {
?image imreg:region ?regionA, ?regionB.
?regionA owl:differentFrom ?regionB.
?regionA imreg:regionDepicts [face:isFaceOf ?personA].
?regionB imreg:regionDepicts [face:isFaceOf ?personB].
}
=>
{
?personA owl:differentFrom ?personB.
}.
knowledge rule

45. execution

46. dbpedia:Koen_De_Bouw
dbpedia:Bruno_Vanden_Broucke
dbpedia:Matthias_Schoenaerts
dbpedia:Bruno_Vanden_Broucke
dbpedia:Koen_De_Graeve

47. _:face owl-s-sparql:bindsVariable "face";
owl-s-sparql:boundTo [face:isFaceOf ?person].
_:person owl-s-sparql:bindsVariable "person";
owl-s-sparql:boundTo ?person.
({?person = dbpedia:Koen_De_Graeve.}
{?person = dbpedia:Bruno_Vanden_Broucke.})
e:disjunction [a e:T].
<Loft.jpg#xywh=45,121,51,51>

48. solution

49. dbpedia:Koen_De_Bouw dbpedia:Matthias_Schoenaerts
dbpedia:Koen_De_Graeve dbpedia:Bruno_Vanden_Broucke

51. The problem
The solution
The problems

52. ἀπὸ μηχανῆς θεός

53. deus ex machina

54. machina ex deo
reasoner

55. What are the research questions?
!
How can semantic technologies
assist with multimedia analysis?

56. What are the hypotheses?
!
This solution does analysis
better than… what?

57. How to evaluate this solution?
!
Is our solution
a good one or a bad one?

58. How can this solution fail?
!
Can (existing) knowledge
always fill the gap?

59. The problem
The solution
The problems

61. Can we link images like we link data?
Can we have a Linked Media Cloud?

62. Linking media, data,
and services
@RubenVerborgh
ruben.verborgh.org