Creating and Sharing Personalized Time-Based Annotations of Videos on the Web

Creating and Sharing
Personalized Time-Based Annotations
of Videos on the Web
Rodrigo Laiola Guimarães
Pablo Cesar
Dick C. A. Bulterman
CWI: Centrum Wiskunde & Informatica
Amsterdam, The Netherlands
DocEng 2010, Manchester-UK
21-24 September 2010

User B: and what the hell was that person
thinking that fell over?
Motivation
2

Supporting Comments on YouTube
3

3
base content

3
base content
navigation suggestion

3
base content
hyperlinks

3
Bottom line: end-users can
provide only a-temporal
text comments!
base content
hyperlinks

Our Approach
4
Authoring ViewingDocument Model

Our Approach
4
Video
URL
Create
Document
Timed-Text
Content
Temporal
Hyperlinks
Selective
Viewing
Contextual
Information

Our Approach
5
Video
URL
Create
Document
Timed-Text
Content
Temporal
Hyperlinks
Selective
Viewing
Contextual
Information

<smil ...>
<head ...> ... </head>
<body>
...
...
</body>
</smil>
<video region=”r1” id=”v1” src=”YouTube video”></video>
Document Model
6
Time
1t

<smil ...>
<head ...> ... </head>
<body>
...
...
</body>
</smil>
Document Model
6
Time
1t
i. Retain base video integrity

<smil ...>
<head ...> ... </head>
<body>
...
...
</body>
</smil>
<smil ...>
<head ...> ... </head>
<body>
...
<seq>
</seq>
...
</body>
</smil>
Document Model
6
Time
1t
t2
<video region=”r1” id=”v3” src=”Truveo video”></video>

<smil ...>
<head ...> ... </head>
<body>
...
...
</body>
</smil>
<smil ...>
<head ...> ... </head>
<body>
...
<seq>
</seq>
...
</body>
</smil>
Document Model
6
Time
1t
t2
ii. Allow multiple-video aggregation
iii. Allow multiple-provider integration

Our Approach
7
Video
URL
Create
Document
Timed-Text
Content
Temporal
Hyperlinks
Selective
Viewing
Contextual
Information

Document Transformations
•Timed-Text Content
8

<smil ...>
<head ...> ... </head>
<body>
...
<seq>
</seq>
...
</body>
</smil>
8
Time
t2
1t

<smil ...>
<head ...> ... </head>
<body>
...
<seq>
</seq>
...
</body>
</smil>
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
<smilText region=”c1”...>
...
...
</smilText>
</par>
</body>
</smil>
8
Time
t2
1t

<smil ...>
<head ...> ... </head>
<body>
...
<seq>
</seq>
...
</body>
</smil>
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
8
Time
t2
1t
Annotation
<clear begin="t1-5s"/>
you see someone falls in and another person...
<clear begin="t1+5s"/>

<smil ...>
<head ...> ... </head>
<body>
...
<seq>
</seq>
...
</body>
</smil>
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
8
Time
t2
1t
Annotation
it looks completely different from a different angle...

<smil ...>
<head ...> ... </head>
<body>
...
<seq>
</seq>
...
</body>
</smil>
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
8
Time
t2
1t
Annotation
iv. Allow timed end-user captions and
comments

Our Approach
9
Video
URL
Create
Document
Timed-Text
Content
Temporal
Hyperlinks
Selective
Viewing
Contextual
Information

•Temporal Hyperlinks
10

10
Time
t2
1t
Annotation
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>

10
Time
t2
1t
Annotation
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
<a href=”Jump to t2”>See the aircraft approaching</a>

10
Time
t2
1t
Annotation
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
<a href=”Another video”>Watch the pilot interview</a>

10
Time
t2
1t
Annotation
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
<a href=”Another video”>Watch the pilot interview</a>
v. Allow timed end-user navigation

Our Approach
11
Video
URL
Create
Document
Timed-Text
Content
Temporal
Hyperlinks
Selective
Viewing
Contextual
Information

•Contextual Information
12

12
Time
t2
1t
Annotation
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>

12
Time
t2
1t
Annotation
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
<smil ...>
...
<par>
<seq> ... </seq>
</smilText>
</par>
...
</smil>
<div>
<metadata name=”author” content=”Rodrigo”>
<metadata name=”audience” content=”all”>
</div>
<div>
<metadata name=”author” content=”Pablo”>
<metadata name=”audience” content=”Rodrigo”>
</div>

12
Time
t2
1t
Annotation
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
<smil ...>
...
<par>
<seq> ... </seq>
</smilText>
</par>
...
</smil>
<div>
</div>
<div>
</div>

12
Time
t2
1t
Annotation
<smil ...>
<head ...> ... </head>
<body>
<par>
<seq>
</seq>
...
...
</smilText>
</par>
</body>
</smil>
<smil ...>
...
<par>
<seq> ... </seq>
</smilText>
</par>
...
</smil>
<div>
</div>
<div>
</div>
vi. Allow micro-personalized time-based
annotations

Our Approach
13
Video
URL
Create
Document
Timed-Text
Content
Temporal
Hyperlinks
Selective
Viewing
Contextual
Information

<smil ...>
...
<par>
<seq> ... </seq>
</smilText>
</par>
...
</smil>
<div>
</div>
<div>
</div>
Selective Viewing
14
Time
t2
1t
Annotation

Ambulant Captioner
15
you see someone fall in and another person...

Ambulant Captioner
15
http://www.ambulantplayer.org/smilTextWebApp/
you see someone fall in and another person...

Predictive Timing Model
0s ttnow

0s ttnow
tend
tnow=

a)
0s ttnow
tend
tnow=t start
tguess=

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of phoneme

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of pauses
duration of phoneme

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of pauses
duration of phoneme
b)
tend
tnow=

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of pauses
duration of phoneme
b)
tend
tnow=t
guess

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of pauses
duration of phoneme
b)
tend
tnow=t
guess
tstart 0s=

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of pauses
duration of phoneme
b)
tend
tnow=t
guess
tstart 0s=
c)
tend
tnow=

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of pauses
duration of phoneme
b)
tend
tnow=t
guess
tstart 0s=
c)
tend
tnow=
t
end‘

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of pauses
duration of phoneme
b)
tend
tnow=t
guess
tstart 0s=
c)
tend
tnow=t
guess
t
end‘

a)
0s ttnow
tend
tnow=t start
tguess=
Number of words
duration of pauses
duration of phoneme
b)
tend
tnow=t
guess
tstart 0s=
c)
tend
tnow=t
guess
t
end‘
t
end‘tstart =

Summary
• Identification and description of a set of steps
that allow end-users to enrich/personalize
third-party video content
17

Summary
• Identification and description of a set of steps
that allow end-users to enrich/personalize
third-party video content
• Report on a predictive timing model for
synchronizing comments with specific events
within a video(s)
17

Conclusions and Future Work
• The contribution of this paper is not restricted to
the SMIL language
18

the SMIL language
• The aggregated annotations and metadata can
yield significant insights into the semantic and
temporal structure of media
18

the SMIL language
• We intend to conduct user studies to evaluate and
validate our approach
18

the SMIL language
• We intend to conduct user studies to evaluate and
validate our approach
• We intend to explore ownership, version control
and multiple-provider integration issues of the
various versions of a document as it is annotated
18

Questions?
19
Rodrigo Laiola Guimarães
rlaiola@cwi.nl
http://www.ambulantplayer.org/smilTextWebApp/

Creating and Sharing Personalized Time-Based Annotations of Videos on the Web

Recommended

Recommended

More Related Content

Similar to Creating and Sharing Personalized Time-Based Annotations of Videos on the Web

Similar to Creating and Sharing Personalized Time-Based Annotations of Videos on the Web (20)

More from Rodrigo Laiola Guimarães

More from Rodrigo Laiola Guimarães (8)

Recently uploaded

Recently uploaded (20)

Creating and Sharing Personalized Time-Based Annotations of Videos on the Web