This document discusses HTML5 interactive video, emphasizing features like overlays, track meta data, and error handling. It covers techniques for enhancing user engagement through navigation options, knowledge retention validation, and social networking incorporation. The document also addresses the use of text tracks and JavaScript for creating dynamic interactions and cues in videos.

1. INTERACTIVE
HTML5 <VIDEO>
November 20, 2013
Charles Hudson
chuckahudson+smw@gmail.com
HTML5 Interactive Video • November 20, 2013
1
42

2. CANVAS
OVERLAYS
CUES
TIMELINE
CANVAS AND VIDEO BINDING
DIGGING DEEP INTO CUES
TRACK META
META DATA TRACK USAGE
HTML5 Interactive Video • November 20, 2013
PLACING CONTROLS OVER
CONTROLLING TIME
SUBTITLES
LEVERAGING SUBTITLES
2
42

3. WHAT IS INTERACTION
FOR WHAT PURPOSE?
• Supplemental information on demand – hotspots, Ajax info.
• Enhanced navigation possibilities – menu, bookmarks, thumbnailing.
• Validating knowledge retention – inline quizzes.
• V-commerce – item in context viewing.
• Improved story telling opportunities.
• More engaging user experiences.
• Incorporation of social networking.
HTML5 Interactive Video • November 20, 2013
3
42

4. CROSS ORIGIN ERRORS
In the examples provided multiple files will be used
including text track files. These files must be served from
the same origin as the video media files to prevent
browser cross origin errors.
HTML5 Interactive Video • November 20, 2013
4
42

5. <VIDEO>
BASIS OF THE INTERACTIONS
HTML5 Interactive Video • November 20, 2013
5
42

6. HTML5 VIDEO ELEMENT
REFRESH
• Added in HTML5 to mainstream media video playback with out plugins.
• Built in controls available.
• Multiple sources to provide various browser support.
• Fallback messaging allowed for unsupported browsers.
<video controls>!
<source src="/video/big_buck_bunny_480p_surround.avi" type='video/mp4’>!
<source src="/video/big_buck_bunny_480p_stereo.ogg" type='video/ogg’>!
<p>HTML5 Video is not supported by this browser.</p>!
</video>!
HTML5 Interactive Video • November 20, 2013
6
42

7. DIGGING DEEP
GOING TO THE EDGE
HTML5 Video structures
are still young.
As we dig deeper this will
be more and more evident.
Present inconsistencies will
be noted.
Browsers of course have
various degrees of support
as we go deeper.
VIDEO
TRACKS
CUES
WATCH THAT LAST STEP…
HTML5 Interactive Video • November 20, 2013
7
42

8. OVERLAYING
ELEMENTS ON TOP OF VIDEO
HTML5 Interactive Video • November 20, 2013
8
42

9. OVERLAYING HTML ELEMENTS
BASIC ELEMENTS
• Leverage CSS position to place elements.
• Canvas an option as well.
• Provides for interactions
– While video loading.
– During playback or timeline navigation.
<div>
– When play ends.
• Can overlay entire video element or partial.
HTML5 Interactive Video • November 20, 2013
<video>
9
42

10. EXAMPLE: HOTSPOTS
Overlaying hotspots on a video.
HTML5 Interactive Video • November 20, 2013
10
42

11. TIMELINE
SYNCHRONIZING MANUAL EVENTS
HTML5 Interactive Video • November 20, 2013
11
42

12. VIDEO API EVENTS & PROPERTIES
SMALL BUT KEY SUBSET
• Key events
play
Video is played.
pause
Video playback is paused.
ended
Video playback has ended.
timeupdate
Video currentTime is updated.
• Properties Include
– currentTime – read/write of the playing pointer on the video media.
HTML5 Interactive Video • November 20, 2013
12
42

13. TIMELINE METHOD / EVENTS
A SAMPLE FLOW
Get currentTime
Returns the current pointer time
Set currentTime
Moves the play pointer
00:00:00:000
Ended event fired
autoplay
Play event fired
pause()
Pause event fired
play()
Play event fired
• Methods and current time fired automatically or manually.
HTML5 Interactive Video • November 20, 2013
13
42

14. TIMEUPDATE EVENT
SYNCHRONIZING WITH THE TIMELINE
• timeupdate fired when currentTime updated.
• By definition;
– Continually fired while media playing.
– Current media position adjusted.
– Frequency not specified or guaranteed.
• In practice, events fires several times per second.
video.addEventListener(’timeupdate', function() {!
console.log(“currentTime = “ + video.currentTime);!
}, false);!
HTML5 Interactive Video • November 20, 2013
14
42

15. EXAMPLE: TIMELINE CONTROL
A DVD style content menu controlling current time and
using overlay actions.
HTML5 Interactive Video • November 20, 2013
15
42

16. <TRACK>
TEXT TRACKS IN PARALLEL
HTML5 Interactive Video • November 20, 2013
16
42

17. TRACK ELEMENT OVERVIEW
SUBELEMENT TO MEDIA
• Track elements are added under the video element.
• Can have multiple tracks for a single video element.
<video controls>!
<source id='mp4' src="/video/trailer.mp4" type='video/mp4’>!
<track kind="subtitles" label="English" src="subtitles_en.vtt" srclang="en" default>!
<track kind="subtitles" label="Italiano" src="subtitles_it.vtt" srclang="it”>!
This browser does not support HTML5 video.!
</video>!
• Tracks accessible by element id attribute or textTracks array.
var allTracks = document.getElementById("sampleVideo").textTracks;!
HTML5 Interactive Video • November 20, 2013
17
42

18. TRACK ELEMENT
ELEMENT TAG
• Track element attributes
– Kind
The kind of track included.
– Src
The source file of the track.
– Srclang
Two letter designation for language.
– Label
User readable label for the text track.
– Default
Default text track designation for use.
• Can be referenced with the id attribute
• Multiple tracks with different kind attributes allowed.
• http://www.w3.org/wiki/HTML/Elements/track
HTML5 Interactive Video • November 20, 2013
18
42

19. 5 KINDS OF TEXT TRACKS
TYING INFORMATION TO VIDEO
SUBTITLES
transcription or
translation of the
dialogue, suitable for
when the soundtrack
is unavailable.
CAPTIONS
transcription or
translation of the
dialogue, sound
effects, relevant
musical cues, and
other relevant audio
information, suitable
for when the
soundtrack is
unavailable.
HTML5 Interactive Video • November 20, 2013
DESCRIPTIONS
CHAPTERS
METADATA
Textual descriptions
of the video
component of the
media resource,
intended for audio
synthesis when the
visual component is
unavailable.
Chapter titles,
intended to be used
for navigating the
media resource.
Tracks intended for
use from script.
19
42

20. TRACK FILES
WEB VIDEO TEXT TRACKS (VTT)
• Mark up layout for text track information.
• Each subtitle cue separated by a blank line.
• Closed captioning when on supersedes subtitles.
• W3C Specification for WebVTT http://dev.w3.org/html5/webvtt/.
WEBVTT FILE INCLUSION
Since the browser is including the WebVTT for the track the
WebVTT must be served from a web server the file cannot be read
from a file:// directive.
HTML5 Interactive Video • November 20, 2013
20
42

21. TRACK FILE FORMAT
VTT FILE STRUCTURE
• Header statement
• Individual Cues
– An identifier
– Start to end time
– Text for cue
WEBVTT FILE!
Subtitle1!
00:00:11.000 --> 00:00:14.000!
Cosa ti porta nella terra dei guardiani?!
Subtitle2!
00:00:18.000 --> 00:00:20.000!
Sto cercando qualcuno.!
Subtitle3!
00:00:36.000 --> 00:00:40.000!
Una pericolosa missione per un cacciatore solitario.!
CUE TIME FORMAT
Time format is critical to proper parsing of the VTT file. If a
start or end time is incorrectly structured such as 00:00:9:000
the behavior is indeterminate. Eg. Chrome will skip this cue.
HTML5 Interactive Video • November 20, 2013
21
42

22. SUBTITLE TRACKS
TIMELINE COORDINATION
• One or more cues handled automatically by html5 player.
• Cues contain an id, start time, end time, and text.
cue1
cue2
startTime 00:01:250
endTime 00:03:250
00:01:000
00:02:000
00:03:000
Cue1 text
displayed
HTML5 Interactive Video • November 20, 2013
startTime 00:05:000
endTime 00:08:000
00:04:000
00:05:000
00:06:000
00:07:000
00:08:000
00:09:000
Cue2 text
displayed
22
42

23. EXAMPLE: SIMPLE SUBTITLES
Localized subtitles for videos using subtitle tracks.
Testing via default attribute.
HTML5 Interactive Video • November 20, 2013
23
42

24. VIDEO LOADEDMETADATA EVENT
WHEN ARE TRACKS AVAILABLE
• Fired when track sources all loaded.
• Track sources are loaded asynchronously.
• Not included in the window load event.
• Must set separate event listener for loadedmetadata!
video.addEventListener('loadedmetadata', function () {
!
!
!// Perform functions on video here.!
}, false);!
HTML5 Interactive Video • November 20, 2013
24
42

25. TRACK MODE
SETTING TRACK VISIBILITY
• The track visibility is set through the mode property.
Disabled
Text track is disabled for the video element.
Hidden
Text track is hidden from being shown.
Showing
Text track is currently being used to show.
HTML5 Interactive Video • November 20, 2013
25
42

26. EXAMPLE: CHANGING TRACKS
Allowing the user to select their language of subtitles.
Leverages pulling all tracks, kind, and mode.
HTML5 Interactive Video • November 20, 2013
26
42

27. METADATA
USING METADATA FOR TRIGGERS
HTML5 Interactive Video • November 20, 2013
27
42

28. METADATA TRACKS
LEVERAGING JAVASCRIPT
• Metadata track cues not shown by video player.
• Many potential uses choreographing functionality.
<video>!
<source …>!
…!
<track kind="metadata" label=”Quiz Cues" src=“quiz-metadata-en.vtt" default>!
</video>!
METADATA TRACK LOADING
To have a metadata track loaded the default attribute must
be used in the <track> element.
HTML5 Interactive Video • November 20, 2013
28
42

29. METADATA TRACKS
JSON INSIDE CUES
• Text field can include complex data.
• Read like normal text property and then parsed.
Bubble1!
00:00:05.000 --> 00:00:09.000!
{!
"title":"Fingerless Gloves",!
"text":"Madonna made it fashionable to wear fingerless gloves,
also called Glovelettes, in the 1980s with the record cover
photo for Like a Virgin.",!
"xpos":"450px",!
"ypos":"350px”!
}!
• But first need to be able to grab each cue.
HTML5 Interactive Video • November 20, 2013
29
42

30. CUES FROM TRACKS
ACCESSING CUES
HTML5 Interactive Video • November 20, 2013
30
42

31. CUES
CUE OBJECT STRUCTURE
• Cues accessed as array fromtextTrack object.
• Properties of each cue can be accessed or modified.
for (var cueIdx = 0; cueIdx < textTrack.cues.length; cueIdx++) {!
console.log(”Cue id:” + textTrack.cues[cueIdx].id);!
…!
}!
id
The id of the specific cue.
startTime
Start time as defined by cue.
endTime
End time as defined by cue.
text
HTML5 Interactive Video • November 20, 2013
Text description of cue.
31
42

32. TRAPPING CUE EVENTS
TWO METHODS TO CATCH CUE CHANGES
• Cue events
– enter
!(onenter)
fired when a cue is entered.
– exit
!(onexit)
fired when a cue is exited.
• Text track cuechange (oncuechange).
– Fired whenever the list of active cues changes.
textTrack.oncuechange = function (){…};!
– oncuechange appears to be more reliable than cuechange!
HTML5 Interactive Video • November 20, 2013
32
42

33. TRAPPING CUE EVENTS
TWO METHODS TO CATCH CUE CHANGES
• activeCues array of one or more active cues.
• If activeCues length is zero when cuechange fired then exiting cue.
console.log(textTrack.activeCues[0].id);!
CUE EVENT RELIABILITY
Cue events are known to be problematic with inconsistent
behaviors (events not firing, events ceasing to fire).
HTML5 Interactive Video • November 20, 2013
33
42

34. EXAMPLE: BUBBLES
Using cue events and cuechange to effect the famous
music video pop up bubbles.
Fraught with trouble. Don’t look down and step away from the
edge.
HTML5 Interactive Video • November 20, 2013
34
42

35. EXAMPLE: BUBBLES NEW
Using what we know with timeline update events to create
a bubble solution.
HTML5 Interactive Video • November 20, 2013
35
42

36. TRACKS ON THE FLY
CREATING TRACKS AND CUES
• In JS can also add a new text track.
• addTextTrack takes the kind, label and option language code.
newTextTrack = myVideo.addTextTrack('chapters','bookmarks');!
newTextTrack.default = true;
!!
• New cues added with addCue(cue).
var newCue = new TextTrackCue(startTime, endTime, ’label');!
newCue.id = 'bookmark1’;!
newCue.pauseOnExit = false;!
newTextTrack.addCue(newCue);!
• Cues can also be removed with removeCue(cue)!
HTML5 Interactive Video • November 20, 2013
36
42

37. EXAMPLE: BOOKMARKS
Create cues dynamically and list out the cue list. Navigate
to bookmarks created by user.
HTML5 Interactive Video • November 20, 2013
37
42

38. COMPLEX INTERACTIONS
SHOWING VIDEO ON CANVAS ELEMENTS
• Take frame images from video and place on canvas.
• Video element is hidden while canvas is shown.
• Leverages
– canvas getContext(“2d”)
– drawImage, getImageData, and putImageData
Dynamic Content
Injection example
Image Copied
<video>
HTML5 Interactive Video • November 20, 2013
<canvas>
38
42

39. COMPLEX OVERLAYING
ADDING SPECIAL EFFECTS
• Timing challenge with direct processing.
• Frame buffer needed for holding frames from timer.
• http://www.craftymind.com/factory/html5video/CanvasVideo.html
• http://www.kaizou.org/2012/09/frame-by-frame-video-effects-using-html5and/
Copy taken
On timer
<video>
Hidden
HTML5 Interactive Video • November 20, 2013
Effect Added
Buffer
<canvas>
<canvas>
Shown
39
42

40. RESOURCES
VARIOUS HTML5 VIDEO RESOURCES
• Documentation
– Whatwg.org video element http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html
• 3rd Party Libraries to aid development (not used here)
– cuepoint.js
cuepoint.org
Jquery plugin for subtitles and creating cue points.
– popcorn.js
popcornjs.org
JS Media framework for easier time-based interaction.
– video.js
videojs.com
Open source html5 video player and library.
HTML5 Interactive Video • November 20, 2013
40
42

41. linkedin.com/in/chuckhudson
Code samples (will be posted)
https://github.com/cahudson/html5video
Q&A
Chuckahudson+smw@gmail.com
HTML5 Interactive Video • November 20, 2013
41
42

42. VIDEO CREDITS
• Thanks to the video providers
– Blender Foundation Open Source Projects http://www.blender.org/features/projects/
HTML5 Interactive Video • November 20, 2013
42
42