Making the HTML5 Video element interactive

13,052 views

Published on

From Streaming Media West Conference
Huntington Beach, CA
November 2013

C202: HOW TO: Making the HTML5 Video Element Interactive
The HTML5 Video element has now become widely used by browsers and supported in a broad set of websites for streaming video content. With some JavaScript and CSS, we can leverage the HTML video element to create highly interactive experiences for the viewer in both traditional and mobile browsing environments. This hands-on session explores the integration of events with the video timeline, creating positioned hotspots with links and dynamic content, and capturing user input. In addition, the session examines supporting mobile platform browsers along with future opportunities with the HTML5 video tag. Get sample code, ideas, and best practices for making the HTML5 video element an engaging interactive experience for your viewers.
Speaker: Chuck Hudson, Co-Author, HTML5 Developer's Cookbook

Published in: Technology
4 Comments
9 Likes
Statistics
Notes
No Downloads
Views
Total views
13,052
On SlideShare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
194
Comments
4
Likes
9
Embeds 0
No embeds

No notes for slide

Making the HTML5 Video element interactive

  1. 1. INTERACTIVE HTML5 <VIDEO> November 20, 2013 Charles Hudson chuckahudson+smw@gmail.com HTML5 Interactive Video • November 20, 2013 1 42
  2. 2. CANVAS OVERLAYS CUES TIMELINE CANVAS AND VIDEO BINDING DIGGING DEEP INTO CUES TRACK META META DATA TRACK USAGE HTML5 Interactive Video • November 20, 2013 PLACING CONTROLS OVER CONTROLLING TIME SUBTITLES LEVERAGING SUBTITLES 2 42
  3. 3. WHAT IS INTERACTION FOR WHAT PURPOSE? •  Supplemental information on demand – hotspots, Ajax info. •  Enhanced navigation possibilities – menu, bookmarks, thumbnailing. •  Validating knowledge retention – inline quizzes. •  V-commerce – item in context viewing. •  Improved story telling opportunities. •  More engaging user experiences. •  Incorporation of social networking. HTML5 Interactive Video • November 20, 2013 3 42
  4. 4. CROSS ORIGIN ERRORS In the examples provided multiple files will be used including text track files. These files must be served from the same origin as the video media files to prevent browser cross origin errors. HTML5 Interactive Video • November 20, 2013 4 42
  5. 5. <VIDEO> BASIS OF THE INTERACTIONS HTML5 Interactive Video • November 20, 2013 5 42
  6. 6. HTML5 VIDEO ELEMENT REFRESH •  Added in HTML5 to mainstream media video playback with out plugins. •  Built in controls available. •  Multiple sources to provide various browser support. •  Fallback messaging allowed for unsupported browsers. <video controls>! <source src="/video/big_buck_bunny_480p_surround.avi" type='video/mp4’>! <source src="/video/big_buck_bunny_480p_stereo.ogg" type='video/ogg’>! <p>HTML5 Video is not supported by this browser.</p>! </video>! HTML5 Interactive Video • November 20, 2013 6 42
  7. 7. DIGGING DEEP GOING TO THE EDGE HTML5 Video structures are still young. As we dig deeper this will be more and more evident. Present inconsistencies will be noted. Browsers of course have various degrees of support as we go deeper. VIDEO TRACKS CUES WATCH THAT LAST STEP… HTML5 Interactive Video • November 20, 2013 7 42
  8. 8. OVERLAYING ELEMENTS ON TOP OF VIDEO HTML5 Interactive Video • November 20, 2013 8 42
  9. 9. OVERLAYING HTML ELEMENTS BASIC ELEMENTS •  Leverage CSS position to place elements. •  Canvas an option as well. •  Provides for interactions –  While video loading. –  During playback or timeline navigation. <div> –  When play ends. •  Can overlay entire video element or partial. HTML5 Interactive Video • November 20, 2013 <video> 9 42
  10. 10. EXAMPLE: HOTSPOTS Overlaying hotspots on a video. HTML5 Interactive Video • November 20, 2013 10 42
  11. 11. TIMELINE SYNCHRONIZING MANUAL EVENTS HTML5 Interactive Video • November 20, 2013 11 42
  12. 12. VIDEO API EVENTS & PROPERTIES SMALL BUT KEY SUBSET •  Key events play Video is played. pause Video playback is paused. ended Video playback has ended. timeupdate Video currentTime is updated. •  Properties Include –  currentTime – read/write of the playing pointer on the video media. HTML5 Interactive Video • November 20, 2013 12 42
  13. 13. TIMELINE METHOD / EVENTS A SAMPLE FLOW Get currentTime Returns the current pointer time Set currentTime Moves the play pointer 00:00:00:000 Ended event fired autoplay Play event fired pause() Pause event fired play() Play event fired •  Methods and current time fired automatically or manually. HTML5 Interactive Video • November 20, 2013 13 42
  14. 14. TIMEUPDATE EVENT SYNCHRONIZING WITH THE TIMELINE •  timeupdate fired when currentTime updated. •  By definition; –  Continually fired while media playing. –  Current media position adjusted. –  Frequency not specified or guaranteed. •  In practice, events fires several times per second. video.addEventListener(’timeupdate', function() {! console.log(“currentTime = “ + video.currentTime);! }, false);! HTML5 Interactive Video • November 20, 2013 14 42
  15. 15. EXAMPLE: TIMELINE CONTROL A DVD style content menu controlling current time and using overlay actions. HTML5 Interactive Video • November 20, 2013 15 42
  16. 16. <TRACK> TEXT TRACKS IN PARALLEL HTML5 Interactive Video • November 20, 2013 16 42
  17. 17. TRACK ELEMENT OVERVIEW SUBELEMENT TO MEDIA •  Track elements are added under the video element. •  Can have multiple tracks for a single video element. <video controls>! <source id='mp4' src="/video/trailer.mp4" type='video/mp4’>! <track kind="subtitles" label="English" src="subtitles_en.vtt" srclang="en" default>! <track kind="subtitles" label="Italiano" src="subtitles_it.vtt" srclang="it”>! This browser does not support HTML5 video.! </video>! •  Tracks accessible by element id attribute or textTracks array. var allTracks = document.getElementById("sampleVideo").textTracks;! HTML5 Interactive Video • November 20, 2013 17 42
  18. 18. TRACK ELEMENT ELEMENT TAG •  Track element attributes –  Kind The kind of track included. –  Src The source file of the track. –  Srclang Two letter designation for language. –  Label User readable label for the text track. –  Default Default text track designation for use. •  Can be referenced with the id attribute •  Multiple tracks with different kind attributes allowed. •  http://www.w3.org/wiki/HTML/Elements/track HTML5 Interactive Video • November 20, 2013 18 42
  19. 19. 5 KINDS OF TEXT TRACKS TYING INFORMATION TO VIDEO SUBTITLES transcription or translation of the dialogue, suitable for when the soundtrack is unavailable. CAPTIONS transcription or translation of the dialogue, sound effects, relevant musical cues, and other relevant audio information, suitable for when the soundtrack is unavailable. HTML5 Interactive Video • November 20, 2013 DESCRIPTIONS CHAPTERS METADATA Textual descriptions of the video component of the media resource, intended for audio synthesis when the visual component is unavailable. Chapter titles, intended to be used for navigating the media resource. Tracks intended for use from script. 19 42
  20. 20. TRACK FILES WEB VIDEO TEXT TRACKS (VTT) •  Mark up layout for text track information. •  Each subtitle cue separated by a blank line. •  Closed captioning when on supersedes subtitles. •  W3C Specification for WebVTT http://dev.w3.org/html5/webvtt/. WEBVTT FILE INCLUSION Since the browser is including the WebVTT for the track the WebVTT must be served from a web server the file cannot be read from a file:// directive. HTML5 Interactive Video • November 20, 2013 20 42
  21. 21. TRACK FILE FORMAT VTT FILE STRUCTURE •  Header statement •  Individual Cues –  An identifier –  Start to end time –  Text for cue WEBVTT FILE! Subtitle1! 00:00:11.000 --> 00:00:14.000! Cosa ti porta nella terra dei guardiani?! Subtitle2! 00:00:18.000 --> 00:00:20.000! Sto cercando qualcuno.! Subtitle3! 00:00:36.000 --> 00:00:40.000! Una pericolosa missione per un cacciatore solitario.! CUE TIME FORMAT Time format is critical to proper parsing of the VTT file. If a start or end time is incorrectly structured such as 00:00:9:000 the behavior is indeterminate. Eg. Chrome will skip this cue. HTML5 Interactive Video • November 20, 2013 21 42
  22. 22. SUBTITLE TRACKS TIMELINE COORDINATION •  One or more cues handled automatically by html5 player. •  Cues contain an id, start time, end time, and text. cue1 cue2 startTime 00:01:250 endTime 00:03:250 00:01:000 00:02:000 00:03:000 Cue1 text displayed HTML5 Interactive Video • November 20, 2013 startTime 00:05:000 endTime 00:08:000 00:04:000 00:05:000 00:06:000 00:07:000 00:08:000 00:09:000 Cue2 text displayed 22 42
  23. 23. EXAMPLE: SIMPLE SUBTITLES Localized subtitles for videos using subtitle tracks. Testing via default attribute. HTML5 Interactive Video • November 20, 2013 23 42
  24. 24. VIDEO LOADEDMETADATA EVENT WHEN ARE TRACKS AVAILABLE •  Fired when track sources all loaded. •  Track sources are loaded asynchronously. •  Not included in the window load event. •  Must set separate event listener for loadedmetadata! video.addEventListener('loadedmetadata', function () { ! ! !// Perform functions on video here.! }, false);! HTML5 Interactive Video • November 20, 2013 24 42
  25. 25. TRACK MODE SETTING TRACK VISIBILITY •  The track visibility is set through the mode property. Disabled Text track is disabled for the video element. Hidden Text track is hidden from being shown. Showing Text track is currently being used to show. HTML5 Interactive Video • November 20, 2013 25 42
  26. 26. EXAMPLE: CHANGING TRACKS Allowing the user to select their language of subtitles. Leverages pulling all tracks, kind, and mode. HTML5 Interactive Video • November 20, 2013 26 42
  27. 27. METADATA USING METADATA FOR TRIGGERS HTML5 Interactive Video • November 20, 2013 27 42
  28. 28. METADATA TRACKS LEVERAGING JAVASCRIPT •  Metadata track cues not shown by video player. •  Many potential uses choreographing functionality. <video>! <source …>! …! <track kind="metadata" label=”Quiz Cues" src=“quiz-metadata-en.vtt" default>! </video>! METADATA TRACK LOADING To have a metadata track loaded the default attribute must be used in the <track> element. HTML5 Interactive Video • November 20, 2013 28 42
  29. 29. METADATA TRACKS JSON INSIDE CUES •  Text field can include complex data. •  Read like normal text property and then parsed. Bubble1! 00:00:05.000 --> 00:00:09.000! {! "title":"Fingerless Gloves",! "text":"Madonna made it fashionable to wear fingerless gloves, also called Glovelettes, in the 1980s with the record cover photo for Like a Virgin.",! "xpos":"450px",! "ypos":"350px”! }! •  But first need to be able to grab each cue. HTML5 Interactive Video • November 20, 2013 29 42
  30. 30. CUES FROM TRACKS ACCESSING CUES HTML5 Interactive Video • November 20, 2013 30 42
  31. 31. CUES CUE OBJECT STRUCTURE •  Cues accessed as array fromtextTrack object. •  Properties of each cue can be accessed or modified. for (var cueIdx = 0; cueIdx < textTrack.cues.length; cueIdx++) {! console.log(”Cue id:” + textTrack.cues[cueIdx].id);! …! }! id The id of the specific cue. startTime Start time as defined by cue. endTime End time as defined by cue. text HTML5 Interactive Video • November 20, 2013 Text description of cue. 31 42
  32. 32. TRAPPING CUE EVENTS TWO METHODS TO CATCH CUE CHANGES •  Cue events –  enter !(onenter) fired when a cue is entered. –  exit !(onexit) fired when a cue is exited. •  Text track cuechange (oncuechange). –  Fired whenever the list of active cues changes. textTrack.oncuechange = function (){…};! –  oncuechange appears to be more reliable than cuechange! HTML5 Interactive Video • November 20, 2013 32 42
  33. 33. TRAPPING CUE EVENTS TWO METHODS TO CATCH CUE CHANGES •  activeCues array of one or more active cues. •  If activeCues length is zero when cuechange fired then exiting cue. console.log(textTrack.activeCues[0].id);! CUE EVENT RELIABILITY Cue events are known to be problematic with inconsistent behaviors (events not firing, events ceasing to fire). HTML5 Interactive Video • November 20, 2013 33 42
  34. 34. EXAMPLE: BUBBLES Using cue events and cuechange to effect the famous music video pop up bubbles. Fraught with trouble. Don’t look down and step away from the edge. HTML5 Interactive Video • November 20, 2013 34 42
  35. 35. EXAMPLE: BUBBLES NEW Using what we know with timeline update events to create a bubble solution. HTML5 Interactive Video • November 20, 2013 35 42
  36. 36. TRACKS ON THE FLY CREATING TRACKS AND CUES •  In JS can also add a new text track. •  addTextTrack takes the kind, label and option language code. newTextTrack = myVideo.addTextTrack('chapters','bookmarks');! newTextTrack.default = true; !! •  New cues added with addCue(cue). var newCue = new TextTrackCue(startTime, endTime, ’label');! newCue.id = 'bookmark1’;! newCue.pauseOnExit = false;! newTextTrack.addCue(newCue);! •  Cues can also be removed with removeCue(cue)! HTML5 Interactive Video • November 20, 2013 36 42
  37. 37. EXAMPLE: BOOKMARKS Create cues dynamically and list out the cue list. Navigate to bookmarks created by user. HTML5 Interactive Video • November 20, 2013 37 42
  38. 38. COMPLEX INTERACTIONS SHOWING VIDEO ON CANVAS ELEMENTS •  Take frame images from video and place on canvas. •  Video element is hidden while canvas is shown. •  Leverages –  canvas getContext(“2d”) –  drawImage, getImageData, and putImageData Dynamic Content Injection example Image Copied <video> HTML5 Interactive Video • November 20, 2013 <canvas> 38 42
  39. 39. COMPLEX OVERLAYING ADDING SPECIAL EFFECTS •  Timing challenge with direct processing. •  Frame buffer needed for holding frames from timer. •  http://www.craftymind.com/factory/html5video/CanvasVideo.html •  http://www.kaizou.org/2012/09/frame-by-frame-video-effects-using-html5and/ Copy taken On timer <video> Hidden HTML5 Interactive Video • November 20, 2013 Effect Added Buffer <canvas> <canvas> Shown 39 42
  40. 40. RESOURCES VARIOUS HTML5 VIDEO RESOURCES •  Documentation –  Whatwg.org video element http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html •  3rd Party Libraries to aid development (not used here) –  cuepoint.js cuepoint.org Jquery plugin for subtitles and creating cue points. –  popcorn.js popcornjs.org JS Media framework for easier time-based interaction. –  video.js videojs.com Open source html5 video player and library. HTML5 Interactive Video • November 20, 2013 40 42
  41. 41. linkedin.com/in/chuckhudson Code samples (will be posted) https://github.com/cahudson/html5video Q&A Chuckahudson+smw@gmail.com HTML5 Interactive Video • November 20, 2013 41 42
  42. 42. VIDEO CREDITS •  Thanks to the video providers –  Blender Foundation Open Source Projects http://www.blender.org/features/projects/ HTML5 Interactive Video • November 20, 2013 42 42

×