Multimedia on the web.
            HTML5 Audio and Video




                    Christian Heilmann, MIT Boston, January 2...
I’m Chris.
What we will cover:
★   Quick history of Multimedia on the web
★   Annoyances with Flash
★   HTML5 audio and video
★   Pai...
Quick history of
Multimedia on the web
First, there were
images.

JPG, GIF, PNG (later),
WBMP, ICO
Connections were bad.

Progressive JPG,
interlaced GIF,
<img src=”foo.jpg”
     lowsrc=”foogrey.gif”
     alt=”a foo, what...
Animation was done
with animated GIFs or
with JavaScript
animation.
Audio was mostly Midi
 background music.




(and if you think about using that today I will hit you!)
Other things we did
were Java Applets.
And then
we got
Real
Player.
And many others...
★ Quicktime  / Quicktime VR
★ Microsoft Media Player
★ Shockwave
★ Acrobat had some image
  editing fea...
They all had the same
issues.
★ End users must install a
  plugin.
★ You need to upgrade the
  plugin constantly.
★ There ...
Another big issue is and
was security – plugins
are one of main attack
vectors for malware.
In the end, one
plugin prevailed
over all the
others: Flash.
With aggressive
marketing, clever
partnerships and a rich
dev environment Flash
became the multimedia
choice for the web.
And there is the DRM
stuff...
Annoyances with Flash
+
Apple and Flash movies means a lot of times your fan will start up.




                                     =
Why would I want yet
another editor to build
web content?
WTF is all this?
<object width="640" height="385">
<param name="movie" value="http://www.youtube.com/
v/SJixW2u4IvQ?fs=1&a...
Flash is a black box   Alien
inside a document.
Therefore it is also an
accessibility problem.
★ Other browsers than IE don’t allow you
  to access Flash with a keyboard....
Flash can talk to
JavaScript via APIs.
http://icant.co.uk/easy-youtube/
You are at the mercy of
the API developers
though – what they
don’t make public, you
can’t reach.
HTML5 audio and video
HTML5 audio and video
make things much
simpler:
★ Betteraccessibility
★ Doing one thing well
★ Much simpler API
★ Allows f...
Screencast of Oprah’s web site




http://www.oprah.com/own
Painful stuff – Codecs
and conversion
So, we understood that
there is a need for an
open high-fidelity
technology if we really
want the web to be a
media.
What we didn’t quite
consider yet is
something that both
accelerated and
hindered innovation for
a long time.
Copyright
Videos on the web are
encoded to make them
smaller – there is no
such things as an “AVI
File”...
They are containers for
video in a certain
compression and audio
in another.
The issue was that all
these compression
formats had their own
copyright and didn’t
work with open
technology like HTML5.
Therefore we needed
new formats:
★ H.264 “MP4”
★ OGG/Vorbis
★ VP8 / WebM
Our job is now to
convert the videos we
record to these open
formats – which can be
annoying.
The main annoyance is
that different browsers
support different
formats, which means
we need to provide
several versions :(
Audacity:
                   http://audacity.sourceforge.net/
WebM tools:
                http://www.webmproject.org/tools...
http://www.mirovideoconverter.com/
http://www.archive.org
Embedding
Embedding audio and
video in an HTML5
document is easy:
<audio src=”foo.ogg”>
	   If your browser didn’t suck, you’d have ...
That doesn’t do
anything yet
<video src=”foo.ogv” controls>
	   If your browser didn’t suck, you’d have video here.

</vid...
Controls differ from
browser to browser...

Firefox

Opera

Safari            Full Screen

Chrome
So to give it to all
browsers...
  <video controls>
    <source src="http://www.archive.org/{...}_512kb.mp4"
            t...
Things to consider:

★ Use  MP4 always as the first
  option to support iOS devices
  (iPads, iPhone and so on...)
★ If yo...
Media queries can save
bandwidth:
    <video controls>
     <source src="http://www.archive.org/{...}_512kb.mp4"
         ...
Other attributes:

★ poster – define a picture to
  show before loading.
★ height/width – oh, well...
★ loop – automatical...
But what if you don’t
like the controls?
Controlling
HTML5’s Media API
gives you control:
★   load() – load a new media.
★   canPlayType(type) – returns probably,
    maybe an...
Video details:
width / height / videoWidth / videoHeight / poster

Controls:
controls / volume / muted

Tracks:
tracks

Ne...
Writing a play button is
simple:
var audio = document.getElementsByTagName('audio')[0];
var play = document.getElementsByC...
However, simply
checking properties is
not safe!
Any JS use gets much
safer by listening to
events.
HTML5 video events:
loadstart / progress / suspend / abort /
error / emptied / stalled / play / pause /
loadedmetadata / l...
video.addEventListener('play', playEvent, false);
video.addEventListener('pause', pausedEvent, false);
video.addEventListe...
http://www.w3.org/2010/05/video/mediaevents.html
You can use these
events to sync
animation or trigger
changes with audio and
video.
http://isithackday.com/spirit-of-indiana/
Transforming




               http://www.flickr.com/photos/jiazi/1061447777
I am a film buff and I like to find easter eggs in
movies. Every pixar movie for example has A113
in it – the room in the ...
http://isithackday.com/mit/transforming-video.html
Using SVG and other
technologies you can
change the look and
feel of videos even
more.
http://people.mozilla.com/~prouget/demos/round/index.xhtml
http://people.mozilla.com/~prouget/demos/mashup/video.xhtml
Realtime changes
http://introducinghtml5.com/examples/ch05/animate-with-
                        video.html
http://people.mozilla.com/~prouget/demos/green/green.xhtml
http://people.mozilla.com/~prouget/demos/
    DynamicContentInjection/play.xhtml
http://people.mozilla.com/~prouget/demos/tracker/
                   tracker.xhtml
http://nerget.com/edgeDetection/
Awesome
audio
stuff
https://developer.mozilla.org/en/Introducing_the_Audio_API_Extension
https://wiki.mozilla.org/Audio_Data_API
http://hacks.mozilla.org/2011/01/html5guitar/
http://audioscene.org/scene-files/humph/sfxr/
Thanks!

Chris Heilmann
@codepo8
http://icant.co.uk
Multimedia on the web - HTML5 video and audio
Multimedia on the web - HTML5 video and audio
Multimedia on the web - HTML5 video and audio
Multimedia on the web - HTML5 video and audio
Multimedia on the web - HTML5 video and audio
Multimedia on the web - HTML5 video and audio
Upcoming SlideShare
Loading in...5
×

Multimedia on the web - HTML5 video and audio

8,408

Published on

A lecture given at MIT in Boston about the benefits and technicalities of open web standards for Video and Audio. Lots of examples how to manipulate live video using CSS3 and Canvas.

Published in: Education
3 Comments
17 Likes
Statistics
Notes
No Downloads
Views
Total Views
8,408
On Slideshare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
350
Comments
3
Likes
17
Embeds 0
No embeds

No notes for slide

Multimedia on the web - HTML5 video and audio

  1. 1. Multimedia on the web. HTML5 Audio and Video Christian Heilmann, MIT Boston, January 2012
  2. 2. I’m Chris.
  3. 3. What we will cover: ★ Quick history of Multimedia on the web ★ Annoyances with Flash ★ HTML5 audio and video ★ Painful stuff – codecs and conversion ★ Embedding ★ Controlling ★ Transforming ★ Realtime changes ★ Awesome audio stuff
  4. 4. Quick history of Multimedia on the web
  5. 5. First, there were images. JPG, GIF, PNG (later), WBMP, ICO
  6. 6. Connections were bad. Progressive JPG, interlaced GIF, <img src=”foo.jpg” lowsrc=”foogrey.gif” alt=”a foo, what else?”>
  7. 7. Animation was done with animated GIFs or with JavaScript animation.
  8. 8. Audio was mostly Midi background music. (and if you think about using that today I will hit you!)
  9. 9. Other things we did were Java Applets.
  10. 10. And then we got Real Player.
  11. 11. And many others... ★ Quicktime / Quicktime VR ★ Microsoft Media Player ★ Shockwave ★ Acrobat had some image editing features. ★ iPix / VRML and many other forgotten ones...
  12. 12. They all had the same issues. ★ End users must install a plugin. ★ You need to upgrade the plugin constantly. ★ There is limited interaction with the rest of the page.
  13. 13. Another big issue is and was security – plugins are one of main attack vectors for malware.
  14. 14. In the end, one plugin prevailed over all the others: Flash.
  15. 15. With aggressive marketing, clever partnerships and a rich dev environment Flash became the multimedia choice for the web.
  16. 16. And there is the DRM stuff...
  17. 17. Annoyances with Flash
  18. 18. + Apple and Flash movies means a lot of times your fan will start up. =
  19. 19. Why would I want yet another editor to build web content?
  20. 20. WTF is all this? <object width="640" height="385"> <param name="movie" value="http://www.youtube.com/ v/SJixW2u4IvQ?fs=1&amp;hl=en_US" /> <param name="allowFullScreen" value="true" /> <param name="allowscriptaccess" value="always" /> <embed src="http://www.youtube.com/v/SJixW2u4IvQ? fs=1&amp;hl=en_US" type="application/x-shockwave- flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385" /> </object>
  21. 21. Flash is a black box Alien inside a document.
  22. 22. Therefore it is also an accessibility problem. ★ Other browsers than IE don’t allow you to access Flash with a keyboard. ★ You are at the mercy of the developers to make their movies keyboard accessible. ★ Which is bad, as audio and video can help a lot of people to understand things.
  23. 23. Flash can talk to JavaScript via APIs.
  24. 24. http://icant.co.uk/easy-youtube/
  25. 25. You are at the mercy of the API developers though – what they don’t make public, you can’t reach.
  26. 26. HTML5 audio and video
  27. 27. HTML5 audio and video make things much simpler: ★ Betteraccessibility ★ Doing one thing well ★ Much simpler API ★ Allows for styling and overlays ★ View-source “hackable”
  28. 28. Screencast of Oprah’s web site http://www.oprah.com/own
  29. 29. Painful stuff – Codecs and conversion
  30. 30. So, we understood that there is a need for an open high-fidelity technology if we really want the web to be a media.
  31. 31. What we didn’t quite consider yet is something that both accelerated and hindered innovation for a long time.
  32. 32. Copyright
  33. 33. Videos on the web are encoded to make them smaller – there is no such things as an “AVI File”...
  34. 34. They are containers for video in a certain compression and audio in another.
  35. 35. The issue was that all these compression formats had their own copyright and didn’t work with open technology like HTML5.
  36. 36. Therefore we needed new formats: ★ H.264 “MP4” ★ OGG/Vorbis ★ VP8 / WebM
  37. 37. Our job is now to convert the videos we record to these open formats – which can be annoying.
  38. 38. The main annoyance is that different browsers support different formats, which means we need to provide several versions :(
  39. 39. Audacity: http://audacity.sourceforge.net/ WebM tools: http://www.webmproject.org/tools/ Evom: http://thelittleappfactory.com/evom/ VLC: http://www.videolan.org/vlc/ Ogg convert: http://oggconvert.tristanb.net/ Firefogg: http://firefogg.org/ TinyOgg: http://tinyogg.com/ ffmpeg: http://ffmpeg.org/
  40. 40. http://www.mirovideoconverter.com/
  41. 41. http://www.archive.org
  42. 42. Embedding
  43. 43. Embedding audio and video in an HTML5 document is easy: <audio src=”foo.ogg”> If your browser didn’t suck, you’d have audio here. </audio> <video src=”foo.ogv”> If your browser didn’t suck, you’d have video here. </video>
  44. 44. That doesn’t do anything yet <video src=”foo.ogv” controls> If your browser didn’t suck, you’d have video here. </video> ★ Controls appear on mouse hover or tabbing ★ Keyboard enabled controls ★ Video can be styled in any way.
  45. 45. Controls differ from browser to browser... Firefox Opera Safari Full Screen Chrome
  46. 46. So to give it to all browsers... <video controls> <source src="http://www.archive.org/{...}_512kb.mp4" type="video/mp4"></source> <source src="http://www.archive.org/{...}nsters.ogv" type="video/ogg"></source> <p>Your browser doesn't like HTML5 video, watch the movie on <a href="http://www.archive.org/{...}_monsters"> archive.org</a>. </p> </video>
  47. 47. Things to consider: ★ Use MP4 always as the first option to support iOS devices (iPads, iPhone and so on...) ★ If you omit the type attribute, the browser loads the meta data of each source!
  48. 48. Media queries can save bandwidth: <video controls> <source src="http://www.archive.org/{...}_512kb.mp4" type="video/mp4" media="(min-device-width:800px)"></source> <source src="http://www.archive.org/{...}_low.mp4" type="video/mp4"></source> <source src="http://www.archive.org/{...}nsters.ogv" type="video/ogg"></source> <p>watch the movie on <a href="{...}_monsters"> archive.org</a>.</p> </video>
  49. 49. Other attributes: ★ poster – define a picture to show before loading. ★ height/width – oh, well... ★ loop – automatically restart ★ preload (auto/none/ metadata)
  50. 50. But what if you don’t like the controls?
  51. 51. Controlling
  52. 52. HTML5’s Media API gives you control: ★ load() – load a new media. ★ canPlayType(type) – returns probably, maybe and “” (really!) ★ play() – play the movie ★ pause() – pause the movie. ★ addTrack(label,kind,language) -for subtitles
  53. 53. Video details: width / height / videoWidth / videoHeight / poster Controls: controls / volume / muted Tracks: tracks Network state: src / currentSrc / networkState / preload / buffered Ready state readyState / seeking Playback state currentTime / startTime / duration / paused / defaultPlayBackRate / playbackRate / played / seekable / ended / autoplay / loop
  54. 54. Writing a play button is simple: var audio = document.getElementsByTagName('audio')[0]; var play = document.getElementsByClassName('play')[0]; play.addEventListener('click',function(e){ var t = e.target; if(audio.paused){ audio.play(); t.innerHTML = 'pause'; } else { audio.pause(); t.innerHTML = 'play'; } e.preventDefault(); });
  55. 55. However, simply checking properties is not safe!
  56. 56. Any JS use gets much safer by listening to events.
  57. 57. HTML5 video events: loadstart / progress / suspend / abort / error / emptied / stalled / play / pause / loadedmetadata / loadeddate / waiting / playing / canplay / canplaythrough / seeking / seeked / timeupdate / ended / ratechange
  58. 58. video.addEventListener('play', playEvent, false); video.addEventListener('pause', pausedEvent, false); video.addEventListener('ended', function () { this.pause(); }, false); function playEvent() { play.innerHTML = 'pause'; } function pausedEvent() { play.innerHTML = 'play'; } play.onclick = function () { if (video.ended) { video.currentTime = 0; } if (video.paused){ video.play(); }else{ video.pause(); } };
  59. 59. http://www.w3.org/2010/05/video/mediaevents.html
  60. 60. You can use these events to sync animation or trigger changes with audio and video.
  61. 61. http://isithackday.com/spirit-of-indiana/
  62. 62. Transforming http://www.flickr.com/photos/jiazi/1061447777
  63. 63. I am a film buff and I like to find easter eggs in movies. Every pixar movie for example has A113 in it – the room in the uni a lot of pixar employees had their animation lectures in. To find things like that I sometimes zoom the screen and scan around. I thought this should be possible in HTML5.
  64. 64. http://isithackday.com/mit/transforming-video.html
  65. 65. Using SVG and other technologies you can change the look and feel of videos even more.
  66. 66. http://people.mozilla.com/~prouget/demos/round/index.xhtml
  67. 67. http://people.mozilla.com/~prouget/demos/mashup/video.xhtml
  68. 68. Realtime changes
  69. 69. http://introducinghtml5.com/examples/ch05/animate-with- video.html
  70. 70. http://people.mozilla.com/~prouget/demos/green/green.xhtml
  71. 71. http://people.mozilla.com/~prouget/demos/ DynamicContentInjection/play.xhtml
  72. 72. http://people.mozilla.com/~prouget/demos/tracker/ tracker.xhtml
  73. 73. http://nerget.com/edgeDetection/
  74. 74. Awesome audio stuff
  75. 75. https://developer.mozilla.org/en/Introducing_the_Audio_API_Extension
  76. 76. https://wiki.mozilla.org/Audio_Data_API
  77. 77. http://hacks.mozilla.org/2011/01/html5guitar/
  78. 78. http://audioscene.org/scene-files/humph/sfxr/
  79. 79. Thanks! Chris Heilmann @codepo8 http://icant.co.uk
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×