"Probably, Maybe, No: The State of HTML5 Audio" - Scott Schiller

  • 2,253 views
Uploaded on

A brief history of audio on the web, its present shape as of mid-2011, where things are going, and some examples of how you might use audio today on the web.

A brief history of audio on the web, its present shape as of mid-2011, where things are going, and some examples of how you might use audio today on the web.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
2,253
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
0
Comments
1
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. “ Probably, Maybe, No”: The State Of HTML5 Audio
  • 2. Hi.
    • Scott Schiller (@schill)
    • HTML Technician (front-end engineering)
    • Photography / DJing / carnivorous plant enthusiast
    • Y! Photos (2005-2007,) Flickr (present)
  • 3. The Point / Things I’m Going To Talk Aboot
  • 4. The Point / Things I’m Going To Talk Aboot
    • Where we:
    • ( were / are / going ) with web audio
  • 5.
    • How we got this far
    The Point / Things I’m Going To Talk Aboot
  • 6.
    • “ Show me something shiny ”
    The Point / Things I’m Going To Talk Aboot
  • 7.
    • Inform + inspire + excite -> go and make shiny things
    The Point / Things I’m Going To Talk Aboot
  • 8. A (largely) silent web
  • 9.
    • Sound is one of the senses.
  • 10.
    • Yet, it seems to be largely missing from the web.
  • 11.
    • WTF?
  • 12.
    • It’s complicated.
  • 13.
    • [ anti-audio rant here ]
  • 14. A Brief History of Web Audio
  • 15.
    • <bgsound=&quot;xfiles.mid&quot; controls=&quot;console&quot; loop=&quot;5&quot;>
    • <!-- inline player UI, often QuickTime -->
    A Brief History of Web Audio
  • 16.
    • <!-- autostart === auto-annoy, hidden evil -->
    • <embed src=&quot;hamsterdance.wav&quot; autostart=true loop=true volume=100 hidden=true>
    • <noembed>
    • <bgsound src=&quot;hamsterdance.wav&quot;>
    • </noembed>
    A Brief History of Web Audio
  • 17.
    • <!-- Adobe Flash embed code, 2011 -->
    • <OBJECT classid=&quot;clsid:D27CDB6E-AE6D-11cf-96B8-444553540000&quot; codebase=&quot;http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0&quot; WIDTH=&quot;550&quot; HEIGHT=&quot;400&quot; id=&quot;myMovieName&quot;>
    • <PARAM NAME=movie VALUE=&quot;flashSoundPlayer.swf&quot;>
    • <PARAM NAME=quality VALUE=high>
    • <PARAM NAME=bgcolor VALUE=#FFFFFF>
    • <EMBED href=&quot;flashSoundPlayer.swf&quot; quality=high bgcolor=#FFFFFF WIDTH=&quot;550&quot; HEIGHT=&quot;400&quot; NAME=&quot;myMovieName&quot; ALIGN=&quot;&quot; TYPE=&quot;application/x-shockwave-flash&quot; PLUGINSPAGE=&quot;http://www.macromedia.com/go/getflashplayer&quot;></EMBED>
    • </OBJECT>
    A Brief History of Web Audio
  • 18. The (Mis-)Truth Is Out There.
  • 19.
    • The scary part...
  • 20.
    • These are from search results. In 2011.
  • 21. <BGSOUND> + <EMBED> Limitations
  • 22.
    • Slow. Buggy. Inconsistent.
    <BGSOUND> + <EMBED> Limitations
  • 23.
    • <embed> loads a plugin/helper binary
    • (QT, RealPlayer, WMP)
    <BGSOUND> + <EMBED> Limitations
  • 24.
    • Parameters differ. loop=true vs. loop=5
    <BGSOUND> + <EMBED> Limitations
  • 25.
    • <embed> sometimes scriptable via JS. Little/no API docs per plugin. (+buggy.) *
    <BGSOUND> + <EMBED> Limitations * Excluding Flash 8+, documented and scriptable via ExternalInterface.
  • 26. Which brings us to…
  • 27.  
  • 28. Ah, the future. So shiny.
  • 29.
    • <audio src=&quot;yayhtml5.mp3&quot; controls>
    • new Audio('yayhtml5.mp3');
  • 30. So simple, too!
  • 31.  
  • 32. Browser format support is varied.
  • 33. (Shock, horror.)
  • 34. So it’s not that simple.
  • 35.
    • <audio src=&quot;html5.mp3&quot; controls>
    • <!– for non-HTML5 browsers… -->
    • Download <a href=&quot;html5.mp3&quot;>html5.mp3</a>.
    • </audio>
  • 36. Furthermore…
  • 37. No single format is supported in all major browsers.* * Not even .WAV, as of March 2011. Boourns.
  • 38. So it’s not really that simple.
  • 39. At least two formats are required.
  • 40.
    • <audio controls>
    • <source src=&quot;html5.mp3&quot; />
    • <source src=&quot;html5.ogg&quot; />
    • Download <a href=&quot;html5.mp3&quot;>html5.mp3</a> or <a href=&quot;html5.ogg&quot;>html5.ogg</a>
    • </audio>
  • 41. AND…
  • 42. Your client wants it to work in IE 6.
  • 43. (As with any new technology, there are always a few “buts”…)
  • 44. If only <audio> were as simple as <img> .
  • 45. This is unlikely, given technical, philosophical and legal complexities.
  • 46. If only <audio> were as simple as <img> . The A/V Codec Fight
  • 47. Format Wars
  • 48.
    • Mozilla + Google (OGG/WEB-M) vs. Apple + Microsoft (AAC/MP4/H.264)
    Format Wars
  • 49.
    • Web-M = “free”*, H.264 = IP/$$$
    Format Wars * Mumble mumble something about MPEG-LA threatening Web-M, allegedly violating IP, etc.
  • 50.
    • Developer headaches, 2+ encodings, confusion
    Format Wars
  • 51. And then there’s mobile.
  • 52. “ Special snowflake” devices
  • 53. Only one sound at a time
  • 54. *Under most circumstances. iOS 4.2 locks this down pretty well. No auto-play allowed*
  • 55. Pop-up blocker-style security model for Audio.play()
  • 56. o nfinish()-> play() allowed, otherwise, fail
  • 57. (As with any new technology, there are always a few “buts”…)
  • 58. So it’s really not that simple.
  • 59. Other Annoyances
  • 60. Occasional Safari/OS X audio load failure.
  • 61. &quot;Safari&quot; (Windows) = No HTML5 A/V support.
  • 62. Download &quot;Safari + QuickTime&quot; = HTML5 A/V.
  • 63. How is HTML5 Audio Implemented?
  • 64.
    • Multiple vendors targeting W3 spec
    How is HTML5 Audio Implemented?
  • 65.
    • &quot;Plug-in&quot; architecture (eg. Safari -> QT Plugin)
    How is HTML5 Audio Implemented?
  • 66.
    • Per spec, user should be able to install plugins
    How is HTML5 Audio Implemented?
  • 67.
    • XiphQT -> Safari (and iTunes) do OGG
    How is HTML5 Audio Implemented?
  • 68.
    • Chrome, Firefox /Mozilla include F/OSS-type libraries for playback
    How is HTML5 Audio Implemented?
  • 69.
    • IE 9: WMP /DirectAudio / ??
    How is HTML5 Audio Implemented?
  • 70. Potential vs. Reality
  • 71.
    • HTML5, like AJAX, is not “magical.” *
    Potential vs. Reality * Unless you’re in marketing, and then maybe it is.
  • 72.
    • The potential is there.
    Potential vs. Reality
  • 73.
    • We'll get there, eventually.
    Potential vs. Reality
  • 74.
    • For now, bugs and quirks will likely:
    Potential vs. Reality
  • 75.
    • For now, bugs and quirks will likely:
    • * Enhance job security
    Potential vs. Reality
  • 76.
    • For now, bugs and quirks will likely:
    • * Enhance job security
    • * Enhance grey hair
    Potential vs. Reality
  • 77. (As with any new technology, there are always a few “buts”…) [ enhance ]
  • 78. Can I Use HTML5 Audio Today?
  • 79. (Might as well) Go for it.
  • 80.  
  • 81. Use responsibly, tread carefully.
  • 82. HTML5 Audio via JavaScript
  • 83.
    • Audio().canPlayType() =
    • 'probably','maybe', 'no' *
    • The most un-computer-like response I’ve ever seen. Reflects the reality of fragmented format support.
    HTML5 Audio via JavaScript * Spec changed 'no' to '', to be a non-truthy return value.
  • 84.
    • Audio().canPlayType('audio/mp3'); Audio().canPlayType('audio/mpeg; codecs=&quot;mp3&quot;'); More detail should = “probably”
    HTML5 Audio via JavaScript
  • 85.
    • Audio can use HTTP partials, range requests. Arbitrary seek is a benefit.
    HTML5 Audio via JavaScript
  • 86.
    • A taste of events: canplay, load, ended, error, progress, timeupdate, waiting
    HTML5 Audio via JavaScript
  • 87. And How About Flash?
  • 88.
    • Single-vendor/platform consistency advantage
    And How About Flash?
  • 89.
    • Overall, best non-HTML5 fallback for MP3/MP4
    And How About Flash?
  • 90. Is Flash Dead Yet?
  • 91. No.
  • 92. http://isflashdeadyet.com/
  • 93. HTML5 Isn’t Going To Kill Flash?
  • 94. No.
  • 95. Flash remains a handy fallback for non-HTML5 UAs and is the de-facto method for “secure” RTMP streaming, etc.
  • 96. So HTML5 And Flash Can Co-Exist…
  • 97. CAN I USE BOTH AT THE SAME TIME TO PLAY AUDIO ON BOTH IE 6 AND THE iPHONE, DEPENDING ON WHICH TECHNOLOGY IS AVAILABLE?
  • 98. Exactly. :)
  • 99. This is why SoundManager 2 exists.
  • 100.
    • JavaScript API for sound
    SoundManager 2
  • 101.
    • HTML5 Audio() and/or Flash as needed (abstracted) ~10 KB (gzip)
    SoundManager 2
  • 102.
    • soundManager.onready(function() { soundManager.play( 'mySound', '/path/to/an.mp3’ );
    • }); JS talks to Flash (ExternalInterface) or HTML5 Audio(), depending on support. Transparent to developer.
    SoundManager 2
  • 103.
    • [ demos ]
    SoundManager 2
  • 104.
    • Why use an abstraction?
    SoundManager 2
  • 105.
    • Why use an abstraction? Support for both HTML4 + HTML5
    SoundManager 2
  • 106.
    • Why use an abstraction? Support for both HTML4 + HTML5 Normalize HTML5 differences
    SoundManager 2
  • 107. SO ULTIMATELY TO PLAY MP3/MP4 EVERYWHERE, YOU MUST HAVE FLASH?
  • 108. Correct. It’s either that, or split formats.
  • 109. THAT IS LAME!
  • 110. Unfortunately, yes. LAME is also a great free MP3 encoder. ;)
  • 111. Bonus Flash Bits
  • 112.
    • Flash audio API: Well-designed, overall
    Bonus Flash Bits
  • 113.
    • Revamped with Flash 8, extended in 9, moreso in 10
    Bonus Flash Bits
  • 114.
    • Some conveniences/extras beyond HTML5 audio
    Bonus Flash Bits
  • 115.
    • Audio metadata access (waveform/spectrum) = nifty visualizations http://flic.kr/p/9btbyE
    Bonus Flash Bits
  • 116. A Glimpse Into The Future
  • 117.
    • HTML5 audio: A good start
    A Glimpse Into The Future
  • 118.
    • Meets basics for sound apps (near-Flash parity)
    A Glimpse Into The Future
  • 119.
    • Pending: Bug finding, fixes, improvements and eventual stability
    A Glimpse Into The Future
  • 120. What’s Next?
  • 121. Mozilla Audio Data API / Webkit “Web Audio” API
  • 122.
    • Access/manipulate raw audio bytes during playback
    A Glimpse Into The Future: Mozilla/Webkit Audio APIs
  • 123.
    • Generate sound entirely from JavaScript
    A Glimpse Into The Future: Mozilla/Webkit Audio APIs
  • 124.
    • Mozilla released their API with Firefox 4 http://videos.mozilla.org/serv/blizzard/audio-slideshow/ https://wiki.mozilla.org/Audio_Data_API
    A Glimpse Into The Future: Mozilla/Webkit Audio APIs
  • 125.
    • Chrome 10+ beta with &quot;Web Audio&quot; enabled in about:flags http://chromium.googlecode.com/svn/trunk/samples/audio/o3d-webgl-samples/pool.html http://chromium.googlecode.com/svn/trunk/samples/audio/simple.html
    A Glimpse Into The Future: Mozilla/Webkit Audio APIs
  • 126. A Glimpse Into The Future: Mozilla/Webkit Audio APIs
    • [ demo? ]
  • 127. Show me the shiny.
  • 128. In late 2010, Technics announced that they were ceasing production of the DJ industry hallmark, the SL-1200 turntable.
  • 129. 1972 - 2010
  • 130. As a DJ / Web Developer, It seemed appropriate to take some sort of action.
  • 131. So I started looking at CSS, which does boxes quite well.
  • 132.  
  • 133. CSS3’s border-radius finally lets us draw circles.
  • 134.  
  • 135. We can also apply background:url() and overflow:hidden …
  • 136.  
  • 137. Mozilla and Webkit give us -moz-transform and –webkit-transform:rotate()
  • 138.  
  • 139.  
  • 140. <input type=“range” … />
  • 141. <input type=“crossfader”> ? ;)
  • 142. Prototype Goals
  • 143.
    • Recreate core turntable UI + behaviours in a browser.
    Prototype Goals
  • 144.
    • HTML5 audio should cover most sound requirements. Load, pause/play, seek etc.
    Prototype Goals
  • 145.
    • Hardware acceleration should keep it fast, too.
    Prototype Goals
  • 146.
    • Initially considered canvas, went with HTML + CSS. Layout advantage, fast render still possible.
    Prototype Findings
  • 147.
    • HTML5 audio works, despite browser quirks (and support issues.) The fix: Flash fallback.
    Prototype Findings
  • 148.
    • Heavily dependent on acceleration. transform:rotate3d() etc. Smooth in Webkit on OS X, some audio lag on Windows. Might be OK on IE 9. Acceleration is still “new.”
    Prototype Findings
  • 149.
    • [ demo ]
    Turntable Prototype
  • 150.
    • How about pitch bending, scratch, EQ effects etc.? (This is where Flash 10 / Audio Data APIs come in.)
    Turntable Prototype
  • 151.
    • [ demo ]
    Turntable Prototype
  • 152.
    • [ demo ]
    Turntable Prototype Layout/box model mode (RGBA outline and background colors)
  • 153.
    • [ demo ]
    Turntable Prototype For fun, an alternate skin (background-color, etc.)
  • 154. The (Near) End
  • 155. In Closing
    • HTML5: Tons of potential.
    • Bugs to be found + fixed.
    • X-platform/device support.
    • Audio Data APIs = shiny.
  • 156. In Closing
    • Use abstractions as needed to get similar experiences in more places.
    • My hope is that SoundManager 2 becomes redundant over time.
  • 157. “ You see, HTML5 audio is like a shark…”
  • 158.  
  • 159. In Closing Creative Commons-Licensed Photos &quot;Butts&quot; by arbyreed - http://flic.kr/p/jvCCh &quot;Untitled&quot; (facepalm) by Zack Sheppard - http://flic.kr/p/7gwXNt &quot;20100216_0082_1600x1067&quot; (hockey photo) by Les_Stockton - http://flic.kr/p/7DL6NU &quot;Snow Phone&quot; by WKeown - http://flic.kr/p/7Ef7Aa &quot;Safari&quot; by aditza121 - http://flic.kr/p/k7kkq &quot;Holly (and shadow) mid jump&quot; by Ali Smiles :) - http://flic.kr/p/5ZKWKY &quot;I can't believe this illustration had no caption&quot; by Ross Harmes - http://flic.kr/p/9jmjyt &quot;Water Slide&quot; by Ross Harmes - http://flic.kr/p/9jJRB7 fin.