This document summarizes new sound and music APIs available on the web, including Web Audio, Web MIDI, and WebRTC. It provides an overview and demos of each API, as well as lessons on how to use them, including generating and manipulating sound with Web Audio, querying and sending MIDI messages with Web MIDI, and getting audio from microphones with WebRTC. The document emphasizes that these new capabilities allow musical instruments and audio hardware to be used directly on the web in real-time.

1. Web & Sound
Playing with Sound APIs recently introduced to the Web

2. Overview + Demo

3. New Sound APIs
◇ Web Audio (http://webaudio.github.io/web-audio-api/)
￭ Real-time audio synthesis
￮ cf. SuperCollider, Pure Data, Max/MSP
￭ SFX (Pan, Delay, Filter, etc)
◇ Web MIDI (http://webaudio.github.io/web-midi-api/)
￭ Real-time communication
￮ Web ⇄ Musical Instruments
● Synths, DJ and lighting controllers
◇ WebRTC (http://w3c.github.io/webrtc-pc/)
￭ Camera & Microphone

4. Web Audio Demo
◇ Web Audio Playground (http://webaudioplayground.appspot.com/)
￭ Straight-forward GUI wrapper
◇ WebPd (https://github.com/sebpiq/WebPd)
￭ Pure Data clone
◇ CoffeeCollider (https://github.com/mohayonao/CoffeeCollider)
￭ Coffee Script + SuperCollider
◇ T’SS JS edition (https://github.com/toyoshim/tss)
￭ Chiptune sound emulation (demo, tsdsp)

5. MIDI is old, and new
◇ 30th anniversary
◇ MIDI is not only SMF but a protocol
￭ SMF: Standard MIDI File
￮ Protocol dump file with timestamps
￭ Real-time communication - lighting and more
￮ 冨田勲×初音ミク「イーハトーヴ」@ YouTube
◇ De facto for music production tools
￭ Software Synth in DAW: Audio Unit, VSTi, DXi
￭ Hardware: DIN, USB, FireWire, RTP, Bluetooth(*)
￮ (*): supported by Yosemite, Windows 10, Android M

6. Web MIDI Demo
◇ Physical controller for online sound systems
￭ Web Audio Synth v2 (http://aikelab.net/websynthv2/)
￭ Web FM Synthesizer (http://www.taktech.org/takm/WebFMSynth/)
￭ Online DAW: audiotool (http://www.audiotool.com/)
￭ Online VJ: MajVj (GGT 2014 demo)
◇ Demo and Productions by YAMAHA
￭ Pocket Miku (Product page & Web App)
￭ Web Music Platform (toyoshim’s fork)
￭ Reface + Soundmondo (Product page)
￮ Social to share your sound and music via Web MIDI

7. WebRTC
◇ microphone
￭ getUserMedia({audio:true}, …)
￮ Can connect to Web Audio via
createMediaStreamSource()
◇ camera
￭ getUserMedia({video: true}... )
￮ Can use it as a texture for WebGL

8. Web Apps be Secure!
◇ Hardware is hard to use securely
￭ What happens on WebGL?
￮ Need GPU process and GPU command validator
￭ Recent APIs to use hardware
￮ Permission prompt (infobar/bubble UI)
￮ Secure origin, e.g. https or localhost, is mandatory
￭ Real P0 issue happened on Web MIDI
￮ Microsoft GS Wavetable SW Synth crashes
● on receiving a special valid sequence
● outside Chrome sandbox, inside the driver

9. How to learn
◇ Web Audio
￭ Getting Started with Web Audio API html5rocks
￭ Web Audio API Ryoya Kawai
￭ Web Audio API 解説 g200kg
◇ Web MIDI
￭ Web MIDI API (x-webmidi編) Ryoya Kawai
￭ ブラウザで電子楽器を作ってみよう HTML5 Experts
￭ 最高に乱暴なWeb MIDI API利用方法 tadfmac

10. Lesson: Web Audio 1
◇ Simple sine wave
var audio = new AudioContext();
var osc = audio.createOscillator();
osc.frequency.value = 440; // in Hz
osc.connect(audio.destination);
osc.start(0);

11. Lesson: Web Audio 2
◇ Other nodes
￭ Audio source
￮ AudioBufferSourceNode (from Audio resources, e.g., mp3, m4a)
￮ MediaElementAudioSourceNode (from DOM media object)
￮ MediaStreamSource (from microphone via getUserMedia)
￮ ScriptProcessorNode (deprecated) => AudioWorkerNodeProcessor (from JS)
￮ OscillatorNode (from various kinds of oscillators)
￭ Effect
￮ GainNode
￮ DelayNode
￮ (Stereo)PannerNode / AudioListener
￮ ConvolverNode (e.g., Vocoder, Reverb, and so on)
￮ DynamicsCompressorNode / WaveShaperNode
￮ BuquadFilterNode (e.g., LPF, HPF, BPF, and so on)
￭ Misc
￮ AnalyserNode (FFT)
￮ ChannelSplitterNode / ChannelMergerNode (Mixer)

12. Lesson: Web MIDI 1
◇ Device Query
var promise = navigator.requestMIDIAccess();
promise.then(function(midi) {
for (var input of midi.inputs.values())
console.log(input); // MIDIInput
for (var output of midi.outputs.values())
console.log(output); // MIDIOutput
});

13. Lesson: Web MIDI 2
◇ MIDIInput
for (var input of midi.inputs.values())
input.onmidimessage = handler;
function handler (e) {
// Single MIDI message in Array.
console.log(e.data);
}

14. Lesson: Web MIDI 3
◇ MIDIOutput
for (var output of midi.outputs.values()) {
// Send single MIDI message or multiple
// MIDI messages.
output.send([0x90, 0x40, 0x7f]); // Note ON
}

15. Lesson: Promise
◇ New replacement for callbacks
foo(successCallback, errorCallback);
↓
foo().then(successCallback, errorCallback);
foo().then(bar)
.then(baz)
.catch(qux);
◇ Chaining
Promise.all([foo(),
bar()])
.then(baz, qux);
◇ Wait multiple async functions

16. Lesson: Maplike
◇ readonly maplike<DOMString, MIDIInput> inputs
◇ readonly maplike<DOMString, MIDIOutput> outputs
◇ keys() … MIDIPort.id is used as an unique key
◇ values() … MIDIInput / MIDIOutput
for (var input of inputs.values()) {
// New syntax for ES6 iterable
}

17. Lesson: Web MIDI 4
◇ Hotplug
midi.onstatechange = function (e) {
var newport = e.port;
if (newport.state == “connected”)
console.log(newport.name + “ is attached”;
else if (newport.state == “disconnected”)
console.log(newport.name + “ is detached”;
};

18. Fine ||

19. References
◇ Slide template
￭ from Slides Carnival
￮ http://www.slidescarnival.com/imogen-free-
presentation-template/399