WebRTC and the Web Audio API allow real-time communication and audio/video processing capabilities directly in the browser. WebRTC provides APIs for capturing media streams from devices and establishing peer-to-peer connections, while dealing with challenges like firewalls and NAT using ICE, STUN and TURN. The Web Audio API enables efficient audio stream manipulation through a processing block model. While building full-featured video chat applications with WebRTC alone can be difficult, these capabilities open opportunities for applications involving real-time media and audio processing directly in the browser.

1. WebRTC and Web Audio API
BY MATTEO AVALLE
Co-Founder & CTO Fluentify.com

2. OUTLINE
• Introduction
• WebRTC
– MediaStream
– RTCPeerConnection & RTCDataChannel
– ICE, STUN, TURN
– Browser support
• Alternatives
• “Abusing” MediaStreams
• Web Audio APIs
• Conclusions
WebRTC and Web Audio API
MATTEO AVALLE

3. Introduction
• How do we usually interact with websites?
– Mouse & Keyboard
– Plugin-based webcam & mic
– External apps
• Can we do better?
– Voice recognition
– Image processing
– Audio/Video chat integrated in webpages
WebRTC and Web Audio API
MATTEO AVALLE

4. WebRTC
• A JavaScript API set
– Audio/Video stream capture
– P2P Communication API
– Real time stream encoding/decoding
• Low level • Modular
WebRTC and Web Audio API
MATTEO AVALLE

5. WebRTC
• The three APIs:
– MediaStream
• Gets data from Webcam, Microphone, etc.
– RTCDataChannel
• Creates generic P2P connections
– RTCPeerConnection
• Handle complex procedures like:
– Audio/Video calls
– Encryption
– Bandwidth management
WebRTC and Web Audio API
MATTEO AVALLE

6. MediaStream
• Available in Chrome, Opera, Firefox, Edge
• Used to “connect” an audio/video stream to:
– A <video> / <canvas> tag
– A WebRTC stream
– ...
• Extremely simple
WebRTC and Web Audio API
MATTEO AVALLE

7. MediaStream: the Mirror Example
WebRTC and Web Audio API
MATTEO AVALLE

8. WebRTC and Web Audio API
MATTEO AVALLE
Note: examples are just
meant to be examples

9. RTCPeerConnection, RTCDataChannel
• Available in Chrome, Opera, Firefox
• They handle:
– P2P connection creation
– Data encoding/encrypion
– Data buffering
WebRTC and Web Audio API
MATTEO AVALLE

10. RTCPeerConnection, RTCDataChannel
WebRTC and Web Audio API
MATTEO AVALLE

11. Issues of P2P
• How do I know how many other guys are present?
• Who they are? (i.e. What are their IP addresses?)
• What if they are behind a NAT router (like 99% of home
users)?
• What if they are behind a firewall?
• What should I do if the connection drops/is bad?
WebRTC and Web Audio API
MATTEO AVALLE

12. ICE, STUN, TURN
• ICE/STUN are:
– “HACKS” to bypass the firewalls
– They use a server to create the initial connection
– The rest of the communication is P2P
• TURN:
– The last resort for too-protected networks
– All the traffic passes through a server
– No more P2P, but the most reliable
WebRTC and Web Audio API
MATTEO AVALLE

13. Browser compatibility
• Chrome, Opera and Firefox have all the APIs, but:
– They require different prefixes
– Certain behaviors are still different
• What about the other browsers?
WebRTC and Web Audio API
MATTEO AVALLE

14. WebRTC and Web Audio API
MATTEO AVALLE
HELP! THIS IS GETTING SO
COMPLICATED!

15. What can we do?
• Start on a journey to become a WebRTC master
• Use third-party libraries
– OpenTok
– EasyRTC
– PeerJS
– XirSys *
– simplewebrtc.com
– Hangouts APIs *
WebRTC and Web Audio API
MATTEO AVALLE

16. OpenTok
P2P? Relayed? You choose, they provide the servers
• DASH “for free” (in terms of coding)
• IE plugin available
• Real-Time performance monitor
• 100% customizable Javascript library
• BUT: it’s kinda expensive, both for small- scale and
large-scale applications
WebRTC and Web Audio API
MATTEO AVALLE

17. WebRTC and Web Audio API
MATTEO AVALLE
Creating video-chat
websites with WebRTC
alone is hard.
BUT THEN...
ARE WEBRTC APIS STILL USEFUL
FOR US TO KNOW?

18. “Abusing” WebRTC APIs
• We can redirect our Media Streams elsewhere
– <video>/<canvas> tags
– Filters (CSS/canvas)
– Background Javascript Workers
• We have a GPU-accelerated videoP codec encoder/
decoder
• We can develop P2P applications
WebRTC and Web Audio API
MATTEO AVALLE

19. “Abusing” WebRTC APIs:
the Filtered Mirror Example
WebRTC and Web Audio API
MATTEO AVALLE

20. Web Audio APIs
• An efficient API to manipulate audio streams
• Most common “slow” functions are hardcoded in the
browser (e.g. FFTransform), you just use them
• “Processing blocks” coding style
WebRTC and Web Audio API
MATTEO AVALLE

21. Web Audio APIs
WebRTC and Web Audio API
MATTEO AVALLE
Audio Capture
OutputJavascript filter
Output the
filtered stream

22. Web Audio APIs: Mp3 Volume bar Example
WebRTC and Web Audio API
MATTEO AVALLE

23. Conclusions
• WebRTC APIs are tremendously powerful
• Maybe too low level to create skype rivals “easily”
• We can do much more than just video chat
WebRTC and Web Audio API
MATTEO AVALLE
What would YOU do with these?

24. THANK YOU
www.fluentify.com/torinojs.zip
WebRTC and Web Audio API
BY MATTEO AVALLE
Co-Founder & CTO Fluentify.com