Download to read offline
Uploaded byMatteo Avalle
Torino js
WebRTC and the Web Audio API allow real-time communication and audio/video processing capabilities directly in the browser. WebRTC provides APIs for capturing media streams from devices and establishing peer-to-peer connections, while dealing with challenges like firewalls and NAT using ICE, STUN and TURN. The Web Audio API enables efficient audio stream manipulation through a processing block model. While building full-featured video chat applications with WebRTC alone can be difficult, these capabilities open opportunities for applications involving real-time media and audio processing directly in the browser.
More Related Content
![JavaOne: Welcome alexa, your personal assistant [con1700]](https://cdn.slidesharecdn.com/ss_thumbnails/welcomealexayourpersonalassistantcon1700-160923181032-thumbnail.jpg?width=640&height=640&fit=bounds)
Torino js
- 1. WebRTC and WebAudio API BY MATTEO AVALLE Co-Founder & CTO Fluentify.com
- 2. OUTLINE • Introduction • WebRTC –MediaStream – RTCPeerConnection & RTCDataChannel – ICE, STUN, TURN – Browser support • Alternatives • “Abusing” MediaStreams • Web Audio APIs • Conclusions WebRTC and Web Audio API MATTEO AVALLE
- 3. Introduction • How dowe usually interact with websites? – Mouse & Keyboard – Plugin-based webcam & mic – External apps • Can we do better? – Voice recognition – Image processing – Audio/Video chat integrated in webpages WebRTC and Web Audio API MATTEO AVALLE
- 4. WebRTC • A JavaScriptAPI set – Audio/Video stream capture – P2P Communication API – Real time stream encoding/decoding • Low level • Modular WebRTC and Web Audio API MATTEO AVALLE
- 5. WebRTC • The threeAPIs: – MediaStream • Gets data from Webcam, Microphone, etc. – RTCDataChannel • Creates generic P2P connections – RTCPeerConnection • Handle complex procedures like: – Audio/Video calls – Encryption – Bandwidth management WebRTC and Web Audio API MATTEO AVALLE
- 6. MediaStream • Available inChrome, Opera, Firefox, Edge • Used to “connect” an audio/video stream to: – A <video> / <canvas> tag – A WebRTC stream – ... • Extremely simple WebRTC and Web Audio API MATTEO AVALLE
- 7. MediaStream: the MirrorExample WebRTC and Web Audio API MATTEO AVALLE
- 8. WebRTC and WebAudio API MATTEO AVALLE Note: examples are just meant to be examples
- 9. RTCPeerConnection, RTCDataChannel • Availablein Chrome, Opera, Firefox • They handle: – P2P connection creation – Data encoding/encrypion – Data buffering WebRTC and Web Audio API MATTEO AVALLE
- 10. RTCPeerConnection, RTCDataChannel WebRTC andWeb Audio API MATTEO AVALLE
- 11. Issues of P2P •How do I know how many other guys are present? • Who they are? (i.e. What are their IP addresses?) • What if they are behind a NAT router (like 99% of home users)? • What if they are behind a firewall? • What should I do if the connection drops/is bad? WebRTC and Web Audio API MATTEO AVALLE
- 12. ICE, STUN, TURN •ICE/STUN are: – “HACKS” to bypass the firewalls – They use a server to create the initial connection – The rest of the communication is P2P • TURN: – The last resort for too-protected networks – All the traffic passes through a server – No more P2P, but the most reliable WebRTC and Web Audio API MATTEO AVALLE
- 13. Browser compatibility • Chrome,Opera and Firefox have all the APIs, but: – They require different prefixes – Certain behaviors are still different • What about the other browsers? WebRTC and Web Audio API MATTEO AVALLE
- 14. WebRTC and WebAudio API MATTEO AVALLE HELP! THIS IS GETTING SO COMPLICATED!
- 15. What can wedo? • Start on a journey to become a WebRTC master • Use third-party libraries – OpenTok – EasyRTC – PeerJS – XirSys * – simplewebrtc.com – Hangouts APIs * WebRTC and Web Audio API MATTEO AVALLE
- 16. OpenTok P2P? Relayed? Youchoose, they provide the servers • DASH “for free” (in terms of coding) • IE plugin available • Real-Time performance monitor • 100% customizable Javascript library • BUT: it’s kinda expensive, both for small- scale and large-scale applications WebRTC and Web Audio API MATTEO AVALLE
- 17. WebRTC and WebAudio API MATTEO AVALLE Creating video-chat websites with WebRTC alone is hard. BUT THEN... ARE WEBRTC APIS STILL USEFUL FOR US TO KNOW?
- 18. “Abusing” WebRTC APIs •We can redirect our Media Streams elsewhere – <video>/<canvas> tags – Filters (CSS/canvas) – Background Javascript Workers • We have a GPU-accelerated videoP codec encoder/ decoder • We can develop P2P applications WebRTC and Web Audio API MATTEO AVALLE
- 19. “Abusing” WebRTC APIs: theFiltered Mirror Example WebRTC and Web Audio API MATTEO AVALLE
- 20. Web Audio APIs •An efficient API to manipulate audio streams • Most common “slow” functions are hardcoded in the browser (e.g. FFTransform), you just use them • “Processing blocks” coding style WebRTC and Web Audio API MATTEO AVALLE
- 21. Web Audio APIs WebRTCand Web Audio API MATTEO AVALLE Audio Capture OutputJavascript filter Output the filtered stream
- 22. Web Audio APIs:Mp3 Volume bar Example WebRTC and Web Audio API MATTEO AVALLE
- 23. Conclusions • WebRTC APIsare tremendously powerful • Maybe too low level to create skype rivals “easily” • We can do much more than just video chat WebRTC and Web Audio API MATTEO AVALLE What would YOU do with these?
- 24. THANK YOU www.fluentify.com/torinojs.zip WebRTC andWeb Audio API BY MATTEO AVALLE Co-Founder & CTO Fluentify.com