Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2016 February - WebRTC Conference japan - English

1,411 views

Published on

WebRTC has been an exciting technology, and extremely fast moving for the past years. While its adoption and its disruptive power are not challenged anymore, the fast evolution pace, and the fast update cycles of the browsers made it difficult to build complex solutions on top of it which would leverage all that webrtc has to offer. Late 2015, the different standard committees and corresponding working groups that compose webRTC have finally reach a consensus, and from the convergence of all their efforts, stable specifications were born.
Through the use of GoToMeeting and other software, we will illustrate first the usual pains that most using webrtc have experienced, and then show how the webrtc APIs, which had started as a peer-to-peer API, were extended with an object model API to provide more options and more controls to this who need it, while keeping the simplicity of P2P for the others. The similitudes between the new Object Model API, and the ORTC API (implemented in edge) will also be illustrated.

Published in: Engineering
  • Be the first to comment

2016 February - WebRTC Conference japan - English

  1. 1. WebRTC Evolution Dr. Alex Gouaillard © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  2. 2. Overview © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 • Early 2015 WebRTC • Object Models APIs • Glue (Transceiver) • Faster ICE • Ports optimizations • Simulcast • Others in webrtc 1.0
  3. 3. Early 2015, peer-to-peer webRTC Model © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 http://www.html5rocks.com/en/tutorials/webrtc/basics/
  4. 4. Signaling server Early 2015, peer-to-peer webRTC Model PeerConnection JavaScript APP(Sender) JavaScript APP (Receiver) PeerConnection Media Stream Data Channel CreateOffer SetLocal SetRemote SetLocal SetRemote Offer (SDP) Answer (SDP) Candidates (SDP) Media Stream Data Channel OnCandidate AddCandidate CreateAnswer ICE Bindings + tests © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  5. 5. Signaling server Early 2015, peer-to-peer webRTC Model OmniPotent PeerConnection PeerConnection JavaScript APP (Sender) JavaScript APP (Receiver) PeerConnectionMedia Stream Data Channel ICE UADTLS RTP Code c CreateOffer Gather Transport SetLocal SetRemote ICE UA DTLS RTP Code c Gather Transport SetLocal SetRemote Offer (SDP) Answer (SDP) Candidates (SDP) Media Stream Data Channel OnCandidate AddCandidate CreateAnswer ICE Bindings + tests © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  6. 6. DONEFROZENINCALLCONNECTIONHANDSHAKE(SDPO/A). new checking connected disconnected failed Completed close new gathering complete CALLER SIG-SERVER CALLEE stable have-local-offer stable Close OFFER ANSWER Create PC Add local stream(s) Create offer <modify sdp> SetLocal(offer) Sending offer Create PC SetRemote(offer) addRemoteStream(s) Add local stream(s) Create answer <modify sdp> SetLocal(answer) Send answer stable Have-remote- offer stable new gathering complete onIceCandidate <Filter candidates> Send candidate <Filter candidates> addIceCandidate onIceCandidate <Filter candidates> Send candidate <Filter candidates> addIceCandidate new checking connected disconnected failed Completed close Close © Temasys Communications, pvt, ltd, 2014 Document provided under CC BY-NC 4.0 CANDIDATES Peer Connection ICE Connection ICE Gathering SetRemote(answer) addRemoteStream BYE
  7. 7. Early 2015, peer-to-peer webRTC Model Open Questions 1. How do I handle bandwidth preferences? 2. How do I check fingerprint (MITM attacks prevention)? 3. How do I handle codecs and their parameters? => Hum, ……. Just modify the SDP and pray it works. 4. How to know which pair of candidate is being used? 5. How do I restrict the ICE gathering (not to expose my IPs) 6. How do I get more info about ICE failures? 7. How do I make the (ICE) handshake faster? 8. How do I get rid of JSEP, SDP, O/A, …..? 9. Can I switch front / self view without renegotiation? => No can’t do. © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  8. 8. Overview © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 • Early 2015 WebRTC • Object Models APIs • Glue (Transceiver) • Faster ICE • Ports optimizations • Simulcast • Others in webrtc 1.0
  9. 9. Q2 2015, New W3C WebRTC charter • => ORTC – WebRTC merge • Notion of Object Model API (ORTC inspired) © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 http://ortc.org/wp-content/uploads/2014/08/ortc.html
  10. 10. Object Models? • Each Object Does only one thing! • Each Object can be probed for capabilities (JS) • Parameters can be get/set dynamically (JS) © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 #8 (Edge) http://ortc.org/wp-content/uploads/2014/08/ortc.html
  11. 11. Late 2015, Object Model API Open Questions Have a final API. • How do I handle bandwidth preferences • How do I check fingerprint (MITM attacks prevention) • How to know which pair of candidate is being used? • How do I get more info about ICE failures? • How do I get rid of JSEP, SDP, O/A, …..? • Can I switch front / self view without renegotiation? Have some API, can do better or more. • How do I handle codecs and their parameters • How do I restrict the ICE gathering (not to expose my IPs) Not addressed yet • How do I make the (ICE) handshake faster? © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  12. 12. Overview © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 • Early 2015 WebRTC • Object Models APIs • Glue (Transceiver) • Faster ICE • Ports optimizations • Simulcast • Others in webrtc 1.0
  13. 13. WebRTC 1.0 = Object Model + JSEP PLACEHOLDER FOR DRAWING © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  14. 14. WebRTC 1.0 = Object Model + JSEP Problems: 1. m-lines in SDP are intrinsically bi-directional 2. m-lines in SDP have attributes that fit neither sender nor receiver object model (mid), or that would be duplicated. How to model the equivalent to a m-line on top of the Object Model (so we can throw it later if we decide to get rid of SDP, JSEP, ….) ? © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  15. 15. Rtp Sender Rtp Receiver Dtls Transport Ice Transport Local Track NET Ice Transport Dtls Transport Rtp Sender WebRTC / Object Model Example 1: Audio only, 1:1 Dtls Transport Ice Transport Rtp Receiver Remote Track Dtls Transport Ice Transport Rtp Transceiver Remote Track Local Track Rtp Transceiver © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0 © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  16. 16. Overview © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 • Early 2015 WebRTC • Object Models APIs • Glue (Transceiver) • Faster ICE • Ports optimizations • Simulcast • Others in webrtc 1.0
  17. 17. What’s your flavor? Full, Trickle, Lite Feature flavor Full Trickle Lite Make connectivity checks Yes Yes NO Answer connectivity checks Yes Yes Yes Controlling or controled Both Both Controlled Candidates send in the Offer/Answer Yes No ? Flag Ice-option: trickle a=ice-lite https://tools.ietf.org/html/draft-ietf-mmusic-trickle-ice-02 https://webrtchacks.com/trickle-ice/ © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 Trickle: From 10s to 1S Q: How to get 100ms?
  18. 18. Faster ICE => more async, less waiting - Start even before the call! - ICE pool - Pre-gathering - Warmup: - ICE, DTLS, media ready before signaling is. - Early Media: - Start sending media with answer - Simplify & fasten the pairing / checking - Default TURN server © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  19. 19. Overview © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 • Early 2015 WebRTC • Object Models APIs • Glue (Transceiver) • Faster ICE • Ports optimizations • Simulcast • Others in webrtc 1.0
  20. 20. RTP/RTCP within 1 PeerConnection Use case: 2 media tracks (audio and video), how many ports do I need? Less ports would useful for - Better connection rate esp. over mobile net. - Optimize turn/media server resources - Client side ports constraints © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  21. 21. RTP/RTCP within a PeerConnection Default 4 A V RTP RTCP http://tools.ietf.org/html/draft-ietf-mmusic-sdp-bundle-negotiation http://datatracker.ietf.org/doc/rfc5761/ © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  22. 22. RTP/RTCP Default 4 Bundle 2 A V RTP RTCP A V RTP RTCP a=group:BUNDLE http://tools.ietf.org/html/draft-ietf-mmusic-sdp-bundle-negotiation http://datatracker.ietf.org/doc/rfc5761/ © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  23. 23. RTP/RTCP Default 4 Bundle 2 Rtcp-mux 2 A V RTP RTCP A V RTP RTCP A V RTP RTCP a=rtcp-muxa=group:BUNDLE http://tools.ietf.org/html/draft-ietf-mmusic-sdp-bundle-negotiation http://datatracker.ietf.org/doc/rfc5761/ © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  24. 24. RTP/RTCP Default 4 Bundle 2 Rtcp-mux 2 Both 1 A V RTP RTCP A V RTP RTCP A V RTP RTCP A V RTP RTCP a=rtcp-muxa=group:BUNDLE http://tools.ietf.org/html/draft-ietf-mmusic-sdp-bundle-negotiation http://datatracker.ietf.org/doc/rfc5761/ © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  25. 25. RTP/RTCP – extrapolation to n tracks Default 2n Bundle n Rtcp-mux n Both 1 A V RTP RTCP A V RTP RTCP A V RTP RTCP A V RTP RTCP a=rtcp-muxa=group:BUNDLE http://tools.ietf.org/html/draft-ietf-mmusic-sdp-bundle-negotiation http://datatracker.ietf.org/doc/rfc5761/ © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  26. 26. Rtp Sender Rtp Receiver Dtls Transport Ice Transport Local Track NET Ice Transport Dtls Transport Rtp Sender WebRTC / Object Model Example 1: Audio only, 1:1 Dtls Transport Ice Transport Rtp Receiver Remote Track Dtls Transport Ice Transport Rtp Transceiver Remote Track Local Track Rtp Transceiver RTP RTCP © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0 © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 Default 2n
  27. 27. Rtp Sender Rtp Receiver Local Track NET Remote Track Rtp Sender Local Track WebRTC / Object Model Example 1: Audio only, 1:1 rtcp-mux Dtls Transport Ice Transport Rtp Receiver Remote Track Dtls Transport Ice Transport Rtp Transceiver Rtp Transceiver © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0 © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 Rtcp-mux n
  28. 28. WebRTC / Object Model Example 2: Audio + Video, 1:1 rtcp-mux Rtp Sender Rtp Receiver Local Track NET Remote Track Rtp Sender Local Track Dtls Transport Ice Transport Rtp Receiver Remote Track Dtls Transport Ice Transport Rtp Transceiver Rtp Transceiver Rtp Sender Rtp Receiver Local Track NET Remote Track Rtp Sender Local Track Dtls Transport Ice Transport Rtp Receiver Remote Track Dtls Transport Ice Transport Rtp Transceiver Rtp Transceiver AUDIO VIDEO © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0 © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 Rtcp-mux n
  29. 29. RtpTransceiver RtpTransceiver Rtp Sender Rtp Sender Rtp Sender Rtp ReceiverTrack Rtp Sender Rtp Receiver Dtls Transport Ice Transport Local Track NET Ice Transport Dtls Transport Track Remote Track Rtp ReceiverTrack Remote Track Rtp Sender Track Local Track WebRTC / Object Model Example 2: Audio + Video, 1:1 rtcp-mux + BUNDLE RtpTransceiver RtpTransceiver © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0 © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 Both 1
  30. 30. Data Channel Data Channel WebRTC / Object Model Generic & Default ! RtpTransceiver RtpTransceiver Rtp Sender Rtp Sender Rtp Sender Rtp ReceiverTrack Rtp Sender Rtp Receiver Dtls Transport Ice Transport Local Track NET Ice Transport Dtls Transport Track Remote Track Rtp ReceiverTrack Remote Track Rtp Sender Track Local Track RtpTransceiver RtpTransceiver Sctp Transport Data Channel Sctp Transport Data Channel © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0 © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 Both 1
  31. 31. Overview © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 • Early 2015 WebRTC • Object Models APIs • Glue (Transceiver) • ICE and Optimizations • Simulcast • Others in webrtc 1.0
  32. 32. Reminder Multiparty vs Simulcast vs SVC © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 A B C Multiparty • Several tracks • Decodable separately • Bandwidth management separated HD SD QVGA Simulcast • Several tracks • Coming from the same source • Decodable separately • Smart bandwidth management possible 1SSRC Δ2 Δ1 QVGA SVC Encoding • Several tracks • Coming from the same source • Not Decodable separately (Except base layer) • Smart bandwidth management mandatory • Less bandwidth, more resilience. 1SSRC E E E E E E SVCEnc. D D D HD SD QVGA
  33. 33. Reminder State of Multiparty signaling © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 From the beginning of webrtc, one have been able to send multiple media streams in separate peer connections, but it consumes much more ports, and there is no sync between those tracks (lip sync, ..). A B C E E E Multicast • Several tracks • Decodable separately • Bandwidth management separated
  34. 34. Reminder State of Multiparty signaling For multiple streams in a single peer connection! © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  35. 35. Reminder Multiparty vs Simulcast vs SVC © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 A B C Multicast • Several tracks • Decodable separately • Bandwidth management separated HD SD QVGA Simulcast • Several tracks • Coming from the same source • Decodable separately • Smart bandwidth management possible 1SSRC Δ2 Δ1 QVGA SVC Encoding • Several tracks • Coming from the same source • Not Decodable separately (Except base layer) • Smart bandwidth management mandatory • Less bandwidth, more resilience. 1SSRC E E E E E E SVCEnc. D D D HD SD QVGA
  36. 36. Simulcast: Use case for webRTC 1.0 - Use Case for webrtc 1.0: SFU - Browser send simulcast, does not receive simulcast (in webrtc 1.0) Desk. HD SFU SD QVGA Cr. Book Mobile Desk. HD SD QVGA © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  37. 37. Simulcast (and SVC later): 3 layers, 3 working groups, 2 standard bodies - Separate the - “on-the-wire” info (1), - JS API (2), - and signaling (3) Signaling server Web server Bob (device) Alice (browser) Media (3) (2) (1) © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0 © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  38. 38. Overview © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0 • Early 2015 WebRTC • Object Models APIs • Glue (Tranceiver) • ICE and Optimizations • Simulcast • Others in webrtc 1.0
  39. 39. Screen Capture API simplification - No more distinction between screen, windows and apps. => no more constraints, flat list. • Screen Capture http://w3c.github.io/mediacapture-screen-share/ © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  40. 40. Record your streams (local), but not all streams (remote). http://www.w3.org/TR/mediastream-recording/ Media Stream Recording (Client side) © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  41. 41. Use <AUDIO> or <VIDEO> as a source ! http://w3c.github.io/mediacapture-fromelement/ Media Capture from DOM Elements && Image Capture (a.k.a. snapshot) © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  42. 42. Send different streams to different outputs http://w3c.github.io/mediacapture-output/ Audio output devices API © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  43. 43. http://www.w3.org/TR/webrtc-stats/ Identifiers for WebRTC's Statistics API Full stats about data channel, codecs, FEC, … © Dr. Alex Gouaillard @ Cosmo Consulting., 2015. Document provided under CC BY-NC 4.0
  44. 44. Transferable DataChannels a.k.a. DataChannels in workers © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  45. 45. SECURITY: IP Address Leakage 4 levels for candidate gathering - All [with consent] - Restricted I (host + 1918) [default] - Restricted II (no host) [pref/extension] - Only proxy/VPN [pref/extension] © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0
  46. 46. Conclusion WebRTC 1.0 is Awesome [Evil Grin] © Dr. Alex Gouaillard @ Citrix Systems, 2015. Document provided under CC BY-NC 4.0

×