Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives
•
5 likes•5,479 views
WebRTC is one of the main trends on the multimedia arena in the last few years. The ability of bringing real-time audio and video to WWW browsers opens new horizons for developers to create context aware customized applications for inter-human communications. However, for WebRTC technologies to work seamlessly in WWW applications, it’s necessary to manage with a number of present and future complex challenges. In this talk, we present the experience of the Kurento Media Server team in creating a WebRTC endpoint for GStreamer. We describe the main problems and limitations basing on current GStreamer status describing which parts of WebRTC standards can be currently implemented with GStreamer and which parts require further evolutions and efforts from the community. We will also describe the plans and drafts that are emerging at different standardization groups, including the WebRTC WG at W3C and the RTCWeb WG at IETF. Basing on this, we will try to forecast how WebRTC technologies in particular, but also how real-time multimedia communications in general, may be evolving in the next couple of years and the activities that the GStreamer community should be considering for adapting to these evolutions. In particular, we shall introduce in detail topics such as the following: • The evolution of ICE (Interactive Connectivity Establishment). • Congestion control for RTC streams. • Implementing WebRTC security securely • Implementing and optimizing the AVPF profile for RTP • Benchmarking WebRTC: stats metrics • Managing sensor data through DataChannels.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives
Similar to Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives (20)
Recently uploaded
Recently uploaded (20)
Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives
- 1. Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives 8-9 October 2015 Dublin, Ireland Conference 2015 Miguel París Jose A. Santos santoscadenas@gmail.com mparisdiaz@gmail.com
- 2. 2 Who we are José Antonio Santos Cadenas ● Software Engineer ● Telematic Systems Master's ● Developer at Naevatec (Madrid, Spain) ● Kurento Media Server (KMS) manager ● Integrating with client APIs ● santoscadenas@gmail.com Miguel París ● Software Engineer ● Telematic Systems Master's ● Researcher at Universidad Rey Juan Carlos (Madrid, Spain) ● Kurento real-time manager ● mparisdiaz@gmail.com ● Twitter: @mparisdiaz
- 3. What is WebRTC 3 ● WebRTC is a new movement bringing RTC to the WWW ● It is based on standards ● Supported by Chrome & Firefox ● Windows Edge added partial support recently ● Safari is rumored to be on the way of providing it ● FOSS implementations for Smartphone platforms (iOS and Android) ● More tan 1B devices currently support WebRTC Developing RTC applications for the WWW Before WebRTC After WebRTC Begin End • Unified APIs • Standards • FOSS • Multiplatform
- 4. WebRTC standards 4 ● Standard APIs: Standardized by the WebRTC WG at W3C Media Capture APIs ● Standardize APIs for media capture PeerConnection APIs (WebRTC 1.0 almost finished) ● Standardize API for P2P media communications ● Media = Audio + Video + Data ● DataChannel API: low latency WebRTC stats ● RTP stats for inbound and outbound directions ● encode/decode stats ● DataChannel stats
- 5. Implementing WebRTC in GStreamer 5 ● Deep interest in the GStreamer community ● Several implementations OpenWebRTC (Ericsson) ● Design for working as part of client applications Kurento Media Server (Kurento.org) ● Design for working as part of media server infrastructures Main differences between them: OpenWebRTC captures media from camera and mic. KMS doesn't OpenWebRTC renders to display and speakers. KMS doesn't OpenWebRTC encodes/decodes audio/video. KMS only if needed.
- 6. 6 <GstPipeline> pipeline0 [>] KmsWebrtcEndpoint kmswebrtcendpoint1 [>] GstRTPRtxQueue rtprtxqueue1 [>] GstRtpVP8Pay rtpvp8pay1 [>] GstRtpOPUSPay rtpopuspay1 [>] KmsWebrtcSession kmswebrtcsession1 [>] KmsRtcpDemux kmsrtcpdemux1 [>] GstRtpSsrcDemux rtpssrcdemux5 [>] KmsWebrtcTransportSinkNice kmswebrtctransportsinknice1 [>] GstNiceSink nicesink1 [>] GstDtlsSrtpEnc dtlssrtpenc1 [>] GstFunnel funnel [>] GstSrtpEnc srtp-encoder [>] GstDtlsEnc dtls-encoder [>] KmsWebrtcTransportSrcNice kmswebrtctransportsrcnice1 [>] GstDtlsSrtpDec dtlssrtpdec1 [>] GstSrtpDec srtp-decoder [>] GstDtlsDec dtls-decoder [>] GstDtlsSrtpDemux dtls-srtp-demux [>] GstNiceSrc nicesrc1 [>] GstRtpBin rtpbin1 [>] GstRtpSsrcDemux rtpssrcdemux4 [>] GstRtpSession rtpsession3 [>] GstRtpSsrcDemux rtpssrcdemux3 [>] GstRtpSession rtpsession2 [>] KmsWebrtcEndpoint kmswebrtcendpoint0 [>] GstRtpVP8Depay rtpvp8depay0 [>] KmsAgnosticBin2 kmsagnosticbin2-1 [>] GstQueue queue3 [>] KmsParseTreeBin kmsparsetreebin1 [>] KmsVp8Parse kmsvp8parse0 [>] GstFakeSink fakesink3 [>] GstTee tee3 [>] GstFakeSink fakesink2 [>] GstTee tee2 [>] GstRTPOpusDepay rtpopusdepay0 [>] KmsAgnosticBin2 kmsagnosticbin2-0 [>] GstQueue queue1 [>] KmsParseTreeBin kmsparsetreebin0 [>] GstOpusParse opusparse0 [>] GstFakeSink fakesink1 [>] GstTee tee1 [>] GstFakeSink fakesink0 [>] GstTee tee0 [>] GstRTPRtxQueue rtprtxqueue0 [>] GstRtpVP8Pay rtpvp8pay0 [>] GstRtpOPUSPay rtpopuspay0 [>] KmsWebrtcSession kmswebrtcsession0 [>] KmsRtcpDemux kmsrtcpdemux0 [>] GstRtpSsrcDemux rtpssrcdemux2 [>] KmsWebrtcTransportSinkNice kmswebrtctransportsinknice0 [>] GstNiceSink nicesink0 [>] GstDtlsSrtpEnc dtlssrtpenc0 [>] GstFunnel funnel [>] GstSrtpEnc srtp-encoder [>] GstDtlsEnc dtls-encoder [>] KmsWebrtcTransportSrcNice kmswebrtctransportsrcnice0 [>] GstDtlsSrtpDec dtlssrtpdec0 [>] GstSrtpDec srtp-decoder [>] GstDtlsDec dtls-decoder [>] GstDtlsSrtpDemux dtls-srtp-demux [>] GstNiceSrc nicesrc0 [>] GstRtpBin rtpbin0 [>] GstRtpJitterBuffer rtpjitterbuffer1 [>] GstRtpPtDemux rtpptdemux1 [>] GstRtpJitterBuffer rtpjitterbuffer0 [>] GstRtpPtDemux rtpptdemux0 [>] GstRtpSsrcDemux rtpssrcdemux1 [>] GstRtpSession rtpsession1 [>] GstRtpSsrcDemux rtpssrcdemux0 [>] GstRtpSession rtpsession0 [>] Legend Element-States: [~] void-pending, [0] null, [-] ready, [=] paused, [>] playing Pad-Activation: [-] none, [>] push, [<] pull Pad-Flags: [b]locked, [f]lushing, [b]locking; upper-case is set Pad-Task: [T] has started task, [t] has paused task proxypad40 [>][bfb] sink [>][bfb] sink_audio [>][bfb] proxypad42 [>][bfb] sink [>][bfb] sink_video [>][bfb] sink [>][bfb] src [>][bfb] send_rtp_sink_1 [>][bfb] proxypad33 [>][bfb] src [>][bfb] src [>][bfb] send_rtp_sink_0 [>][bfb] proxypad31 [>][bfb] send_rtp_src_0 [>][bfb] sink [>][bfb] rtp_src [>][bfb] rtcp_src [>][bfb] rtcp_sink [>][bfb] sink [>][bfb] src_1 [>][bfb] recv_rtp_sink_1 [>][bfb] rtcp_src_1 [>][bfb] recv_rtcp_sink_1 [>][bfb] src_421259003 [>][bfb] recv_rtp_sink_0 [>][bfb] rtcp_src_421259003 [>][bfb] recv_rtcp_sink_0 [>][bfb] proxypad44 [>][bfb] proxypad45 [>][bfb] proxypad46 [>][bfb] proxypad47 [>][bfb] sink [>][bfb] proxypad34 [>][bfb] rtp_sink_0 [>][bfb] rtp_sink_0 [>][bfb] src [>][bfb] proxypad36 [>][bfb] rtcp_sink_0 [>][bfb] rtcp_sink_0 [>][bfb] proxypad37 [>][bfb] rtp_sink_1 [>][bfb] rtp_sink_1 [>][bfb] proxypad39 [>][bfb] rtcp_sink_1 [>][bfb] rtcp_sink_1 [>][bfb] proxypad29 [>][bfb] funnelpad5 [>][bfb] src [>][bfb] funnelpad6 [>][bfb] funnelpad7 [>][bfb] funnelpad8 [>][bfb] funnelpad9 [>][bfb] rtp_src_0 [>][bfb] rtcp_src_0 [>][bfb] rtp_src_1 [>][bfb] rtcp_src_1 [>][bfb] src [>][bfb][T] proxypad28 [>][bfb] sink [>][bfb] sink [>][bfb] rtp_src [>][bfb] proxypad26 [>][bfb] proxypad27 [>][bfb] rtcp_src [>][bfb] rtp_sink [>][bfb] rtp_src [>][bfb] rtcp_sink [>][bfb] rtcp_src [>][bfb] sink [>][bfb] rtp_src [>][bfb] dtls_src [>][bfb] src [>][bfb][T] send_rtp_sink [>][bfb] send_rtp_sink [>][bfb] recv_rtp_sink [>][bfb] recv_rtcp_sink [>][bfb] recv_rtp_sink [>][bfb] recv_rtcp_sink [>][bfb] proxypad30 [>][bfb] proxypad32 [>][bfb] send_rtp_src_1 [>][bfb] proxypad35 [>][bfb] send_rtcp_src_0 [>][bfb] proxypad38 [>][bfb] send_rtcp_src_1 [>][bfb] sink [>][bfb] rtcp_sink [>][bfb] send_rtp_src [>][bfb] send_rtcp_src [>][bfb] recv_rtp_src [>][bfb] sync_src [>][bfb] sink [>][bfb] rtcp_sink [>][bfb] send_rtp_src [>][bfb] send_rtcp_src [>][bfb] recv_rtp_src [>][bfb] sync_src [>][bfb] proxypad14 [>][bfb] sink [>][bfb] sink_audio [>][bfb] audio_src_0 [>][bfb] proxypad15 [>][bfb] sink [>][bfb] sink_video [>][bfb] proxypad24 [>][bfb] proxypad25 [>][bfb] video_src_0 [>][bfb] sink [>][bfb] src [>][bfb] sink [>][bfb] proxypad23 [>][bfb] src_0 [>][bfb] sink [>][bfb] proxypad43 [>][bfb]sink [>][bfb] src [>][bfb][T] sink [>][bfb] src [>][bfb] sink [>][bfb] src_0 [>][bfb] sink [>][bfb] src_2 [>][bfb] sink [>][bfb] src_0 [>][bfb] src_1 [>][bfb] sink [>][bfb] src [>][bfb] sink [>][bfb] proxypad19 [>][bfb] src_0 [>][bfb] sink [>][bfb] proxypad41 [>][bfb]sink [>][bfb] src [>][bfb][T] sink [>][bfb] src [>][bfb] sink [>][bfb] src_0 [>][bfb] sink [>][bfb] src_2 [>][bfb] sink [>][bfb] src_0 [>][bfb] src_1 [>][bfb] sink [>][bfb] src [>][bfb] send_rtp_sink_1 [>][bfb] proxypad7 [>][bfb] src [>][bfb] src [>][bfb] send_rtp_sink_0 [>][bfb] proxypad5 [>][bfb] send_rtp_src_0 [>][bfb] sink [>][bfb] rtp_src [>][bfb] rtcp_src [>][bfb] rtcp_sink [>][bfb] sink [>][bfb] src_1442068093 [>][bfb] recv_rtp_sink_0 [>][bfb] rtcp_src_1442068093 [>][bfb] recv_rtcp_sink_0 [>][bfb] src_836061664 [>][bfb] recv_rtp_sink_1 [>][bfb] rtcp_src_836061664 [>][bfb] recv_rtcp_sink_1 [>][bfb] proxypad16 [>][bfb] proxypad17 [>][bfb] proxypad20 [>][bfb] proxypad21 [>][bfb] sink [>][bfb] proxypad8 [>][bfb] rtp_sink_0 [>][bfb] rtp_sink_0 [>][bfb] src [>][bfb] proxypad10 [>][bfb] rtcp_sink_0 [>][bfb] rtcp_sink_0 [>][bfb] proxypad11 [>][bfb] rtp_sink_1 [>][bfb] rtp_sink_1 [>][bfb] proxypad13 [>][bfb] rtcp_sink_1 [>][bfb] rtcp_sink_1 [>][bfb] proxypad3 [>][bfb] funnelpad0 [>][bfb] src [>][bfb] funnelpad1 [>][bfb] funnelpad2 [>][bfb] funnelpad3 [>][bfb] funnelpad4 [>][bfb] rtp_src_0 [>][bfb] rtcp_src_0 [>][bfb] rtp_src_1 [>][bfb] rtcp_src_1 [>][bfb] src [>][bfb][T] proxypad2 [>][bfb] sink [>][bfb] sink [>][bfb] rtp_src [>][bfb] proxypad0 [>][bfb] proxypad1 [>][bfb] rtcp_src [>][bfb] rtp_sink [>][bfb] rtp_src [>][bfb] rtcp_sink [>][bfb] rtcp_src [>][bfb] sink [>][bfb] rtp_src [>][bfb] dtls_src [>][bfb] src [>][bfb][T] send_rtp_sink [>][bfb] send_rtp_sink [>][bfb] recv_rtp_sink [>][bfb] recv_rtcp_sink [>][bfb] recv_rtp_sink [>][bfb] recv_rtcp_sink [>][bfb] proxypad4 [>][bfb] proxypad6 [>][bfb] send_rtp_src_1 [>][bfb] proxypad9 [>][bfb] send_rtcp_src_0 [>][bfb] proxypad12 [>][bfb] send_rtcp_src_1 [>][bfb] proxypad18 [>][bfb] recv_rtp_src_0_1442068093_111 [>][bfb] proxypad22 [>][bfb] recv_rtp_src_1_836061664_100 [>][bfb] sink [>][bfb] src [>][bfb][T] sink_rtcp [>][bfb] sink [>][bfb] src_100 [>][bfb] sink [>][bfb] src [>][bfb][T] sink_rtcp [>][bfb] sink [>][bfb] src_111 [>][bfb] sink [>][bfb] src_836061664 [>][bfb] rtcp_sink [>][bfb] rtcp_src_836061664 [>][bfb] send_rtp_src [>][bfb] send_rtcp_src [>][bfb] recv_rtp_src [>][bfb] sync_src [>][bfb] sink [>][bfb] src_1442068093 [>][bfb] rtcp_sink [>][bfb] rtcp_src_1442068093 [>][bfb] send_rtp_src [>][bfb] send_rtcp_src [>][bfb] recv_rtp_src [>][bfb] sync_src [>][bfb] Kurento WebRtcEp The pipeline
- 7. Interactive Connectivity Establishment ICE I 7 ● Using libnice (NiceSrc and NiceSink) Involved RFCs: RFC 5245: Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols RFC 5389: Session Traversal Utilities for NAT (STUN) RFC 5766: Traversal Using Relays around NAT (TURN): Relay Extensions to Session Traversal Utilities for NAT (STUN)
- 8. Interactive Connectivity Establishment ICE II 8 ● Our experience with libnice 0.1.7 version seem to be stable, but lack many required features 0.1.10 version incorporate relevant features but seem not to provide the appropriate stability yet ● Leaks of TCP sockets ● Connection states ● Connectivity problems (Fails 1/1000) Fixed in 0.1.13 ? (we have to check it)
- 9. ICE III 9 ● What next needed in ICE for WebRTC? IPv6 shall be mandatory in WebRTC. We need to check IPv6 support in libnice draft-ietf-rtcweb-transports-09 (mandates IPv6 support) HTTP CONNECT protocol for TURN might be important in corporate networks (connectivity) draft-hutton-httpbis-connect-protocol-00 STUN consent freshness may become mandatory for WebRTC. This may require some adjustments on libnice for supporting consent negotiation (security) Draft-ietf-rtcweb-stun-consent-freshness-16 ICE stated published through events ICE state should be reported as specified in the WebRTC 1.0 WG draft
- 10. DTLS / SRTP I 10 ● Using dtlssrtp(dec|enc) bins dtls(dec|enc) manages DTLS handshake srtp(dec|enc) decrypts/encrypts RTP packets ● Involved RFCs: RFC 5764: Datagram Transport Security (DTLS) Extension to Establish Keys for the Secure Real-time Protocol (SRTP) RFC 3711: The Secure Real-time Transport Protocol (SRTP)
- 11. DTLS / SRTP II 11 ● What next for WebRTC? New cryptographic framework for group communications This is under definition at the Privacy Enhanced RTP Conferencing (PERC) IETF WG Only early drafts here draft-jones-perc-private-media-framework-00 draft-jones-perc-private-media-reqts-00 draft-mattsson-perc-srtp-cloud-00 draft-westerlund-perc-webrtc-use-case-00 PERC shall provide in the next few months (improve efficiency) Enabling unicast group communications without requiring infrastructure to decipher New mechanisms enabling unicast group communications without requiring infrastructure to cipher once per RTC session. These may require re-factoring of current DTLS stack DTLS state published through events. Full connectivity requires ICE connectivity + successful DTLS handshake. DTLS is being incorporated by the W3C in the WebRTC 1.0 spec as event generator dealing with connection health
- 12. Congestion control I 12 ● This is an essential feature for providing satisfactory QoE for WebRTC users How does it work? Bandwidth estimator Based on packet delay (useful RTP ExtHdr: abs-send-time) Based on packet loss REMB (Receiver Estimated Maximum Bitrate) GStreamer does not provide any kind of features for this off the self How Kurento estimates bandwidth? on-sending-rtcp Get packet-loss, bytes-sent, etc. from RtpSession stats Advantages: simple Drawbacks Reactive and not predictive Requires packet loss to detect BW constraints Getting RtpSession stats quite inefficient
- 13. Congestion control II 13 ● For having satisfactory congestion control we should look to the following drafts draft-alvestrand-rmcat-remb-03: REMB mechanism implemented by Chrome draft-ietf-rmcat-cc-requirements-09: Congestion Control Requirements for Interactive Real-Time Media draft-ietf-avtcore-rtp-circuit-breakers-10: Multimedia Congestion Control: Circuit Breakers for Unicast RTP Sessions ● In summary, congestion control should: be predictive be efficient be fair "break the circuit" (drop the BW) in case of evidence of huge congestion This is to avoid breaking other types of traffic.
- 14. DataChannels 14 ● Kurento has DataChannels support ● Protocols: DTLS-SCTP ● Drafts draft-ietf-rtcweb-data-channel draft-ietf-rtcweb-data-protocol draft-ietf-mmusic-data-channel-sdpneg Impl. based on sctp(dec|enc) provided by Ericsson What have Kurento done? Fixed problems when using Gstreamer 1.4 Fixed a lot of bugs Implementation of the DataChannel protocol AppSrc, AppSink
- 15. Thank you Suggestions, comments and complains: mparisdiaz@gmail.com santoscadenas@gmail.com http://www.kurento.org http://www.github.com/kurento info@kurento.org Twitter: @kurentoms http://www.nubomedia.eu http://www.fi-ware.org http://ec.europa.eu