WebRTC is one of the main trends on the multimedia arena in the last few years. The ability of bringing real-time audio and video to WWW browsers opens new horizons for developers to create context aware customized applications for inter-human communications. However, for WebRTC technologies to work seamlessly in WWW applications, it’s necessary to manage with a number of present and future complex challenges. In this talk, we present the experience of the Kurento Media Server team in creating a WebRTC endpoint for GStreamer. We describe the main problems and limitations basing on current GStreamer status describing which parts of WebRTC standards can be currently implemented with GStreamer and which parts require further evolutions and efforts from the community. We will also describe the plans and drafts that are emerging at different standardization groups, including the WebRTC WG at W3C and the RTCWeb WG at IETF. Basing on this, we will try to forecast how WebRTC technologies in particular, but also how real-time multimedia communications in general, may be evolving in the next couple of years and the activities that the GStreamer community should be considering for adapting to these evolutions. In particular, we shall introduce in detail topics such as the following: • The evolution of ICE (Interactive Connectivity Establishment). • Congestion control for RTC streams. • Implementing WebRTC security securely • Implementing and optimizing the AVPF profile for RTP • Benchmarking WebRTC: stats metrics • Managing sensor data through DataChannels.

1. Implementing a WebRTC
endpoint in GStreamer:
challenges, problems and
perspectives
8-9 October 2015
Dublin, Ireland
Conference 2015
Miguel París
Jose A. Santos
santoscadenas@gmail.com
mparisdiaz@gmail.com

2. 2
Who we are
José Antonio Santos Cadenas
●
Software Engineer
●
Telematic Systems Master's
●
Developer at Naevatec (Madrid,
Spain)
●
Kurento Media Server (KMS)
manager
●
Integrating with client APIs
●
santoscadenas@gmail.com
Miguel París
●
Software Engineer
●
Telematic Systems Master's
●
Researcher at Universidad Rey
Juan Carlos (Madrid, Spain)
●
Kurento real-time manager
●
mparisdiaz@gmail.com
●
Twitter: @mparisdiaz

3. What is WebRTC
3
●
WebRTC is a new movement bringing RTC to the WWW
●
It is based on standards
●
Supported by Chrome & Firefox
●
Windows Edge added partial support recently
●
Safari is rumored to be on the way of providing it
●
FOSS implementations for Smartphone platforms (iOS and Android)
●
More tan 1B devices currently support WebRTC
Developing
RTC applications
for the WWW
Before WebRTC After WebRTC
Begin End
• Unified APIs
• Standards
• FOSS
• Multiplatform

4. WebRTC standards
4
●
Standard APIs: Standardized by the WebRTC WG at W3C
 Media Capture APIs
●
Standardize APIs for media capture
 PeerConnection APIs (WebRTC 1.0 almost finished)
●
Standardize API for P2P media communications
●
Media = Audio + Video + Data
●
DataChannel API: low latency
 WebRTC stats
●
RTP stats for inbound and outbound directions
●
encode/decode stats
●
DataChannel stats

5. Implementing WebRTC in GStreamer
5
●
Deep interest in the GStreamer community
●
Several implementations
 OpenWebRTC (Ericsson)
●
Design for working as part of client applications
 Kurento Media Server (Kurento.org)
●
Design for working as part of media server infrastructures
 Main differences between them:

OpenWebRTC captures media from camera and mic. KMS doesn't

OpenWebRTC renders to display and speakers. KMS doesn't

OpenWebRTC encodes/decodes audio/video. KMS only if needed.

6. 6
<GstPipeline>
pipeline0
[>]
KmsWebrtcEndpoint
kmswebrtcendpoint1
[>]
GstRTPRtxQueue
rtprtxqueue1
[>]
GstRtpVP8Pay
rtpvp8pay1
[>]
GstRtpOPUSPay
rtpopuspay1
[>]
KmsWebrtcSession
kmswebrtcsession1
[>]
KmsRtcpDemux
kmsrtcpdemux1
[>] GstRtpSsrcDemux
rtpssrcdemux5
[>]
KmsWebrtcTransportSinkNice
kmswebrtctransportsinknice1
[>]
GstNiceSink
nicesink1
[>]
GstDtlsSrtpEnc
dtlssrtpenc1
[>]
GstFunnel
funnel
[>]
GstSrtpEnc
srtp-encoder
[>]
GstDtlsEnc
dtls-encoder
[>]
KmsWebrtcTransportSrcNice
kmswebrtctransportsrcnice1
[>]
GstDtlsSrtpDec
dtlssrtpdec1
[>]
GstSrtpDec
srtp-decoder
[>]
GstDtlsDec
dtls-decoder
[>]
GstDtlsSrtpDemux
dtls-srtp-demux
[>]
GstNiceSrc
nicesrc1
[>]
GstRtpBin
rtpbin1
[>]
GstRtpSsrcDemux
rtpssrcdemux4
[>]
GstRtpSession
rtpsession3
[>]
GstRtpSsrcDemux
rtpssrcdemux3
[>]
GstRtpSession
rtpsession2
[>]
KmsWebrtcEndpoint
kmswebrtcendpoint0
[>]
GstRtpVP8Depay
rtpvp8depay0
[>]
KmsAgnosticBin2
kmsagnosticbin2-1
[>]
GstQueue
queue3
[>]
KmsParseTreeBin
kmsparsetreebin1
[>]
KmsVp8Parse
kmsvp8parse0
[>]
GstFakeSink
fakesink3
[>]
GstTee
tee3
[>]
GstFakeSink
fakesink2
[>]
GstTee
tee2
[>]
GstRTPOpusDepay
rtpopusdepay0
[>]
KmsAgnosticBin2
kmsagnosticbin2-0
[>]
GstQueue
queue1
[>]
KmsParseTreeBin
kmsparsetreebin0
[>]
GstOpusParse
opusparse0
[>]
GstFakeSink
fakesink1
[>]
GstTee
tee1
[>]
GstFakeSink
fakesink0
[>]
GstTee
tee0
[>]
GstRTPRtxQueue
rtprtxqueue0
[>]
GstRtpVP8Pay
rtpvp8pay0
[>]
GstRtpOPUSPay
rtpopuspay0
[>]
KmsWebrtcSession
kmswebrtcsession0
[>]
KmsRtcpDemux
kmsrtcpdemux0
[>]
GstRtpSsrcDemux
rtpssrcdemux2
[>]
KmsWebrtcTransportSinkNice
kmswebrtctransportsinknice0
[>]
GstNiceSink
nicesink0
[>]
GstDtlsSrtpEnc
dtlssrtpenc0
[>]
GstFunnel
funnel
[>]
GstSrtpEnc
srtp-encoder
[>]
GstDtlsEnc
dtls-encoder
[>]
KmsWebrtcTransportSrcNice
kmswebrtctransportsrcnice0
[>]
GstDtlsSrtpDec
dtlssrtpdec0
[>]
GstSrtpDec
srtp-decoder
[>]
GstDtlsDec
dtls-decoder
[>]
GstDtlsSrtpDemux
dtls-srtp-demux
[>]
GstNiceSrc
nicesrc0
[>]
GstRtpBin
rtpbin0
[>]
GstRtpJitterBuffer
rtpjitterbuffer1
[>] GstRtpPtDemux
rtpptdemux1
[>]
GstRtpJitterBuffer
rtpjitterbuffer0
[>] GstRtpPtDemux
rtpptdemux0
[>]
GstRtpSsrcDemux
rtpssrcdemux1
[>]
GstRtpSession
rtpsession1
[>]
GstRtpSsrcDemux
rtpssrcdemux0
[>]
GstRtpSession
rtpsession0
[>]
Legend
Element-States: [~] void-pending, [0] null, [-] ready, [=] paused, [>] playing
Pad-Activation: [-] none, [>] push, [<] pull
Pad-Flags: [b]locked, [f]lushing, [b]locking; upper-case is set
Pad-Task: [T] has started task, [t] has paused task
proxypad40
[>][bfb]
sink
[>][bfb]
sink_audio
[>][bfb]
proxypad42
[>][bfb]
sink
[>][bfb]
sink_video
[>][bfb]
sink
[>][bfb]
src
[>][bfb]
send_rtp_sink_1
[>][bfb]
proxypad33
[>][bfb]
src
[>][bfb]
src
[>][bfb]
send_rtp_sink_0
[>][bfb]
proxypad31
[>][bfb]
send_rtp_src_0
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
rtcp_src
[>][bfb] rtcp_sink
[>][bfb]
sink
[>][bfb]
src_1
[>][bfb]
recv_rtp_sink_1
[>][bfb]
rtcp_src_1
[>][bfb] recv_rtcp_sink_1
[>][bfb]
src_421259003
[>][bfb] recv_rtp_sink_0
[>][bfb]
rtcp_src_421259003
[>][bfb] recv_rtcp_sink_0
[>][bfb]
proxypad44
[>][bfb]
proxypad45
[>][bfb]
proxypad46
[>][bfb]
proxypad47
[>][bfb]
sink
[>][bfb]
proxypad34
[>][bfb]
rtp_sink_0
[>][bfb]
rtp_sink_0
[>][bfb]
src
[>][bfb]
proxypad36
[>][bfb]
rtcp_sink_0
[>][bfb]
rtcp_sink_0
[>][bfb]
proxypad37
[>][bfb]
rtp_sink_1
[>][bfb]
rtp_sink_1
[>][bfb]
proxypad39
[>][bfb]
rtcp_sink_1
[>][bfb]
rtcp_sink_1
[>][bfb]
proxypad29
[>][bfb]
funnelpad5
[>][bfb]
src
[>][bfb]
funnelpad6
[>][bfb]
funnelpad7
[>][bfb]
funnelpad8
[>][bfb]
funnelpad9
[>][bfb]
rtp_src_0
[>][bfb]
rtcp_src_0
[>][bfb]
rtp_src_1
[>][bfb]
rtcp_src_1
[>][bfb]
src
[>][bfb][T]
proxypad28
[>][bfb]
sink
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
proxypad26
[>][bfb]
proxypad27
[>][bfb]
rtcp_src
[>][bfb]
rtp_sink
[>][bfb]
rtp_src
[>][bfb]
rtcp_sink
[>][bfb]
rtcp_src
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
dtls_src
[>][bfb]
src
[>][bfb][T]
send_rtp_sink
[>][bfb]
send_rtp_sink
[>][bfb]
recv_rtp_sink
[>][bfb]
recv_rtcp_sink
[>][bfb]
recv_rtp_sink
[>][bfb]
recv_rtcp_sink
[>][bfb]
proxypad30
[>][bfb]
proxypad32
[>][bfb]
send_rtp_src_1
[>][bfb]
proxypad35
[>][bfb]
send_rtcp_src_0
[>][bfb]
proxypad38
[>][bfb]
send_rtcp_src_1
[>][bfb]
sink
[>][bfb]
rtcp_sink
[>][bfb]
send_rtp_src
[>][bfb]
send_rtcp_src
[>][bfb]
recv_rtp_src
[>][bfb]
sync_src
[>][bfb]
sink
[>][bfb]
rtcp_sink
[>][bfb]
send_rtp_src
[>][bfb]
send_rtcp_src
[>][bfb]
recv_rtp_src
[>][bfb]
sync_src
[>][bfb]
proxypad14
[>][bfb]
sink
[>][bfb]
sink_audio
[>][bfb]
audio_src_0
[>][bfb]
proxypad15
[>][bfb]
sink
[>][bfb]
sink_video
[>][bfb]
proxypad24
[>][bfb]
proxypad25
[>][bfb]
video_src_0
[>][bfb]
sink
[>][bfb]
src
[>][bfb]
sink
[>][bfb]
proxypad23
[>][bfb]
src_0
[>][bfb]
sink
[>][bfb]
proxypad43
[>][bfb]sink
[>][bfb]
src
[>][bfb][T]
sink
[>][bfb]
src
[>][bfb]
sink
[>][bfb]
src_0
[>][bfb]
sink
[>][bfb]
src_2
[>][bfb]
sink
[>][bfb]
src_0
[>][bfb]
src_1
[>][bfb]
sink
[>][bfb]
src
[>][bfb]
sink
[>][bfb]
proxypad19
[>][bfb]
src_0
[>][bfb]
sink
[>][bfb]
proxypad41
[>][bfb]sink
[>][bfb]
src
[>][bfb][T]
sink
[>][bfb]
src
[>][bfb]
sink
[>][bfb]
src_0
[>][bfb]
sink
[>][bfb]
src_2
[>][bfb]
sink
[>][bfb]
src_0
[>][bfb]
src_1
[>][bfb]
sink
[>][bfb]
src
[>][bfb]
send_rtp_sink_1
[>][bfb]
proxypad7
[>][bfb]
src
[>][bfb]
src
[>][bfb]
send_rtp_sink_0
[>][bfb]
proxypad5
[>][bfb]
send_rtp_src_0
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
rtcp_src
[>][bfb]
rtcp_sink
[>][bfb]
sink
[>][bfb]
src_1442068093
[>][bfb]
recv_rtp_sink_0
[>][bfb]
rtcp_src_1442068093
[>][bfb]
recv_rtcp_sink_0
[>][bfb]
src_836061664
[>][bfb]
recv_rtp_sink_1
[>][bfb]
rtcp_src_836061664
[>][bfb]
recv_rtcp_sink_1
[>][bfb]
proxypad16
[>][bfb]
proxypad17
[>][bfb]
proxypad20
[>][bfb]
proxypad21
[>][bfb]
sink
[>][bfb]
proxypad8
[>][bfb]
rtp_sink_0
[>][bfb]
rtp_sink_0
[>][bfb]
src
[>][bfb]
proxypad10
[>][bfb]
rtcp_sink_0
[>][bfb]
rtcp_sink_0
[>][bfb]
proxypad11
[>][bfb]
rtp_sink_1
[>][bfb]
rtp_sink_1
[>][bfb]
proxypad13
[>][bfb]
rtcp_sink_1
[>][bfb]
rtcp_sink_1
[>][bfb]
proxypad3
[>][bfb]
funnelpad0
[>][bfb]
src
[>][bfb]
funnelpad1
[>][bfb]
funnelpad2
[>][bfb]
funnelpad3
[>][bfb]
funnelpad4
[>][bfb]
rtp_src_0
[>][bfb]
rtcp_src_0
[>][bfb]
rtp_src_1
[>][bfb]
rtcp_src_1
[>][bfb]
src
[>][bfb][T]
proxypad2
[>][bfb]
sink
[>][bfb]
sink
[>][bfb] rtp_src
[>][bfb]
proxypad0
[>][bfb]
proxypad1
[>][bfb]
rtcp_src
[>][bfb]
rtp_sink
[>][bfb]
rtp_src
[>][bfb]
rtcp_sink
[>][bfb]
rtcp_src
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
dtls_src
[>][bfb]
src
[>][bfb][T]
send_rtp_sink
[>][bfb]
send_rtp_sink
[>][bfb]
recv_rtp_sink
[>][bfb]
recv_rtcp_sink
[>][bfb]
recv_rtp_sink
[>][bfb]
recv_rtcp_sink
[>][bfb]
proxypad4
[>][bfb]
proxypad6
[>][bfb]
send_rtp_src_1
[>][bfb]
proxypad9
[>][bfb]
send_rtcp_src_0
[>][bfb]
proxypad12
[>][bfb]
send_rtcp_src_1
[>][bfb]
proxypad18
[>][bfb]
recv_rtp_src_0_1442068093_111
[>][bfb]
proxypad22
[>][bfb]
recv_rtp_src_1_836061664_100
[>][bfb]
sink
[>][bfb] src
[>][bfb][T]
sink_rtcp
[>][bfb]
sink
[>][bfb]
src_100
[>][bfb]
sink
[>][bfb] src
[>][bfb][T]
sink_rtcp
[>][bfb]
sink
[>][bfb]
src_111
[>][bfb]
sink
[>][bfb]
src_836061664
[>][bfb]
rtcp_sink
[>][bfb]
rtcp_src_836061664
[>][bfb]
send_rtp_src
[>][bfb]
send_rtcp_src
[>][bfb]
recv_rtp_src
[>][bfb]
sync_src
[>][bfb]
sink
[>][bfb]
src_1442068093
[>][bfb]
rtcp_sink
[>][bfb]
rtcp_src_1442068093
[>][bfb]
send_rtp_src
[>][bfb]
send_rtcp_src
[>][bfb]
recv_rtp_src
[>][bfb]
sync_src
[>][bfb]
Kurento WebRtcEp
The pipeline

7. Interactive Connectivity Establishment
ICE I
7
●
Using libnice (NiceSrc and NiceSink)

Involved RFCs:
 RFC 5245: Interactive Connectivity Establishment
(ICE): A Protocol for Network Address Translator (NAT)
Traversal for Offer/Answer Protocols
 RFC 5389: Session Traversal Utilities for NAT (STUN)
 RFC 5766: Traversal Using Relays around NAT (TURN):
Relay Extensions to Session Traversal Utilities for NAT
(STUN)

8. Interactive Connectivity Establishment
ICE II
8
●
Our experience with libnice
 0.1.7 version seem to be stable, but lack many
required features
 0.1.10 version incorporate relevant features but seem
not to provide the appropriate stability yet
●
Leaks of TCP sockets
●
Connection states
●
Connectivity problems (Fails 1/1000)
 Fixed in 0.1.13 ? (we have to check it)

9. ICE III
9
●
What next needed in ICE for WebRTC?
 IPv6 shall be mandatory in WebRTC.

We need to check IPv6 support in libnice

draft-ietf-rtcweb-transports-09 (mandates IPv6 support)
 HTTP CONNECT protocol for TURN might be important in
corporate networks (connectivity)

draft-hutton-httpbis-connect-protocol-00
 STUN consent freshness may become mandatory for
WebRTC. This may require some adjustments on libnice for
supporting consent negotiation (security)

Draft-ietf-rtcweb-stun-consent-freshness-16
 ICE stated published through events

ICE state should be reported as specified in the WebRTC
1.0 WG draft

10. DTLS / SRTP I
10
●
Using dtlssrtp(dec|enc) bins
 dtls(dec|enc) manages DTLS handshake
 srtp(dec|enc) decrypts/encrypts RTP packets
●
Involved RFCs:
 RFC 5764: Datagram Transport Security (DTLS)
Extension to Establish Keys for the Secure Real-time
Protocol (SRTP)
 RFC 3711: The Secure Real-time Transport Protocol
(SRTP)

11. DTLS / SRTP II
11
●
What next for WebRTC?
 New cryptographic framework for group communications

This is under definition at the Privacy Enhanced RTP Conferencing (PERC) IETF
WG

Only early drafts here
 draft-jones-perc-private-media-framework-00
 draft-jones-perc-private-media-reqts-00
 draft-mattsson-perc-srtp-cloud-00
 draft-westerlund-perc-webrtc-use-case-00

PERC shall provide in the next few months (improve efficiency)
 Enabling unicast group communications without requiring infrastructure to
decipher
 New mechanisms enabling unicast group communications without
requiring infrastructure to cipher once per RTC session.

These may require re-factoring of current DTLS stack
 DTLS state published through events.

Full connectivity requires ICE connectivity + successful DTLS handshake.

DTLS is being incorporated by the W3C in the WebRTC 1.0 spec as event
generator dealing with connection health

12. Congestion control I
12
●
This is an essential feature for providing satisfactory QoE for WebRTC users

How does it work?
 Bandwidth estimator

Based on packet delay (useful RTP ExtHdr: abs-send-time)

Based on packet loss
 REMB (Receiver Estimated Maximum Bitrate)

GStreamer does not provide any kind of features for this off the self

How Kurento estimates bandwidth?
 on-sending-rtcp
 Get packet-loss, bytes-sent, etc. from RtpSession stats
 Advantages: simple
 Drawbacks

Reactive and not predictive

Requires packet loss to detect BW constraints

Getting RtpSession stats quite inefficient

13. Congestion control II
13
●
For having satisfactory congestion control we should look to the following
drafts
 draft-alvestrand-rmcat-remb-03: REMB mechanism implemented by
Chrome
 draft-ietf-rmcat-cc-requirements-09: Congestion Control Requirements
for Interactive Real-Time Media
 draft-ietf-avtcore-rtp-circuit-breakers-10: Multimedia Congestion Control:
Circuit Breakers for Unicast RTP Sessions
●
In summary, congestion control should:
 be predictive
 be efficient
 be fair
 "break the circuit" (drop the BW) in case of evidence of huge congestion

This is to avoid breaking other types of traffic.

14. DataChannels
14
●
Kurento has DataChannels support
●
Protocols: DTLS-SCTP
●
Drafts
 draft-ietf-rtcweb-data-channel
 draft-ietf-rtcweb-data-protocol
 draft-ietf-mmusic-data-channel-sdpneg

Impl. based on sctp(dec|enc) provided by Ericsson

What have Kurento done?
 Fixed problems when using Gstreamer 1.4
 Fixed a lot of bugs
 Implementation of the DataChannel protocol

AppSrc, AppSink

15. Thank you
Suggestions, comments and complains:
mparisdiaz@gmail.com
santoscadenas@gmail.com
http://www.kurento.org
http://www.github.com/kurento
info@kurento.org
Twitter: @kurentoms
http://www.nubomedia.eu
http://www.fi-ware.org
http://ec.europa.eu