Kurento is an open source software that simplifies the creation of real-time communication applications involving audio and video streams. It provides a server that abstracts compatibility issues between senders and receivers and allows for manipulation or redistribution of streams. The server includes endpoints for stream input/output and filters for processing or transforming media as it flows through the pipeline. Example applications demonstrated by Kurento include an RTP receiver that redirects streams to a browser and a magic mirror that applies computer vision to detect and overlay images on a face in real-time video.

1. Real-Time Media Stream Processing using Kurento
Juan Navarro Moreno
Kurento Software Developer
jnavarro@naevatec.com
github.com/j1elo
www.kurento.org

2. Overview – What is Kurento?
●
A server that simplifies creation of streaming applications.
●
Focused on processing of Audio/Video media streams.
●
Supporting WebRTC communications.
●
Simple workflow: input → process → output.
Kurento is an Open Source project.
Hosted on GitHub: https://github.com/Kurento

3. Overview – Why Kurento?
●
Abstracts compatibility issues between sender and receiver.
●
Manipulates or redistributes the streams.
●
Extracts information from the streams.
Input stream
Kurento
Media
Server
Output stream
Product-specific data
Eg.: FIWARE data-store

4. Overview – Why Kurento?
Examples:
●
Abstraction of incompatible
video codecs
●
Multi-point distribution of media
●
Processing / Storing media

5. Overview – Why Kurento?

6. Overview – How does Kurento work?
A server with two main components:
●
Endpoints: where data flows In/Out.
●
Filters: composable modules where data is processed or
transformed.
Media stream
Src
Sink Media stream
Input
Endpoint
Output
Endpoint

7. Overview – How does Kurento work?
Endpoints and Filters are linked together, mix-and-matched to form a
Pipeline.
Input
Endpoint
Src
Sink
Output
Endpoint
Src
Sink
Src
Sink
Src
Sink

8. Overview – How does Kurento work?
The server is controlled with an RPC API.
●
Client applications manipulate Endpoints and Filters through this API.
●
Ready-made client SDKs for Java, Node.js and in-browser JavaScript.
Components of a Real-World application:
●
Kurento Media Server
●
Client Application – usually a server too.
●
User Interface – common case is a web page.

9. Overview – How does Kurento work?

10. Getting technical

11. Component overview
Kurento Media Server (aka. “KMS”)
●
Core: Basic functionality and RPC API.
●
Elements: Endpoints used for input and
output of streams.
●
Filters: Implementation of all stream-
processing modules.

12. KMS Elements
Protocols and Codecs Media Repository
WebRtcEndpoint
RtpEndpoint
HttpEndpoint
PlayerEndpoint
RecorderEndpoint

13. KMS Elements
Different Endpoints for different needs:
●
WebRtcEndpoint – Full support of WebRTC standard.
•
Currently compatible with Chrome and Firefox (Safari and Edge are
work-in-progress).
•
All WebRTC lingo: SDP, (Trickle-)ICE, STUN, TURN, Google REMB.
●
RtpEndpoint – For RTP and SRTP streams.
•
Supports port auto-discovery as an alternative to ICE.
●
HttpEndpoint – Accepts GET/POST requests (eg. file uploading to KMS).

14. KMS Elements
Special Endpoints:
●
PlayerEndpoint – Retrieves content from either of the file system, HTTP
servers, or RTSP sources. Input-only.
●
RecorderEndpoint – Redirects streams to storage. Output-only.

15. KMS Filters
Process or transform data as it flows through filters.
●
Could be a simple transformation. Eg: Convert video to black and white.
●
Could be a complex task, involving external libraries. Eg: Apply
Computer Vision algorithms and extract features from the video.
●
Imagination is the limit.

16. KMS Filters
Computer Vision Generic filter
Zbar
FaceDetector
PlateDetector
GStreamerFilter

17. Pipelines
●
All elements get created and linked through specific RPC commands
from the Client Application.
●
As seen before, complex topologies are possible.

18. Pipelines
SinkSink
SRCSRC
SinkSink
SRCSRCSinkSink
SinkSinkSRCSRC
WebRtcEndpoint
AR Filter
RecorderEndpoint
HttpGetEndpoint
RtpEndpoint
MP4 fle stored
in media
repository
Web application
using HTML5
<video> tag
RTP full duplex
client video
phone
WebRTC full
duplex client
video application
SinkSink

19. Client Application
●
Implements or includes an RPC client.
●
Orchestrates the creation of the Pipeline, with Elements and Filters.
●
WebRtcEndpoint follows the standard way of configuration via SDP
Negotiation (SDP Offer/Answer Model).
•
The Client Application is in charge of passing around ICE Candidates.
●
RtpEndpoint also uses SDP for configuration, but no ICE.
●
Other Endpoints have custom RPC methods for configuration.

20. Client Application
●
Provides access to external sources such as FIWARE data-stores.
●
Proceeds with the rest of the business logic.
●
Kurento provides a FIWARE integration package for Java applications.
●
Also there are multiple example applications available.

21. Client Application
Application
Server
Application
Server
Media
Server
Media
ServerClientClient
Media
Negotiation
phase
11
Media channel
preparation
phase
(optional)
22
Media
exchange
phase
33
Specifc app logic
Pipeline
building
Pipeline
managed
media
SDP Ofer
(Mangled) SDP Ofer
SDP Answer(Mangled) SDP Answer
ICE Candidates
ICE Candidates

22. A closer look to some of the underlying technologies

23. A closer look – GStreamer media library
Kurento is powered under the hood by the GStreamer project:
https://gstreamer.freedesktop.org
GStreamer is in charge of all media handling:
●
Decoding / encoding of input / output video & audio.
●
Establishment of communication streams (eg. RTP Sessions).
●
Special interest: Agnostic bin.

24. A closer look – GStreamer Agnostic bin
Kurento abstracts codec compatibility issues between Endpoints.
●
Achieved through an “invisible” intermediate filter – ‘agnosticbin’.
●
Applies on-the-fly transcoding iff the codecs are not compatible.
WebRtcEndpoint WebRtcEndpoint
Agnostic media
adapter, “hidden”
behind every
connection
Agnostic media
adapter, “hidden”
behind every
connection
H.264VP8
SinkSink
SRCSRC
SinkSink
SRCSRC

25. A closer look – SDP Offer/Answer
●
The standard signaling format for WebRTC.
●
Used to configure codecs & advanced settings in WebRtcEndpoint and
RtpEndpoint.
●
SDP started simple but has grown to become a mess.
See: https://webrtchacks.com/sdp-anatomy/
(The problem is, the IETF RTCWEB working group was/is dominated by players from the SIP world who
wanted to re-use their existing code.)

26. A closer look – SDP Offer/Answer
v=0
o=- 0 0 IN IP4 127.0.0.1
s=-
c=IN IP4 127.0.0.1
t=0 0
m=audio 9 RTP/AVPF 96
a=rtpmap:96 opus/48000/2
a=sendonly
a=direction:active
a=ssrc:445566 cname:user@example.com
m=video 9 RTP/AVPF 103
a=rtpmap:103 H264/90000
a=sendonly
a=direction:active
a=ssrc:112233 cname:user@example.com
v=0
o=- 372 372 IN IP4 192.168.1.15
s=Kurento Media Server
c=IN IP4 192.168.1.15
t=0 0
m=audio 41654 RTP/AVPF 96
a=rtpmap:96 opus/48000/2
a=recvonly
a=direction:passive
a=ssrc:3224 cname:user225@host-9b2
m=video 61134 RTP/AVPF 103
a=rtpmap:103 H264/90000
a=recvonly
a=direction:passive
a=ssrc:2717 cname:user225@host-9b2
SDP Offer to RtpEndpoint: SDP Answer from RtpEndpoint:

27. Application examples

28. A closer look – Application example: RTP Receiver
Kurento RTP Player is a recent addition to the Kurento Tutorials:
●
Receives an audio/video stream.
●
Redirects it and sends to a browser through WebRTC.
●
http://doc-kurento.readthedocs.io/en/latest/user/tutorials.html#rtp-receiver
RTP
producer

29. A closer look – Application example: RTP Player
void start(String browserSdpOffer)
{
// Create and link Endpoints
MediaPipeline pipeline = kurento.createMediaPipeline();
RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
rtpEndpoint.connect(webRtcEndpoint);
// Configure the RtpEndpoint
String fakeSdpOffer = "...";
String kmsSdpAnswer = rtpEndpoint.processOffer(fakeSdpOffer);
String browserSdpAnswer = webRtcEndpoint.processOffer(browserSdpOffer);
webRtcEndpoint.gatherCandidates();
}

30. A closer look – Application example: Magic Mirror
●
Receives a video stream from a browser.
●
Applies Computer Vision to detect a face and overlay a picture.
●
Sends back the modified video to the same Endpoint it came from.
●
http://doc-kurento.readthedocs.io/en/latest/user/tutorials.html#webrtc-magic-mirror

31. A closer look – Application example: Magic Mirror
void start()
{
// Create and link Endpoints, Filters
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
FaceOverlayFilter faceOverlayFilter =
new FaceOverlayFilter.Builder(pipeline).build();
faceOverlayFilter.setOverlayedImage("mario-hat.png");
webRtcEndpoint.connect(faceOverlayFilter);
faceOverlayFilter.connect(webRtcEndpoint);
webRtcEndpoint.gatherCandidates();
}

32. Closing & Questions

33. Thank you!
http://fiware.org
Follow @FIWARE on Twitter
Juan Navarro Moreno
Kurento Software Developer
jnavarro@naevatec.com
github.com/j1elo
www.kurento.org