Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

FIWARE Global Summit - Real-time Media Stream Processing Using Kurento


Published on

Presentation by Juan Navarro Moreno
Kurento Software Developer, Naevatec

FIWARE Global Summit
8-9 May, 2018
Porto, Portugal

Published in: Technology
  • Be the first to comment

  • Be the first to like this

FIWARE Global Summit - Real-time Media Stream Processing Using Kurento

  1. 1. Real-Time Media Stream Processing using Kurento Juan Navarro Moreno Kurento Software Developer
  2. 2. Overview – What is Kurento? ● A server that simplifies creation of streaming applications. ● Focused on processing of Audio/Video media streams. ● Supporting WebRTC communications. ● Simple workflow: input → process → output. Kurento is an Open Source project. Hosted on GitHub:
  3. 3. Overview – Why Kurento? ● Abstracts compatibility issues between sender and receiver. ● Manipulates or redistributes the streams. ● Extracts information from the streams. Input stream Kurento Media Server Output stream Product-specific data Eg.: FIWARE data-store
  4. 4. Overview – Why Kurento? Examples: ● Abstraction of incompatible video codecs ● Multi-point distribution of media ● Processing / Storing media
  5. 5. Overview – Why Kurento?
  6. 6. Overview – How does Kurento work? A server with two main components: ● Endpoints: where data flows In/Out. ● Filters: composable modules where data is processed or transformed. Media stream Src Sink Media stream Input Endpoint Output Endpoint
  7. 7. Overview – How does Kurento work? Endpoints and Filters are linked together, mix-and-matched to form a Pipeline. Input Endpoint Src Sink Output Endpoint Src Sink Src Sink Src Sink
  8. 8. Overview – How does Kurento work? The server is controlled with an RPC API. ● Client applications manipulate Endpoints and Filters through this API. ● Ready-made client SDKs for Java, Node.js and in-browser JavaScript. Components of a Real-World application: ● Kurento Media Server ● Client Application – usually a server too. ● User Interface – common case is a web page.
  9. 9. Overview – How does Kurento work?
  10. 10. Getting technical
  11. 11. Component overview Kurento Media Server (aka. “KMS”) ● Core: Basic functionality and RPC API. ● Elements: Endpoints used for input and output of streams. ● Filters: Implementation of all stream- processing modules.
  12. 12. KMS Elements Protocols and Codecs Media Repository WebRtcEndpoint RtpEndpoint HttpEndpoint PlayerEndpoint RecorderEndpoint
  13. 13. KMS Elements Different Endpoints for different needs: ● WebRtcEndpoint – Full support of WebRTC standard. • Currently compatible with Chrome and Firefox (Safari and Edge are work-in-progress). • All WebRTC lingo: SDP, (Trickle-)ICE, STUN, TURN, Google REMB. ● RtpEndpoint – For RTP and SRTP streams. • Supports port auto-discovery as an alternative to ICE. ● HttpEndpoint – Accepts GET/POST requests (eg. file uploading to KMS).
  14. 14. KMS Elements Special Endpoints: ● PlayerEndpoint – Retrieves content from either of the file system, HTTP servers, or RTSP sources. Input-only. ● RecorderEndpoint – Redirects streams to storage. Output-only.
  15. 15. KMS Filters Process or transform data as it flows through filters. ● Could be a simple transformation. Eg: Convert video to black and white. ● Could be a complex task, involving external libraries. Eg: Apply Computer Vision algorithms and extract features from the video. ● Imagination is the limit.
  16. 16. KMS Filters Computer Vision Generic filter Zbar FaceDetector PlateDetector GStreamerFilter
  17. 17. Pipelines ● All elements get created and linked through specific RPC commands from the Client Application. ● As seen before, complex topologies are possible.
  18. 18. Pipelines SinkSink SRCSRC SinkSink SRCSRCSinkSink SinkSinkSRCSRC WebRtcEndpoint AR Filter RecorderEndpoint HttpGetEndpoint RtpEndpoint MP4 fle stored in media repository Web application using HTML5 <video> tag RTP full duplex client video phone WebRTC full duplex client video application SinkSink
  19. 19. Client Application ● Implements or includes an RPC client. ● Orchestrates the creation of the Pipeline, with Elements and Filters. ● WebRtcEndpoint follows the standard way of configuration via SDP Negotiation (SDP Offer/Answer Model). • The Client Application is in charge of passing around ICE Candidates. ● RtpEndpoint also uses SDP for configuration, but no ICE. ● Other Endpoints have custom RPC methods for configuration.
  20. 20. Client Application ● Provides access to external sources such as FIWARE data-stores. ● Proceeds with the rest of the business logic. ● Kurento provides a FIWARE integration package for Java applications. ● Also there are multiple example applications available.
  21. 21. Client Application Application Server Application Server Media Server Media ServerClientClient Media Negotiation phase 11 Media channel preparation phase (optional) 22 Media exchange phase 33 Specifc app logic Pipeline building Pipeline managed media SDP Ofer (Mangled) SDP Ofer SDP Answer(Mangled) SDP Answer ICE Candidates ICE Candidates
  22. 22. A closer look to some of the underlying technologies
  23. 23. A closer look – GStreamer media library Kurento is powered under the hood by the GStreamer project: GStreamer is in charge of all media handling: ● Decoding / encoding of input / output video & audio. ● Establishment of communication streams (eg. RTP Sessions). ● Special interest: Agnostic bin.
  24. 24. A closer look – GStreamer Agnostic bin Kurento abstracts codec compatibility issues between Endpoints. ● Achieved through an “invisible” intermediate filter – ‘agnosticbin’. ● Applies on-the-fly transcoding iff the codecs are not compatible. WebRtcEndpoint WebRtcEndpoint Agnostic media adapter, “hidden” behind every connection Agnostic media adapter, “hidden” behind every connection H.264VP8 SinkSink SRCSRC SinkSink SRCSRC
  25. 25. A closer look – SDP Offer/Answer ● The standard signaling format for WebRTC. ● Used to configure codecs & advanced settings in WebRtcEndpoint and RtpEndpoint. ● SDP started simple but has grown to become a mess. See: (The problem is, the IETF RTCWEB working group was/is dominated by players from the SIP world who wanted to re-use their existing code.)
  26. 26. A closer look – SDP Offer/Answer v=0 o=- 0 0 IN IP4 s=- c=IN IP4 t=0 0 m=audio 9 RTP/AVPF 96 a=rtpmap:96 opus/48000/2 a=sendonly a=direction:active a=ssrc:445566 m=video 9 RTP/AVPF 103 a=rtpmap:103 H264/90000 a=sendonly a=direction:active a=ssrc:112233 v=0 o=- 372 372 IN IP4 s=Kurento Media Server c=IN IP4 t=0 0 m=audio 41654 RTP/AVPF 96 a=rtpmap:96 opus/48000/2 a=recvonly a=direction:passive a=ssrc:3224 cname:user225@host-9b2 m=video 61134 RTP/AVPF 103 a=rtpmap:103 H264/90000 a=recvonly a=direction:passive a=ssrc:2717 cname:user225@host-9b2 SDP Offer to RtpEndpoint: SDP Answer from RtpEndpoint:
  27. 27. Application examples
  28. 28. A closer look – Application example: RTP Receiver Kurento RTP Player is a recent addition to the Kurento Tutorials: ● Receives an audio/video stream. ● Redirects it and sends to a browser through WebRTC. ● RTP producer
  29. 29. A closer look – Application example: RTP Player void start(String browserSdpOffer) { // Create and link Endpoints MediaPipeline pipeline = kurento.createMediaPipeline(); RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build(); WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); rtpEndpoint.connect(webRtcEndpoint); // Configure the RtpEndpoint String fakeSdpOffer = "..."; String kmsSdpAnswer = rtpEndpoint.processOffer(fakeSdpOffer); String browserSdpAnswer = webRtcEndpoint.processOffer(browserSdpOffer); webRtcEndpoint.gatherCandidates(); }
  30. 30. A closer look – Application example: Magic Mirror ● Receives a video stream from a browser. ● Applies Computer Vision to detect a face and overlay a picture. ● Sends back the modified video to the same Endpoint it came from. ●
  31. 31. A closer look – Application example: Magic Mirror void start() { // Create and link Endpoints, Filters MediaPipeline pipeline = kurento.createMediaPipeline(); WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); FaceOverlayFilter faceOverlayFilter = new FaceOverlayFilter.Builder(pipeline).build(); faceOverlayFilter.setOverlayedImage("mario-hat.png"); webRtcEndpoint.connect(faceOverlayFilter); faceOverlayFilter.connect(webRtcEndpoint); webRtcEndpoint.gatherCandidates(); }
  32. 32. Closing & Questions
  33. 33. Thank you! Follow @FIWARE on Twitter Juan Navarro Moreno Kurento Software Developer