Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The future of multimedia communications and services: Kurento and it's role


Published on

This is a presentation specifically created for the GSMA interest group on WebRTC. This presentations introduces Kurento from the perspective of operators. Kurento is a multimedia development framework. It has been created to ease the life of multimedia application developers. Using multimedia capabilities such as embedding a video onto your app or establishing a video conferencing link between two clients may be tricky, but there is no rocket science there. However, for applications requiring more advanced features things quickly get unmanageable. If you have been involved in multimedia projects, you probably know that features such as interoperable group communications, different communication roles (e. g.. publishers/viewers), video transforming and transcoding, video storage and tagging, integration into legacy video/voice infrastructures, computer vision, augmented reality, integration with external systems and databases and many others, pose quite a complex challenge, which usually requires huge expertise and effort. Specially when real-time communications are involved. If this is your case, Kurento will help you.

Published in: Internet, Technology, Education
  • Be the first to comment

The future of multimedia communications and services: Kurento and it's role

  1. 1. Presentation for the GSMA WebRTC interest group on April 7th, 2014 The future of multimedia communications and services: Kurento and it’s role Luis Lopez
  2. 2. Who am I? • Luis Lopez – Associate Professor at ETSIT/URJC • • – Director of the FUN-LAB research group • Future Networks Laboratory • – Leader of the initiative • – Software engineer, software developer and Open Source Software Advocate 2
  3. 3. What does Kurento mean? 3
  4. 4. Kurento’s History 2004 •Kurento is born as a tool for transferring research results to industry 2012 •Kurento goes open source LGPL v2.1. Professional services provided by Naevatec (startup) June’13 •1M€ of investment for providing multimedia capabilities to FI-WARE Dec13 •Kurento Featured at WebRTC Expo (Paris) Feb’14 •3.3M€ of investment for making Kurento an elastic cloud platform (NUBOMEDIA) April’14 •Kurento featured at IMS World Conference (Barcelone) July’14 •Kurento featured at DevCon5 (New York) Sep’14 •500K€ of investment for maintenance of Kurento FI- WARE APIs 4
  5. 5. Kurento team • 18 people: 15 developers, 1 sysadmin, 2 coordination 5
  6. 6. Real-time communications: our vision Numberofusers General purpose RTC multimedia Services Specific purpose RTC multimedia services Commodity, no business opportunities here Who cares about the immense minority? Many business opportunities here 6
  7. 7. The opportunity: from services to platforms and APIs 7
  8. 8. Who shares this vision: The WebRTC movement Numberofusers General purpose RTC multimedia Services Specific purpose RTC multimedia services Commodity, no business opportunities here Who cares about the immense minority? Many business opportunities here Disruption direction of WebRTC 8
  9. 9. Before and after WebRTC: the perspective of the application developer Developing the client side Before WebRTC After WebRTC Begin End • Unified APIs • Standards • FOSS • Multiplatform 9
  10. 10. Why developers are important for the WebRTC movement? The effect of WWW developers on different technological areas 10
  11. 11. Should telcos take the hint? WebRTC WWW development community RCS/IMS development community WebRTC makes multimedia RTC to be just another feature of your WWW application 11
  12. 12. What’s the relationship of WebRTC with Kurento? Developing the client side Developing the infrastructure side Before WebRTC After WebRTC Next natural step… Begin End Begin End Begin End • Unified APIs • Standards • FOSS • Multiplatform • Unified APIs • Standards • FOSS • Multiplatform • Unified APIs • Standards • FOSS • Multiplatform 12
  13. 13. But is the infrastructure important? What RTC applications will be demanded by the market on the next few years? 13
  14. 14. Kurento vision for Future RTC multimedia services Media is here Media got there Media got there Analyze Transform Store Transport Enrich Augment Adapt Sensors Context Events Media is here 14
  15. 15. Enrich, analyze, adapt, augment: What’s the problem? Complexity 15
  16. 16. Future Internet Multimedia Infrastructure Simple Development APIs The Kurento equation for Future Internet infrastructures 16
  17. 17. What’s Kurento? • – Developer platform • Infrastructures + APIs • Kurento is not a service – Future Internet • Generalized multimedia – Audio/Video/Sensor • Advanced media processing – Computer Vision, Augmented reality • Interoperability – WebRTC, RTP, HTTP, all codecs, etc. – Open Source Software • LGPL v2.1 17
  18. 18. Kurento Media Server: The nucleus of Kurento • KMS is a middleware for media streams – Receives the stream – Process the stream – Issues the stream Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS 18
  19. 19. Media API: The API for accessing KMS capabilities Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS Java Media API JavaScript Media API Other languages Applications define the processing of streams getting through KMS 19
  20. 20. Media API: media elements and media pipelines SinkSRC Sink SRC SRCSink Sink Media Element • Provides a specific media functionality › Send/receive media › Process media › Transform media • Exchange media through › Sources › Sinks Media pipeline • Chain of media elements implementing the desired media logic. • The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Media Element Sink SRC 20
  21. 21. The Lego Game: Developers create applications connecting media elements Protocols & Codecs Computer Vision Augmented Reality Mul sensory Mul media Media Repository IPTV Integra on Group Communica ons Sink% H pEndpoint Sink%SRC% RtpEndpoint SRC$ DataChannelEndpointSink%SRC% WebRtcEndpoint Sink% SRC% BarCodeReader Sink% SRC% PointerTracker Sink% SRC% FaceDetector Sink% SRC% FaceOverlay Sink% SRC% ChromaFilter Sink% SRC% HeartRateBlender Sink% RecorderEndpoint SRC$ PlayerEndpoint Sink% CdnUpload Sink% IPTVConnector Mixer SinkSRC Sink SRC Sink SRCSink Sink Application 2 Sink SRC Sink SRCSink SinkSRC Sink Application 3 SinkSRC SRCSink SRC Sink SinkSRC Application 1 Toolbox of media elements 21
  22. 22. Media API trivial example MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build(); HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(httpEndpoint); httpEndpoint.getUrl(); //URL where the media is made available Media Pipeline HttpGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint 22
  23. 23. The signaling: adapting the WWW model to the multimedia services Process WWW request - DDBB access - Authentication - XML processing - Etc. HTTP request: I want this resource HTTP response: The resource Process media request - Media API - DDBB access - Authentication - XML processing - Etc. Signaling request: I want this media Signaling response: The media is here This is how you create WWW applications (Servlets, ASP, PHP, Rails, etc.) This is how you create Kurento enabled applications: Multimedia RTC is just another feature of your application Developers create this 23
  24. 24. The Content Handler: reacting to signaling events KMS SinkSRC Sink SRC Sink SRCSink Media API REST API (Open API protocol) The Content Handler Equivalent to a Servlet/ASP/PHP script Exposes APIs for specifying: - when receiving “this request”… - … execute “this logic” Developer can use the media API Code building the media pipeline and executing the application logic the developer wants 24
  25. 25. Kurento Application Server, the container of Handlers • Is an extension of Java EE technologies. • Compatible with all Java EE Servlet containers • Hold Handlers – Your specific application code • Receives signaling requests: – I want “this media” in this way … • Dispatches request to the appropriate handler – @annotations based mapping • Generate an answer showing “where and how to find the media” – URL where media is located – Negotiated SDP Java EE compatible container HTTP Servlet SIP Servlet Web service s Kurento REST API Specific handler implementations Media API DD.BB. Kurento Application Server (KAS) Other java APIs. 25
  26. 26. Putting it altogether: the Architecture Kurento Media Server (KMS) Receive Video Augmented Reality Send Video Computer Vision Video Playing and Recording Java EE compatible container HTTP Servlet SIP Servlet Web service s Kurento REST API Specific handler implementations Signaling and WWW traffic Media Media Signaling and WWW traffic Media API DD.BB. Kurento Application Server (KAS) Other java APIs. 26
  27. 27. Application execution flow Client Code Application Server (KAS) Media Server (KMS) I want this media in this way … (JSON-RPC request) Commands requesting the creation of a pipeline What you want is here … (JSON-RPC answer) Media negotiation phase Media exchange phase 1 2 Specific application logic at the server-side (Content Handler) Media pipeline creation Media exchange between client and server 27
  28. 28. Kurento in the context of IMS KMS = MRF (Media Resource Function) KAS = AS (Application Server) S-CSCF (Serving- Call Session Control Function Media Control Media Events Context and other information services UE (User Equipment) Application logic & orchestration Media capabilities (pipeline) Media elements can generate events through deep analysis of media • IVR • Sentiment analysis • Face recognition • Object tracking • Etc. Application logic can use context and other external information services for controlling media elements or dynamically modifying media pipelines 28
  29. 29. Kurento use case: WebRTC video conference with “secret” participant• The “connection” primitive acts from source to destination – You can connect how many sinks as you to a source – Not-connected sources or sinks don’t break anything. • In this example – User 1 and 2 maintain a video conference • webRtcEndpoint1.connect(webRtcEndpoint2), webRtcEndpoint2.connect(webRtcEndpoint1) – User 3 can “see” that video conference, but he cannot participate • webRtcEndpoint1.connect(webRtcEndpoint3), webRtcEndpoint2.connect(webRtcEndpoint3’) 29 Media Pipeline WebRTC audio streaming SinkSRC WebRTC video streaming SRCSinkSRCSink WebRTC audio streaming WebRtcEndpoint 1 WebRtcEndpoint 3 WebRtcEndpoint 2SRCSink WebRTC audio streaming WebRtcEndpoint 3’ User 1 User 2 User 3
  30. 30. Kurento use case: WebRTC bridge/SBC between WebRTC and RTP domains • Kurento transparently adapts codecs and formats. Just connect: – webRtcEndpoint.connect(rtpEndpoint) – rtpEndpoint.connect(webRtcEndpoint) • Support for the most common audio and video codecs – VP8, H.264, H.263, OPUS, AMR, Speex, G.711, etc. 30 Media Pipeline WebRTC audio streaming SinkSRC SRCSink Media Pipeline SinkSRC SRCSink WebRTC video streaming RTP audio streaming RTP video streaming
  31. 31. 31 SinkSRC SRC Sink SinkSRC Kurento use case: interoperable MCU SRCSink • Group communications – Dispatcher • Media router • N to M – Mixer • Composite mixed media • N to 1
  32. 32. Kurento use case: Video conference recorded and distributed (CDN) • Kurento makes possible to interconnect different types of endpoints in your application – RecorderEndpoint • Records media in the desired format – HttpEndpoint • Exposes the media flow through HTTP pseudostreaming (HTML5 <video> tag) 32 SinkSRC Media Pipeline Sink SRCSink SinkSRC Sink Media to file or repository CDN or Web browser
  33. 33. Kurento use case: video-conference with augmented reality • The infrastructure exposes advances processing capabilities • You can connect them following the same scheme – playerEndpoint.connect(jackVaderFilter) – jackVaderFilter.connect(httpEndpoint) • Use RTC endpoints if you want real-time media processing – WebRtcEndpoint, RtpEndpoint 33 Media Pipeline Media from file or URI Sink SRC SRC Sink JackVaderFilter Result:
  34. 34. Kurento use case: WebRTC video game Result: SinkSRC Sink Sink SRC Sink SRC Sink SRC Sink SRC WebRtcEndpoint MirrorFilter PointerDetectorFilter ChromaFilter FaceOverlayFilter Media Pipeline 34 • Beyond video-conferencing: receiving events from the media stream – Immersive and personalized advertising – Games and entertainment Events (buttons) Game or advertisement logic Change faceChange background
  35. 35. Media Pipeline Media Pipeline IoT and context aware applications: hot IVR plugin 35 • Pipelines are dynamic, you can hot modify them using context or sensor information – Ex. when device is moving faster than 40 Kh, add IVR system • “Record this call”, “Redirect to my secretary”, “Play file one”, Etc. • Voice commands can modify the pipeline themselves – “Record this call”  add a RecorderEndpoint – “Play file 1”  add a player – Ex. When security sensor fires, activate face-recognition/person-detection, etc SinkSRC SRCSink Sink SRC SinkSRC SRCSink IvrFilter VoiceEvents Dynamic pipeline modification Depending on context or other events
  36. 36. Media Pipeline Smart-cities applications 36 • Advanced computer vision filters for – Crowd detection – Car plate recognition – Intrusion detection – Traffic control – Etc. CrowdEvents Sink SRC Sink SRC CrowdDetector RtspPlayer
  37. 37. Other scenarios 37 • There are opportunities on many verticals – eHealth – Logistics – Security – Social networks – Jounalisms • Creation of additional Media Elements is simple – Integrate your own IVRs – Integrate your own Computer Vision capabilities – Integrate your own repositories – Etc. • Potential applications are unlimited – Just combine the media elements
  38. 38. Thank you Suggestions, comments and complains: Twitter: @Kurentoms 38