Developing rich multimedia applications with Kurento: a tutorial for JavaScript Developers
Upcoming SlideShare
Loading in...5
×
 

Developing rich multimedia applications with Kurento: a tutorial for JavaScript Developers

on

  • 1,759 views

This presentation was carried out at DevCon5'14 (New York) for introducing the Kurento new JavaScript developer APIs. These APIs make possible to create rich video applications supporting WebRTC and ...

This presentation was carried out at DevCon5'14 (New York) for introducing the Kurento new JavaScript developer APIs. These APIs make possible to create rich video applications supporting WebRTC and HTTP pseudo-streaming (video tag) and leveraging Kurento Media Server capabilities, which include computer vision, augmented reality, group communications and recording.

Statistics

Views

Total Views
1,759
Views on SlideShare
998
Embed Views
761

Actions

Likes
2
Downloads
10
Comments
0

8 Embeds 761

https://pulse.neudesic.com 615
https://twitter.com 93
https://pre.naevatec.com 20
http://www.slideee.com 16
https://www.linkedin.com 8
http://naevatec.urbimedia.es 5
http://geekple.com 2
http://pulse.neudesic.com 2
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Developing rich multimedia applications with Kurento: a tutorial for JavaScript Developers Developing rich multimedia applications with Kurento: a tutorial for JavaScript Developers Presentation Transcript

  • www.kurento.org Developing rich multimedia applications with Kurento Developing rich multimedia applications with Kurento: a tutorial for JavaScript developers lulop@kurento.org http://www.kurento.org http://www.nubomedia.eu http://www.fi-ware.org http://ec.europa.eu
  • www.kurento.org WebRTC technologies from the perspective of a developer 2 Developing the client side Developing the infrastructure side Before WebRTC First wave of WebRTC technologies Second wave of WebRTC technologies Begin End Begin End • APIs • Standards • FOSS • Coherent APIs • Standards • FOSS • Multiplatform Begin End • APIs • Standards • FOSS • Platforms • Out of the box capabilities 2012 2014
  • www.kurento.org What’s a WebRTC infrastructure 3 Peer-to-Peer WebRTC Application (without media infrastructure) WebRTC video stream WebRTC Application with media infrastructure Media infrastructure
  • www.kurento.org Some WebRTC media infrastructure providers 4
  • www.kurento.org Media infrastructure common capabilities 5 Transcoding media infrastructure VP8 H.264 MCU media infrastructures Recording media infrastructure
  • www.kurento.org Can’t we go beyond? Media is here Media got there Media got there Analyze Transform Store Transcoding MCU Recording Enrich Augment Adapt Sensors Context Events Media is here What WebRTC infrastructures are • Transcoding • MCU • Recording What WebRTC infrastructures should be • Flexible processing • Augmented reality • Blending • Mixing • Analyzing • Etc. 6
  • www.kurento.org •Interoperable media exchange (multiplatform/multiprotocol) •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities Capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision, augmented reality, mixing, blending, etc. Multimedia infrastructure • REST API • JavaScript API • Java API APIs • LGPL 2.1 Is distributed through a flexible FOSS license What’s Kurento 7
  • www.kurento.org Future Internet Multimedia Infrastructure Simple Development APIs Kurento: the equation 8
  • www.kurento.org Kurento Media Server (KMS): the nucleus of Kurento • KMS is a middleware for media streams – Receives the stream – Process the stream – Issues the stream Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS 9
  • www.kurento.org The Media API: The API for accessing KMS capabilities Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS Java Media API JavaScript Media API REST API Applications define the processing of streams getting through KMS 10
  • www.kurento.org Media API: Media Elements and Media Pipelines SinkSRC Sink SRC SRCSink Sink Media Element • Provides a specific media functionality › Send/receive media › Process media › Transform media • Exchange media through › Sources › Sinks Media pipeline • Chain of media elements implementing the desired media logic. • The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Media Element Sink SRC 11
  • www.kurento.org The Lego Game: Developers create applications connecting media elements Protocols & Codecs Computer Vision Augmented Reality Mul sensory Mul media Media Repository IPTV Integra on Group Communica ons Sink% H pEndpoint Sink%SRC% RtpEndpoint SRC$ DataChannelEndpointSink%SRC% WebRtcEndpoint Sink% SRC% BarCodeReader Sink% SRC% PointerTracker Sink% SRC% FaceDetector Sink% SRC% FaceOverlay Sink% SRC% ChromaFilter Sink% SRC% HeartRateBlender Sink% RecorderEndpoint SRC$ PlayerEndpoint Sink% CdnUpload Sink% IPTVConnector Mixer SinkSRC Sink SRC Sink SRCSink Sink Application 2 Sink SRC Sink SRCSink SinkSRC Sink Application 3 SinkSRC SRCSink SRC Sink SinkSRC Application 1 Toolbox of media elements 12
  • www.kurento.org Media API: trivial example (JavaScript) kwsMedia.create("MediaPipeline", function(error, pipeline){ pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){ pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){ playerEndpoint.connect(httpGetEndpoint, function(error){ httpGetEndpoint.getUrl(function(error, url){ videoInput.src = url; }); playerEndpoint.play(function(error){ }); }); }); }); }); Media Pipeline (inside KMS) HttpGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint 13
  • www.kurento.org Kurento Architecture: putting it all together Kurento Media Server (KMS) Receive Video Augmented Reality Send Video Computer Vision Video Playing and Recording Kurento WS JSON-RPC API Media Media Kurento Media Connector (KMC) 14 SIP Servlet SOAP HTTP Servlet REST Browser Developer code with specific application logic (pipelines) Media capabilities of the browser Kurento Media API Specific logic
  • www.kurento.org Application execution flow Client Code Media Connector (KMC) Media Server (KMS) Crate pipeline/element… (JSON-RPC) Commands requesting the creation of a pipeline Reference to media object … (JSON-RPC) Pipeline cereation phase Media exchange phase 1 2 Specific logic (e.g. AAA) Media pipeline creation Media exchange between client and server 15
  • www.kurento.org What about Node.js? 16 Kurento Media Server (KMS) Receive Video Augmented Reality Send Video Computer Vision Video Playing and Recording Kurento WS JSON-RPC API Media Media Kurento Media Connector (KMC) SIP Servlet SOAP HTTP Servlet REST Node.js Pipeline creation + AAA Kurento Media API Browser Browser-side code
  • www.kurento.org Let’s develop with Kurento • What you need – A Kurento instance • You can install your own Kurento instance – http://www.kurento.org/docs/current/ • You can launch a Kurento instance at the FI-LAB (FI-WARE project) – http://lab.fi-ware.org – FI-LAB is a cloud infrastructure making possible to execute instances from images or blueprints • kws-media-api – https://github.com/kurento/kws-media-api – http://github.com/kurento/kws-utils » Node.js • npm install kws-media-api • npm install kws-utils » Browser • bower install kws-media-api • bower install kws-utils • Getting help – Kurento web site • http://www.kurento.org – Kurento mailing list • https://groups.google.com/forum/#!forum/kurento – Twitter • @kurentoms 17
  • www.kurento.org Kurento Hello World: Playing a file with an HTML5 client Media Pipeline HttpGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint Kurento WS JSON-RPC API Create / excec … Ref / answer … 18 Kurento Media Connector (KMC)
  • www.kurento.org Playing a file with an HTML5 client: JavaScript code const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change this const file_uri = “http://uri.to.video.file/”; //change this var videoInput = document.getElementById("videoInput"); //video tag element KwsMedia(ws_uri, function(kwsMedia){ //instantiate kws-media-api manager kwsMedia.create("MediaPipeline", function(error, pipeline){ //instantiate pipeline pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){ pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){ playerEndpoint.connect(httpGetEndpoint, function(error){ httpGetEndpoint.getUrl(function(error, url){ videoInput.src = url; }); playerEndpoint.play(); }); }); }); }); }); Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer 19
  • www.kurento.org Playing a file with an HTML5 client: HTML code <!DOCTYPE html> <html> <head> <meta charset="US-ASCII"> <title>Insert title here</title> <script type="text/javascript" src="../api/kws-media-api.js"></script> <script type="text/javascript" src="demo.js"></script> </head> <body> <button id="playButton">Click to start playing</button><br> <video id="videoInput" autoplay></video> </body> </html> Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer 20
  • www.kurento.org Playing a file with an HTML5 client: See the example working https://www.youtube.com/watch?v=Mp6UEL9U-v4 21
  • www.kurento.org WebRTC loopback Kurento Media Server WebRTC Streaming SinkSRC 22 WebRtcEndpoint
  • www.kurento.org WebRTC loopback: JavaScript code 23 Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change this var videoInput = document.getElementById("videoInput"); var videoOutput = document.getElementById("videoOutput"); var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError); function onOffer(offer){ KwsMedia(ws_uri, function(kwsMedia){ kwsMedia.create("MediaPipeline", function(error, pipeline){ pipeline.create("WebRtcEndpoint", function(error, webRtc){ webRtc.processOffer(offer, function(error, answer){ webRtcPeer.processSdpAnswer(answer); }); webRtc.connect(webRtc); }); }); }); }; 23
  • www.kurento.org WebRTC loopback: HTML code 24 <!DOCTYPE html> <html> <head> <meta charset="US-ASCII"> <title>Insert title here</title> <script type="text/javascript" src="../api/adapter.js"></script> <script type="text/javascript" src="../api/kws-media-api.js"></script> <script type="text/javascript" src="../api/kws-utils.js"></script> <script type="text/javascript" src="demo.js"></script> </head> <body> <button id="launchButton">Click me ...</button><br> <div style="float: left; border: 2px solid black; margin: 5px;"> <h1>Local stream</h1> <video id="videoInput" autoplay></video> </div> <div style="float:left; border: 2px solid black; margin: 5px;"> <h1>Remote stream</h1> <video id="videoOutput" autoplay></video> </div> </body> </html> Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback 24
  • www.kurento.org WebRTC loopback: see the example working 25 https://www.youtube.com/watch?v=hUChVutQ2og 25
  • www.kurento.org WebRTC with FaceOverlay Media Pipeline WebRTC Streaming SinkSRC 26 Sink SRC WebRtcEndpoint FaceOverlayFilter
  • www.kurento.org WebRTC with FaceOverlay: JavaScript code 27 Source: https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlay var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput , onOffer, onError); function onOffer(offer) { KwsMedia(ws_uri, function(kwsMedia) { kwsMedia.create("MediaPipeline", function(error, pipeline) { pipeline.create("WebRtcEndpoint", function(error, webRtc) { pipeline.create("FaceOverlayFilter”, function(error, filter) { //hat_uri is the uri pointing to the overlay image to draw on top of detected faces //These percent variables need to be defined to adjust overlay image to face box filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent); webRtc.connect(filter, function(error) { filter.connect(webRtc, function(error) { webRtc.processOffer(offer, function(error, answer) { webRtcPeer.processSdpAnswer(answer); }); …//many }); omited here for simplicity }); 27
  • www.kurento.org WebRTC recorder 28 28 Media Pipeline WebRTC Streaming SinkSRC Sink Media to file or URI Media Pipeline WebRTC Streaming SinkSRC Media from file or URI SRC WebRtcEndpoint WebRtcEndpoint RecorderEndpoint PlayerEndpoint
  • www.kurento.org WebRTC Recorder: JavaScript code webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError); function onOffer(offer) { KwsMedia(ws_uri, function(kwsMedia) { kwsMedia.create('MediaPipeline', function(error, pipeline) { pipeline.create('RecorderEndpoint', {uri : file_uri}, function(error, recorder) { pipeline.create('WebRtcEndpoint', function(error, webRtv) { webRtc.connect(recorder, function(error) { recorder.record(); webRtc.connect(webRtc, function(error) { webRtc.processOffer(offer, function(error, answer) { webRtcPeer.processSdpAnswer(answer); }); ... }); //many }); omited here for simplicity } 29 29 Source: https://github.com/Kurento/kws-tutorial/tree/develop/Recorder
  • www.kurento.org WebRTC with Recorder: see the example working 30 https://www.youtube.com/watch?v=wp5A4NZ-kd4 30
  • www.kurento.org But … is this code beautiful? 31 31 Developers love spaghetti for eating, but not for code From that code structure to this code structure
  • www.kurento.org Avoiding the callback hell • The hidden secret of the content API – Every call to a kws-media-api returns a promise • var p1= kwsMedia.create(“MediaPipeline”); • p1.then(function(pipeline){…}) – Promises • A proxy to a value not knows when the proxy is created • Allows associating handlers to – Value successfully obtained – Error • https://developer.mozilla.org/en- US/docs/Web/JavaScript/Reference/Global_Objects/Promise • http://www.html5rocks.com/en/tutorials/es6/promises/ – Example 32 32
  • www.kurento.org Are promises really useful for our objectives? • Developing with promises – WebRTC loopack with FaceOverlay • Source – https://github.com/Kurento/kws-media- api/tree/develop/example/WebRtcEndpoint-FaceOverlayFilter_promises_2 • Does structure improve? – Not much – Problems • Nesting avoiding, but chaining isn’t much beautiful • Nesting eliminated => closures eliminated – Scope of variables (pipelines, elements, etc.) lost. – Need to pass variables along on each call • What’s the point of using promises then? – Continue reading if you are the kind of guy of like living on the edge … 33 33
  • www.kurento.org Generators+co: the callback hell killer • Generators – Cooperative (non preemtive) simulation of synchronous behavior on iterator functions by maintaining execution state • https://developer.mozilla.org/en- US/docs/Web/JavaScript/Guide/Iterators_and_Generators – Allow multiple entry points for suspending and resuming execution at certain locations. – Available on EC6 (Harmony) • Co – Combines promises and generators – Uses promises to “connect” generator exit and reentrance – https://github.com/visionmedia/co 34
  • www.kurento.org Revisiting the WebRTC with loopback example var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError); function onOffer(offer){ co(function*(){ var kwsMedia = yield KwsMedia(ws_uri); var pipeline = yield kwsMedia.create("MediaPipeline"); var webRtc = yield pipeline.create("WebRtcEndpoint"); var filter = yield pipeline.create("FaceOverlayFilter"); //you need to define the hat_uri and percents variables yield filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent); var answer = yield webRtc.processOffer(offer); webRtcPeer.processSdpAnswer(answer); yield webRtc.connect(filter); yield filter.connect(webRtc); })(); } 35 Source code of previous examples using generators: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlayGenerator https://github.com/Kurento/kws-tutorial/tree/develop/RecorderGenerator
  • www.kurento.org Isn’t it beautiful ? 36
  • www.kurento.org Let’s do some more complex stuff: a WebRTC game Media Pipeline SinkSRC Sink SRC Sink SRC 37 Sink SRC Sink WebRtcEndpoint MirrorFilter PointerDetectorFilter FaceOverlayFilter RecorderEndpoint
  • www.kurento.org WebRTC game: code 38 Source: https://github.com/Kurento/kws-tutorial/tree/develop/RecorderFilterGenerator 38
  • www.kurento.org WebRTC game: see the example working 39 https://www.youtube.com/watch?v=5eJRnwKxgbY 39
  • www.kurento.org Summary and conclusions • Kurento – Open Source Softare • LGPL v2.1 – Multimedia infrastructure for HTML5 • WebRTC • HTML5 – Advanced media processing capabilities • Recording, MCUs, Transcoding, Computer Vision, Augmented Reality, Mixing, Blending. • kws-media-api – Access Kurento capabilities from JavaScript • Media elements and media pipelines • Like a “Lego game” – Beautiful code • Promises • Generators • CO 40
  • www.kurento.org Thank you 41 Suggestions, comments and complains: lulop@kurento.org http://www.kurento.org http://www.github.com/kurento Twitter: @Kurentoms http://www.nubomedia.eu http://www.fi-ware.org http://ec.europa.eu