Your SlideShare is downloading. ×
0
www.kurento.org
Developing rich multimedia applications with Kurento
Developing rich multimedia applications with
Kurento:...
www.kurento.org
WebRTC technologies from the
perspective of a developer
2
Developing
the client side
Developing the
infras...
www.kurento.org
What’s a WebRTC infrastructure
3
Peer-to-Peer WebRTC Application (without media infrastructure)
WebRTC vid...
www.kurento.org
Some WebRTC media infrastructure
providers
4
www.kurento.org
Media infrastructure common
capabilities
5
Transcoding media infrastructure
VP8 H.264
MCU media infrastruc...
www.kurento.org
Can’t we go beyond?
Media is
here
Media got
there
Media got
there
Analyze
Transform
Store
Transcoding
MCU
...
www.kurento.org
•Interoperable media exchange (multiplatform/multiprotocol)
•WebRTC, RTP, HTTP (video tag), etc.
•Standard...
www.kurento.org
Future
Internet
Multimedia
Infrastructure
Simple
Development
APIs
Kurento: the equation
8
www.kurento.org
Kurento Media Server (KMS): the
nucleus of Kurento
• KMS is a middleware for media streams
– Receives the ...
www.kurento.org
The Media API: The API for accessing
KMS capabilities
Send
Receive
Analyze
Augment
Enrich
Transform
Transc...
www.kurento.org
Media API: Media Elements and Media
Pipelines
SinkSRC
Sink
SRC
SRCSink
Sink
Media Element
• Provides a sp...
www.kurento.org
The Lego Game: Developers create
applications connecting media elements
Protocols
& Codecs
Computer
Vision...
www.kurento.org
Media API: trivial example (JavaScript)
kwsMedia.create("MediaPipeline", function(error, pipeline){
pipeli...
www.kurento.org
Kurento Architecture: putting it all
together
Kurento Media Server (KMS)
Receive
Video
Augmented
Reality
S...
www.kurento.org
Application execution flow
Client
Code
Media Connector
(KMC)
Media
Server (KMS)
Crate pipeline/element…
(J...
www.kurento.org
What about Node.js?
16
Kurento Media Server (KMS)
Receive
Video
Augmented
Reality
Send
Video
Computer
Visi...
www.kurento.org
Let’s develop with Kurento
• What you need
– A Kurento instance
• You can install your own Kurento instanc...
www.kurento.org
Kurento Hello World: Playing a file
with an HTML5 client
Media Pipeline
HttpGetEndpoint
Media from
file or...
www.kurento.org
Playing a file with an HTML5 client:
JavaScript code
const ws_uri = “ws://your.kurento.server:8888/thrift/...
www.kurento.org
Playing a file with an HTML5 client:
HTML code
<!DOCTYPE html>
<html>
<head>
<meta charset="US-ASCII">
<ti...
www.kurento.org
Playing a file with an HTML5 client:
See the example working
https://www.youtube.com/watch?v=Mp6UEL9U-v4
21
www.kurento.org
WebRTC loopback
Kurento Media Server
WebRTC
Streaming
SinkSRC
22
WebRtcEndpoint
www.kurento.org
WebRTC loopback: JavaScript code
23
Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoo...
www.kurento.org
WebRTC loopback: HTML code
24
<!DOCTYPE html>
<html>
<head>
<meta charset="US-ASCII">
<title>Insert title ...
www.kurento.org
WebRTC loopback: see the example
working
25
https://www.youtube.com/watch?v=hUChVutQ2og
25
www.kurento.org
WebRTC with FaceOverlay
Media Pipeline
WebRTC
Streaming
SinkSRC
26
Sink
SRC
WebRtcEndpoint
FaceOverlayFilt...
www.kurento.org
WebRTC with FaceOverlay: JavaScript
code
27
Source: https://github.com/Kurento/kws-tutorial/tree/develop/F...
www.kurento.org
WebRTC recorder
28
28
Media Pipeline
WebRTC
Streaming
SinkSRC
Sink
Media
to
file or
URI
Media Pipeline
Web...
www.kurento.org
WebRTC Recorder: JavaScript code
webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, o...
www.kurento.org
WebRTC with Recorder: see the
example working
30
https://www.youtube.com/watch?v=wp5A4NZ-kd4
30
www.kurento.org
But … is this code beautiful?
31
31
Developers love spaghetti for eating,
but not for code
From that code ...
www.kurento.org
Avoiding the callback hell
• The hidden secret of the content API
– Every call to a kws-media-api returns ...
www.kurento.org
Are promises really useful for our
objectives?
• Developing with promises
– WebRTC loopack with FaceOverla...
www.kurento.org
Generators+co: the callback hell killer
• Generators
– Cooperative (non preemtive) simulation of synchrono...
www.kurento.org
Revisiting the WebRTC with loopback
example
var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput,...
www.kurento.org
Isn’t it beautiful ?
36
www.kurento.org
Let’s do some more complex stuff: a
WebRTC game
Media Pipeline
SinkSRC
Sink
SRC
Sink
SRC
37
Sink
SRC
Sink
...
www.kurento.org
WebRTC game: code
38
Source:
https://github.com/Kurento/kws-tutorial/tree/develop/RecorderFilterGenerator
...
www.kurento.org
WebRTC game: see the example
working
39
https://www.youtube.com/watch?v=5eJRnwKxgbY
39
www.kurento.org
Summary and conclusions
• Kurento
– Open Source Softare
• LGPL v2.1
– Multimedia infrastructure for HTML5
...
www.kurento.org
Thank you
41
Suggestions, comments and complains:
lulop@kurento.org
http://www.kurento.org
http://www.gith...
Upcoming SlideShare
Loading in...5
×

Developing rich multimedia applications with Kurento: a tutorial for JavaScript Developers

3,190

Published on

This presentation was carried out at DevCon5'14 (New York) for introducing the Kurento new JavaScript developer APIs. These APIs make possible to create rich video applications supporting WebRTC and HTTP pseudo-streaming (video tag) and leveraging Kurento Media Server capabilities, which include computer vision, augmented reality, group communications and recording.

Published in: Technology

Transcript of "Developing rich multimedia applications with Kurento: a tutorial for JavaScript Developers"

  1. 1. www.kurento.org Developing rich multimedia applications with Kurento Developing rich multimedia applications with Kurento: a tutorial for JavaScript developers lulop@kurento.org http://www.kurento.org http://www.nubomedia.eu http://www.fi-ware.org http://ec.europa.eu
  2. 2. www.kurento.org WebRTC technologies from the perspective of a developer 2 Developing the client side Developing the infrastructure side Before WebRTC First wave of WebRTC technologies Second wave of WebRTC technologies Begin End Begin End • APIs • Standards • FOSS • Coherent APIs • Standards • FOSS • Multiplatform Begin End • APIs • Standards • FOSS • Platforms • Out of the box capabilities 2012 2014
  3. 3. www.kurento.org What’s a WebRTC infrastructure 3 Peer-to-Peer WebRTC Application (without media infrastructure) WebRTC video stream WebRTC Application with media infrastructure Media infrastructure
  4. 4. www.kurento.org Some WebRTC media infrastructure providers 4
  5. 5. www.kurento.org Media infrastructure common capabilities 5 Transcoding media infrastructure VP8 H.264 MCU media infrastructures Recording media infrastructure
  6. 6. www.kurento.org Can’t we go beyond? Media is here Media got there Media got there Analyze Transform Store Transcoding MCU Recording Enrich Augment Adapt Sensors Context Events Media is here What WebRTC infrastructures are • Transcoding • MCU • Recording What WebRTC infrastructures should be • Flexible processing • Augmented reality • Blending • Mixing • Analyzing • Etc. 6
  7. 7. www.kurento.org •Interoperable media exchange (multiplatform/multiprotocol) •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities Capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision, augmented reality, mixing, blending, etc. Multimedia infrastructure • REST API • JavaScript API • Java API APIs • LGPL 2.1 Is distributed through a flexible FOSS license What’s Kurento 7
  8. 8. www.kurento.org Future Internet Multimedia Infrastructure Simple Development APIs Kurento: the equation 8
  9. 9. www.kurento.org Kurento Media Server (KMS): the nucleus of Kurento • KMS is a middleware for media streams – Receives the stream – Process the stream – Issues the stream Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS 9
  10. 10. www.kurento.org The Media API: The API for accessing KMS capabilities Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS Java Media API JavaScript Media API REST API Applications define the processing of streams getting through KMS 10
  11. 11. www.kurento.org Media API: Media Elements and Media Pipelines SinkSRC Sink SRC SRCSink Sink Media Element • Provides a specific media functionality › Send/receive media › Process media › Transform media • Exchange media through › Sources › Sinks Media pipeline • Chain of media elements implementing the desired media logic. • The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Media Element Sink SRC 11
  12. 12. www.kurento.org The Lego Game: Developers create applications connecting media elements Protocols & Codecs Computer Vision Augmented Reality Mul sensory Mul media Media Repository IPTV Integra on Group Communica ons Sink% H pEndpoint Sink%SRC% RtpEndpoint SRC$ DataChannelEndpointSink%SRC% WebRtcEndpoint Sink% SRC% BarCodeReader Sink% SRC% PointerTracker Sink% SRC% FaceDetector Sink% SRC% FaceOverlay Sink% SRC% ChromaFilter Sink% SRC% HeartRateBlender Sink% RecorderEndpoint SRC$ PlayerEndpoint Sink% CdnUpload Sink% IPTVConnector Mixer SinkSRC Sink SRC Sink SRCSink Sink Application 2 Sink SRC Sink SRCSink SinkSRC Sink Application 3 SinkSRC SRCSink SRC Sink SinkSRC Application 1 Toolbox of media elements 12
  13. 13. www.kurento.org Media API: trivial example (JavaScript) kwsMedia.create("MediaPipeline", function(error, pipeline){ pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){ pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){ playerEndpoint.connect(httpGetEndpoint, function(error){ httpGetEndpoint.getUrl(function(error, url){ videoInput.src = url; }); playerEndpoint.play(function(error){ }); }); }); }); }); Media Pipeline (inside KMS) HttpGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint 13
  14. 14. www.kurento.org Kurento Architecture: putting it all together Kurento Media Server (KMS) Receive Video Augmented Reality Send Video Computer Vision Video Playing and Recording Kurento WS JSON-RPC API Media Media Kurento Media Connector (KMC) 14 SIP Servlet SOAP HTTP Servlet REST Browser Developer code with specific application logic (pipelines) Media capabilities of the browser Kurento Media API Specific logic
  15. 15. www.kurento.org Application execution flow Client Code Media Connector (KMC) Media Server (KMS) Crate pipeline/element… (JSON-RPC) Commands requesting the creation of a pipeline Reference to media object … (JSON-RPC) Pipeline cereation phase Media exchange phase 1 2 Specific logic (e.g. AAA) Media pipeline creation Media exchange between client and server 15
  16. 16. www.kurento.org What about Node.js? 16 Kurento Media Server (KMS) Receive Video Augmented Reality Send Video Computer Vision Video Playing and Recording Kurento WS JSON-RPC API Media Media Kurento Media Connector (KMC) SIP Servlet SOAP HTTP Servlet REST Node.js Pipeline creation + AAA Kurento Media API Browser Browser-side code
  17. 17. www.kurento.org Let’s develop with Kurento • What you need – A Kurento instance • You can install your own Kurento instance – http://www.kurento.org/docs/current/ • You can launch a Kurento instance at the FI-LAB (FI-WARE project) – http://lab.fi-ware.org – FI-LAB is a cloud infrastructure making possible to execute instances from images or blueprints • kws-media-api – https://github.com/kurento/kws-media-api – http://github.com/kurento/kws-utils » Node.js • npm install kws-media-api • npm install kws-utils » Browser • bower install kws-media-api • bower install kws-utils • Getting help – Kurento web site • http://www.kurento.org – Kurento mailing list • https://groups.google.com/forum/#!forum/kurento – Twitter • @kurentoms 17
  18. 18. www.kurento.org Kurento Hello World: Playing a file with an HTML5 client Media Pipeline HttpGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint Kurento WS JSON-RPC API Create / excec … Ref / answer … 18 Kurento Media Connector (KMC)
  19. 19. www.kurento.org Playing a file with an HTML5 client: JavaScript code const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change this const file_uri = “http://uri.to.video.file/”; //change this var videoInput = document.getElementById("videoInput"); //video tag element KwsMedia(ws_uri, function(kwsMedia){ //instantiate kws-media-api manager kwsMedia.create("MediaPipeline", function(error, pipeline){ //instantiate pipeline pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){ pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){ playerEndpoint.connect(httpGetEndpoint, function(error){ httpGetEndpoint.getUrl(function(error, url){ videoInput.src = url; }); playerEndpoint.play(); }); }); }); }); }); Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer 19
  20. 20. www.kurento.org Playing a file with an HTML5 client: HTML code <!DOCTYPE html> <html> <head> <meta charset="US-ASCII"> <title>Insert title here</title> <script type="text/javascript" src="../api/kws-media-api.js"></script> <script type="text/javascript" src="demo.js"></script> </head> <body> <button id="playButton">Click to start playing</button><br> <video id="videoInput" autoplay></video> </body> </html> Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer 20
  21. 21. www.kurento.org Playing a file with an HTML5 client: See the example working https://www.youtube.com/watch?v=Mp6UEL9U-v4 21
  22. 22. www.kurento.org WebRTC loopback Kurento Media Server WebRTC Streaming SinkSRC 22 WebRtcEndpoint
  23. 23. www.kurento.org WebRTC loopback: JavaScript code 23 Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change this var videoInput = document.getElementById("videoInput"); var videoOutput = document.getElementById("videoOutput"); var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError); function onOffer(offer){ KwsMedia(ws_uri, function(kwsMedia){ kwsMedia.create("MediaPipeline", function(error, pipeline){ pipeline.create("WebRtcEndpoint", function(error, webRtc){ webRtc.processOffer(offer, function(error, answer){ webRtcPeer.processSdpAnswer(answer); }); webRtc.connect(webRtc); }); }); }); }; 23
  24. 24. www.kurento.org WebRTC loopback: HTML code 24 <!DOCTYPE html> <html> <head> <meta charset="US-ASCII"> <title>Insert title here</title> <script type="text/javascript" src="../api/adapter.js"></script> <script type="text/javascript" src="../api/kws-media-api.js"></script> <script type="text/javascript" src="../api/kws-utils.js"></script> <script type="text/javascript" src="demo.js"></script> </head> <body> <button id="launchButton">Click me ...</button><br> <div style="float: left; border: 2px solid black; margin: 5px;"> <h1>Local stream</h1> <video id="videoInput" autoplay></video> </div> <div style="float:left; border: 2px solid black; margin: 5px;"> <h1>Remote stream</h1> <video id="videoOutput" autoplay></video> </div> </body> </html> Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback 24
  25. 25. www.kurento.org WebRTC loopback: see the example working 25 https://www.youtube.com/watch?v=hUChVutQ2og 25
  26. 26. www.kurento.org WebRTC with FaceOverlay Media Pipeline WebRTC Streaming SinkSRC 26 Sink SRC WebRtcEndpoint FaceOverlayFilter
  27. 27. www.kurento.org WebRTC with FaceOverlay: JavaScript code 27 Source: https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlay var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput , onOffer, onError); function onOffer(offer) { KwsMedia(ws_uri, function(kwsMedia) { kwsMedia.create("MediaPipeline", function(error, pipeline) { pipeline.create("WebRtcEndpoint", function(error, webRtc) { pipeline.create("FaceOverlayFilter”, function(error, filter) { //hat_uri is the uri pointing to the overlay image to draw on top of detected faces //These percent variables need to be defined to adjust overlay image to face box filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent); webRtc.connect(filter, function(error) { filter.connect(webRtc, function(error) { webRtc.processOffer(offer, function(error, answer) { webRtcPeer.processSdpAnswer(answer); }); …//many }); omited here for simplicity }); 27
  28. 28. www.kurento.org WebRTC recorder 28 28 Media Pipeline WebRTC Streaming SinkSRC Sink Media to file or URI Media Pipeline WebRTC Streaming SinkSRC Media from file or URI SRC WebRtcEndpoint WebRtcEndpoint RecorderEndpoint PlayerEndpoint
  29. 29. www.kurento.org WebRTC Recorder: JavaScript code webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError); function onOffer(offer) { KwsMedia(ws_uri, function(kwsMedia) { kwsMedia.create('MediaPipeline', function(error, pipeline) { pipeline.create('RecorderEndpoint', {uri : file_uri}, function(error, recorder) { pipeline.create('WebRtcEndpoint', function(error, webRtv) { webRtc.connect(recorder, function(error) { recorder.record(); webRtc.connect(webRtc, function(error) { webRtc.processOffer(offer, function(error, answer) { webRtcPeer.processSdpAnswer(answer); }); ... }); //many }); omited here for simplicity } 29 29 Source: https://github.com/Kurento/kws-tutorial/tree/develop/Recorder
  30. 30. www.kurento.org WebRTC with Recorder: see the example working 30 https://www.youtube.com/watch?v=wp5A4NZ-kd4 30
  31. 31. www.kurento.org But … is this code beautiful? 31 31 Developers love spaghetti for eating, but not for code From that code structure to this code structure
  32. 32. www.kurento.org Avoiding the callback hell • The hidden secret of the content API – Every call to a kws-media-api returns a promise • var p1= kwsMedia.create(“MediaPipeline”); • p1.then(function(pipeline){…}) – Promises • A proxy to a value not knows when the proxy is created • Allows associating handlers to – Value successfully obtained – Error • https://developer.mozilla.org/en- US/docs/Web/JavaScript/Reference/Global_Objects/Promise • http://www.html5rocks.com/en/tutorials/es6/promises/ – Example 32 32
  33. 33. www.kurento.org Are promises really useful for our objectives? • Developing with promises – WebRTC loopack with FaceOverlay • Source – https://github.com/Kurento/kws-media- api/tree/develop/example/WebRtcEndpoint-FaceOverlayFilter_promises_2 • Does structure improve? – Not much – Problems • Nesting avoiding, but chaining isn’t much beautiful • Nesting eliminated => closures eliminated – Scope of variables (pipelines, elements, etc.) lost. – Need to pass variables along on each call • What’s the point of using promises then? – Continue reading if you are the kind of guy of like living on the edge … 33 33
  34. 34. www.kurento.org Generators+co: the callback hell killer • Generators – Cooperative (non preemtive) simulation of synchronous behavior on iterator functions by maintaining execution state • https://developer.mozilla.org/en- US/docs/Web/JavaScript/Guide/Iterators_and_Generators – Allow multiple entry points for suspending and resuming execution at certain locations. – Available on EC6 (Harmony) • Co – Combines promises and generators – Uses promises to “connect” generator exit and reentrance – https://github.com/visionmedia/co 34
  35. 35. www.kurento.org Revisiting the WebRTC with loopback example var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError); function onOffer(offer){ co(function*(){ var kwsMedia = yield KwsMedia(ws_uri); var pipeline = yield kwsMedia.create("MediaPipeline"); var webRtc = yield pipeline.create("WebRtcEndpoint"); var filter = yield pipeline.create("FaceOverlayFilter"); //you need to define the hat_uri and percents variables yield filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent); var answer = yield webRtc.processOffer(offer); webRtcPeer.processSdpAnswer(answer); yield webRtc.connect(filter); yield filter.connect(webRtc); })(); } 35 Source code of previous examples using generators: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlayGenerator https://github.com/Kurento/kws-tutorial/tree/develop/RecorderGenerator
  36. 36. www.kurento.org Isn’t it beautiful ? 36
  37. 37. www.kurento.org Let’s do some more complex stuff: a WebRTC game Media Pipeline SinkSRC Sink SRC Sink SRC 37 Sink SRC Sink WebRtcEndpoint MirrorFilter PointerDetectorFilter FaceOverlayFilter RecorderEndpoint
  38. 38. www.kurento.org WebRTC game: code 38 Source: https://github.com/Kurento/kws-tutorial/tree/develop/RecorderFilterGenerator 38
  39. 39. www.kurento.org WebRTC game: see the example working 39 https://www.youtube.com/watch?v=5eJRnwKxgbY 39
  40. 40. www.kurento.org Summary and conclusions • Kurento – Open Source Softare • LGPL v2.1 – Multimedia infrastructure for HTML5 • WebRTC • HTML5 – Advanced media processing capabilities • Recording, MCUs, Transcoding, Computer Vision, Augmented Reality, Mixing, Blending. • kws-media-api – Access Kurento capabilities from JavaScript • Media elements and media pipelines • Like a “Lego game” – Beautiful code • Promises • Generators • CO 40
  41. 41. www.kurento.org Thank you 41 Suggestions, comments and complains: lulop@kurento.org http://www.kurento.org http://www.github.com/kurento Twitter: @Kurentoms http://www.nubomedia.eu http://www.fi-ware.org http://ec.europa.eu
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×