Developing rich multimedia applications with Kurento: a tutorial for Java Developers
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Developing rich multimedia applications with Kurento: a tutorial for Java Developers

  • 1,810 views
Uploaded on

This presentation contains a tutorial devoted to showing how Java developers can create rich multimedia applications with Kurento. Java developers will find natural Kurento development model, which......

This presentation contains a tutorial devoted to showing how Java developers can create rich multimedia applications with Kurento. Java developers will find natural Kurento development model, which is based on standard Java EE technologies and is inspired on the WWW Servlet model.

If you have ever developed a Web application, you may be familiar with this scheme. At the browser, HTML and JavaScript code is in charge of user interaction and generates HTTP requests to the server. This code is usually programmed with the help of APIs such as jQuery, DOM, XHR or others. Upon reception, HTTP requests are processed by some kind of server side technology (e. g.. PHP, Java, Ruby, etc.) using service APIs providing features such as DB access, communications, transactions, XML parsing, and others. As a result, an HTTP response is issued and sent back to the client. Following this scheme, both server and client side APIs are just capabilities simplifying developer work and providing abstractions for programming faster and more efficiently.

Kurento technologies adapt to the Web development model so that, from a programmer perspective, Kurento can be seen just as an additional set of APIs. Developers does not need to learn novel programming schemes and can reuse all their knowledge and previous background on WWW application development. When you need multimedia, just use Kurento APIs. For the rest, use your preferred APIs or reuse previous code. Kurento APIs have been designed for simplicity and Web developers will find them familiar and intuitive. Most of the low level details related to codecs, formats, protocols, profiles and containers are abstracted by the framework. Programmers just concentrate on specifying the sequence of processing steps that they want to execute on the media flows.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,810
On Slideshare
1,769
From Embeds
41
Number of Embeds
3

Actions

Shares
Downloads
25
Comments
0
Likes
3

Embeds 41

https://twitter.com 25
http://www.slideee.com 14
https://www.linkedin.com 2

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. www.kurento.org Real-time Multimedia Stream Processing Developing rich multimedia applications with KurentoDeveloping rich multimedia applications with Kurento: a tutorial for Java developers lulop@kurento.org http://www.kurento.org
  • 2. www.kurento.org Multimedia infrastructures for the Future Internet Media is here Media got there Media got there Analyze Transform Store Transport Enrich Augment Adapt Sensors Context Events Media is here 2
  • 3. www.kurento.org Enrich, augment, adapt, analyze, transform, store: what’s the problem? Complexity 3
  • 4. www.kurento.org Future Internet Multimedia Infrastructure Simple Development APIs Kurento: the equation 4
  • 5. www.kurento.org •Interoperable media exchange (multiplatform/multiprotocol) •WebRTC, RTP, HTTP (video tag), etc. •Process media (Computer vision, augmented reality, media indexing, etc.) •Media and metadata recording and recovery •Transform and adapt media (H.264, H.263, VP8, Ogg, and others) •Media routing and mixing •Etc. Multimedia infrastructure • REST API • JavaScript API • Java API APIs • LGPL 2.1 Is distributed through a flexible FOSS license What’s Kurento 5
  • 6. www.kurento.org Kurento Media Server (KMS): the nucleus of Kurento • KMS is a middleware for media streams – Receives the stream – Process the stream – Issues the stream Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS 6
  • 7. www.kurento.org The Media API: The API for accessing KMS capabilities Send Receive Analyze Augment Enrich Transform Transcode Record Process Replicate Media Source Media Sink KMS Java Media API JavaScript Media API REST API Applications define the processing of streams getting through KMS 7
  • 8. www.kurento.org Media API: Media Elements and Media Pipelines SinkSRC Sink SRC SRCSink Sink Media Element • Provides a specific media functionality › Send/receive media › Process media › Transform media • Exchange media through › Sources › Sinks Media pipeline • Chain of media elements implementing the desired media logic. • The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Media Element Sink SRC 8
  • 9. www.kurento.org Media API: trivial example (Java) MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build(); HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(httpEndpoint); httpEndpoint.getUrl(); //URL where the media is made available Media Pipeline (inside KMS) HttpGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint 9
  • 10. www.kurento.org Is that enough? Think about the WWW development model Process WWW request - DDBB access - Authentication - XML processing - Etc. HTTP request: I want this resource HTTP response: The resource • Intuition behind the WWW model – Client asks what it wants • GET /this_resource HTTP/1.1 – Server side APIs execute the associated processing You develop this using … • Servlet • JSP • ASP • PHP • Ruby on Rails • CGI • Etc. 10
  • 11. www.kurento.org We need an equivalent model: the role of the Signaling Plane • We need “something” with the capability of negotiating what, when and how media is exchanged • A protocol for “saying” – I want “this media” … • Identification of the media to exchange – File in hard-drive, IP camera, user, etc. – in “this way” … • Identification of the processing of media – Augmented, analyzed, etc. – with “this format” … • Quality – Codec, screen-size, frame-rate, etc. – at “this moment” • Stream control – Play, stop, start, pause, etc. 11
  • 12. www.kurento.org Don’t get it? think about WWW development again … Process WWW request - DDBB access - Authentication - XML processing - Etc. HTTP request: I want this resource HTTP response: The resource Process media request - Media API - DDBB access - Authentication - XML processing - Etc. Signaling request: I want this media Signaling response: The media is here This is how you create WWW applications (Servlets, ASP, PHP, Rails, etc.) This is how you create Kurento enabled applications: Multimedia RTC is just another feature of your application You, as a programmer, create this 12
  • 13. www.kurento.org Abstracting the signaling plane: The Content Handler KMS SinkSRC Sink SRC Sink SRCSink Media API REST API (Open API protocol) The Content Handler Equivalent to a Servlet/ASP/PHP script Exposes APIs for specifying: - when receiving “this request”… - … execute “this logic” Developer can use the media API Code building the media pipeline and executing the application logic the developer wants 13
  • 14. www.kurento.org Kurento Application Server: The container of Handlers • Is an extension of Java EE technologies. • Compatible with all Java EE Servlet containers • Hold Handlers – Your specific application code • Receives signaling requests: – I want “this media” in this way … • Dispatches request to the appropriate handler – @annotations based mapping • Generate an answer showing “where and how to find the media” – URL where media is located – Negotiated SDP Java EE compatible container HTTP Servlet SIP Servlet Web service s Kurento REST API Specific handler implementations Media API DD.BB. Kurento Application Server (KAS) Other java APIs. 14
  • 15. www.kurento.org Kurento Architecture: putting it all together Kurento Media Server (KMS) Receive Video Augmented Reality Send Video Computer Vision Video Playing and Recording Java EE compatible container HTTP Servlet SIP Servlet Web service s Kurento REST API Specific handler implementations Signaling and WWW traffic Media Media Signaling and WWW traffic Media API DD.BB. Kurento Application Server (KAS) Other java APIs. 15
  • 16. www.kurento.org Application execution flow Client Code Application Server (KAS) Media Server (KMS) I want this media in this way … (JSON-RPC request) Commands requesting the creation of a pipeline What you want is here … (JSON-RPC answer) Media negotiation phase Media exchange phase 1 2 Specific application logic at the server-side (Content Handler) Media pipeline creation Media exchange between client and server 16
  • 17. www.kurento.org Content Handler: trivial example //Specify the type of service provided by this handler: Player, Recorder, WebRTC, RTP, etc. @HttpPlayerService(path = "/player”) //Mapping of handler specified in path public class MyPlayerHandler extends HttpPlayerHandler { @Override public void onContentRequest(HttpPlayerSession contentSession) { //Thie client wants the media this handler provides //Create the pipeline for providing the media } @Override public void onContentStarted(HttpPlayerSession contentSession) { //Media started flowing, you can execute additional actions } @Override Public void onSessionTerminated(HttpPlayerSession contentSenssion){ //Media exchange termianted, you can collect your resources } 17
  • 18. www.kurento.org Let’s develop with Kurento • What you need – A Kurento instance • You can install your own Kurento instance • You can launch a Kurento instance at the FI-LAB (FI-WARE project) – http://lab.fi-ware.org • Getting help – Kurento web site • http://www.kurento.org – Kurento mailing list • https://groups.google.com/forum/#!forum/kurento – Twitter • @kurentoms 18
  • 19. www.kurento.org Kurento Hello World: Playing a file with an HTML5 client Media Pipeline HttpGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint Media API REST API (Open API protocol) Handler codeI want “this media” Media is “at this URL” 19
  • 20. www.kurento.org Playing a file with an HTML5 client: Handler code@HttpPlayerService(path = "/player”) public class MyPlayerHandler extends HttpPlayerHandler { @Override public void onContentRequest(HttpPlayerSession contentSession) throws Exception { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "http://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setAttribute("player", playerEndpoint); HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(httpEndpoint); contentSession.start(httpEndpoint); } @Override public void onContentStarted(HttpPlayerSession contentSession) { PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player"); playerEndpoint.play(); } } Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/java/com/kurento/tutorial/MyPlayerHandler.java 20
  • 21. www.kurento.org Playing a file with an HTML5 client: client code<!DOCTYPE html> <html> <head> <script src="./js/kws-content-api.js"></script> <script> var conn; function start() { var options = { remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsContentPlayer("./player", options); } function terminate() { conn.terminate(); } </script> </head> <body> <button onclick="start();">Start</button> <button onclick="terminate();">Terminate</button> <br /> <video id="remoteVideo" autoplay></video> </body> </html> Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/webapp/player.html 21
  • 22. www.kurento.org Playing a file with an HTML5 client: See the example working https://www.youtube.com/watch?v=n5BQlhYgGSo 22
  • 23. www.kurento.org Media Pipeline Adding Computer Vision HttpGetEndpoint Media from file or URI HTTP media streaming Sink SRC PlayerEndpoint SRC Sink JackVaderFilter Media API REST API (Open API protocol) Handler codeI want “this media” Media is “at this URL” 23
  • 24. www.kurento.org Adding Computer Vision: Handler code @HttpPlayerService(path = "/playerWithFilter”) public class MyPlayerHandler extends HttpPlayerHandler { @Override public void onContentRequest(HttpPlayerSession contentSession) throws Exception { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "http://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setAttribute("player", playerEndpoint); JackVaderFilter filter = mp.newJackVaderFilter().build(); HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build(); filter.connect(httpEndpoint); playerEndpoint.connect(filter); contentSession.start(httpEndpoint); } @Override public void onContentStarted(HttpPlayerSession contentSession) { PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getAttribute("player"); playerEndpoint.play(); } } Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java 24
  • 25. www.kurento.org Adding Computer Vision: Client code <!DOCTYPE html> <html> <head> <script src="./js/kws-content-api.js"></script> <script> var conn; function start() { var options = { remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", options); } function terminate() { conn.terminate(); } </script> </head> <body> <button onclick="start();">Start</button> <button onclick="terminate();">Terminate</button> <br /> <video id="remoteVideo" autoplay></video> </body> </html> Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/webapp/playerFilter.html 25
  • 26. www.kurento.org Adding Computer Vision: See the example working https://www.youtube.com/watch?v=yJAQs23eoXw 26
  • 27. www.kurento.org WebRTC loopback Media Pipeline WebRTC Streaming Media API REST API (Open API protocol) Handler codeI want “this media (SDP)” Media is “at here (SDP)” SinkSRC 27
  • 28. www.kurento.org WebRTC loopback: Handler code 28 Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/java/com/kurento/tutorial/MyWebRtcLoopback.java @WebRtcContentService(path = "/webRtcLoopback") public class MyWebRtcLoopback extends WebRtcContentHandler { @Override public void onContentRequest(WebRtcContentSession contentSession) throws Exception { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); webRtcEndpoint.connect(webRtcEndpoint); contentSession.start(webRtcEndpoint); } } 28
  • 29. www.kurento.org WebRTC loopback: client code 29 <!DOCTYPE html> <html> <head> <script src="./js/kws-content-api.js"></script> <script src="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script> var conn; function start() { var options = { localVideoTag : "localVideo", remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsWebRtcContent("./webRtcLoopback", options); } function terminate() { conn.terminate(); } </script> </head> <body> <button id="start" onclick="start();">Start</button> <button id="terminate" onclick="terminate();">Terminate</button> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video> </body> </html> Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/webapp/webrtcLoopback.html 29
  • 30. www.kurento.org WebRTC loopback: see the example working 30 https://www.youtube.com/watch?v=HaVqO06uuNA 30
  • 31. www.kurento.org WebRTC recording Media Pipeline WebRTC Streaming Media API REST API (Open API protocol) Handler codeI want “this media (SDP)” Media is “at here (SDP)” SinkSRC Sink Media to file or URI 31
  • 32. www.kurento.org WebRTC recorder: Handler code 32 Source https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java Source of handler playing the recorded video: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/java/com/kurento/tutorial/MyPlayerRecording.java @WebRtcContentService(path = "/webRtcRecorder") public class MyWebRtcRecorder extends WebRtcContentHandler { @Override public void onContentRequest(WebRtcContentSession contentSession) throws Exception { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); RecorderEndpoint recorderEndpoint = mp.newRecorderEndpoint("file:///tmp/recording").build(); contentSession.setAttribute("recorder", recorderEndpoint); webRtcEndpoint.connect(webRtcEndpoint); webRtcEndpoint.connect(recorderEndpoint); contentSession.start(webRtcEndpoint); } @Override public void onContentStarted(WebRtcContentSession contentSession) { RecorderEndpoint recorderEndpoint = (RecorderEndpoint) contentSession.getAttribute("recorder"); recorderEndpoint.record(); } } 32
  • 33. www.kurento.org WebRTC recording: client code 33 <!DOCTYPE html> <html> <head> <script src="./js/kws-content-api.js"></script> <script src="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script> var conn; function start() { var options = { localVideoTag : "localVideo", remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsWebRtcContent("./webRtcRecorder", options); } function terminate() { conn.terminate(); } </script> </head> <body> <button id="start" onclick="start();">Start</button> <button id="terminate" onclick="terminate();">Terminate</button> <a href="./playerRecording">Play recorded content</a> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video> </body> </html> Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/webapp/webrtcRecorder.html 33
  • 34. www.kurento.org WebRTC recording: see the example working 34 https://www.youtube.com/watch?v=ogN81PGkMuE 34
  • 35. www.kurento.org WebRTC one-to-many (Handler omitted for simplicity) Media Pipeline SinkSRC SRCSinkSRCSinkSRCSink 35
  • 36. www.kurento.org WebRTC one-to-many: Handler code 36 Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/java/com/kurento/tutorial/MyWebRtcOneToMany.java @WebRtcContentService(path = "/webRtcOneToMany") public class MyWebRtcOneToMany extends WebRtcContentHandler { private WebRtcEndpoint firstWebRtcEndpoint; private String sessionId; @Override public synchronized void onContentRequest(WebRtcContentSession contentSession) throws Exception { if (firstWebRtcEndpoint == null) { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); firstWebRtcEndpoint = mp.newWebRtcEndpoint().build(); sessionId = contentSession.getSessionId(); contentSession.releaseOnTerminate(firstWebRtcEndpoint); firstWebRtcEndpoint.connect(firstWebRtcEndpoint); //Loopback contentSession.start(firstWebRtcEndpoint); } else { MediaPipeline mp = firstWebRtcEndpoint.getMediaPipeline(); WebRtcEndpoint newWebRtcEndpoint = mp.newWebRtcEndpoint().build(); contentSession.releaseOnTerminate(newWebRtcEndpoint); newWebRtcEndpoint.connect(firstWebRtcEndpoint); firstWebRtcEndpoint.connect(newWebRtcEndpoint); //Latest client gives media to the master endpoint contentSession.start(newWebRtcEndpoint); } } } 36
  • 37. www.kurento.org WebRTC one-to-many: client code 37 <!DOCTYPE html> <html> <head> <script src="./js/kws-content-api.js"></script> <script src="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script> var conn; function start() { var options = { localVideoTag : "localVideo", remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsWebRtcContent("./webRtcOneToMany", options); } function terminate() { conn.terminate(); } </script> </head> <body> <button id="start" onclick="start();">Start</button> <button id="terminate" onclick="terminate();">Terminate</button> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video> </body> </html> Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/webapp/webrtcOneToMany.html 37
  • 38. www.kurento.org WebRTC one-to-many: see the example working 38 https://www.youtube.com/watch?v=TBkrl3fmHWI 38
  • 39. www.kurento.org WebRTC game (Handler omitted for simplicity) Media Pipeline SinkSRC Sink SRC Sink SRC 39
  • 40. www.kurento.org WebRTC game: Handler code 40 Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/java/com/kurento/tutorial/MyWebRtcWithFilters.java @WebRtcContentService(path = "/webRtcWithFilters") public class MyWebRtcWithFilters extends WebRtcContentHandler { @Override public void onContentRequest(WebRtcContentSession contentSession)throws Exception { final MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); final WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); final PointerDetectorFilter pointerDetectorFilter = mp.newPointerDetectorFilter().build(); final FaceOverlayFilter faceOverlayFilter = mp.newFaceOverlayFilter().build(); PointerDetectorWindowMediaParam start = new PointerDetectorWindowMediaParam("start", 100, 100, 280, 380); start.setImage("http://ci.kurento.com/imgs/start.png"); pointerDetectorFilter.addWindow(start); pointerDetectorFilter.addWindowInListener(new MediaEventListener<WindowInEvent>() { public void onEvent(WindowInEvent event) { faceOverlayFilter.setOverlayedImage( "http://ci.kurento.com/imgs/mario-wings.png", -0.35F, -1.2F, 1.6F, 1.6F); } }); webRtcEndpoint.connect(pointerDetectorFilter); pointerDetectorFilter.connect(faceOverlayFilter); faceOverlayFilter.connect(webRtcEndpoint); contentSession.start(webRtcEndpoint); } } 40
  • 41. www.kurento.org WebRTC game: client code 41 <!DOCTYPE html> <html> <head> <script src="./js/kws-content-api.js"></script> <script src="https://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script> var conn; function start() { var options = { remoteVideoTag : "remoteVideo" }; conn = new kwsContentApi.KwsWebRtcContent("./webRtcWithFilters", options); } function terminate() { conn.terminate(); } </script> </head> <body> <button id="start" onclick="start();">Start</button> <button id="terminate" onclick="terminate();">Terminate</button> <br /> <video id="remoteVideo" autoplay></video> </body> </html> Source: https://github.com/Kurento/kurento-media-framework/blob/develop/kmf-samples/kmf- tutorial/src/main/webapp/webrtcFilters.html 41
  • 42. www.kurento.org WebRTC game: see the example working 42 https://www.youtube.com/watch?v=5eJRnwKxgbY 42
  • 43. www.kurento.org Collaborations welcome http://www.github.com/kurento Thank you very much for your attention Suggestions, comments and complains: lulop@kurento.org 43