Developing rich multimedia applications with FI-WARE.

1,103 views
1,000 views

Published on

FI-WARE will deliver a novel service infrastructure, building upon elements called Generic Enablers (GEs), which offer reusable and commonly shared functions making it easier to develop Future Internet Applications in multiple sectors. This presentation provides an overview of Kurento: the FI-WARE Generic Enabler that will ease development of advanced multimedia stream processing applications.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,103
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
37
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Developing rich multimedia applications with FI-WARE.

  1. 1. Open APIs for Open Minds The Stream Oriented Generic Enabler – SO GE Developing rich multimedia applications with FI-WARE.
  2. 2. Speakers 1  Luis López • Leader of the Kurento project • As. Prof. at URJC • Coordinator of the Stream Oriented GE at FI-WARE  Jose Antonio Santos • Chief Architect of Kurento project • Senior Developer at Naevatec • Researcher at FI-WARE project
  3. 3. The Stream Oriented Generic Enabler – SO GE 2 • Send and receive media (Multidevice/multiprotocol) • Process media (Computer vision, augmented reality, media indexing, etc.) • Transform and adapt media (H.264, H.263, VP8, Ogg, and others) Provides multimedia capabilities to the FI-WARE infrastructure • No need of protocol/codec low level understanding Exposes those capabilities through a simple to use API • LGPL 2.1 Is distributed through a flexible FOSS licence • Sorry for the instabilities… Is still work in progress
  4. 4. How media APIs (usually) work 3 Client side - WWW browser - Native API Server side - Media repositories - Media capabilities Hey, I want to send/receive media in this way Here you have the media you requested API API
  5. 5. The SO-GE API is a WWW inspired API 4  Dynamic WWW page development • Lots of capabilities (ex. Java EE)  Multimedia development with the SO-GE • Don’t want to loose any capability Create HTML - DDBB access - Transactions - Security tools - Etc. HTTP request: I want this resource HTTP response: The resource Create Media - Media API - DDBB access - Transactions - Security tools - Etc. HTTP request: I want this media HTTP response: The media is here
  6. 6. The SO-GE Media API: media elements and pipelines 5  Media Element • Provides a specific media functionality › Building block › Send/receive media › Process media › Transform media • The Media API provides a toolbox of media elements ready to be used. • New media elements can be added  Media pipeline • Chain of media elements implementing the desired media logic. • The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Media Element Sink SRC File Source SRC Ogg-demuxer Sink SRC Vorbis-decoder Sink SRC Audio-sink Sink Theora-decoder Sink SRC Video-sink Sink
  7. 7. The global vision of a SO-GE application 6 SO-GE Media Server Infrastructure Decode Video Augmented Reality Encode Video Computer Vision Video Playing and Recording
  8. 8. Examples of media elements: HttpEndPoint 7 HttpEndPoint  Function • Sends/receives media basing on HTTP transports (HTML5 video tag) • Associates media to a temporary-one- time-use URL  Formats • WebM • H.264 support under development  Input parameters • None HttpEndPoint Media Source Media Sink HTTP POST requests HTTP answers
  9. 9. Example of media elements: RtpEndPoint 8 RtpEndPoint  Function • Sends/receives media basing on RTP transports (Real Time Communications). • Negotiates media parameterization through SDPs  Formats • H.263, MPEG4, H.264, VP8  Input parameters • Input SDP RtpEndPoint Media Source Media Sink RTP input media RTP output media
  10. 10. Example of media elements: PlayerEndPoint 9 PlayerEndPoint  Function • Plays media from file or URI.  Formats • WebM • MP4 (H.264) • 3GPP • AVI • Etc.  Input parameters • File URI PlayerEndPoint Media Source Media from file or URI
  11. 11. Example of media elements: RecorderEndPoint 10 RecorderEndPoint  Function • Records media into a file.  Formats • WebM • H.264 formats under development  Input parameters • File path RecorderEndPoint Media Source Media Sink Media to file
  12. 12. Example of media elements: JackVaderFilter 11 JackVaderFilter  Function • Example of filter • Recognizes faces • Adds a Jack Sparrow or Darth Vader hat onto faces  Formats • Raw  Input parameters • None JackVaderFilter Sink SRC Media Stream Augmented Media stream
  13. 13. Example of media elements: ZBarFilter 12 ZBarFilter  Function • Example of filter • Recognizes Bar and QR codes • Generates events  Formats • Raw  Input parameters • None ZBarFilter Sink SRC Media Stream Media stream Event
  14. 14. Developing media application with the SO-GE  Runtime environment • Install Kurento Media Server (soon available at FI-LAB) • Install JBoss Java EE container (soon available at FI-LAB)  Development environment • Maven (repository: http://repository.kurento.com/archiva/repository/snapshots) • Soon available at maven central › kmf-content-api › Provides handers › kmf-media-api › Provides media elements and pipelines • Eclipse • Javascript libraries (available at the same repository) › kws-content-api 13
  15. 15. My first application with the SO-GE: play a video 14 HttpEndPoint Media Source Media Sink PlayerEndPoint Media Source Media from file or URI HTTP media streaming Client side Server side
  16. 16. My first application with the SO-GE: play a video  Source code … 15
  17. 17. My second application: including a simple filter 16 HttpEndPoint Media Source Media Sink PlayerEndPoint Media Source HTTP media streaming JackVaderFilter Sink SRC Client side Server side
  18. 18. My second application: including a simple filter  Source code … 17
  19. 19. Making it more fun: introducing real time media 18 Media Source Media Sink Real time media HTTP media streaming JackVaderFilter Sink SRC RtpEndPoint Media Source Media Sink Client side Client sideServer Side
  20. 20. Making it more fun: introducing real time media  Source code … 19
  21. 21. Programming with media events: the ZBarFilter 20 Real time media ZBarFilter Sink SRC RtpEndPoint Media Source Media Sink HttpEndPoint Media Source Media Sink Media events to browser Client side Client sideServer Side
  22. 22. Programming with media events: the ZBarFilter  Source code … 21
  23. 23. Digging into the media API: implementing your own filters: requirements  Requirements • GStreamer › Based on GObject and GLib › C programing skills needed • OpenCV or your preferred image processing software • C++ basic knowledge • Thrift basic knowledge 22
  24. 24. Digging into the media API: implementing your own filters: GStreamer media element implementation  Download media server source code  Look for sub-module gst-kurento-plugins  Enter into src/filters folder  Create your filter as couple of filtername.c filtername.h files • Your filter must inherit from GstVideoFilter • You have to implement virtual method transform_frame_ip › This method gives you each frame to be processed • Be aware of init and finalize functions if you use attributes that need to be freed 23
  25. 25. Digging into the media API: implementing your own filters: Integration in media server  Modify kms-interface/mediaServer.thrift file adding a new value to enum FilterType  Create a C++ file to manage your GStreamer filter (by convention we use the same name in camel case) • This class inherits from kurento::Filter class • Its constructor creates a “filterelement” gstreamer element and add your filter name as a parameter (it will be created automatically) • Destructor should free this element  Modify kurento::MediaPipeline::createFilter method in order to add your filter type in the switch statement 24
  26. 26. Facing the FI-WARE challenge using the SO-GE  Getting started • Use the FI-WARE Catalogue (ready since 15 minutes ago hopefully!) › http://catalogue.fi-ware.eu/enablers/stream-oriented-kurento/ • Ask the SO-GE team around the FI-WARE stand  Augmenting videos in novel ways • Sensor information on live stream videos › Pollution/noise/traffic  Red-filtering level • Combine audio-visual & sensor information through augmented reality › Thermometers, charts, gauges on top of a media flow.  Make a camera to become an advanced sensor using computer vision • Traffic status › Car counter • Weather › Sky/ground color  Conceive novel WWW and Smartphone interaction GUIs • Up to your imagination 25
  27. 27.  http://fi-ppp.eu  http://fi-ware.eu  Follow @Fiware on Twitter ! Thanks ! 26
  28. 28. Technical details: Javascript KwsContent API  Creating a media connection • conn = new KwsContent(uri, options); • Options › remoteVideoTag: id of the video tag rendering the remote stream › Others for WebRtc (currently not available) • Semantics › Connects to URI and requests media › Upon media reception, renders it on the specified video tag  Adding event handlers • conn.onstart( function … • conn.onremotestream (function … • conn.onterminate( function … • conn.onmediaevent( function … 27
  29. 29. The SO-GE Content API: handlers  Handler • Is implemented by the application developer (equivalent to a Servlet) • Defines the sequence of instructions executed when a request for media is received  Four types • PlayerHandler › Handles requests for playing media with HTTP transport › Compatible with the HTML5 video tag • RecorderHandler › Handlers request for recording media with HTML transport › Compatible with HTTP file upload mechanism • RtpMediaHandler › Handlers request for establishing RTP media sessions • WebRtcMediaHandler (under development) 28
  30. 30. Applica on logic Applica on Server (JBosss/Mobicents)Media Server Flash Video Server HTTP ServeletSIP Servlet Media element proxies Media Session (signaling) Server side signaling plane IP Network (remote clients) Media Session (media plane) SIPSignaling Media events RESTfulSignaling RAWHTTPSignaling OtherSignaling Other JEE SOAPSignaling Media applica on (App. logic) Media Repository Input Element Decoder AR Filter CV Filter Encoder Adaptor Output Element Incomingmediastream Outgoingmediastream Media message bus Control commands Technical details: the SO-GE Architecture 29
  31. 31. PlayerHanlderinternaldetails: sequencediagram 30 Player Handler Servlet Media ServerClient Player Handler (Applica on logic) PlayRequest Applica on Server POST Content URL create() onMediaRequest(playHandler) Addi onal logic play(source) create(h pEndPoint) connect(source, h pEndPoint) h pEndPoint.getURL Assigned URL (JSON object) 200 OK (JSON object) build media pipeline createPipeline create(media-element) create, connect, etc. GET URL Media

×