Using Dialogic® Media Toolkit API for Image Overlay on Video ...

586 views
551 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
586
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Using Dialogic® Media Toolkit API for Image Overlay on Video ...

  1. 1. Small Logo Using Dialogic ® Application Note Media Toolkit API for Image Overlay on Video Streams
  2. 2. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Executive Summary Dialogic Multimedia Platform Software for AdvancedTCA Release 2.0 and Dialogic Multimedia Kit Software Release ® ® 1.0 for PCIe support the ability to overlay images on video streams using the image overlay feature implemented in the Dialogic Media Toolkit API library. This application note demonstrates how the image overlay feature (called the ® “MTK API” in this application note) can be used by an application so that a user can play an image (YUV or JPEG) over streaming video. Another application note discusses using an open source project for converting text into a graphics image, which can then be used by the image overlay feature.
  3. 3. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Table of Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Application Use Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Video Conference Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 About Dialogic Media Toolkit API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 ® MTK API Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Before Using the MTK API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 MTK API Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Creating and Applying Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Applying the Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Bounding Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Modifying the Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Other Functionalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Timers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Cleaning Handles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Demo Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Building the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Starting the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Determining Image Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Appendix A: Dialogic Media Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 ® Interdevice Connnections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 For More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1
  4. 4. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Introduction •  Displaying stream control icons for Play, Pause, Fast-   Forward, etc. Dialogic Multimedia Platform Software for AdvancedTCA ® Release 2.0 (MMP Software 2.0 for ATCA) and Dialogic ® •  Displaying call waiting notifications   Multimedia Kit Software Release 1.0 for PCIe (MMK Software Some of the use cases listed above involve using text, and 1.0 for PCIe) support the ability to overlay images on video other use cases use images as overlays on a video stream. streams by using the image overlay feature provided by the This application note focuses on image overlays on video Dialogic Media Toolkit API library. ® streams. The text-to-image conversion is discussed in another This application note demonstrates how the MTK API can be application note, Using Dialogic Media Toolkit API and ® used by an application to manipulate video streams. Various ImageMagick for Image Overlay on Video Streams (see the For application use cases involving image overlay are provided, More Information section). and sample applications that demonstrate this functionality Video Conference Examples are provided for download (see the For More Information section). Figures 1 through 4 show some examples of image overlay on a video conference. Participants in this video conference are Application Use Cases SIP peers connected to ipmB1C1 and ipmB1C2. Additionally, a 3G terminal is participating in the conference via the The following are a few examples of application use cases for using image overlays: m3gB1C1 device. It is also possible to have a multimedia device in the video conference, such as for the purposes of •  Applying a company logo to a video stream (single video   recording the video conference or playing some media into the stream or a conference) video conference, represented as mmB1C1 in the figures. •  Applying conferee name, title, or a conference title   Image overlays in this example video conference can be applied •  Displaying time left notifications, for example, “5 minutes   in a way so that some or all participants see the overlays via the left in conference” direction parameter. This parameter will be discussed in the •  Displaying video clip name or clip owner information   Overlay Direction section. Conference ipmB1C1 m3gB1C1 eSM_OVERLAY_DIRECTION_DEVICE Overlay to apply mmB1C1 ipmB1C2 Input to Video Conference Output from Video Conference Video Conference Output Received by Device’s Network Peer Input to Device from Network Peer Figure 1. All Participants See the Overlay for ipmB1C1 Participant 2
  5. 5. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Overlay to apply Conference eSM_OVERLAY_DIRECTION_NETWORK m3gB1C1 ipmB1C1 mmB1C1 ipmB1C2 Input to Video Conference Output from Video Conference Video Conference Output Received by Device’s Network Peer Input to Device from Network Peer Figure 2. Overlay Only Seen by ipmB1C1 Participant Overlay to apply Conference eSM_OVERLAY_DIRECTION_NETWORK eSM_OVERLAY_DIRECTION_NETWORK m3gB1C1 ipmB1C1 eSM_OVERLAY_DIRECTION_NETWORK eSM_OVERLAY_DIRECTION_NETWORK mmB1C1 ipmB1C2 Input to Video Conference Output from Video Conference Video Conference Output Received by Device’s Network Peer Input to Device from Network Peer Figure 3. Each Participant Sees a Different Overlay 3
  6. 6. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Overlay to apply Conference eSM_OVERLAY_DIRECTION_NETWORK eSM_OVERLAY_DIRECTION_NETWORK m3gB1C1 ipmB1C1 eSM_OVERLAY_DIRECTION_NETWORK eSM_OVERLAY_DIRECTION_NETWORK mmB1C1 ipmB1C2 Input to Video Conference Output from Video Conference Video Conference Output Received by Device’s Network Peer Input to Device from Network Peer Figure 4. All Participants See the Same Overlay About Dialogic® Media Took Kit API Advanced video features of MMP Software 2.0 for ATCA and MMK Software 1.0 for PCIe include image overlays and video conference party layouts. The Dialogic Media Toolkit API library’s image overlay feature (the Dialogic Media Toolkit API is called the “MTK API” in this application note) provides those features. The Dialogic Media Toolkit API library consists of general-purpose structures and attribute templates, as well as API functions for building and manipulating media-related items, such as video layouts and bitmaps. The MTK API allows manipulation of an incoming media stream to a device or outgoing media stream from a device - that is, it allows a user to place an image (YUV or JPEG) over a streaming video. The user creates a template of the image and defines the area on the video screen in which the image will be displayed. More information about the MTK API can be obtained from the API reference, Dialogic Media Toolkit API: Library Reference (see the For More Information section). ® Image overlay operations are applicable to MM, IPM, and M3G devices, each of which has audio and video ports that are either internal device connection ports (evolution of the CT-Bus) or external network ports (such as audio and video RTP streams of IPM devices). See Appendix A: Dialogic Media Devices section for an overview of various media devices and how they can be ® connected to each other and/or to the external network. MTK API Family The MTK API family is a collection of APIs that allow an application to manipulate video streams. The APIs are categorized as follows: •    tk_xxx() — General purpose functions used in creating and destroying image, file, bitmap, and frame templates. These m templates are then used by other APIs in the MTK API family. This category also contains functions that perform general error handling functionality. •   b_xxx() — Layout builder functions. These functions are used for creating video conference layouts. This API is not l discussed in this application note. •    m_xxx() — Stream manipulation functions. These functions allow overlays to be applied to a stream. The stream could be s coming into a device or going out from it. •    b_xxx() — Overlay builder functions. These functions, in conjunction with the mtk_xxx functions are used for creating and o manipulating image overlay templates. 4
  7. 7. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Before Using the MTK API Before an overlay can be applied, the following steps would need to performed to set the application variables. 1. Determine the image attributes. The image attributes are: image type and image dimensions. The Dialogic MTK API does not have functions that help determine these attributes. See the Determining Image Attributes section for one method of many that aid in determining image attributes. The MMP Software 2.0 for ATCA firmware supports YUV and JPEG image types. 2. Determine image scaling. An image can be applied at 100% scale (no scaling) or some other scaling. For better quality, the image should not be scaled. If the image is scaled higher or lower, the aspect ratio of the image should be retained. 3. Determine the position within a frame of a video stream where the image should be overlaid. The position can be either a percentage of a video stream’s frame dimensions or an absolute position. The upper left-hand corner of the frame is location (0,0). Figure 5 shows a frame and the coordinate system orientation. The overlay position can also be specified as a horizontal and vertical justification. Therefore, the application does not need to calculate position locations; it can use the API to specify the position justification as Left, Right, Top, Bottom, or Center for both horizontal and vertical settings. See the Bounding Frame Justification section for examples. If any attributes are to be applied in terms of absolute coordinates, the application needs to separately track the video stream resolution (QCIF or CIF). The application must have connected devices together using the dev_PortConnect(DMFL_TRANSCODE_ON). If a device is connected to another resource using the dev_Connect() function or dev_PortConnect(DMFL_TRANSCODE_NATIVE), the overlay operation cannot be performed. (0,0) (N,0) (0,M) (N,M) Figure 5. Coordinate System MTK API Concepts The MTK API is not bound to a Dialogic device initially. The application first uses the MTK API to obtain various handles and then ® eventually, the media is applied to a Dialogic device. Specifically, before an overlay can be applied to a media stream, applications must create two types of templates: overlay templates and bounding frame templates. These templates aid in associating the image data and placement of the image on a media stream. 5
  8. 8. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams The overlay template initialization activity associates an image 1. Create one or more overlay template(s). or image buffer to some handles, sets the fill style of the image in an area of the stream that will contain the image, and sets 2. Create one or more bounding frame template(s). the overlay visibility duration once the overlay is active on a media stream. 3. Bind an overlay template to a bounding frame. The bounding frame template represents the section of the 4. Apply an overlay template to a device. video frame in which the overlay appears in the media stream. Steps 1, 2, and 3 can be done in advance of when the overlay Creation of the overlay and bounding frame templates are is needed to be applied to a device. The figures show N overlay shown in the upper-left and upper-right rectangles of Figures and M bounding frame templates being created in advance. 6 and 7, respectively. Thus, different overlay schemes can be created in advance Creating and Applying Overlays and particular templates can be applied during application An application may use a file as the source of an image, or it runtime, as is done by Step 4. may load the image contents into a buffer and use that image buffer. Figure 6 and Figure 7 show the four steps required One or two overlays may be applied to a media device in a for creating an overlay and adding the overlay onto a device’s single sm_AddOverlays() function call. A device can support a media stream: maximum of two overlays at a given time. 1. Create N overlay templates 2. Create M bounding frame templates (M) hImageTemplate = hBoundingFrameTemplate = mtk_CreateImageTemplate(eMTK_IMAGE_FORMAT_JPEG) mtk_CreateFrameTemplate() hMedia = mtk_CreateMediaFileTemplate(hImageTemplate, mtk_SetFramePosition(boundingFrameTemplate,...) <file name>) mtk_SetFrameSize(boundingFrameTemplate,...) hOvlTemplate = ob_CreateImageOverlayTemplate(hMedia) ob_SetOverlayDuration(hOvlTemplate,...) ob_SetOverlayFillStyle(hOvlTemplate) 3. Bind the overlay templates to bounding frames hBoundingSnapshot = ob_SetOverlayBoundingFrame(hOvlTemplate,boundingFrameTemplate) 4. Apply up to 2 template(s) to media device sm_AddOverlays(SRL_DEVICE,{direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2) Figure 6. Adding JPEG Image Overlay 6
  9. 9. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams 1. Create N overlay templates 2. Create M bounding frame templates (M) hImageTemplate = hBoundingFrameTemplate = mtk_CreateImageTemplate(eMTK_IMAGE_FORMAT_YUV) mtk_CreateFrameTemplate() mtk_SetYUImageFormat(hImageTemplate, mtk_SetFramePosition(boundingFrameTemplate,...) eMTK_YUV_IMAGE_FORMAT_420) hMedia = mtk_SetFrameSize(boundingFrameTemplate,...) mtk_CreateMediaFileTemplate(hImageTemplate, <file name>) hOvlTemplate = ob_CreateImageOverlayTemplate(hMedia) ob_SetOverlayDuration(hOvlTemplate,...) ob_SetOverlayFillStyle(hOvlTemplate) 3. Bind the overlay templates to bounding frames hBoundingSnapshot = ob_SetOverlayBoundingFrame(hOvlTemplate,boundingFrameTemplate) 4. Apply up to 2 template(s) to media device sm_AddOverlays(SRL_DEVICE,{direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2) Figure 7. Adding YUV Image Overlay Applying the Overlays The same overlay template can be used to apply the same overlay to different media devices. Figure 8 shows that the SMEV_ ADD_OVERLAY event, returned to the application for Device A, contains hDeviceASnapshot. This snapshot handle is a reference to the parameters in the overlay template when the overlay is added to the given device. Any modification of the overlay or deletion of the overlay must be done using this device-specific snapshot handle. For the same reason, changes to the original overlay template are not made to the added overlay on the device. Note: If a device is connected to another device using dev_Connect() function or dev_PortConnect(DMFL_TRANSCODE_NATIVE), the overlay cannot be applied. In this instance, the dev_PortConnect(DMFL_TRANSCODE_ON) must be used prior to applying the overlay. 7
  10. 10. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Device A Device B Device C sm_AddOverlays(hDeviceA, {direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2) hDeviceASnapshot1 ... SMEV_ADD_OVERLAY hDeviceASnapshot2 sm_AddOverlays(hDeviceB, {direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2) hDeviceBSnapshot1 ... SMEV_ADD_OVERLAY hDeviceBSnapshot2 sm_AddOverlays(hDeviceC, {direction,hOvlTemplate}1 ... {direction, hOvlTemplate}2) hDeviceCSnapshot1 ... SMEV_ADD_OVERLAY hDeviceCSnapshot2 Figure 8. Applying Overlay Overlay Duration An overlay can have a finite duration or infinite duration on each media stream generated by the associated device. The ob_ SetOverlayDuration() function is used to either specify a duration in units of milliseconds for finite durations or MTK_INFINITE for infinite durations. Finite Duration If an overlay has a finite duration, the overlay will be overlaid from the beginning of any stream generated by the device until the specified time has elapsed, at which time it will disappear from the stream. Note that the application is not notified of an overlay expiry. After the expiry, the overlay is no longer visible on the media stream. The application must remove the overlay if the user does not wish for the overlay to be seen in subsequent streams generated by the device. Overlays with finite duration may help in creating animation effects. This is discussed in the Animation section. Infinite Duration An overlay can have an infinite duration. The overlay will be visible on each media stream generated by the device for the full duration of each stream until the application removes it from the device. Overlay Lifetime An overlay applied on a device is not removed from that device until the sm_RemoveOverlays() function is called. If an overlay has a finite duration, the sm_RemoveOverlays() must still be called by the application as discussed in the previous Overlay Duration section. 8
  11. 11. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams If an application closes a device without removing the overlay, the overlay is still associated with that device in the firmware. If a new application process is started, the overlay will still be present on that device. The same is true if the previous application process exited abnormally; the overlay will still be present on the same device when it is opened in a new application process. The sm_RemoveAllOverlays() function is provided for such scenarios. The application should call this API at the start of a new process to ensure that no overlays from a previous application are still visible. If the firmware is restarted, overlays will be removed. See Figure 9 for a diagram of this discussion. Device A Application Process Starts Add Overlay Close Device Application Process Exits New Application Process sm_RemoveAllOverlays() ... Figure 9. Overlay Lifetime Overlay Direction Overlay direction allows an overlay to be visible on either the outgoing or incoming media stream of a device. Two different directions are defined: •    SM_OVERLAY_DIRECTION_DEVICE — The overlay is placed on the data stream between the device in question and any e other device with which it is connected. •    SM_OVERLAY_DIRECTION_NETWORK — The overlay is placed on the data stream between the device in question and the e external network. For a media device that does not normally have a network interface, such as a multimedia device, the external network is considered the sink (that part of the system that receives information) associated with that device, such as a recorded file. Therefore, if a mm_Record(hMMDevice, …) operation is pending, applying an overlay to this multimedia device handle with direction as eSM_OVERLAY_DIRECTION_NETWORK will result in a recorded file with the overlay. If mm_Play(hMMDevice,…) 9
  12. 12. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams was issued and an overlay applied with direction eSM_OVERLAY_DIRECTION_DEVICE, another device that is listening to this multimedia device will receive a media stream with an overlay. Removing Overlays The same overlay can be applied to different media devices. The overlay template handle used during the sm_AddOverlays() function is not the same handle that is used to remove the overlay. When overlay(s) are added, the resultant SMEV_ADD_OVERLAY event is generated and sent to the application. This event contains an overlay snapshot handle for each overlay added to that specific device. If the application needs to remove an overlay, the snapshot handle returned in the SMEV_ADD_OVERLAY event is used in the sm_RemoveOverlays() function. See Figure 10 for an example. Device A Device B Device C sm_RemoveOverlays(hDeviceA, hDeviceASnapshot1 ... hDeviceASnapshot2) SMEV_REMOVE_OVERLAY sm_RemoveOverlays(hDeviceB, hDeviceBSnapshot1 ... hDeviceBSnapshot2) SMEV_REMOVE_OVERLAY sm_RemoveOverlays(hDeviceC, hDeviceCSnapshot1 ... hDeviceCSnapshot2) SMEV_ADD_OVERLAY Figure 10. Removing Overlays Bounding Frames A bounding frame is a region that specifies the placement of an image on top of the media stream. The bounding frame has a position and size and is associated to an image (or image buffer) to be overlaid. Bounding Frame Position The position of a bounding frame is set using mtk_SetFramePosition(MTK_FRAME_HANDLE a_hFrame, float a_x, float a_y, eMTK_POSITION_TYPE a_ePositionType) on the handle returned by the mtk_CreateFrameTemplate() function. The meaning of a_x and a_y arguments depends on the a_ePositionType parameter. 10
  13. 13. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Position type can be: •  eMTK_POSITION_TYPE_PIXEL — Pixel based coordinates; (0,0) is upper, left-hand corner   •  eMTK_POSITION_TYPE_PERCENT — Percentage based coordinates; (0,0) is upper, left-hand corner   •    MTK_POSITION_TYPE_JUSTIFICATION — Justification-based coordinates. See Bounding Frame Justification section for e examples. Bounding Frame Size The size of a bounding frame is set using mtk_SetFrameSize(MTK_FRAME_HANDLE a_hFrame, float a_w, float a_h, eMTK_SIZE_ TYPE a_eSizeType) on the handle returned by the mtk_CreateFrameTemplate() function. The meaning of a_w and a_h arguments depends on the a_eSizeType parameter. The size type can be: •  eMTK_SIZE_TYPE_PIXEL — pixel-based sizing   •  eMTK_SIZE_TYPE_PERCENT — percentage of the bounding frame-based sizing   The size of a bounding frame impacts the quality of the resultant overlay. The image will be stretched to fit the bounding frame. Note that the bounding frame is not visible — only the image to be overlaid is visible on top of the media stream. Figure 11 shows examples of an image being applied to a media stream and the results based on the bounding frame size. Ex Image Bounding Frame Media Stream Overlay Result 1 2 3 Figure 11. Bounding Frames 11
  14. 14. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams •    xample 1 — If the bounding frame does not have the same dimensions as the image, then the image will be stretched and E may be distorted. •  Example 2 — If the bounding frame is the same size as the image, then scaling distortion is not present.   •    xample 3 — If the image is larger than the bounding frame, then the image will be scaled down to fit the bounding frame. E This too may result in distortion. A main concern is that the proportion of the image and the frame should match closely. Bounding Frame Justification An application can specify the position in terms of justification instead of basing it on the base frame. Some justification examples are shown in Figure 12. (0,0) (N,0) A B C D E F G H I (0,M) (N,M) Figure 12. Overlay Justification Table 1 maps the example image overlay locations in Figure 13 to the justification parameters. Image Horizontal Justification Vertical Justification A MTK_JUSTIFY_LEFT MTK_JUSTIFY_TOP B MTK_JUSTIFY_CENTER MTK_JUSTIFY_TOP C MTK_JUSTIFY_RIGHT MTK_JUSTIFY_TOP D MTK_JUSTIFY_LEFT MTK_JUSTIFY_CENTER E MTK_JUSTIFY_CENTER MTK_JUSTIFY_CENTER F MTK_JUSTIFY_RIGHT MTK_JUSTIFY_CENTER G MTK_JUSTIFY_LEFT MTK_JUSTIFY_BOTTOM H MTK_JUSTIFY_CENTER MTK_JUSTIFY_BOTTOM I MTK_JUSTIFY_RIGHT MTK_JUSTIFY_BOTTOM Table 1. Overlay Justification 12
  15. 15. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Modifying the Overlays An application may want to modify an overlay, such as an overlay’s position, size, or duration. If a different image is required on a device, the application first removes an existing overlay by calling the sm_RemoveOverlays() function and then applies a new overlay using the sm_AddOverlays()function. If an overlay’s position or size is to be changed, a call to the sm_RemoveOverlays() function is not required. Overlay location changes are accomplished by resetting the bounding frame on the overlay snapshot and calling the sm_AddOverlays() function on the snapshot. This is discussed in following section. Bounding Frame Related Modifications Several modifications can be made to an overlay that has already been applied to a media device. Some modifications are performed by manipulating the bounding frame set on the overlay. Figure 13 shows a modification of an image’s placement on a media stream. The initial image was overlaid in the bottom-right corner. The image is being moved from bottom-right to upper-left of the video frame. After applying the initial overlay using the sm_AddOverlays() function, the application received SMEV_ADD_OVERLAY event. The event contained hDeviceSnapshot. Now the application wants to change the position of the image. Since the overlay snapshot is a separate overlay from the template used to add the overlay, it has a separate bounding frame than the original template. To modify the bounding frame associated with the overlay snapshot, the application developer would need to make a call to the ob_GetOverlayBoundingFrame() function. Then mtk_SetFramePosition() and mtk_SetFrameSize() functions can be used with that newly acquired handle. The application must then use hDeviceSnapshot (from SMEV_ADD_OVERLAY) to apply the bounding frame change to the specific media device’s media stream. Note that since the snapshot’s bounding frame was modified, no other overlay’s bounding frames are affected. A call to the sm_AddOverlay() function must be called on those devices using their own hDeviceSnapshot overlay handles and those overlays’ bounding frames. Note that in the example shown in Figure 14, an existing overlay need not be removed. Also, the overlay and bounding frame templates do not need to be rebound. Other Functionalities Animation Concepts presented so far in this application note can be used to create rudimentary animation effects. Some of the building block concepts that can be used to create animation effects are: •  Creating a series of Overlay Templates and Bounding Frame Templates as shown in Steps 1 and 2 of Figure 6   •  Bounding frame modification: position   •  Bounding frame modification: size   Timers Timers are the key to creating animation effects. Timers are not provided by Dialogic APIs. A timer facility to drive MTK API for ® animation effects can be created for an application. 13
  16. 16. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams hImageTemplate, hMedia, hOvlTemplate, hBoundingFrameTemplate sm_AddOverlays() → SMEV_ADD_OVERLAY → hDeviceSnapshot1… hDeviceSnapshot2 mtk_DestroyMediaTemplate(hImageTemplate), mtk_DestroyMediaTemplate(hMedia), ob_DestroyOverlayTemplate(hOvlTemplate), mtk_DestroyFrameTemplate(hBoundingFrameTemplate), hSnapshotBoundingFrame = ob_GetOverlayBoundingFrame(hDeviceSnapshot1) sm_RemoveOverlays(hDevice, hDeviceSnapshot1… No need for these steps. hDeviceSnapshot2) mtk_SetFramePosition(hSnapshotBoundingFrame, …) mtk_SetFrameSize(hSnapshotBoundingFrame, …) hBoundingSnapshot = ob_SetOverlayBoundingFrame(hOvlTemplate, boundingFrameTemplate) sm_AddOverlays(SRL_DEVICE, {direction, hDeviceSnapshot})… Figure 13. Bounding Frame Modification Using Timers Although a maximum of two overlays may be added at a time, an application can create a series of overlay templates and bounding frame templates. Then, the application may remove one or both overlays that are currently applied and add the next overlay(s) from the series of pre-created templates. Using application-level timers, the application can remove or modify overlays as discussed in previous sections. Cleaning Handles MTK API provides other features besides image overlay. Due to this flexibility, many handles are tracked by the MTK library. These handles should be closed by the application at an appropriate time. Figure 14 shows when different handles can be closed. Some handles (hBoundingSnapshot and hDeviceSnapshot) are not required to be closed. The MTK library takes care of closing the snapshot handles, such as those created by calls to the ob_ CreateImageOverlay() function, and in the event data associated with calls to the sm_AddOverlays() function. 14
  17. 17. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams hImageTemplate, hMedia, hOvlTemplate, hBoundingFrameTemplate, hBoundingSnapshot sm_AddOverlays() → SMEV_ADD_OVERLAY → hDeviceSnapshot1… hDeviceSnapshot2 mtk_DestroyMediaTemplate(hImageTemplate), mtk_DestroyMediaTemplate(hMedia), ob_DestroyOverlayTemplate(hOvlTemplate), mtk_DestroyFrameTemplate(hBoundingFrameTemplate), hBoundingSnapshot(No action required) ... sm_RemoveOverlays(hDevice, hDeviceSnapshot1… hDeviceSnapshot2) ... Figure 14. Handle Cleanup Demo Application A sample application is provided to demonstrate the image overlay functionality of MTK API. This application is called img_ovl. It sets up a chain of devices as shown in Figure 15. This chain of devices includes two MM and two IPM devices. The MM and IPM devices are routed to each other using the dev_PortConnect() function with DMFL_TRANSCODE_ON flag. The two IPM devices stream to each other using four RTP streams. Each IPM transmits a video and audio RTP stream for a total of two streams per IP media device. One of the MM devices (mmB1C1) used for playing a media file contains audio and video. This media stream is transmitted to peer IPM device (ipmB1C1). This IPM device in turn sends RTP streams to neighboring IPM device (ipmB1C2) using the TCP/IP stack. The receiving IPM device then transmits the media stream to the second MM device (mmB1C2). This second MM device records the incoming media stream to audio and video files. The sample application applies the overlay on mmB1C1 using direction of eSM_OVERLAY_DIRECTION_DEVICE. This results in the overlay on top of media stream flowing from mmB1C1 to ipmB1C1. No other overlays are added by the demo application. 15
  18. 18. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams File I/O: Device Connections: mm_Play dev_PortConnect(DMFL_TRANSCODE_ON) Ai Ao RAi Ao Ai IP media RAo MM device device (mmB1C1) Vi Vo (ipmB1C1) RVi Vo Vi RVo RTP Streams File I/O: Device Connections: IP mm_Record dev_PortConnect(DMFL_TRANSCODE_ON) Ai Ao RAi Ao Ai IP media RAo MM device device (mmB1C2) Vi Vo (ipmB1C2) RVi Vo Vi RVo Figure 15. Demo Application Device Setup Building the Application make clean The img_ovl application can be built with or without the USE _ IMAGE _ MAGICK=1 make support of the ImageMagick library. To build img_ovl with the ImageMagick library, see the Using Dialogic Media Toolkit ® Starting the Application API and ImageMagick for Image Overlay on Video Streams The overlay that is applied depends on the command line application note (see the For More Information section). option. The application takes “-o” command line option. Briefly, ImageMagick allows an application to convert text If option 1 is selected, srcMedia/hello.jpg file is used for string to an image file. If ImageMagick support is enabled, the overlay. If option 2 or 3 is selected, UTF-8 encoded file img_ovl can convert UTF-8 encoded files or ASCII to images in the srcMedia directory is converted into an image using and use those images as overlays. The demo is provided with ImageMagick and that image is applied as overlay. If option Hindi and Chinese strings in UTF-8 encoded files. Those files 4 is selected, MM channel’s name is converted into an image are in srcMedia directory. using ImageMagick and applied. To build the application without ImageMagick support, type: Application built without ImageMagick support: make clean o - [0, 1] == Overlay option make 0 == No overlay To build the application with ImageMagick support, first follow 1 == hardcoded overlay, see source instructions in the Using Dialogic Media Toolkit API and ® code ImageMagick for Image Overlay on Video Streams application note to install various packages. Then type: 16
  19. 19. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Application built with ImageMagick support: Determining Image Attributes o - [0, 1, 2, or 3] == Overlay option Many commercial and open source tools are available to help 0 == No overlay edit images as well as identify the image attributes. One tool 1 == hardcoded overlay, see source is “identify” and it is included with ImageMagick (see the code application note listed in the For More Information section). 2 == Hindi overlay Sample output of “identify”: 3 == Chinese overlay [root@localhost export]# identify 4 == Channel name as overlay -verbose hello.jpg The demo application also takes “-cN” option, where N Image: hello.jpg represents number of MM+IPM pairs to open. By default, two pairs are opened, as shown in Figure 11. Format: JPEG (Joint Photographic Experts Group JFIF format) Application Runtime <snip> The img_ovl application takes keyboard entries for several tasks. When the application starts, it opens the devices, Geometry: 100x100+0+0 initializes the image overlays, and connects the devices using the dev_PortConnect() function. Once the initialization is done, <snip> keyboard entry is allowed. Valid keypress options are: Compression: JPEG s – Start mm_Play(), ipm_StartMedia(), mm_Record(), apply overlays <snip> t – Stop activity started by “s”. Overlays applied to devices are [root@localhost export]# also removed Summary r – After “s” and before “t”, remove an overlay that has already been applied. This application note presented the steps for enabling the creation of overlay and border templates, applying overlays o – After “s” and before “t”, add an overlay and modifying overlays, and binding overlay templates and q – Quit the application border templates. API call sequences were presented, along with a demo application. Image overlays can be applied into A user may want to experiment by changing the code such or out of media streams of Dialogic media devices such as ® that overlay is applied to video streams received by mmB1C2 or by applying the overlay on top video streams entering or MM, IPM, and M3G. The reader can opt to implement various exiting the IPM devices. application use cases listed in the Introduction. 17
  20. 20. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams Appendix A: Dialogic® Media Devices Appendix A provides background on the Dialogic media devices. A variety of media devices stream audio and video: •   PM devices — IPM devices source and receive RTP streams. Because there can be many instances of IPM devices in a I system, IPM devices are enumerated using ipmBxCy naming scheme, where x represents a virtual board and y represents a channel with a unique RTP network address. IPM devices are network devices. •    M devices — Multimedia devices provide file and buffer input/output to other devices. MM devices are a resource for the M network devices. Multiple MM devices can be connected to each other as well. •    3G devices — M3G devices are capable of controlling 3g324m connections. M3G devices rely on other transport channels M such as a dti channel (DS0) for 3G Release 99 networks or ipm channel (NbUP over RTP) for 3G Release 4. A common feature of these devices is that they have audio and video ports. These ports are used to stream media between these devices. The devices can also stream from/to the external network. Some device types can stream directly to the external network, whereas other device types are treated as resources that are used by the network devices for streaming media. Interdevice Connections Devices with common port types can be connected to each other via the Device Management API. An example is shown in Figure 16. MM and IPM devices have audio and video input and output ports; these are labeled Ai, Ao, Vi, and Vo ports. Ai and Ao are the device’s audio input and output ports respectively. Similarly, the Vi and Vo ports are the video input and output ports of the device. The input port of a device is connected to output port of peer device. File I/O Device Connections RTP Streams A Ao RAi Ao Ai IP media RAo MM device device IP (mmBxCy) V Vo (ipmBxCy) RVi Vo Vi RVo Figure 16. SIP Video Portal Another example of device connections is shown in Figure 17. In this figure, a 3G caller is connected to a video portal service. The video portal service is enabled by connecting a dti device with m3g device. This dti ← → m3g connection provides the transport of multiplexed 3g-324m data between the 3G caller and the m3g device. The m3g device then de-multiplexes the 3g-324m data into audio, video, and control channels. The audio and video data is then connected to an MM device for multimedia playback and/or recording. 18
  21. 21. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams In Figure 17, the external network is Release 99-based, which uses TDM for bearers. The DS0 is connected using the CTBus ports of dti and m3g devices. If the external network was IP-based Release 4/5, the 3g-324m stream would be carried over RTP. In that case, an IPM channel would be used for 3g-324m transport. See Figure 18 for the port connections between various devices. File I/O 324mi Ao Ai TDM 324mo Mux3G Ai Ao device MM device PSTN device CTbus (mmBxCy) RJ45 (dtiBxTy) (m3gBnTm) Vo Vi CTbus Vi Vo Figure 17. 3G Video Portal, Release 99 File I/O 324m Ao A 324mo 324mo Mux3G Ai Ao NbUPi IP media MM device IP device 324m device NbUPo (ipmBxCy) (m3gBnTm) Vo (mmBxCy) V CTbus Vi Vo Figure 18. 3G Video, Release 4/5 19
  22. 22. Application Note Using Dialogic Media Toolkit API for Image ® Overlay on Video Streams For More Information A Zip file containing the sample applications can be downloaded at — http://www.dialogic.com/goto/?11034 Related Application Note Using Dialogic Media Toolkit API and ImageMagick for Image Overlay on Video Streams — ® http://www.dialogic.com/goto/?11011 Reference Material Dialogic Media Toolkit API: Library Reference — ® http://www.dialogic.com/manuals/docs/media_toolkit_api_v2.pdf 20
  23. 23. Small Logo www.dialogic.com Dialogic Corporation 9800 Cavendish Blvd., 5th floor Montreal, Quebec CANADA H4M 2V9 INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH PRODUCTS OF DIALOGIC CORPORATION OR ITS SUBSIDIARIES (“DIALOGIC”). NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN A SIGNED AGREEMENT BETWEEN YOU AND DIALOGIC, DIALOGIC ASSUMES NO LIABILITY WHATSOEVER, AND DIALOGIC DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/ OR USE OF DIALOGIC® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY INTELLECTUAL PROPERTY RIGHT OF A THIRD PARTY. Dialogic products are not intended for use in medical, life saving, life sustaining, critical control or safety systems, or in nuclear facility applications. Dialogic may make changes to specifications, product descriptions, and plans at any time, without notice. Dialogic is a registered trademark of Dialogic Corporation. Dialogic’s trademarks may be used publicly only with permission from Dialogic. Such permission may only be granted by Dialogic’s legal department at 9800 Cavendish Blvd., 5th Floor, Montreal, Quebec, Canada H4M 2V9. Any authorized use of Dialogic’s trademarks will be subject to full respect of the trademark guidelines published by Dialogic from time to time and any use of Dialogic’s trademarks requires proper acknowledgement. The names of actual companies and products mentioned herein are the trademarks of their respective owners. Dialogic encourages all users of its products to procure all necessary intellectual property licenses required to implement their concepts or applications, which licenses may vary from country to country. This document discusses one or more open source products, systems and/or releases. Dialogic is not responsible for your decision to use open source in connection with Dialogic products (including without limitation those referred to herein), nor is Dialogic responsible for any present or future effects such usage might have, including without limitation effects on your products, your business, or your intellectual property rights. Any use case(s) shown and/or described herein represent one or more examples of the various ways, scenarios or environments in which Dialogic products can be used. Such use case(s) are non-limiting and do not represent recommendations of Dialogic as to whether or how to use Dialogic products. Copyright © 2008 Dialogic Corporation All rights reserved. 08/08 11033-01

×