Structural Alignment-Based Temporal Concealment of Packet-Loss in Video Ajit S. Bopardikar, Odd Inge Hillestad and Andrew Perkis Centre for Quantifiable Quality of Service in Communication Systems Trondheim, Norway.
Publications Based On This Work
A. S. Bopardikar, O. I. Hillestad and A. Perkis, “Temporal error concealment algorithm based on structural alignment for packet video,” International Conference on Signal Processing and Communications (SPCOM) , Bangalore, India, December 2004.
A. S. Bopardikar, O. I. Hillestad and A. Perkis, ”Structural alignment-based temporal concealment of packet-loss in video” in review, International Conference on Acoustics Speech and Signal Processing , Philadelphia, PA, USA, May 2005.
This Presentation Is About….
A novel temporal algorithm to conceal packet-loss related distortion in video.
Algorithm based on the insight that the spatial region around the area in the previous frame that yields the best concealment will in general be highly correlated with the corresponding area around the region to be concealed in the present frame.
Packet-loss related artifacts in block-based video compression schemes.
Spatial and temporal concealment methods.
BMA-type temporal concealment methods.
Results and discussions.
Flowgraph for Video Stream Packetization Video stream RTP/UDP IP packets Internet/ Simulation Test net Smart mapping Flows File format
Example: MPEG-2 Bitstream
Each packet can be decoded independently irrespective of the errors in decoding previous packets.
Loss of packet means loss of spatial and motion vector information for that part of the frame.
Packet-loss Related Artifacts in Block-Based Coding Schemes
Concealment Methods for Packet-Loss Related Errors in Video
Spatial Concealment Methods
Bilinear interpolation based methods.
Projection onto convex sets (POCS).
Requires no knowledge of the previous frame(s).
Simple and can cause blurring.
Better results can be obtained using temporal methods.
Temporal Concealment Methods
Makes use of the high degree of correlation that generally exists between consecutive frames.
The idea is conceal artifacts using ’best-fitting’ parts from the previous frame which is assumed to be error-free.
Temporal Replacement (TR)
Boundary Matching Algorithm (BMA).
Replaces lost and damaged macroblocks (MBs) in the present frame with MBs from the same spatial location in the previous, error-free frame.
In presence of significant motion, TR can give rise to visible discontinuities at boundaries of replaced MBs.
Boundary Matching Algorithm
Motion vector information typically lost with the packet.
These motion vectors need to be estimated for effective concealment.
BMA assembles a set of candidate motion vectors and then chooses a best fit from among the MBs in the previous frame associated with them.
Best fit MB minimizes the sum of mean squared error (MSE) or the sum of absolute difference (SAD) between its boundaries and the boundaries adjacent to them from the top, bottom and left MBs around the area to be concealed.
Based on the high degree of correlation that can exist between spatial region around the best-fit candidate MB in the previous frame and the corresponding region around the MB-area to be concealed in the present frame.
Insert candidate MB (M -1C ) in the area to be concealed.
Compute x and y Sobel gradient maps for the two composite blocks.
Compute the overall magnitude gradient map.
In the gradient map, compute the SAD between the corresponding red and green boundaries of the surrounding MBs.
Choose the MB from the previous frame that minimizes the sum of the three SADs for concealment.
Full search adopted to search for the best fitting MB.
The area in the previous frame for the search is 52x52 pixels centered on the spatial position of the MB to be concealed.
All video clips in SIF (352x240 pixels, 30 frames per second) MPEG-2 coded at 1.5Mbps.
Packet-loss simulated using NTT DoCOMo software. PLR = 0.001.
Performance compared to a BMA-like algorithm.
Simulations: Results for Frame 346 of ’Table Tennis’ sequence. Original PL ( 13.84 dB) BMA (28.91 dB) Proposed (36.34 dB)