This document proposes a new motion-compensated error concealment scheme for MPEG-4 video transmission. It begins with an introduction to error concealment approaches and the problem of error propagation in entropy-coded MPEG-4 video. It then describes the proposed scheme which uses a modified spatial concealment method for corrupted IVOPs and a motion-compensated approach for PVOPs. For PVOPs, it first determines a predicted motion vector for each corrupted macroblock and then uses different search patterns tailored to small, medium, and large motion blocks to find the best matching region for concealment. Simulation results show the proposed scheme achieves better quality than several comparison methods in terms of PSNR.

1. A New Motion-Compensated Error Concealment Scheme for MPEG-4 Video
Transmission
Ching-Tung Hsu and Jin-Jang Leou
Department ofComputer Science and Information Engineering
National Chung Cheng University
Chiayi, Taiwan 621, Republic ofChina
E-mail. jjleou@,cs.ccu.edu.tw
Abstract-For entropy-coded NMPEG-4 video object planes error concealment approach [7-13] can be classified into
(VOPs), a transmission error in a codeword will not only affect three categories: (1) spatial concealment [7-8], (2) temporal
the underlying codeword but may also affect subsequent concealment [8-12], and (3) hybrid concealment [8], [13]. In
codewords, resulting in a great degradation of the received this study, a new motion-compensated error concealment
VOPs. In this study, after an error detection scheme is scheme is proposed.
employed to detect all the corrupted MPEG-4 VOPs and
macroblocks (MBs). A modified version of the spatial error The paper is organized as follows. The proposed scheme
concealment scheme 181 is employed to conceal corrupted is addressed in Section II. Simulation results are included in
IVOPs and the proposed motion-compensated error Section III, followed by concluding remarks.
concealment scheme is used to conceal corrupted PVOPs.
Here, a new fitness function for error concealment is II. PROPOSED SCHEME
proposed and a predicted motion vector (PMV) for each In this study, frame-based MPEG-4 videos are treated
corrupted MB is first determined by the spatially neighboring and each video sequence consists of 100 VOPs (frames), in
MVs around the corrupted MB and the temporally motion- which the first VOP is intra-coded IVOP, followed by 99
projected overlapping MBs in the previous VOP. With the inter-coded PVOPs. After all the corrupted MBs are
PMV being the central point, three rood search patterns are determined by an error detection scheme [13], the corrupted
developed for motion-compensated error concealment of
small-, medium-, and large-motion corrupted MBs. Finally, bocks (MBsin each cpted oP concealed by a
error concealment refinement using Lagrange interpolation is modified version ofthe spatial error concealment scheme [8]
performed on all the initially concealed MBs so that the final and the corrupted MBS in PVOPs will be concealed by the
error concealment results can be improved. proposed scheme.
A. Proposed Fitness Function FF()
I. INTRODUCTION To evaluate the fitness (goodness) of a candidate
For entropy-coded MPEG-4 video object planes (VOPs), concealed MB, a new fitness function, FF(), combining
a transmission error in a codeword will not only affect the three fitness measures, namely, DBME, MD, and VD, is
underlying codeword but also may affect subsequent proposed in this study and described as follows.
codewords, resulting in a great degradation of the received For a corrupted MB, x(p, q), p, q 1, 2. 16, if the
VOPs [1]. To limit error propagation, MPEG-4 inserts some coFresponding ith candidate concealed MB is denoted as
resynchronization makers into compressed video bitstreams 1 2 ... 16 the four (uer left bottom and right
and the section between two consecutive resynchronization q), prq = ofx.p,qh are (p, leftp,bqo, gndq)maesis called a video packet (VP) [2]. Although error neighboring MBs of x(p, q) are xu/(p, q), XL(P, q), XB(P, q),makersois calle video when any Althonizatior and XR(P, q), respectively. The conventional boundary
propagation can be terminated when any resynchronization matching error (BME) for the ith candidate concealed MB ymaker is correctly received, a transmission error will affect ingier (d
the underlying codeword and subsequent codewords within sgivenby 16
the corrupted VP, as an illustrated example shown in Fig. 1. BME(x,y)y= [xu(16,q)- yi (1,q)]2 + Y[XL(p16)-yi(p,1)]2
In general, error resilient approaches include two 16 q=1 16 p=
categories [1], namely, (1) the error resilient encoding + , [xB(1,q)- y (16,q)]2 +,[XR (p,1)- y (p,16)]2. (1)
approach [3-6] and (2) the error concealment approach [7- q=1 p=l
13]. In terms of the information used for concealment, the In this study, a new BME using boundary pixel directions
+ This work was supported in part by National Science Council and (orientations), namely, directional BME (DBME), is
Ministry of Economic Affairs, Taiwan, Republic of China under Grants proposed. DBME is defined as the sum of absolute
NSC 94-2213-E-194-006 and 94-EC-17-A-02-S1-032.
0-7803-9390-2/06/$20.00 ©)2006 IEEE 2873 ISCAS 2006

2. differences between the ith candidate concealed MB yi and PMV= Median(believable spatially neighboring MVs,
all the "believable" (correctly-received or concealed) four- temporally motion-projected overlapping MVs, MVavg), (3)
connected neighboring MBs using boundary pixel directions
(orientations). As an illustrated example shown in Fig. 2, the MVavg = Mean(believable spatially neighboring MVs,
boundary pixel directions are determined by using two temporally motion-projected overlapping MVs). (4)
bottom rows ofthe upper four-connected neighboring pixels,
xu(l5, 1), xu(l5, 2), ..., xu(15, 16); xu(16, 1), xu(16, 2), * ., Note that the PMV will be quantized to be the quantized
xu(16, 16), and two extra pixels, XUL(15, 16) and XUR(15, 1) PMV at 1/2-pixel accuracy and in Eqs. (3) and (4), the
(if they exist). For the pixel xu(16, i), there are three Median() and Mean() functions are applied individually on
(diagonal, vertical, and anti-diagonal) directional differences, the x andy components ofMVs.
di = xu(16, i) - xu(15, i-i), d2 xu(16, i) - xu(15, i), d3 =
xu(16, i) - xu(15, i+±), i =
2, 3,.15. In particular, the three On the other hand, partial probability distributions of the
directional differences ofxu(16, 1) ared = xu(16, 1) - XUL(i5, sum of absolute component differences between the
16), d2= xu(16, 1) - xu(15, 1), d3 xu(16, 1) - xu(15, 2), and quantized PMV at 1/2-pixel accuracy, (PMVX, PMVy),
that ofxu(16, 16) are di = xu(16, 16) - xu(15, 15), d2= xu(16, determined by the proposed scheme and the best MV, (Xb, yb),
16) - xu(15, 16), d3= xu(16, 16) - XuR(l5, 1), wherexuL(i, j) determined by the full search motion estimation (ME)
and XUR(i, j) denote the upper-left and upper-right algorithm at 1/2-pixel accuracy with the search range R= 16,
neighboring MBs of the corrupted MB, respectively. For the and QP = 18 for the three video sequences, "Table Tennis,"
pixel xu(16, i), if dk= min (di, d2, d3), the boundary pixel "Foreman," and "Stefan," are illustrated in Table 1, where d
direction ofxu(16, i) is dk. For example, ifthe pixel direction lxb-PMVxl + Yb-PMVyl. In Table 1, the probability that d
ofxu(16, i) is dl, d2, or d3, then xu(16, i) -yk(l, i+), XU(16, 2.5 pixels is larger than 90.0000. That is, the difference
i) - yk(1, i), or xU(16, i) - yk(l, i-i)l will be included in between (Xb, yb) and (PMVX, PMVy) can be solved
DBME computation, respectively. In particular, if either the satisfactorily by an adaptive local search ME procedure.
boundary pixel direction ofxu(16, 1) is d3 or that ofxu(6, 16) In this study, within the adaptive local search NM
is di, then they are ignored in DBME computation. procedure, the best MVs of small-, medium-, and large-
The corresponding DBMEs between the ith candidate motion MBs can be searched over small, medium, and large
concealed MB and the other "believable" four-connected numbers of checking points using different rood search
neighboring MBs can be similarily computed. DBME (yi) of patterns. Here, three kinds of rood search patterns in the
the ith candidate concealed MB yiis the weighted sum of the previous VOP, namely, 5-, 9-, and 12-point rood search
above-mentioned DBMEs. Second, the mean difference (MD) patterns, are developed, in which the quantized PMV is used
is defined as the mean difference between a candidate as the central point of each rood search pattern in the
concealed MB y1 and its "believable" eight-connected previous VOP. If PMV = pMV2+ PMV2 , IPMM is used
neighboring MBs. Third, the variance difference (VD) is y
defined as the variance difference between a candidate to determine the size of the rood search pattern. If
concealed MB yi and its "believable" eight-connected IPMi <T.7, the corrupted MB is determined as a small-
neighboring MBs. Finally, the proposed fitness function, motion MB, where T, is a predefined threshold. Then, as
FF(), for error concealment is given by shown in Fig. 3, the 5-point rood search pattern containing
five checking points with the PMV being its central point is
employed, i.e., five candidate concealed MBs in the previous
FF(y,) = DBME(y,) ± MD(y,) ± VD(y,). (2) VOP are generated for the corrupted MB. Finally, the
optimal candidate concealed MB having the smallest FF() is
Note that the candidate concealed MB having the smallest used to conceal the corrupted MB.
fitness function value is the "best" candidate concealed MB.
If TI ! PMV IZ Th, the MB is determined as a medium-
B. Proposed Motion-Compensated Error Concealment motion MB, where Th is also a predefined threshold. Then
Schemefor Inter-Coded PVOPs two rood search patterns shown in Fig. 4 are employed
In this study, the optimal candidate concealed MB for a "sequentially." The medium rood search pattern (step size =
corrupted MB in a PVOP is searched over all the motion- 2) contains 5 checking points with the PMV being its central
compensated MBs in the previous VOP. To speed up the point and then the small rood search pattern (step size = 1)
search process, a fast motion-compensated search algorithm contains 5 checking points with its central point being the
is proposed and described as follows. First, a predicted best checking point in the previous medium rood search
motion vector (PMV) for the corrupted MB is first pattern. In this case, at most 9 checking points (candidate
determined by using the spatially neighboring motion vectors concealed MBs) will be examined.
(MVs) around the corrupted MB and the corresponding f
"temporally neighboring" MVls in the previous vOPr whose t
"motion-projected" MBs in the current VOP overlap the large-motion MB. Then, as shown in Fig. 5, three rood
corrupted MB, namely, temporally motion-projected search patterns are employed "sequentially." The large rood
overlapping MVs. That is, search pattern (step size =3) containing 5 checking points
2874

3. with the PMV being its central point, the medium rood the proposed scheme are better than those of the four
search pattern (step size = 1) containing 5 checking point comparison schemes, namely, Zero-S, SWBM, SMVM, and,
with its central point being the best checking point in the MVRI. The proposed scheme can recover high-quality
previous large rood search pattern, and the small rood search MPEG-4 VOPs from their corresponding corrupted MPEG-4
pattern (step size = 1) containing four checking points with VOPs.
the "major" point being the best checking point in the
previous medium search pattern. In this case, overall at most REFERENCES
12 checking points (candidate concealed MBs) will be
examined. [1] Y. Wang and Q. F. Zhu, "Error control and concealment for videocommunication: a review," Proceedings ofthe IEEE, vol. 86, no. 5,
In this study, after all the corrupted MBs are initially pp. 974-997,May 1998.
concealed, error concealment refinement using Lagrange [2] ISO/IEC JTC1/SC29/WG11 N3093, "MPEG-4 video verification
MV interpolation [12] is performed on all initially concealed model version 17.0," July 2000.
MBs. This will improve the video quality of concealed [3] J. Cai and C. W. Chen, "Robust joint source-channel coding for
PVOPs. image transmission over wireless channels," IEEE Trans. on Circuitsand Systemsfor Video Technology, vol. 10, no. 6, pp. 962-966, Sept.
2000.
III. SIMULATION RESULTS [4] D. W. Redmill and N. G. Kingsbury, "The EREC: an error-resilient
technique for coding variable-length blocks of data," IEEE Trans. on
Several CIF (352x288 in size) video sequences with Image Processing, vol. 5, no. 4, pp. 565-574, April 1996.
different packet loss rates (PLRs) are used to evaluate the [5] U. Horn, K. Stuhlmuller, M. Link, and B. Girod, "Robust Internet
performance ofthe proposed scheme. Here the peak signal to video transmission based on scalable coding and unequal error
noise ratio (PSNR) is employed as the objective performance protection," Signal Processing: Image Communication, vol. 15, pp.
measure and the two thresholds, T, and Th, are empirically set 77-94, 1999.
to 8 and 16, respectively. To evaluate the performance ofthe [6] Y. Wang and S. Lin, "Error-resilient video coding using multiple
proposedscheme, four approaches
description motion compensation," IEEE Trans. on Circuits and
proposedl scheme, four comparison approaches are Systems for Video Technology, vol. 12, no. 6, pp. 438-452 , June
implemented. They are: (1) Zero-S (Zero-substitution); (2) 2002.
error concealment using subblock-based motion [7] Z. Wang, Y. Yu, and D. Zhang, "Best neighborhood matching: an
compensation with weighted boundary matching [10] information loss restoration technique for block-based image coding
(denoted by SWBM); (3) error concealment using selective systems," IEEE Trans. on Image Processing, vol. 7, no. 7, pp. 1056-
motion vector matching [9] (denoted by SMVM); (4) error 1061, July 1998.
concealment using motion vector rational interpolation [11] [8] L. W. Kang and J. J. Leou, "A hybrid error concealment scheme for
(denoted by MVRI).
MPEG-2 video transmission based on best neighborhood matching(denotedbyMVRI). algorithm," J Visual Communication and Image Representation, vol.
In terms of PSNRseq (dB), the simulation results for the 16, pp. 288-310, 2005.
"Stefan" sequence with different PLRs of the four [9] B. Yan and K. W. Ng, "A novel selective motion vector matching
Stefan. sequene wh
dalgorithm for error concealment in MPEG-4 video transmission over
comparison schemes and the proposed scheme are listed in error-prone channels," IEEE Trans. on Consumer Electronics, vol.
Table 2. As a subjective measure ofthe quality of concealed 49, no. 4, pp. 1416-1423, Nov. 2003.
PVOPs, the error-free and concealed PVOPs for the "Stefan" [10] Y. Xu and Y. Zhou, "H.264 video communication based refined error
sequence with PLR = 15% of the four comparison error concealment schemes," IEEE Trans. on Consumer Electronics, vol.
concealment schemes and the proposed scheme are shown in 50, no. 4, pp. 1135-1141, Nov. 2004.
Fig. 6. [11] s. Tsekeridou, F. A. Cheikh, M. Gabbouj, and I. Pitas, "Vector
rational interpolation schemes for erroneous motion field estimation
applied to MPEG-2 error concealment," IEEE Trans. on Multimedia,
IV. CONCLUDING REMARKS vol. 6, no. 6, pp. 876-885, Dec. 2004.
Based on the simulation results obtained in this study, [12] J. Zheng and L. P. Chau, "A temporal error concealment algorithm
severalobservations can be found.,1) The smoother te
' for H.264 using Lagrange interpolation," Proceedings of the IEEE,
several observations can be found. (1) The smoother the vol.2,pp. 133-136,May2004.
Original VOPs are, the better the corresponding error
vl
,p.1316 a 04
original VOPs are,the-better the corresponding error
[13] W. J. Chu and J. J. Leou, "Detection and concealment oftransmission
concealment results are. (2) The parameters used in the errors in H.261 images," IEEE Trans. on Circuits and Systems for
proposed scheme, such as T, and Th, will influence the Video Technology, vol. 8, no. 1, pp. 74-84, Feb. 1998.
performance of the proposed scheme. If T, and Th are set to
small values, most corrupted MBs will be determined as
"large-motion" MBs which will be concealed by examining
a larger number of candidate concealed MBs. This will
increase the computational complexity of the proposed
scheme (with the better conceaealmt results), and vice versa.
Hence, all the thresholds are empirically selected,
considering the trade-off between concealment performance(a(b
and computational complexity. (3) As shown in Table 1,(a(b
PMVs by the proposed scheme are indeedly very good. (4) Fig. 1. The original and corrupted MPEG-4 VOPs of the 15th VOP ofthe
As shown in Table 2 and Fig. 6, the concealment results of "Foreman" sequence with PLR =l10%: (a) the original VOP; (b) the
corrupted VOP.
2875

4. 5,1 6)1 xu (65,2) I xu(65,3) I xu(65,4) Ixu(65I()1 xu(l5,6)6Ix(I5(7)1 xu(l5,8) 1 xu(l5,9) 1 xu(l5,l0) 1 x ,(15,12) xI I5I 3 I x4 1 6 (15,16) (15.l)
,,(16,1) ,X(]6,2) ,X(16,3) ,X(]6,4) ,X(16,.5) ,X(]6,6) ,X(]6,7) ,X(]6,8) ,X(]6,9) ,X(16,10) ,X(16,11) ,X(16,12) ,X(16,13) ,X(16,14) ,X(16,15) ,X(16,16)
x(l,l) (1,2) x(1,3) x(1,4) x(1,5) x(1,6) ,(1,7) x(1,8)x(1,9)x(1,10)x(1,11) x(1,12) x(1,13) x(1,14) x(1,1.5) ,(1,16)l
Fig. 2. The relationship between the boundary pixels of a 16x 16 candidate concealed MB and its upper, upper-left, and upper-right neighboring MBs, xu,
xuL, and xuR, respectively.
Fig. 3. The 5-point rood search pattern for a small-motion corrupted MB
with the PMV being its central point (1/2-pixel accuracy).l
(c) (d)
(e) ~~~~~~~~(f)
Fig. 6. The error-free and concealed MPEG-4 VOPs of a PVOP (the 63th
VOP) within the "Stefan" sequence with PLR = 15%: (a) the error-free
VOP; (b)-(f) the concealed VOPs by Zero-S, SWBM, SMVM, MVRI, and
the proposed scheme, respectively.
Fig. 4. The medium rood search pattern (step size 2, 0) and the small
rood search pattern (step size =1, for a medium-motion corrupted MB Table 1. Partial probability distributions ofthe sum of absolute component
with the PMV being the central point *ofthe medium rood search pattern differences between the quantized PMV at 1/2-pixel accuracy by the
(1/2-pixel accuracy). proposed scheme and the best quantized MV by the full search ME
algorithm at 1/2-pixel accuracy with the search range R 16 and QP = 18.
d=O d' 0.5 d: 1 d' 1.5 d': 2 d': 2.5
sequence| | | pixel pixel pixel pixel pixel pixel
Table
41.18% 54.99% 71.74% 80.69% 85.63% 93.58%
Foreman 37.34% 59.23% 74.84% 82.62% 88.53% 96.62%
--- - Stefan 36.80% 64.51% 73.02% 77.32% 80.58% 90.25%
Table 2. The simulation results, PSNRseq (dB), for the "Stefan" sequence
1 T -1 T-1 1 T w e1 with different PLRs of the four comparison schemes and the proposed
1 1 -1 T 1 1 T 1 scheme.
Fig. 5. The large rood search pattern (step size 3, *), the medium rood _ PLR_ PSNRseq (dB)
search pattern (step size = 1, U), and the small rood search pattern (step ____ Zero-S SWBM SMVM MVRI Proposed
size = 1, A) for a large-motion corrupted MB with the PMV being the 0% 35.25 35.25 35.25 35.25 35.25
central point * ofthe large rood search pattern (1/2-pixel accuracy). 1% 14.89 31.47 31.77 32.04 32.47
5% 9.59 25.16 25.53 { 25.83 26.35
10% 8.61 22.18 22.46 22.77 23.33
15% 8.01 20.73 21.01 21.29 21.79
2876