This paper introduces a novel bidirectional bias correction scheme for gradient-based shift estimation (GBSE), which addresses the systematic bias caused by noise and aliasing in image registration. The method improves the accuracy of shift estimation by computing a bias matrix from multiple subpixel shifts and correcting for estimation biases that are linearly proportional to the true shift. Experimental results demonstrate that the proposed approach effectively reduces bias and enhances precision, making it suitable for various computer vision applications.

1. BIDIRECTIONAL BIAS CORRECTION FOR GRADIENT-BASED SHIFT ESTIMATION
Tuan Q. Pham Matthew Duggan
Canon Information Systems Research Australia (CiSRA)
1 Thomas Holt drive, North Ryde, NSW 2113, Australia
http://www.cisra.com.au
ABSTRACT of the image content and the distribution of the motion vec-
tors. However, none of the above methods considers bias due
This paper characterizes the bias of a Gradient-Based Shift
to noise or aliasing. This explains why [8] fails to produce a
Estimator (GBSE) and proposes a novel scheme to correct for
satisfactory result for low Signal-to-Noise Ratio images (e.g.,
the bias. The bias of GBSE comes from an inaccurate es-
SNR < 25dB).
timate of the image gradient energy, which is due to noise,
To our knowledge, only iterative GBSE can reduce the
aliasing and built-in low-pass filtering. For subpixel shift,
estimation bias to almost zero [5]. The method successively
the bias is linearly proportional to the true shift. The linear
shifts one image closer to the other using a previously esti-
bias factor can be blindly estimated using two or more sub-
mated shift. Since the displacement between the two images
pixel shift estimates of the same image pair, each estimate is
gets smaller after each iteration, so is the bias. However, the
performed on a slightly shifted version of the second image.
method requires multiple passes over the input images, each
Compared to the original GBSE, the new method reduces the
with an expensive motion compensation operation. It is there-
bias by half at the same time improving the precision.
fore not suitable for many real-time applications.
Index Terms— image registration, gradient-based shift In the remaining part of this paper, we will show that the
estimation, bidirectional bias correction, optical flow. bias of GBSE mainly comes from a poor estimation of the im-
age gradient. Whether it is due to image prefiltering or noise
1. INTRODUCTION or aliasing, this introduces an estimation bias linearly propor-
tional to the true shift. Using at most four shift estimations on
Shift estimation is a basic building block in many com- single-pixel shifted versions of the same image pair, the bias
puter vision systems. The task is to estimate an apparent can be computed and compensated for.
translational motion between two images of a moving scene
or two images captured by a moving camera. There are 2. GRADIENT-BASED SHIFT ESTIMATION
various methods to handle this task, for example, cross-
correlation, phase correlation, and Gradient-Based Shift Esti- Gradient-based shift estimation, a.k.a. optic flow computa-
mation (GBSE). GBSE [1], a.k.a. differential-based method tion [1], is a least-squares method to solve for a small shift
or optic flow, is an efficient spatial-domain method for sub- v between two images I1 and I2 . The method makes use of
pixel shift estimation. However, GBSE has been shown to the Taylor series expansion of a slightly shifted version of the
be biased [2, 3, 4]. The bias is due to a truncation of the reference image I1 :
Taylor series approximation within the algorithm as well as
I2 (x) = I1 (x + v) = I1 (x) + v I(x) + ε (1)
noise [5]. This paper estimates the bias of GBSE directly
from the input images and hence produces a bias-reduced where x is the image coordinate, I is the first-order deriva-
shift estimation. tives of I1 , and ε represents the sum of higher-order polyno-
Previous attempts to reduce the bias of GBSE mainly fo- mial terms (ε is negligible for ||v|| < 1). The solution is ob-
cus on the design of gradient filters. Because finite gradi- tained by minimizing an approximated Mean Squared Error
ents are more accurate at low frequencies, the image is often (MSE) of the two images with respect to the unknown shift
pre-filtered prior to differentiation. Simoncelli [6] designed a v:
2
general-purpose gradient filter by matching it with the deriva- ε2 = (I2 − I1 − v I) (2)
tive of the pre-filter. Elad et.al. [7] designed two pre-filters
and a gradient filter for GBSE by minimizing the error of Tay- where denotes an average operator over the whole image.
lor series approximation. Robinson and Milanfar [8] designed For 1-D signals, this yields a least-squares solution:
a gradient filter to minimize the bias due to Taylor series ap- It I
proximation. Both the latter methods use a prior knowledge v= (3)
I2

2. where I is the spatial gradient image and It = I2 − I1 is the ˆ ˆ
but E[b] = b. Equating b and b, the estimated shift v is ˆ
temporal gradient image. linearly proportional to the true shift v via a bias matrix:
For 2-D images, minimizing (2) yields a linear system:
ˆv
Aˆ = Av ⇒ ˆ ˆ
v = A−1 Av = Mv (6)
Av = b (4)
2 Similar to the bias factor in 1D, this bias matrix M is inde-
Ix Ix Iy
where A = 2 is the second moment pendent of the second image I2 .
Ix Iy Iy
It Ix a b ˆ a + ∆a b + ∆b
matrix, b = is the spatio-temporal gradient cor- Denote A = and A = ,
It Iy b c b + ∆ b c + ∆c
vx where ∆a , ∆b and ∆c are noise-induced offsets of the gra-
relation vector, and v = is the unknown shift. The dient auto-correlation and cross-correlation terms a, b and c,
vy
solution of this linear system is v = A−1 b. respectively, the bias matrix M can be rewritten as:
1 + (a∆c − b∆b )/∆ (b∆c − c∆b )/∆
3. BIAS AND PRECISION OF GBSE M= (7)
(−a∆b + b∆a )/∆ 1 + (c∆a − b∆b )/∆
Due to Taylor series approximation and inaccurate gradient where ∆ = (a+∆a )×(c+∆c )−(b+∆b )2 is the determinant
estimation, GBSE has a systematic error (i.e. bias) and a ˆ
of A. Note that the diagonal terms of M are close to one
stochastic variation about the biased value (i.e. nonzero preci- and the off-diagonal terms are close to zero because a, c >>
sion). Of all the mentioned bias sources, Taylor series approx- b and ∆a , ∆c >> ∆b (b and ∆b are statistically zero for
imation incurs the least bias, which is proportional to a cubic infinitely large images). If the off-diagonal terms of M are
power of the subpixel shift [5]. We therefore only model the approximated by zeros, this 2-D bias matrix model reduces to
bias due to inaccurate gradient estimation in this analysis. a 1-D bias factor model for each shift component along the
x− and y−directions.
3.1. 1-D bias factor model
The 1-D shift estimator in (3) is biased if the gradient I is 4. BIDIRECTIONAL BIAS CORRECTION
computed from a noisy image I1 = I + n1 . This is because
the denominator of the solution is overestimated: Because both the bias factor in 1-D and the bias matrix in 2-
E ˆ2
I = I +E 2
n 2
(5) D depend only on the reference image, all shift estimations
against a common reference image share the same bias fac-
As a result, t is underestimated by a constant factor k = S+NS tor/matrix. We can therefore derive this bias factor/matrix
where S = var(I ) is the variance of the uncorrupted image directly from the input images using additional shift estima-
gradient and N = var(n ) is the variance of the noise gra- tions. The additional shift is estimated between a shifted ver-
dient. Theoretically, S and N can be estimated from a noisy sion of I2 by a known translation against a fixed reference
image, hence the shift bias can be corrected [9]. In practice, image I1 .
however, this is a difficult task because noise is not always
Gaussian nor is it uncorrelated with the signal. Furthermore, 4.1. 1-D bias factor correction
a noise-corrected value ˆ
I 2 − var(nx ) is not necessarily a
good approximation of the true gradient energy I 2 when GBSE assumes that the two input images I1 and I2 are already
a pre-filter is used or when the image is aliased or small. aligned to one pixel accuracy. By shifting image I2 one pixel
Different from bias, which mainly comes from an incor- towards image I1 , the displacement from I1 (x) to I2 (x±1) is
rect estimation of the solution’s denominator, precision is due also less than one (the sign of ±1 equals −sign(v0 ) where v0
to stochastic variation in the nominator. Even when the two is an initial shift estimate between I1 and I2 ). In the example
images are perfectly aligned, noise introduces a nonzero shift illustrated by Figure 1, a plus sign is used because v0 ≤ 0.
estimation (3) because It I = (n2 − n1 )(I + n ) = 0. Because the same reference image I1 (x) is used, the two es-
This gradient correlation term only reduces to zero for in- timated shifts are biased by the same factor:
finitely large images.
v0 = findshift(I1 (x), I2 (x)) = kv (8)
3.2. 2-D bias matrix model v1 = findshift(I1 (x), I2 (x + 1)) = k(v + 1) (9)
In 2-D, the diagonal entries of the second moment matrix A Subtracting (8) from (9) resolves the bias factor:
is overestimated in the presence of noise:
2
Ix + var(nx ) Ix Iy + cov(nx , ny ) k = v1 − v0 (10)
ˆ
E A = 2 =A,
Ix Iy + cov(nx , ny ) Iy + var(ny )

3. Actual shifts
using Moore-Penrose pseudo-inverse. The bias-reduced shift
v v+1
estimate is then computed as: v = M−1 v00 .
I2(x) I1(x) = I2(x – v) I2(x + 1)
4.2.1. Weighted least-squares estimation
There are two problems with the estimation of the bias matrix
ˆ
in (18). First, the resulting second moment matrix A = A.M
First shift estimation Second shift estimation
is not symmetric. Second, all shift measurements vij are
v0 = kv v1 = k(v + 1) treated equally despite their varying levels of accuracy. Be-
cause GBSE estimates small shift more accurately, each shift
Fig. 1. Bidirectional 1-D shift bias factor estimation. estimation should be weighted by its inverse magnitude.
ˆ
Pre-multiplying A to (15-17) yields:
I2(x,y) I2(x+1,y)
ˆ
A.(v10 − v00 ) ≈ A.[1 0]T = [a b]T (19)
v00 v10
ˆ
A.(v01 − v00 ) ≈ A.[0 1]T = [b c]T (20)
ˆ
A.(v11 − v00 ) ≈ A.[1 1]T = [a + b b + c]T (21)
I1(x,y) = I2(x–vx, y–vy)
If the left-hand side of the above equations is assigned to
v10 v11
[p1 q1 ]T , [p2 q2 ]T and [p3 q3 ]T , with associated weights
−2 −2
w1 = |v10 | + |v00 | , w2 = |v01 | + |v00 | and
−2 1
w3 = |v11 | + |v00 | , respectively , a weighted least-
I2(x,y+1) I2(x+1,y+1) squares solution of [a b c]T aims to minimize:
= w1 (a − p1 )2 + w1 (b − q1 )2 + w2 (b − p2 )2 (22)
Fig. 2. Bidirectional 2-D shift bias matrix estimation.
+w2 (c − q2 ) + w3 (a + b − p3 ) + w3 (b + c − q3 )2
2 2
4.2. 2-D bias matrix correction Setting the derivatives of against a, b and c to zeros yields:
   
Similar to the 1-D case, the 2-D bias matrix can be derived w1 + w3 w3 0 a
from multiple subpixel shift estimates between I1 and shifted  w3 w1 + w2 + 2w3 w3 . b  =
versions of I2 . As illustrated in Figure 2, four following sub- 0 w3 w2 + w3 c
 
pixel shift estimates can be computed for the case of v00 < 0: w1 p1 + w3 p3
 w1 q1 + w2 p2 + w3 (p3 + q3 )  , (23)
v00 = findshift(I1 , I2 (x, y)) = Mv (11)
w2 q2 + w3 q3
T
v10 = findshift(I1 , I2 (x+1, y)) = M(v+[1 0] ) (12)
from which a solution [a b c]T can be derived.
v01 = findshift(I1 , I2 (x, y+1)) = M(v+[0 1]T ) (13)
v11 = findshift(I1 , I2 (x+1, y+1))=M(v+[1 1]T ) 5. EXPERIMENT
(14)
In this section, the performance of the bias-corrected GBSE in
Subtracting (11) from the above equations eliminates the
section 4.2.1 is compared against those of other GBSE’s. Fif-
unknown shift v:
teen shift estimates are computed from a 90 × 90 × 16 noisy
v10 − v00 = M.[1 0]T (15) and highly aliased image sequence2 whose first frame is the
T reference. Since the ground-truth motion vectors are avail-
v01 − v00 = M.[0 1] (16)
able, the bias and precision of each estimator can be computed
T
v11 − v00 = M.[1 1] (17) from the average and standard deviation of the estimation er-
rors. Figure 3 plots these results in the form of a rectangle
It is clear that the bias matrix M can be resolved from any for each shift estimator. Each rectangle is centered at the av-
two of the equations above. As a result, a minimum of three eraged error [∆v x , ∆v y ] with its dimensions being twice the
subpixel shift estimates are required to estimate M. However, standard deviation of the errors [std(∆vx ), std(∆vy )]. By
if all four estimates (11-14) are available, M can be estimated definition, an estimator is less biased if its corresponding rect-
by least-squares from: angle is closer to the origin. The estimator is also precise if
1 0 1 the area of the rectangle is small.
M. = v10−v00 v01−v00 v11−v00
0 1 1 1 |v| and v denote absolute value and norm of vector v, respectively
(18) 2 EIA sequence available at http://www.soe.ucsc.edu/∼milanfar/DataSets/

4. 0.15
nal GBSE is nonlinear considering the truncation error of the
GBSE (1 iteration of [1]) Taylor series approximation [5]. In these cases, more estima-
CLS (appendix E of [9])
0.1 bidirectional (this paper)
tion points are needed to fit a higher-order bias model.
iterative GBSE [1] Finally, the bidirectional bias correction scheme can be
applied to the registration of images under more complex
∆ty ± std(∆ty)
0.05
transformations models such as rigid, affine, projective trans-
formations and optical flow.
0
7. REFERENCES
−0.05
[1] B.D. Lucas and T. Kanade, “An iterative image registra-
−0.1
−0.2 −0.15 −0.1 −0.05 0 0.05 tion technique with an application to stereo vision,” in
∆t ± std(∆t ) Proc. of DARPA’81, 1981, pp. 121–130.
x x
[2] J. Kearney, W. Thompson, and D. Boley, “Optical flow
Fig. 3. Performance of different shift estimators.
estimation: an error analysis of gradient-based methods
with local optimization,” PAMI, vol. 9, no. 2, pp. 229–
According to figure 3, the bidirectional bias-corrected 244, 1987.
GBSE is the least biased as well as most precise shift es- [3] J.W. Brandt, “Analysis of bias in gradient-based optical
timator. The Corrected Least Squares (CLS) estimator [9] flow estimation,” in Proc. IEEE Asilomar Conf. Signals
has a similar level of precision with the original GBSE in Syst. Comput., 1995, pp. 721–725.
equation (4) because both are single-iteration GBSE. The
CLS method is slightly less biased than GBSE due to a noise- [4] M.D. Robinson and P. Milanfar, “Fundamental perfor-
corrected second moment matrix as described in section 3.1. mance limits in image registration,” IEEE Trans. on IP,
Surprisingly, the iterative GBSE [5] performs worst with the vol. 13, no. 9, pp. 1185–1199, 2004.
largest bias and standard deviation. This is most likely due to [5] T.Q. Pham, M. Bezuijen, L.J. van Vliet, K. Schutte, and
a high level of aliasing in the test sequence, which prevents a C.L. Luengo Hendriks, “Performance of optimal reg-
good motion compensation during the iterative estimation. istration estimators,” in Visual Information Processing
XIV, 2005, vol. 5817 of SPIE, pp. 133–144.
6. CONCLUSION AND FUTURE RESEARCH [6] E.P. Simoncelli, “Design of multi-dimensional deriva-
tives filters,” in Proc. ICIP, 1994.
We have presented a general scheme to correct for the bias
of an n−parameter gradient-based registration estimator us- [7] M. Elad, P. Teo, and Y. Hel-Or, “On the design of opti-
ing at least n+1 estimations of the registration between trans- mal filters for gradient-based motion estimation,” JMIV,
formed versions of the input images. Before each estimation, vol. 23, no. 3, pp. 345–265, 2005.
the second image is pre-transformed by a set of predefined [8] M.D. Robinson and P. Milanfar, “Bias minimizing filter
transformation parameters. The predefined parameters are design for gradient-based image registration,” SP:IC,
chosen such that for each registration parameter, there are at vol. 20, no. 6, pp. 554–568, 2005.
least one estimate above and one estimate below the true value
(hence bidirectional). For 1D shift estimation, a coordinate [9] H. Ji and C. Fermueller, “Noise causes slant underesti-
shift of the second image by one pixel is required. For 2D mation in stereo and motion,” Vision Research, vol. 46,
shift estimation, two coordinate shifts of the second image, pp. 3105–3120, 2006.
each by one pixel along a sampling direction, are sufficient. [10] F. Scarano, “On the stability of iterative PIV im-
For future work, the bidirectional bias correction scheme age interrogation methods,” in Proc. of 12th Interna-
can be applied to other shift estimation methods, as long as tional Symposium on Applications of Laser Techniques
the bias model is known. Cross-correlation, for example, may to Fluid Mechanics, 2004.
have a linear bias model because its iterative version con-
verges monotonically [10]. Phase correlation may also pro- [11] M. Shimizu and M. Okutomi, “Precise subpixel estima-
duce a biased estimate because of mismatched border wrap tion on area-based matching,” Systems and Computers
and non-overlapping areas between the input images. in Japan, vol. 33, no. 7, pp. 1–10, 2002.
Furthermore, the estimation bias should not be restricted [12] V. Argyriou and T. Vlachos, “Estimation of sub-pixel
to a linear model. Bias of a cross-correlation peak due to motion using gradient cross-correlation,” Electronics
parabola fitting, for example, is nonlinear [11], so does the Letters, vol. 39, no. 13, pp. 980–982, 2003.
bias of gradient correlation [12]. Even the bias of the origi-