The document discusses an approach to upscaling video using a back iteration algorithm. It begins with an abstract describing how the back iteration algorithm is similar to back-projection algorithms used in tomography. It then discusses how the back iteration algorithm is implemented iteratively on individual video frames to upscale the video. Key aspects of the algorithm include motion estimation, intensity calculation, and registering frames at sub-pixel accuracy. The document provides details on the mathematical model and implementation of the back iteration algorithm for video upscaling. It presents results of applying the algorithm and concludes with discussing opportunities for future improvements.

1. Approach to upscale video for super resolution
using back iteration algorithm
Tanmay Chatterjee Ravi Kumar Mahua Bhattacharya
IIITM Gwalior IIITM Gwalior IIITM Gwalior
tanmaychatterjee@outlook.com ravikumar851@gmail.com mb@iiitm.ac.in
Abstract: The back iteration algorithm developed based
on the image registration method has been used to
improve the gray scale and color images and further
improved for video upscaling. Back iteration algorithm
is similar to back-projection developed in tornography.
Execution of iterative calculation to expand the picture
determination, together with a system for picture
enlistment with sub-pixel correctness, is exhibited in this
paper. In this way this calculation is further utilized for
handling casing by edge to enhance and again arrange
once more to feature. N-low resolution images are taken
as the input form, followed by denoise and use motion
estimation and intensity calculation for improving the
pixels and producing the N or a single high resolution
image. MATLAB based implementation has been done
for the same.
Keywords - Back iteration algorithm, upscaling, MATAB,
image registration
I INTRODUCTION
Picture determination is controlled by numerous
variables like the optics, the thickness and the spatial
reaction of the finder components. Expanding the
determination of a picture includes utilizing the data or
the subtle elements accessible from one or a lot of
people low determination pictures and attempt to
enhance the perspective proportion or the amount of
pixels utilizing that information. SUPER-
RESOLUTION (SR) is a thoroughly investigated
subject in image processing where a majority of the
work is focused on the creation of one high-resolution
(HR) still image from low-resolution (LR) images. In
our today’s technologies, we are coming up with
devices supporting high resolution displays and hence,
the goal of this project is to propose an optimized
algorithm which could take n number of inputs in the
form of low resolution images and de-noise it and
thereby replace the pixels in the input using motion
estimation and intensity calculation. Many of the
video up-scaling methods involve the use of
spatiotemporal methodology to super resolute the
images combining them thereby to amplify the video.
As indicated by Wikipedia, Superresolution is a class
of systems that improve the determination of an
imaging framework. In some SR systems-termed
optical SR-the diffraction furthest reaches of
frameworks is transcended, while in others-
geometrical SR-the determination of computerized
imaging sensors is upgraded. A feature scaler is a
framework which changes over feature indicators
starting with one showcase determination then onto
the next; ordinarily, scalers are utilized to change over
a sign from a more level determination, (for example,
480pstandard definition) to a higher determination,
(for example, 1080ihigh definition), a procedure
known as "upconversion" or "upscaling" (by
difference, changing over from high to low
determination is known as "downconversion" or
"downscaling"). Video scalers are regularly found
inside buyer gadgets, for example, Tvs, feature
diversion comforts, and DVD or Blu-beam circle
players, however can likewise be found in other AV
supplies, (for example, feature altering and TV
television gear). Feature scalers can additionally be
totally separate gadgets, frequently giving basic
feature exchanging abilities. These units are regularly
found as a feature of home theater or anticipated
presentation frameworks. They are regularly joined
with other feature transforming gadgets or calculations
to make a feature processor that enhances the clear
meaning of feature indicators.
II BACKGROUND AND PREVIOUS WORK
A typical methodology for spatial picture and feature
upscaling is to utilize direct channels with minimal
backing, for example, from the group of cubic
channels [11]. In this part, our center is on multi-

2. outline strategies, which empower determination
upgrade in spatial upscaling, and permit transient edge
introduction (casing rate upconversion). In spite of the
fact that numerous calculations have been proposed
for picture and feature addition, spatial upscaling and
casing insertion (transient upscaling) are by and large
treated independently. The three sorts of required
upscalings are deinterlacing (DI), which is making the
never recorded lines in intertwined feature, otherwise
called joined to dynamic change, feature super
determination (VSR), which is expanding the spatial
determination of each one edge in the feature
throughput, and transient super determination (TSR),
which changes the casing rate of an arrangement by
making completely new casings at the fleeting
positions where they are required. To recoup HR
pictures, additional data must be included [2].
Typically two sorts of additional data could be joined
together: Prior data on the sort of expected results and
additional low determination (LR) information
pictures. The previous identifies with introduction,
while the last constitutes the principle thought behind
SR that is taking numerous LR pictures and circuits
them into a HR picture. These different sources may
start from various perspectives or be progressive edges
from a picture grouping. It is intriguing to note that
supplying a few LR sees for point of interest upgrade
additionally happens in the human visual system
(HVS). The information determination on the retina is
not as high as the determination saw by the HVS after
it has transformed the data. The eye continually makes
little, fast eye developments (Rems), supplying the
HVS with various LR sees from which it can build a
more point by point HR view. It has been bantered for
quite a while whether these Rems accomplish more
than simply prevent the visual information from
blurring, yet it has not been demonstrated up to this
point by barbe [3] that the Rems are critical to the SR
impact of the HVS.
A late work by Brox et al. [10] proposed a feature-to-
feature super determination calculation utilizing a 9
segment sifting strategy, in which nearby picture
structures are arranged into vertical, flat, and inclining
edges, compositions, and even regions by vector
quantization [6] (including logged off taking in), and
set up a suitable channel for each one structure class
already. At that point, with the allotment channel, they
interject the missing pixels and recoup a high
determination feature outline. An alternate late
approach in [8] utilizes a versatile Wiener channel and
has a low computational intricacy when utilizing a
worldwide translational movement model. This is
average for some routine super-determination
techniques, which subsequently regularly don't think
about more unpredictable movement. The three types
of needed upscalings are deinterlacing (DI), which is
creating the never recorded lines in interlaced video,
also known as interlaced to progressive conversion,
video super resolution (VSR), which is increasing the
spatial resolution of each frame in the video
throughput, and temporal super resolution (TSR),
which changes the frame rate of a sequence by creating
fully new frames at the temporal positions where they
are needed. One of the most popular algorithm for
image upscaling is the back iteration algorithm for
super resolution which got introduced in 1991 [7].
This algorithm has been modified accordingly to
implement it for the video upscaling by iterating the
algorithm for repetitive iterations and resolving each
frame. The two most common methods adapted for
video upscaling are spatiotemporal self-similarity and
motion estimation and intensity calculation. On a
thorough read, it can be analyzed that the two methods
are not applicable on a greater scale since the video
upscaling can be categorized into three categories.
Deinterlacing (DI), Video super-resolution (VSR) and
temporal super-resolution (TSR). Hence the proposed
solution involves using the variational methods for
super-resolution.
III BACK ITERATION ALGORITHM
Improving the quality of image using back iteration
involves the motion estimation as well as the looking
into the relative scene locations senses by each pixel
in the observed images. This information is available
in image regions where local deformation can be
described by some parametric function. In
tomography, images are reconstructed from their
projections in many directions. The imaging process,
yielding and observed monochrome image sequence
{gk} is modeled by
gk(m,n) = σ(h(f(x,y)) + η(x,y))
Where
 gk is the kth observed image frame,
 f is the original scene,
 h is a blurring operator,
 ηk is an additive noise term,
 σk is a nonlinear function which digitizes
and decimates the image into pixels and
quantizes the resulting pixel values from

3. intensities into gray levels. σk includes the
displacement of the kth frame.
 (x,y) is the center of the receptive field (in f)
of the detector whose output is gk(m,n).
The scene location (x, y) of the center of the
receptive field for the observed location (m, n) is
computed by
x=xk
o
+ sxm cosθk – syn sinθk
y=yk
o
+ sxm cosθk + syn sinθk
Where
 (xk
o
, yk
o
) is the translation of the kth
frame,
 Θk is the rotation of the kth frame about
the origin,
 sx and sy are the sampling rates in the x
and y directions, respectively.
The Back iteration algorithm mentioned here
attempts to reconstruct a higher resolution image
fˊ, which approximates f as accurately as possible.
It is assumed that the acceleration of the camera
while imaging a single frame is negligible. One of
the important factors for back iteration image
registration is the imaging process. These are the
estimation of relative displacements of the input
images at subpixel accuracy, as well as the blur in
the imaging process. As images are recorded in
discrete time intervals, the displacements between
image vectors may not be sufficiently small for
the motion recovery method. So, the process
being followed for iteration for the two images g1
and g2 is as:
1. For start, assumption is being made that there
is no motion between the frames.
2. Then all the possible motion estimations are
being made and the delta margin is added to
the existing motion estimate.
3. Overlap the frame g2 over g1 and as Step 2 is
repeated g2 gets closer to g1, the error margin
reduces and the motion estimators become
more accurate.
The super-resolution algorithm starts with an
initial guess f(0)
for the high resolution image, then
the imaging process is simulated to obtain a set of
low-resolution images {gk
(0)
} corresponding to
the observed input images {gk}. Now if f(0)
were
the correct high-resolution image then the
simulated images {gk
(0)
} should be identical to the
observed images {gk}. The difference in the
images {gk - gk
(0)
} is then computed, and used to
improve the initial guess by “back projecting”
each vale in the different images onto its receptive
field in f(0)
.
The iterative update scheme to estimate the high-
resolution image f is expressed by
f (n+1)
(x) = f(n)
(x) + ∑yєUkYk,x (gk(y) – gk
(n)
(y))
[(hxy
PB
)2
/ c∑ yˊєUkYk,x hxyˊ
BP
]
Where
 f(n)
is the image equation after nth
iteration,
 Yk,x denotes the set {y є gk | y is
influenced by x},
 c is a (constant) normalizing factor,
 hxy
PB
= hxy
BP
(x-zy) BP is back projection.
IV ALGORITHM IMPLEMENTATION FOR
VIDEO UPSCALING
Here are some of the various methods involved in our
implementation through which we can enhance a set
of grayscale LR images to reproduce a single HR
grayscale image.
A. S&A
This method applies the Shift-and-Add (S&A)
method. The pixels that are not defined in the S&A
step (under-determined case) is estimated by a linear
interpolation step. Later a pop-up menu is generated to
set the Lucy deconvolution methods (MATLAB
subroutines are called to perform these deconvolution
methods). User has the choice of setting the number of
iteration for Lucy and Blind Lucy methods, and the
Noise to Signal ( 1/SNR) value for all these methods.
User can modify the PSF kernel either by typing a
numerical matrix or by providing a MATLAB
function that produces the desired blurring kernel (e.g.
the "fspecial" command).
B. Static Multi-frame Demosaicing and color Super-
Resolution
This method produces a single color HR image from a
sequence of color LR frames. To use this functionality,
the user clicks on the Color SR radio button before
loading the data. This guarantees that the input data
will be read as a color sequence and also enables some
of the relevant push-buttons. After choosing an

4. appropriate motion estimation method, by clicking on
the Demosaic/Color SR button, a pop-up window
appears. The parameters in this window are related to
the parameters in the above equations of Reference
[8]. Bilateral. Reg. Spatial Decay Coefficients,
Bilateral. Regularization Kernel Size, Luminance
Regulation. Factor, Chrominance Regulation. Factor,
Orientation Regulation. Factor and Step size.
C. Grayscale Video Simulation
This option produces a sequence of down-sampled,
noisy, blurred grayscale frames from one single HR
image. That is, to produce each LR image, a section of
this HR image is selected, then blurred by a user
defined PSF, down-sampled by a factor (r) defined by
user and some additive noise is added. To construct the
next LR frame, the section of the HR frame used for
making the previous image is shifted and the same
degradation process is applied to this shifted image.
The user first loads a single grayscale image. Then by
clicking on the BW Video Simulator button a pop-up
window appears. Resolution Factor edit-box defines
the down-sampling factor (r) in each direction.
Number of Output Frames edit-box defines how many
LR frames are desired to be generated from this
sequence. Signal to Noise Ratio edit-box defines the
amount of additive Gaussian noise (in units of dB) that
would be added to the LR sequence.
Output Width and Output Height edit-boxes define the
width and height of each generated LR frames
(referred to the width and height of the resulting LR
frames by the symbols W" and H" respectively,
throughout this document). Note that the LR frames
are produced by moving a shifting window (of width r
X W and height r X H) over the input image. Selecting
smaller values for W and H results in a smaller window
with more space to move on the original HR frame,
increasing the maximum number of realizable LR
frames. The user has the option of choosing between
two related motion generating mechanisms. If the
Motion by Hand check box is checked, then a window
with the image of HR input frame pop-ups. By
consecutive clicking on this image, the user defines a
trace (path) representing the motion of a camera on
this image.
V RESULTS AND OUTPUT
Figure 1 the user interface of the upscaling resolution
implementation
Figure 2 Upscaled images using iteration algorithm
Figure 3 Video frame to frame upscaling sample

5. Figure 4 Sample of upscaling a particular frame after 1st
, 2nd
and 3rd
iteration
VI CONCLUSION AND FUTURE
IMPROVEMENTS
The variational methods for deinterlacing, video super
resolution and temporal super resolution presented in
this thesis all produce high quality results and are
ready for use in software or hardware products. There
are however some remaining problems, that can be
solved in the future. We devote separate sections to
each problem, but first a short introduction of each of
them.
 The Modelling Problem: Currently we use
total variation as the probability function in
all terms of our framework and although it is
a good model of image sequences we should
be able to do better.
 The Flow Problem: We have found that the
use of state of the art variation optical flow
methods helps produce high quality motion
compensated upscaling results, but the flow
computation is in our opinion the element of
variational upscaling methods with the
greatest potential for improvements.
 Speedup: It is no secret that our current
implementations are slow and to achieve
real-time running speedups are required.
 Integrated Upscaling Systems: We now do
upscaling in three separate steps even though
all three build on the same basic technology.
When two or all three steps are applied to a
video throughput integrations are possible.
 Streamed Processing: When taking our
methods out in the real world, we are no
longer processing short test sequences but
have to process continuous streams of video
throughput.
 Benchmarking: To give non-reputable
documentation of the performance of any
method for video processing, one needs to
benchmark against what is otherwise
considered state of the art within a field.
REFERNCES

6. [1] G. Aubert and P. Kornprobst, “Mathematical
Problems in Image Processing: Partial Differential
Equations and the Calculus of Variations”, 2nd ed.,
ser. Applied Mathematical Sciences. Springer-Verlag,
2006, vol. 147.
[2] S. Baker and T. Kanade, “Limits on Super-
Resolution and How to Break Them." IEEE Trans. on
Pattern Analysis and Machine Intelligence, vol. 24,
no. 9, pp. 1167-1183, 2002.
[3] D. F. Barbe, “Charge-Coupled Devices" in Topics
in Applied Physics, D. F. Barbe, Ed. Springer, 1980.
[4] E. Bellers and G. de Haan, “De-interlacing. A Key
Technology for Scan Rate Conversion”, Elsevier
Sciences Publishers, Amsterdam, 2010.
[5] M. Biswas, S. Kumar, and T. Nguyen,
“Performance Analysis of Motion- Compensated De-
Interlacing Systems," IEEE Transactions on Image
Processing, vol. 15, no. 9, pp. 2596-2609, 2006.
[6] M. Biswas and T. Nguyen, “A Novel De-
Interlacing Technique Based on Phase Plane
Correlation Motion Estimation," International
Symposium on Circuits and Systems, ISCAS, vol. 2, pp.
604-607, 2003.
[7] D. Bordwell and K. Thompson, “Film History: An
Introduction” McGraw-Hill, 1991.
[8] S. Borman and R. Stevenson, “Spatial Resolution
Enhancement of Low- Resolution Image Sequences:
A Comprehensive Review with Directions for Future
Research", Laboratory for Image and Sequence
Analysis (LISA), University of Notre Dame, Tech.
Rep., July 2008.
[9] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert,
“High Accuracy Optical Flow Estimation Based on a
Theory for Warping," in Proceedings of the 8th
European Conference on Computer Vision, T. Pajdla
and J. Matas, Eds., vol. 4. Prague, Czech Republic:
Springer {Verlag, 2004, pp. 25-36.
[10] T. Brox, J. Weickert, B. Burgeth, and P. Mreazek,
“Nonlinear structure tensors," Image and Vision
Computing, vol. 24, no. 1, pp. 41-55, Jan. 2006.
[11] A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger,
and C. SchnÄorr, “Variational Optic Flow
Computation in Real-Time," IEEE Trans. on Image
Processing, vol. 14, no. 5, pp. 608-615, 2005