1. Photo Synthesis: Image Stitching and Registration
Keenan Johnstone, Travis Gray, Dr. Mark Eramian, Dr. Kevin Stanley, Dr. Ian Stavness
Problem
Agricultural researchers are beginning to use aerial drones to take multi-spectral
images (images that focus on a narrow spectrum of light) of their crops. However,
doing so results in some unusual challenges.
• The cameras that take images at the different spectrums usually have a
different lense for each specrum.
• The different cameras also don't all capture at the same time.
Together, this means that the images it take will all be slightly misaligned.
Registration vs. Stitching
SolutionTo compensate for this misalignment, we needed to register the images
across the spectrums. That is, we needed to line up the different images so
that the plants are in the same place across all images.
• Fourier transforms were used to take the multiple images and calculate
the optimal phase shift to align the images.
• The results were aligned, but there were several cases of blurred edges
around certain parts of the image.
• Due to this error, we used a more accurate method in which keypoints
in the image were found and matched between the images.
• This allowed us to create a homography between spectrums
(A 3x3 matrix that is an instruction set on how to scale, rotate, and
warp a set of pixels).
• With these matrices, we found much better results, where images were
aligned with subpixel accuracy.
P
2
IRC
This multispectral camera is actually made up of 5 separate cameras, each of which take a shot at a slightly
different time. Red, Green, Blue, Near-Infared, and Rededge.
Images taken in the different spectrums:
Blue, Green, Red, Near Infared, and Rededge respectively
The resulting image after using a homography on the blue and red specturms to align with the
green scpectrum. As you can see from the outside edges, the blue spectrum
had to be shifted to the right by quite a bit, and the red had to be shifted up.
Another application of this key point matching is the ability to stitch together images.
Drones have to take multiple, separate images of a field to be able to get the entire
thing in decent quality. This means that many smaller images will need to be stitched
together to create a large composite image of the entire field.
This can be done by the same keypoint matching as before, but now applied to find
where the overlap of smaller images are, and combine those points to create a bigger
image from the multiple.
These two images show where the matched key points exist in images where some overlap occer.
The bottom of the left image also appears in the top of the right image.
These two images, now stitched together.
The homography relating the right image to the left image.
The result of stitching multiple images together
Constructing a system that can register and stitch drone images has
streamlined the process of plant image analysis by reducing human input.
This is important because all image analysis methods on plants require
aligned colour channels and a complete view of the region of interest.
Future improvements include accounting for discolouration during
registration, and applying orthorectification techniques to the drone
images before stitching.
Conclusion