SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching

Nagoya University
Parallel & Distributed Systems Lab.
パターン・映像情報処理特論 (2016.5.6)
SkyStitch: a Cooperative Multi-UAV-based Real-time
Video Surveillance System with Stitching
Yuki KITSUKAWA
yuki@ertl.jp
Edahiro & Kato Lab.
Graduate School of Information Science
Nagoya University
1

Nagoya University
About Paper
p Title
SkyStitch: a Cooperative Multi-UAV-based Real-time Video
Surveillance System with Stitching
p Authors
Xiangyun Meng, Wei Wang, Ben Leong
(National University of Singapore)
p Conference
Proceedings of the 23rd ACM international conference on Multimedia,
2015
URL - https://www.comp.nus.edu.sg/~bleong/publications/acmmm15-skystitch.pdf
2

Nagoya University
Contents
1. Introduction
2. Related Work
3. Real-time Video Stitching
I. Offloading Feature Extraction
II. Exploiting Flight Status Information
III. Optimizing RANSAC
IV. Optimizing Video Stitching Quality
V. Multiple Video Sources
4. System Implementation
5. Evaluation
I. Stitching Time
II. Stitching Quality
6. Conclusion
3

Nagoya University
Contents
1. Introduction
2. Related Work
5. Evaluation
I. Stitching Time
6. Conclusion
4

Nagoya University
Introduction
Applications using UAV-based surveillance system
ü firefighting
ü search and rescue
ü Border monitoring
To cover large target area…
Multiple UAVs are used to capture videos and put them together
(stitch) consistently.
5
SkyStitch: a Cooperative Multi-UAV-based Real-time
Video Surveillance System with Stitching
Xiangyun Meng, Wei Wang and Ben Leong
School of Computing
National University of Singapore
idmmeng@nus.edu.sg, weiwang@comp.nus.edu.sg, benleong@comp.nus.edu.sg
ABSTRACT
Recent advances in unmanned aerial vehicle (UAV) technologies
have made it possible to deploy an aerial video surveillance system
to provide an unprecedented aerial perspective for ground moni-
toring in real time. Multiple UAVs would be required to cover a
large target area, and it is difﬁcult for users to visualize the over-
all situation if they were to receive multiple disjoint video streams.
To address this problem, we designed and implemented SkyStitch,
a multiple-UAV video surveillance system that provides a single
and panoramic video stream to its users by stitching together mul-
tiple aerial video streams. SkyStitch addresses two key design chal-
lenges: (i) the high computational cost of stitching and (ii) the difﬁ-
culty of ensuring good stitching quality under dynamic conditions.
To improve the speed and quality of video stitching, we incorporate
several practical techniques like distributed feature extraction to re-
Ground station
Video streaming
Video capturing
Video capturing
Video stitching
Figure 1: Illustration of SkyStitch. Each quadcopter transmits a live
video stream to the ground station, which stitches them together to
provide a single composite view of the target area.
For example, a UAV could be deployed to hover over a bush-
Illustration of SkyStitch
UAV – Unmanned Aerial Vehicle
http://www.robotshop.com/en/550mm-rtf-quadcopter-uav.html

Nagoya University
Introduction
SkyStitch - a quadcopter-based HD video stitching system for surveillance
Challenges
1. computational cost
2. video stitching (distortion, frame drops, moving objects…)
Solutions
1. Reduce workload of the ground station by offloading pre-
processing to the UAVs
2. Exploit flight information to speed up feature matching
3. GPU-based RANSAC implementation
4. Utilize optical flow and Kalman filtering
5. Closing loops when stitching multiple video sources
6

Nagoya University
Contents
1. Introduction
2. Related Work
5. Evaluation
I. Stitching Time
6. Conclusion
7

Nagoya University
Contents
1. Introduction
2. Related Work
5. Evaluation
I. Stitching Time
6. Conclusion
8

Nagoya University
Real-time Video Stitching
Image Stitching – finding the transformation (H: homography matrix)
between two images
Estimate H using feature-based method
i. feature extraction
ii. feature matching
iii. Robust homography estimation
9
𝑥"
𝑦"
1 &
~H ) 𝑥 𝑦 1 &
mputational
ess into sev-
Vs. In this
g the work-
l bottleneck
ation is that
mation like
ht controller
perspective
and optical
term stabil-
ﬁlter-based
H
2. Feature matching 4. Compositing3. Estimation of H1. Feature extraction
Image alignment
Figure 2: Overview of image stitching.
Overview of image stitching

Nagoya University
Offloading Feature Extraction
Traditional stitching system – Transmits only video frames
Feature extraction is performed on ground station.
It would become bottleneck when the # of UAVs increase.
SkyStitch – Transmits video frames and corresponding features
Feature extraction is performed on UAVs*.
* 1,000 features extraction from 1280×1024 on NVIDIA Jetson TK1(192 CUDA cores)
- less than 35 ms
10

Nagoya University
Exploiting Flight Status Information
Difficulty in achieving fast and accurate stitching
1. perspective distortion due to wind turbulence
→ Orthorectification of Perspective Distortion
2. time-consuming feature matching between two images
ex) two images with 1,000 features - 1,000,000 combination!
→ Search Region Reduction for Feature Matching
11

Nagoya University
Orthorectification of Perspective Distortion
Orthorectification – Transform the image so that it looks as if the camera is pointed vertically
downwards.
ü Transformation matrix – downwards camera direction to the tilted direction
R =
1 0 0
0 cos 𝜃1 sin 𝜃1
0 −sin 𝜃1 cos 𝜃1
cos 𝜃5 0 −sin 𝜃5
0 1 0
sin 𝜃5 0 cos 𝜃5
cos 𝜃6 sin 𝜃6 0
−sin 𝜃6 cos 𝜃6 0
0 0 1
ü Orthorectification matrix
M = KBR:;B:;K:;
K: transform the pixel into world coodinates
(camera intrinsic matrix)
B: transform camera into flight controller
Orthorectified coordinates can be expressed by
𝑥"
𝑦"
𝑧′ &
~M 𝑥 𝑦 1 &
12
Pitch
Yaw
Roll
Heading
Figure 3: Flight attitude of quadcopter:
roll, pitch and yaw.
N
Flight attitude of quadcopter: roll, pitch and yaw
Roll Pitch Yaw
obtain from flight controller

Nagoya University
Search Region Reduction for Feature Estimation
To reduce feature matching time, GPS location is exploited to find the mapping of pixel
locations between two images.
1. transform the pixel locations from pixel coordinate to a world coordinate system
2. Two image in world coordinate systems can be correlated
using the difference in the GPS locations.
13
P
r
Feature point transformed from Image 1
Bins to be searched for the matched feature
Image 2
Figure 5: Search region reduction for fea-
ture matching.
Search region reduction for
feature matching
ch
opter:
BA
P P
A B
P
Actual scenario
North
Pixel coordinates (after orthorectification)
(a) Mapping a pixel location from one image to another.
CameraCamera
GPS difference
P
(b) Merging coordinate systems.
Figure 4: Overlap determination using ﬂight status information.
, yw, zw) can be obtained by, To further improve RANSAC performance when the number of
Overlap determination using flight status information
Pixel coordinate (in the picture)
𝑥"
, 𝑦"
, 𝑧′
World coordinate (on the ground)
𝑥?, 𝑦?, 𝑧? = 𝑠
𝑥′
𝑧′
−
𝑊
2
, 𝑠
𝑦′
𝑧′
−
𝐻
2
, ℎ
𝑠 = 2ℎ tan
𝜃G
2
𝑊H
s: # of meters representedby the side of a pixel
W×H: image resolution (in pixel)
𝜃G: angle of horizontal field of view in the camera
h: quadcopter altitude

Nagoya University
Optimizing RANSAC
RANSAC (Random Sample Consensus) – method of removing outliers
i. Select four matched features to compute homography matrix H
ii. Apply estimated H on all the matched features
iii. Repeat i-ii for N iterations and find the one with the most inliers
Large N improves H but also increases computation time
GPU-accelerated RANSAC
Ø Gauss-Jordan elimination instead of SVD (Singular Value Decomposition)
Ø Maximize parallelism and minimize IO
Ø Improve score calculation (use the distribution of inliers)
14
P
r
point transformed from Image 1
e searched for the matched feature
2
earch region reduction for fea-
ng.
H
H
H
H
H
2 3 41
2 3 41
2 3 41
2 3 41
2 3 41
...
...
...
...
qp
qp
qp
qp
qp
qp
qp
qp
qp
Uploaded
from CPU
Uploaded
from CPU
...
Reprojection
Error
Pair
Pair
ScoresInlier maskAll matchescorrespondences
4−point
homographies
Candidate
Gauss Jordan
Elimination Pair
...
Pair
DownloadedDownloaded
from GPU from GPU
...
...
...
i1, j1
ik, jk
i1, j1
ik, jk
Hn−1
Hn−1
H0
H0
Figure 6: RANSAC GPU Pipeline
Camera 1 Camera 2 Optical FlowStitching
H1 H2
RANSAC GPU Pipeline

Nagoya University
Optimizing Video Stitching Quality
Causes of stitching failures
ü small overlapping area between two images
ü concentration of inliers selected by RANSAC in a small region
How to detect ill-formed H and prevent them from affecting the perceptual experience?
Ø # of inliers produced by H
If insufficient, discard the current frame
Ø the angle between the two image planes - 𝜃
H = R + tn&
(R: 3×3 rotation matrix between two images)
cos 𝜃 = 𝑟KK (angle around z-axis)
If 𝜃 > 5°, H is erronous
15
P
r
Feature point transformed from Image 1
Bins to be searched for the matched feature
Image 2
Figure 5: Search region reduction for fea-
ture matching.
H
H
H
H
H
2 3 41
2 3 41
2 3 41
2 3 41
2 3 41
...
Uploaded
from CPU
...
correspondences
4−point
homographies
Candidate
Gauss Jordan
Elimination
Figu
(a) Inaccurate rotation. Notice the
broken and bent touchline of the
football ﬁeld, below the running
track.
(b) Perspective distortion. Notice
that the left part of the image is
erroneously warped and cannot be
perfectly aligned with the right im-
age.
Figure 7: Examples of perceptual distortion. Each blue cross rep-
resents a feature point.
When there are sufﬁcient inliers, we also check whether H badly
warps the images and introduces perspective distortion. Let θ be the
Fram
Fram
Cam
Figur
optica
forme
was n
ing to
3.4.
Example of perceptual distortion

Nagoya University
Optimizing Video Stitching Quality – Recovering from Failed Stitching
Prediction of H by utilizing the optical flow of adjacent frames to reduce
frame drops due to ill-formed H
HOP; = MQ,OP;FQ,OP;MQ,O
:; HOM;,OF;,OP;
:; M;,OP;
:;
F: 3×3 homography matrix of optical flow
M: orthorectification matrix
Hn : transformation matrix of the previous stitching
16
ge 1
d feature
ion for fea-
H
H
H
H
H
2 3 41
2 3 41
2 3 41
2 3 41
2 3 41
...
...
...
...
qp
qp
qp
qp
qp
qp
qp
qp
qp
Uploaded
from CPU
Uploaded
from CPU...
Reprojection
Error
Pair
Pair
4−point
homographies
Candidate
Gauss Jordan
Elimination Pair
...
Pair
from GPU from GPU
...
...
...
i1, j1
ik, jk
i1, j1
ik, jk
Hn−1
Hn−1
H0
H0
(b) Perspective distortion. Notice
that the left part of the image is
erroneously warped and cannot be
perfectly aligned with the right im-
age.
ual distortion. Each blue cross rep-
Frame Frame
Frame Frame
Camera 1 Camera 2
F1,n+1 F2,n+1
Hn+1
Hn
n n
n+1 n+1
Figure 8: Predict Hn+1 using
optical ﬂow when Hn+1 is ill-
formed.
Optical FlowStitching
Translation
Matched
features
Solve
Measurement Prediction
MEKF
H1 H2
n2t2R2R1t1n1
R′
H′
t′
Figure 9: Fusing information
from both stitching and optical
ﬂow.
Predict Hn+1 using optical flow when Hn+1 is ill-formed

Nagoya University
Optimizing Video Stitching Quality – Fusing Stitching and Prediction
Filtering techniques to achieve a better estimate by considering both stitching and prediction
results
i. Two homography matrices are decomposed into rotations and translations
ii. Rotation matrices(R1,R2) are converted quaternions and fed into a MEKF
(Multiplicative Extended Kalman Filter)
iii. Compute a least-squares solution of translation t’ using matched features based on
the filtered rotation R’
iv. Form a new homography matrix H’
combining R’ and t’
17
H
H
H
H
H
2 3 41
2 3 41
2 3 41
2 3 41
2 3 41
...
...
...
...
qp
qp
qp
qp
qp
qp
qp
qp
qp
Uploaded
from CPU
Uploaded
from CPU
...
Reprojection
Error
Pair
Pair
4−point
homographies
Candidate
Gauss Jordan
Elimination Pair
...
Pair
from GPU from GPU
...
...
...
i1, j1
ik, jk
i1, j1
ik, jk
Hn−1
Hn−1
H0
H0
ve distortion. Notice
part of the image is
warped and cannot be
ned with the right im-
Frame Frame
Frame Frame
Camera 1 Camera 2
F1,n+1 F2,n+1
Hn+1
Hn
n n
n+1 n+1
Figure 8: Predict Hn+1 using
optical ﬂow when Hn+1 is ill-
formed.
Optical FlowStitching
Translation
Matched
features
Solve
Measurement Prediction
MEKF
H1 H2
n2t2R2R1t1n1
R′
H′
t′
Figure 9: Fusing information
from both stitching and optical
ﬂow.
Fusing information from both stitching and optical flow

Nagoya University
Optimizing Video Stitching Quality – GPS Drift Compensation
GPS drift error can be estimated by comparing the ground truth with the transformation
→ Feed this estimated GPS error to feature matching and reduce search region
18

Nagoya University
Multiple Video Sources
Stitching multiple video frames can be done by iteratively performing
pairwise stitching based on grid-like pattern.
ex) Stitching 6 images
Pairwise stitching <I1,I2>,<I2,I3>,<I3,I4>…<I6,I1>
19
I1 I2 I3
I6 I5 I4
Figure 10: A loop sequence of
stitching 6 images.
Figure 11: Our self-as
quadcopter.
(Section 3.2.2), we can estimate the GPS error. Subseq
A Loop sequence of stitching 6 images.

Nagoya University
Contents
1. Introduction
2. Related Work
5. Evaluation
I. Stitching Time
6. Conclusion
20

Nagoya University
System Implementation
21
Components Model
Flight controller APM 2.6
GPS receiver uBlox LEA-6H
Motor SunnySky X3108S
Electric speed controller Hobby Wing 30AOPTO
Propeller APC 12×3.8
Airframe Hobby KingX550
On-board computer NVIDIA Jetson TK1
Camera PointGrey BlackFly 14S2C
Lens Computer H0514-MP2
WiFi adapter Compex WLE350NX
uence of Figure 11: Our self-assembled
quadcopter.
n estimate the GPS error. Subsequently, we
S error to feature matching and reduce r by
Table 1: Major components used in our quadc
weight (including components not listed) is 2.1 ki
total cost is about USD1,200.
Components Mod
Flight controller APM
GPS receiver uBlox LE
Motor SunnySky
Electronic speed controller HobbyWing 3
Propeller APC 12
Airframe HobbyKin
On-board computer NVIDIA Jet
Self-assembled quadcopter
Video Synchronization
If not synchronized…
Ghosting – a mobile object on the ground appear at different locations
from differnet cameras
→ Hardware-based video synchronization using GPS receiver
Major components used in the quadcopter.
Total weight: 2.1 kilograms, Total cost: USD 1,200

Nagoya University
Contents
1. Introduction
2. Related Work
5. Evaluation
I. Stitching Time
6. Conclusion
22

Nagoya University
Evaluation
Venue
ü Grass field – rich in features
ü Running track – not have many good features
Altitude – 20m
Benchmark for comparison - OpenCV (CPU and GPU) stitching algorithms
23

Nagoya University
Stitching Time – Execution Time of Individual Step
24
0
50
100
150
200
250
300
2 4 6 8 10 12
Featureextractiontime(ms)
Number of video sources
benchmark, CPU
benchmark, GPU
SkyStitch
(a) Feature extraction.
0
2
4
6
8
10
12
14
16
18
20
2 4 6 8 10 12
Featurematchingtime(ms)
benchmark, GPU
SkyStitch
(b) Feature matching.
0
20
40
60
80
100
120
140
160
180
2 4 6 8 10 12
RANSACtime(ms)
benchmark, CPU
benchmark, GPU
SkyStitch
(c) RANSAC.
Figure 12: Execution time of each step of video stitching. The number of features extracted from every frame is 1,000.
5
10
15
20
25
30
35
40
Stitchingrate(fps)
SkyStitch
Benchmark GPU
Benchmark CPU
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
benchmark, GPU
SkyStitch
0
0.5
1
1.5
2
benchmark, GPU
SkyStitch
Execution time of each step of video stitching
SkyStitch:scalable to many video
sources because feature extraction is
offloaded to each quadcopter
Behchmarks:proportional to the # of
video sources
×4
(×70 over CPU)
# of features: 1,000
Overlap: 80%
12 video sources (2×6 grid)
N>7: slopes becomes steeper due to
2×6 grid pattern
×18
×24
Gap CPU-GPU is not large.
- SVD solver not run GPU efficiently
- Many CPU-GPU memory transcation

Nagoya University
Stitching Time – Stitching Rate
0
50
100
2 4 6 8 10 12
Featureext
(a) Feature extraction.
0
2
4
6
8
2 4 6 8 1
Featurema
Number of video sourc
(b) Feature matching.
Figure 12: Execution time of each step of video stitching. The number
0
5
10
15
20
25
30
35
40
2 4 6 8 10 12
Stitchingrate(fps)
SkyStitch
Benchmark GPU
Benchmark CPU
Figure 13: Stitching rate (in terms of frames per second) with re-
spect to the number of video sources.
time of the remaining steps (i.e., feature matching, RANSAC and
image compositing). Then the execution time per stitching is equal
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
500
(a) Imp
Figure 1
executio
25
Stitching rate (frames per second)
×4 over GPU
×16 over CPU
Effect of offloading feature extraction to the UAVs
exeution time = W
𝑚𝑎𝑥 𝑇[]1^_], 𝑇`]a[1b ⋯SkyStitch
𝑇[]1^_] + 𝑇`]a[1b ⋯Benchmarks
Bottleneck of SkyStitch shifts to the
ground station

Nagoya University
Stitching Time – Impact of Feature Count and Overlap Percentage
26
6 8 10 12
mber of video sources
eature matching.
0
2 4 6 8 10 12
(c) RANSAC.
ing. The number of features extracted from every frame is 1,000.
e-
nd
al
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
500 600 700 800 900 1000
Number of Features
benchmark, GPU
SkyStitch
(a) Impact of feature count.
0
0.5
1
1.5
2
20 30 40 50 60 70 80
Featurematchingtime(ms) Overlap Percentage
benchmark, GPU
SkyStitch
(b) Impact of overlap.
Figure 14: Impact of feature count and overlap percentage on the
execution time of feature matching.
rate of increase is slower than than benchmark
Benchmark – pairwise comparison all the features
SkyStitch – search in a small area
Because when the overlap is small,
there are more stitching failures which
lead to a larger search radius.
GPU: Independent of the overlapping area

Nagoya University
Stitching Quality – Number of Inliers
27
-3
-2
-1
0
1
2
0 50 100 150 200 250
Angle(d Frames
(a) Roll
-3
-2
-1
0
1
2
0 50 100 150 20
Angle(d
Frames
(b) Pitch
Figure 15: Jerkiness of rotation components in co
0
50
100
150
200
250
300
350
400
20 30 40 50 60 70 80 90
Numberofinliers
Overlap Percentage
SkyStitch
benchmark
(a) Grass ﬁeld.
0
50
100
150
200
250
300
350
400
20 30 40 50 60 70 80 90
Numberofinliers
Overlap Percentage
SkyStitch
benchmark
(b) Running track.
Figure 16: Impact of overlap percentage on the number of inliers.
The number of features extracted from every frame is 1,000.
100 100
(a) N
Figu
Impact of overlap percentage on the number of inliers
SkyStitch:only searches in a small target area and thus it is less likely to have false matching
Benchmark:pairwise comparison among all the features and the probability of false matching is high

Nagoya University
Stitching Quality – Success Rate of Stitching
28
0
50
100
150
20 30 40 50 60 70 80 90
Nu
Overlap Percentage
(a) Grass ﬁeld.
0
50
100
150
20 30 40 50 60 70 80 90
Nu
Overlap Percentage
(b) Running track.
Figure 16: Impact of overlap percentage on the number of inliers.
The number of features extracted from every frame is 1,000.
0
20
40
60
80
100
20 30 40 50 60 70 80 90
SuccessRate
Overlap Percentage
SkyStitch
benchmark
(a) Grass ﬁeld.
0
20
40
60
80
100
20 30 40 50 60 70 80 90
SuccessRate
Overlap Percentage
SkyStitch
benchmark
(b) Running track.
Figure 17: Impact of overlap percentage on success rate.
make things more intuitive, we convert the rotation matrix into
(a
Fi
(a) U
Figure
We fo
the succ
wise sti
multi-im
video so
97%, wh
Impact of overlap percentage on succes rate
SkyStitch has higher success rate than the benchmark, because failed
stitching can be recovered using optical flow.

Nagoya University
Stitching Quality – Frame Jerkiness
29
Jerkiness of rotation components in computed homographies
-3
-2
-1
0
1
2
3
4
5
0 50 100 150 200 250
Angle(deg)
Frames
Only Stitching
Only Optical Flow
Kalman Filtered
(a) Roll
-3
-2
-1
0
1
2
3
4
5
0 50 100 150 200 250
Angle(deg)
Frames
Only Stitching
Only Optical Flow
Kalman Filtered
(b) Pitch
-3
-2
-1
0
1
2
3
4
5
0 50 100 150 200 250
Angle(deg)
Frames
Only Stitching
Only Optical Flow
Kalman Filtered
(c) Yaw
Figure 15: Jerkiness of rotation components in computed homographies
200
250
300
350
400
rofinliers
SkyStitch
benchmark
200
250
300
350
400
rofinliers
SkyStitch
benchmark
Only stitching: produces very significant jerkiness in pitch and roll
Only optical flow: much smoother but suffers from drift especially in yaw and pitch
Kalman filtered: angles are not prone to either of the problems

Nagoya University
Stitching Quality – Multiple Video Sources
30
Effect of loop closing for 6-image stitching0 150 200 250
rames
Pitch
-3
-2
-1
0 50 100 150 200 250
Frames
(c) Yaw
mponents in computed homographies
(a) No loop closing. (b) With loop closing.
Figure 18: Effect of loop closing for 6-image stitching.
The misalignment is mitigated by applying loop closing method.

Nagoya University
Stitching Quality – Elimination of Ghosting Artefacts
31
Effect of video synchronization on ghost artefacts
(a) No loop closing. (b) With loop closing.
Figure 18: Effect of loop closing for 6-image stitching.
(a) Unsynchronized frames. (b) Synchronized frames.
Figure 19: Effect of video synchronization on ghosting artefacts.
The image is stitched from two video frames synchronized by GPS.

Nagoya University
Contents
1. Introduction
2. Related Work
5. Evaluation
I. Stitching Time
6. Conclusion
32

Nagoya University
Conclusion
SkyStitch – a multi-UAV-based video surveillance system that supports real-time stitching
Ø Feature extraction offloading
Ø Utilization of flight status information
Ø Efficient GPU-based RANSAC implementation
Ø Jerkiness mitigation by Kalman filtering
4 times faster than state-of-the-art GPU accelerated method & Good stitching quality
Future Work
ü Other types of features (color features, low-dimensional feature…)
ü Performance in challenging environment (rough ground with buildings…)
34

SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching

Similar to SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching (20)

Recently uploaded

Recently uploaded (20)

SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching