SlideShare a Scribd company logo
1 of 34
Download to read offline
Nagoya University
Parallel & Distributed Systems Lab.
パターン・映像情報処理特論 (2016.5.6)
SkyStitch: a Cooperative Multi-UAV-based Real-time
Video Surveillance System with Stitching
Yuki KITSUKAWA
yuki@ertl.jp
Edahiro & Kato Lab.
Graduate School of Information Science
Nagoya University
1
Nagoya University
Parallel & Distributed Systems Lab.
About Paper
p Title
SkyStitch: a Cooperative Multi-UAV-based Real-time Video
Surveillance System with Stitching
p Authors
Xiangyun Meng, Wei Wang, Ben Leong
(National University of Singapore)
p Conference
Proceedings of the 23rd ACM international conference on Multimedia,
2015
URL - https://www.comp.nus.edu.sg/~bleong/publications/acmmm15-skystitch.pdf
2
Nagoya University
Parallel & Distributed Systems Lab.
Contents
1. Introduction
2. Related Work
3. Real-time Video Stitching
I. Offloading Feature Extraction
II. Exploiting Flight Status Information
III. Optimizing RANSAC
IV. Optimizing Video Stitching Quality
V. Multiple Video Sources
4. System Implementation
5. Evaluation
I. Stitching Time
II. Stitching Quality
6. Conclusion
3
Nagoya University
Parallel & Distributed Systems Lab.
Contents
1. Introduction
2. Related Work
3. Real-time Video Stitching
I. Offloading Feature Extraction
II. Exploiting Flight Status Information
III. Optimizing RANSAC
IV. Optimizing Video Stitching Quality
V. Multiple Video Sources
4. System Implementation
5. Evaluation
I. Stitching Time
II. Stitching Quality
6. Conclusion
4
Nagoya University
Parallel & Distributed Systems Lab.
Introduction
Applications using UAV-based surveillance system
ü firefighting
ü search and rescue
ü Border monitoring
To cover large target area…
Multiple UAVs are used to capture videos and put them together
(stitch) consistently.
5
SkyStitch: a Cooperative Multi-UAV-based Real-time
Video Surveillance System with Stitching
Xiangyun Meng, Wei Wang and Ben Leong
School of Computing
National University of Singapore
idmmeng@nus.edu.sg, weiwang@comp.nus.edu.sg, benleong@comp.nus.edu.sg
ABSTRACT
Recent advances in unmanned aerial vehicle (UAV) technologies
have made it possible to deploy an aerial video surveillance system
to provide an unprecedented aerial perspective for ground moni-
toring in real time. Multiple UAVs would be required to cover a
large target area, and it is difficult for users to visualize the over-
all situation if they were to receive multiple disjoint video streams.
To address this problem, we designed and implemented SkyStitch,
a multiple-UAV video surveillance system that provides a single
and panoramic video stream to its users by stitching together mul-
tiple aerial video streams. SkyStitch addresses two key design chal-
lenges: (i) the high computational cost of stitching and (ii) the diffi-
culty of ensuring good stitching quality under dynamic conditions.
To improve the speed and quality of video stitching, we incorporate
several practical techniques like distributed feature extraction to re-
Ground station
Video streaming
Video capturing
Video capturing
Video stitching
Figure 1: Illustration of SkyStitch. Each quadcopter transmits a live
video stream to the ground station, which stitches them together to
provide a single composite view of the target area.
For example, a UAV could be deployed to hover over a bush-
Illustration of SkyStitch
UAV – Unmanned Aerial Vehicle
http://www.robotshop.com/en/550mm-rtf-quadcopter-uav.html
Nagoya University
Parallel & Distributed Systems Lab.
Introduction
SkyStitch - a quadcopter-based HD video stitching system for surveillance
Challenges
1. computational cost
2. video stitching (distortion, frame drops, moving objects…)
Solutions
1. Reduce workload of the ground station by offloading pre-
processing to the UAVs
2. Exploit flight information to speed up feature matching
3. GPU-based RANSAC implementation
4. Utilize optical flow and Kalman filtering
5. Closing loops when stitching multiple video sources
6
Nagoya University
Parallel & Distributed Systems Lab.
Contents
1. Introduction
2. Related Work
3. Real-time Video Stitching
I. Offloading Feature Extraction
II. Exploiting Flight Status Information
III. Optimizing RANSAC
IV. Optimizing Video Stitching Quality
V. Multiple Video Sources
4. System Implementation
5. Evaluation
I. Stitching Time
II. Stitching Quality
6. Conclusion
7
Nagoya University
Parallel & Distributed Systems Lab.
Contents
1. Introduction
2. Related Work
3. Real-time Video Stitching
I. Offloading Feature Extraction
II. Exploiting Flight Status Information
III. Optimizing RANSAC
IV. Optimizing Video Stitching Quality
V. Multiple Video Sources
4. System Implementation
5. Evaluation
I. Stitching Time
II. Stitching Quality
6. Conclusion
8
Nagoya University
Parallel & Distributed Systems Lab.
Real-time Video Stitching
Image Stitching – finding the transformation (H: homography matrix)
between two images
Estimate H using feature-based method
i. feature extraction
ii. feature matching
iii. Robust homography estimation
9
𝑥"
	𝑦"
	1 &
~H ) 𝑥	𝑦	1 &
mputational
ess into sev-
Vs. In this
g the work-
l bottleneck
ation is that
mation like
ht controller
perspective
and optical
term stabil-
filter-based
H
2. Feature matching 4. Compositing3. Estimation of H1. Feature extraction
Image alignment
Figure 2: Overview of image stitching.
Overview of image stitching
Nagoya University
Parallel & Distributed Systems Lab.
Offloading Feature Extraction
Traditional stitching system – Transmits only video frames
Feature extraction is performed on ground station.
It would become bottleneck when the # of UAVs increase.
SkyStitch – Transmits video frames and corresponding features
Feature extraction is performed on UAVs*.
* 1,000 features extraction from 1280×1024 on NVIDIA Jetson TK1(192 CUDA cores)
- less than 35 ms
10
Nagoya University
Parallel & Distributed Systems Lab.
Exploiting Flight Status Information
Difficulty in achieving fast and accurate stitching
1. perspective distortion due to wind turbulence
→ Orthorectification of Perspective Distortion
2. time-consuming feature matching between two images
ex) two images with 1,000 features - 1,000,000 combination!
→ Search Region Reduction for Feature Matching
11
Nagoya University
Parallel & Distributed Systems Lab.
Orthorectification of Perspective Distortion
Orthorectification – Transform the image so that it looks as if the camera is pointed vertically
downwards.
ü Transformation matrix – downwards camera direction to the tilted direction
R =
1 0 0
0 cos 𝜃1 sin 𝜃1
0 −sin 𝜃1 cos 𝜃1
cos 𝜃5 0 −sin 𝜃5
0 1 0
sin 𝜃5 0 cos 𝜃5
cos 𝜃6 sin 𝜃6 0
−sin 𝜃6 cos 𝜃6 0
0 0 1
ü Orthorectification matrix
M = KBR:;B:;K:;
K: transform the pixel into world coodinates
(camera intrinsic matrix)
B: transform camera into flight controller
Orthorectified coordinates can be expressed by
𝑥"
	𝑦"
	𝑧′ &
~M 𝑥	𝑦	1 &
12
Pitch
Yaw
Roll
Heading
Figure 3: Flight attitude of quadcopter:
roll, pitch and yaw.
N
Flight attitude of quadcopter: roll, pitch and yaw
Roll Pitch Yaw
obtain from flight controller
Nagoya University
Parallel & Distributed Systems Lab.
Search Region Reduction for Feature Estimation
To reduce feature matching time, GPS location is exploited to find the mapping of pixel
locations between two images.
1. transform the pixel locations from pixel coordinate to a world coordinate system
2. Two image in world coordinate systems can be correlated
using the difference in the GPS locations.
13
P
r
Feature point transformed from Image 1
Bins to be searched for the matched feature
Image 2
Figure 5: Search region reduction for fea-
ture matching.
Search region reduction for
feature matching
ch
opter:
BA
P P
A B
P
Actual scenario
North
Pixel coordinates (after orthorectification)
(a) Mapping a pixel location from one image to another.
CameraCamera
GPS difference
P
(b) Merging coordinate systems.
Figure 4: Overlap determination using flight status information.
, yw, zw) can be obtained by, To further improve RANSAC performance when the number of
Overlap determination using flight status information
Pixel coordinate (in the picture)
𝑥"
, 𝑦"
, 𝑧′
World coordinate (on the ground)
𝑥?, 𝑦?, 𝑧? = 𝑠
𝑥′
𝑧′
−
𝑊
2
, 𝑠
𝑦′
𝑧′
−
𝐻
2
, ℎ
𝑠 = 2ℎ tan
𝜃G
2
𝑊H
s: # of meters representedby the side of a pixel
W×H: image resolution (in pixel)
𝜃G: angle of horizontal field of view in the camera
h: quadcopter altitude
Nagoya University
Parallel & Distributed Systems Lab.
Optimizing RANSAC
RANSAC (Random Sample Consensus) – method of removing outliers
i. Select four matched features to compute homography matrix H
ii. Apply estimated H on all the matched features
iii. Repeat i-ii for N iterations and find the one with the most inliers
Large N improves H but also increases computation time
GPU-accelerated RANSAC
Ø Gauss-Jordan elimination instead of SVD (Singular Value Decomposition)
Ø Maximize parallelism and minimize IO
Ø Improve score calculation (use the distribution of inliers)
14
P
r
point transformed from Image 1
e searched for the matched feature
2
earch region reduction for fea-
ng.
H
H
H
H
H
2 3 41
2 3 41
2 3 41
2 3 41
2 3 41
...
...
...
...
qp
qp
qp
qp
qp
qp
qp
qp
qp
Uploaded
from CPU
Uploaded
from CPU
...
Reprojection
Error
Pair
Pair
ScoresInlier maskAll matchescorrespondences
4−point
homographies
Candidate
Gauss Jordan
Elimination Pair
...
Pair
DownloadedDownloaded
from GPU from GPU
...
...
...
i1, j1
ik, jk
i1, j1
ik, jk
Hn−1
Hn−1
H0
H0
Figure 6: RANSAC GPU Pipeline
Camera 1 Camera 2 Optical FlowStitching
H1 H2
RANSAC GPU Pipeline
Nagoya University
Parallel & Distributed Systems Lab.
Optimizing Video Stitching Quality
Causes of stitching failures
ü small overlapping area between two images
ü concentration of inliers selected by RANSAC in a small region
How to detect ill-formed H and prevent them from affecting the perceptual experience?
Ø # of inliers produced by H
If insufficient, discard the current frame
Ø the angle between the two image planes - 𝜃
H = R + tn&
(R: 3×3 rotation matrix between two images)
cos 𝜃 = 𝑟KK (angle around z-axis)
If 𝜃 > 5°, H is erronous
15
P
r
Feature point transformed from Image 1
Bins to be searched for the matched feature
Image 2
Figure 5: Search region reduction for fea-
ture matching.
H
H
H
H
H
2 3 41
2 3 41
2 3 41
2 3 41
2 3 41
...
Uploaded
from CPU
...
correspondences
4−point
homographies
Candidate
Gauss Jordan
Elimination
Figu
(a) Inaccurate rotation. Notice the
broken and bent touchline of the
football field, below the running
track.
(b) Perspective distortion. Notice
that the left part of the image is
erroneously warped and cannot be
perfectly aligned with the right im-
age.
Figure 7: Examples of perceptual distortion. Each blue cross rep-
resents a feature point.
When there are sufficient inliers, we also check whether H badly
warps the images and introduces perspective distortion. Let θ be the
Fram
Fram
Cam
Figur
optica
forme
was n
ing to
3.4.
Example of perceptual distortion
Nagoya University
Parallel & Distributed Systems Lab.
Optimizing Video Stitching Quality – Recovering from Failed Stitching
Prediction of H by utilizing the optical flow of adjacent frames to reduce
frame drops due to ill-formed H
HOP; = MQ,OP;FQ,OP;MQ,O
:; HOM;,OF;,OP;
:; M;,OP;
:;
F: 3×3 homography matrix of optical flow
M: orthorectification matrix
Hn : transformation matrix of the previous stitching
16
ge 1
d feature
ion for fea-
H
H
H
H
H
2 3 41
2 3 41
2 3 41
2 3 41
2 3 41
...
...
...
...
qp
qp
qp
qp
qp
qp
qp
qp
qp
Uploaded
from CPU
Uploaded
from CPU...
Reprojection
Error
Pair
Pair
ScoresInlier maskAll matchescorrespondences
4−point
homographies
Candidate
Gauss Jordan
Elimination Pair
...
Pair
DownloadedDownloaded
from GPU from GPU
...
...
...
i1, j1
ik, jk
i1, j1
ik, jk
Hn−1
Hn−1
H0
H0
Figure 6: RANSAC GPU Pipeline
(b) Perspective distortion. Notice
that the left part of the image is
erroneously warped and cannot be
perfectly aligned with the right im-
age.
ual distortion. Each blue cross rep-
Frame Frame
Frame Frame
Camera 1 Camera 2
F1,n+1 F2,n+1
Hn+1
Hn
n n
n+1 n+1
Figure 8: Predict Hn+1 using
optical flow when Hn+1 is ill-
formed.
Optical FlowStitching
Translation
Matched
features
Solve
Measurement Prediction
MEKF
H1 H2
n2t2R2R1t1n1
R′
H′
t′
Figure 9: Fusing information
from both stitching and optical
flow.
Predict Hn+1 using optical flow when Hn+1 is ill-formed
Nagoya University
Parallel & Distributed Systems Lab.
Optimizing Video Stitching Quality – Fusing Stitching and Prediction
Filtering techniques to achieve a better estimate by considering both stitching and prediction
results
i. Two homography matrices are decomposed into rotations and translations
ii. Rotation matrices(R1,R2) are converted quaternions and fed into a MEKF
(Multiplicative Extended Kalman Filter)
iii. Compute a least-squares solution of translation t’ using matched features based on
the filtered rotation R’
iv. Form a new homography matrix H’
combining R’ and t’
17
H
H
H
H
H
2 3 41
2 3 41
2 3 41
2 3 41
2 3 41
...
...
...
...
qp
qp
qp
qp
qp
qp
qp
qp
qp
Uploaded
from CPU
Uploaded
from CPU
...
Reprojection
Error
Pair
Pair
ScoresInlier maskAll matchescorrespondences
4−point
homographies
Candidate
Gauss Jordan
Elimination Pair
...
Pair
DownloadedDownloaded
from GPU from GPU
...
...
...
i1, j1
ik, jk
i1, j1
ik, jk
Hn−1
Hn−1
H0
H0
Figure 6: RANSAC GPU Pipeline
ve distortion. Notice
part of the image is
warped and cannot be
ned with the right im-
Frame Frame
Frame Frame
Camera 1 Camera 2
F1,n+1 F2,n+1
Hn+1
Hn
n n
n+1 n+1
Figure 8: Predict Hn+1 using
optical flow when Hn+1 is ill-
formed.
Optical FlowStitching
Translation
Matched
features
Solve
Measurement Prediction
MEKF
H1 H2
n2t2R2R1t1n1
R′
H′
t′
Figure 9: Fusing information
from both stitching and optical
flow.
Fusing information from both stitching and optical flow
Nagoya University
Parallel & Distributed Systems Lab.
Optimizing Video Stitching Quality – GPS Drift Compensation
GPS drift error can be estimated by comparing the ground truth with the transformation
→ Feed this estimated GPS error to feature matching and reduce search region
18
Nagoya University
Parallel & Distributed Systems Lab.
Multiple Video Sources
Stitching multiple video frames can be done by iteratively performing
pairwise stitching based on grid-like pattern.
ex) Stitching 6 images
Pairwise stitching <I1,I2>,<I2,I3>,<I3,I4>…<I6,I1>
19
I1 I2 I3
I6 I5 I4
Figure 10: A loop sequence of
stitching 6 images.
Figure 11: Our self-as
quadcopter.
(Section 3.2.2), we can estimate the GPS error. Subseq
A Loop sequence of stitching 6 images.
Nagoya University
Parallel & Distributed Systems Lab.
Contents
1. Introduction
2. Related Work
3. Real-time Video Stitching
I. Offloading Feature Extraction
II. Exploiting Flight Status Information
III. Optimizing RANSAC
IV. Optimizing Video Stitching Quality
V. Multiple Video Sources
4. System Implementation
5. Evaluation
I. Stitching Time
II. Stitching Quality
6. Conclusion
20
Nagoya University
Parallel & Distributed Systems Lab.
System Implementation
21
Components Model
Flight controller APM 2.6
GPS receiver uBlox LEA-6H
Motor SunnySky X3108S
Electric speed controller Hobby Wing 30AOPTO
Propeller APC 12×3.8
Airframe Hobby KingX550
On-board computer NVIDIA Jetson TK1
Camera PointGrey BlackFly 14S2C
Lens Computer H0514-MP2
WiFi adapter Compex WLE350NX
uence of Figure 11: Our self-assembled
quadcopter.
n estimate the GPS error. Subsequently, we
S error to feature matching and reduce r by
Table 1: Major components used in our quadc
weight (including components not listed) is 2.1 ki
total cost is about USD1,200.
Components Mod
Flight controller APM
GPS receiver uBlox LE
Motor SunnySky
Electronic speed controller HobbyWing 3
Propeller APC 12
Airframe HobbyKin
On-board computer NVIDIA Jet
Self-assembled quadcopter
Video Synchronization
If not synchronized…
Ghosting – a mobile object on the ground appear at different locations
from differnet cameras
→ Hardware-based video synchronization using GPS receiver
Major components used in the quadcopter.
Total weight: 2.1 kilograms, Total cost: USD 1,200
Nagoya University
Parallel & Distributed Systems Lab.
Contents
1. Introduction
2. Related Work
3. Real-time Video Stitching
I. Offloading Feature Extraction
II. Exploiting Flight Status Information
III. Optimizing RANSAC
IV. Optimizing Video Stitching Quality
V. Multiple Video Sources
4. System Implementation
5. Evaluation
I. Stitching Time
II. Stitching Quality
6. Conclusion
22
Nagoya University
Parallel & Distributed Systems Lab.
Evaluation
Venue
ü Grass field – rich in features
ü Running track – not have many good features
Altitude – 20m
Benchmark for comparison - OpenCV (CPU and GPU) stitching algorithms
23
Nagoya University
Parallel & Distributed Systems Lab.
Stitching Time – Execution Time of Individual Step
24
0
50
100
150
200
250
300
2 4 6 8 10 12
Featureextractiontime(ms)
Number of video sources
benchmark, CPU
benchmark, GPU
SkyStitch
(a) Feature extraction.
0
2
4
6
8
10
12
14
16
18
20
2 4 6 8 10 12
Featurematchingtime(ms)
Number of video sources
benchmark, GPU
SkyStitch
(b) Feature matching.
0
20
40
60
80
100
120
140
160
180
2 4 6 8 10 12
RANSACtime(ms)
Number of video sources
benchmark, CPU
benchmark, GPU
SkyStitch
(c) RANSAC.
Figure 12: Execution time of each step of video stitching. The number of features extracted from every frame is 1,000.
5
10
15
20
25
30
35
40
Stitchingrate(fps)
SkyStitch
Benchmark GPU
Benchmark CPU
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Featurematchingtime(ms)
benchmark, GPU
SkyStitch
0
0.5
1
1.5
2
Featurematchingtime(ms)
benchmark, GPU
SkyStitch
Execution time of each step of video stitching
SkyStitch:scalable to many video
sources because feature extraction is
offloaded to each quadcopter
Behchmarks:proportional to the # of
video sources
×4
(×70 over CPU)
# of features: 1,000
Overlap: 80%
12 video sources (2×6 grid)
N>7: slopes becomes steeper due to
2×6 grid pattern
×18
×24
Gap CPU-GPU is not large.
- SVD solver not run GPU efficiently
- Many CPU-GPU memory transcation
Nagoya University
Parallel & Distributed Systems Lab.
Stitching Time – Stitching Rate
0
50
100
2 4 6 8 10 12
Featureext
Number of video sources
(a) Feature extraction.
0
2
4
6
8
2 4 6 8 1
Featurema
Number of video sourc
(b) Feature matching.
Figure 12: Execution time of each step of video stitching. The number
0
5
10
15
20
25
30
35
40
2 4 6 8 10 12
Stitchingrate(fps)
Number of video sources
SkyStitch
Benchmark GPU
Benchmark CPU
Figure 13: Stitching rate (in terms of frames per second) with re-
spect to the number of video sources.
time of the remaining steps (i.e., feature matching, RANSAC and
image compositing). Then the execution time per stitching is equal
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
500
Featurematchingtime(ms)
(a) Imp
Figure 1
executio
25
Stitching rate (frames per second)
×4 over GPU
×16 over CPU
Effect of offloading feature extraction to the UAVs
exeution	time = W
𝑚𝑎𝑥 𝑇[]1^_], 𝑇`]a[1b 			⋯SkyStitch
𝑇[]1^_] + 𝑇`]a[1b			⋯Benchmarks
Bottleneck of SkyStitch shifts to the
ground station
Nagoya University
Parallel & Distributed Systems Lab.
Stitching Time – Impact of Feature Count and Overlap Percentage
26
6 8 10 12
mber of video sources
eature matching.
0
2 4 6 8 10 12
Number of video sources
(c) RANSAC.
ing. The number of features extracted from every frame is 1,000.
e-
nd
al
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
500 600 700 800 900 1000
Featurematchingtime(ms)
Number of Features
benchmark, GPU
SkyStitch
(a) Impact of feature count.
0
0.5
1
1.5
2
20 30 40 50 60 70 80
Featurematchingtime(ms) Overlap Percentage
benchmark, GPU
SkyStitch
(b) Impact of overlap.
Figure 14: Impact of feature count and overlap percentage on the
execution time of feature matching.
rate of increase is slower than than benchmark
Benchmark – pairwise comparison all the features
SkyStitch – search in a small area
Because when the overlap is small,
there are more stitching failures which
lead to a larger search radius.
GPU: Independent of the overlapping area
Nagoya University
Parallel & Distributed Systems Lab.
Stitching Quality – Number of Inliers
27
-3
-2
-1
0
1
2
0 50 100 150 200 250
Angle(d Frames
(a) Roll
-3
-2
-1
0
1
2
0 50 100 150 20
Angle(d
Frames
(b) Pitch
Figure 15: Jerkiness of rotation components in co
0
50
100
150
200
250
300
350
400
20 30 40 50 60 70 80 90
Numberofinliers
Overlap Percentage
SkyStitch
benchmark
(a) Grass field.
0
50
100
150
200
250
300
350
400
20 30 40 50 60 70 80 90
Numberofinliers
Overlap Percentage
SkyStitch
benchmark
(b) Running track.
Figure 16: Impact of overlap percentage on the number of inliers.
The number of features extracted from every frame is 1,000.
100 100
(a) N
Figu
Impact of overlap percentage on the number of inliers
SkyStitch:only searches in a small target area and thus it is less likely to have false matching
Benchmark:pairwise comparison among all the features and the probability of false matching is high
Nagoya University
Parallel & Distributed Systems Lab.
Stitching Quality – Success Rate of Stitching
28
0
50
100
150
20 30 40 50 60 70 80 90
Nu
Overlap Percentage
(a) Grass field.
0
50
100
150
20 30 40 50 60 70 80 90
Nu
Overlap Percentage
(b) Running track.
Figure 16: Impact of overlap percentage on the number of inliers.
The number of features extracted from every frame is 1,000.
0
20
40
60
80
100
20 30 40 50 60 70 80 90
SuccessRate
Overlap Percentage
SkyStitch
benchmark
(a) Grass field.
0
20
40
60
80
100
20 30 40 50 60 70 80 90
SuccessRate
Overlap Percentage
SkyStitch
benchmark
(b) Running track.
Figure 17: Impact of overlap percentage on success rate.
make things more intuitive, we convert the rotation matrix into
(a
Fi
(a) U
Figure
We fo
the succ
wise sti
multi-im
video so
97%, wh
Impact of overlap percentage on succes rate
SkyStitch has higher success rate than the benchmark, because failed
stitching can be recovered using optical flow.
Nagoya University
Parallel & Distributed Systems Lab.
Stitching Quality – Frame Jerkiness
29
Jerkiness of rotation components in computed homographies
-3
-2
-1
0
1
2
3
4
5
0 50 100 150 200 250
Angle(deg)
Frames
Only Stitching
Only Optical Flow
Kalman Filtered
(a) Roll
-3
-2
-1
0
1
2
3
4
5
0 50 100 150 200 250
Angle(deg)
Frames
Only Stitching
Only Optical Flow
Kalman Filtered
(b) Pitch
-3
-2
-1
0
1
2
3
4
5
0 50 100 150 200 250
Angle(deg)
Frames
Only Stitching
Only Optical Flow
Kalman Filtered
(c) Yaw
Figure 15: Jerkiness of rotation components in computed homographies
200
250
300
350
400
rofinliers
SkyStitch
benchmark
200
250
300
350
400
rofinliers
SkyStitch
benchmark
Only stitching: produces very significant jerkiness in pitch and roll
Only optical flow: much smoother but suffers from drift especially in yaw and pitch
Kalman filtered: angles are not prone to either of the problems
Nagoya University
Parallel & Distributed Systems Lab.
Stitching Quality – Multiple Video Sources
30
Effect of loop closing for 6-image stitching0 150 200 250
rames
Pitch
-3
-2
-1
0 50 100 150 200 250
Frames
(c) Yaw
mponents in computed homographies
(a) No loop closing. (b) With loop closing.
Figure 18: Effect of loop closing for 6-image stitching.
The misalignment is mitigated by applying loop closing method.
Nagoya University
Parallel & Distributed Systems Lab.
Stitching Quality – Elimination of Ghosting Artefacts
31
Effect of video synchronization on ghost artefacts
(a) No loop closing. (b) With loop closing.
Figure 18: Effect of loop closing for 6-image stitching.
(a) Unsynchronized frames. (b) Synchronized frames.
Figure 19: Effect of video synchronization on ghosting artefacts.
The image is stitched from two video frames synchronized by GPS.
Nagoya University
Parallel & Distributed Systems Lab.
Contents
1. Introduction
2. Related Work
3. Real-time Video Stitching
I. Offloading Feature Extraction
II. Exploiting Flight Status Information
III. Optimizing RANSAC
IV. Optimizing Video Stitching Quality
V. Multiple Video Sources
4. System Implementation
5. Evaluation
I. Stitching Time
II. Stitching Quality
6. Conclusion
32
33
Nagoya University
Parallel & Distributed Systems Lab.
Conclusion
SkyStitch – a multi-UAV-based video surveillance system that supports real-time stitching
Ø Feature extraction offloading
Ø Utilization of flight status information
Ø Efficient GPU-based RANSAC implementation
Ø Jerkiness mitigation by Kalman filtering
4 times faster than state-of-the-art GPU accelerated method & Good stitching quality
Future Work
ü Other types of features (color features, low-dimensional feature…)
ü Performance in challenging environment (rough ground with buildings…)
34

More Related Content

What's hot

chapter19. Terrestrial and Close-Range Photogrammetry.pdf
chapter19. Terrestrial and Close-Range Photogrammetry.pdfchapter19. Terrestrial and Close-Range Photogrammetry.pdf
chapter19. Terrestrial and Close-Range Photogrammetry.pdf
ssuser3f7a17
 
GIS history & milestones (epm107_2007)
GIS history & milestones (epm107_2007)GIS history & milestones (epm107_2007)
GIS history & milestones (epm107_2007)
esambale
 

What's hot (20)

Glonass
GlonassGlonass
Glonass
 
GNSS(GLOBAL NAVIGATION SATELLITE SYSTEM)
GNSS(GLOBAL NAVIGATION SATELLITE SYSTEM)GNSS(GLOBAL NAVIGATION SATELLITE SYSTEM)
GNSS(GLOBAL NAVIGATION SATELLITE SYSTEM)
 
Introduction to geomatics
Introduction to geomaticsIntroduction to geomatics
Introduction to geomatics
 
chapter19. Terrestrial and Close-Range Photogrammetry.pdf
chapter19. Terrestrial and Close-Range Photogrammetry.pdfchapter19. Terrestrial and Close-Range Photogrammetry.pdf
chapter19. Terrestrial and Close-Range Photogrammetry.pdf
 
Global Navigation Satellite System (GNSS) update
Global Navigation Satellite System (GNSS) updateGlobal Navigation Satellite System (GNSS) update
Global Navigation Satellite System (GNSS) update
 
Basic of Remote Sensing
Basic of Remote SensingBasic of Remote Sensing
Basic of Remote Sensing
 
Presentation on Drones and Open Street Maps
Presentation on Drones and Open Street MapsPresentation on Drones and Open Street Maps
Presentation on Drones and Open Street Maps
 
The GRASS GIS software (with QGIS) - GIS Seminar
The GRASS GIS software (with QGIS) - GIS SeminarThe GRASS GIS software (with QGIS) - GIS Seminar
The GRASS GIS software (with QGIS) - GIS Seminar
 
Global Positioning System (GPS)
Global Positioning System (GPS) Global Positioning System (GPS)
Global Positioning System (GPS)
 
Drone and its application
Drone and its applicationDrone and its application
Drone and its application
 
70.mobile gis
70.mobile gis70.mobile gis
70.mobile gis
 
GAGAN ISRO
GAGAN ISROGAGAN ISRO
GAGAN ISRO
 
03 cameras &amp; their geometry
03 cameras &amp; their geometry03 cameras &amp; their geometry
03 cameras &amp; their geometry
 
Lecture01: Introduction to Photogrammetry
Lecture01: Introduction to PhotogrammetryLecture01: Introduction to Photogrammetry
Lecture01: Introduction to Photogrammetry
 
Global Positioning System
Global Positioning SystemGlobal Positioning System
Global Positioning System
 
GIS
GISGIS
GIS
 
Basic remote sensing and gis
Basic remote sensing and gisBasic remote sensing and gis
Basic remote sensing and gis
 
GIS history & milestones (epm107_2007)
GIS history & milestones (epm107_2007)GIS history & milestones (epm107_2007)
GIS history & milestones (epm107_2007)
 
MODIS (Moderate Resolution Imaging Spectrometer)
MODIS (Moderate Resolution Imaging Spectrometer)MODIS (Moderate Resolution Imaging Spectrometer)
MODIS (Moderate Resolution Imaging Spectrometer)
 
Titan Rover Senior Design Project Critical Design Review 2021
Titan Rover Senior Design Project Critical Design Review 2021Titan Rover Senior Design Project Critical Design Review 2021
Titan Rover Senior Design Project Critical Design Review 2021
 

Viewers also liked

Barbero juan presentació_competic2
Barbero juan presentació_competic2Barbero juan presentació_competic2
Barbero juan presentació_competic2
juan barrod
 
Lau lagun eta erdi eta papa noelen kasua. irati bengoetxea
Lau lagun eta erdi eta papa noelen kasua. irati bengoetxeaLau lagun eta erdi eta papa noelen kasua. irati bengoetxea
Lau lagun eta erdi eta papa noelen kasua. irati bengoetxea
abalanzategi
 
Rowena_Simbulan_Black_Belt_Certificate - Technical IT Recruiting
Rowena_Simbulan_Black_Belt_Certificate - Technical IT RecruitingRowena_Simbulan_Black_Belt_Certificate - Technical IT Recruiting
Rowena_Simbulan_Black_Belt_Certificate - Technical IT Recruiting
Rowena Simbulan
 

Viewers also liked (14)

merge
mergemerge
merge
 
Barbero juan presentació_competic2
Barbero juan presentació_competic2Barbero juan presentació_competic2
Barbero juan presentació_competic2
 
Lau lagun eta erdi eta papa noelen kasua. irati bengoetxea
Lau lagun eta erdi eta papa noelen kasua. irati bengoetxeaLau lagun eta erdi eta papa noelen kasua. irati bengoetxea
Lau lagun eta erdi eta papa noelen kasua. irati bengoetxea
 
Invitacion luis cooper
Invitacion luis cooperInvitacion luis cooper
Invitacion luis cooper
 
Virus y antivirus
Virus y antivirus Virus y antivirus
Virus y antivirus
 
Robust Vehicle Localization in Urban Environments Using Probabilistic Maps
Robust Vehicle Localization in Urban Environments Using Probabilistic MapsRobust Vehicle Localization in Urban Environments Using Probabilistic Maps
Robust Vehicle Localization in Urban Environments Using Probabilistic Maps
 
HM-MMA Powerpoint Indemnification Agreements - 2014 12
HM-MMA Powerpoint Indemnification Agreements - 2014 12HM-MMA Powerpoint Indemnification Agreements - 2014 12
HM-MMA Powerpoint Indemnification Agreements - 2014 12
 
Discharge of the contract
Discharge of the contractDischarge of the contract
Discharge of the contract
 
методе дефинисања
методе дефинисањаметоде дефинисања
методе дефинисања
 
Maternal and child health interventions in ghana
Maternal and child health  interventions in ghanaMaternal and child health  interventions in ghana
Maternal and child health interventions in ghana
 
Ch.2.a healthy diet
Ch.2.a healthy dietCh.2.a healthy diet
Ch.2.a healthy diet
 
Rowena_Simbulan_Black_Belt_Certificate - Technical IT Recruiting
Rowena_Simbulan_Black_Belt_Certificate - Technical IT RecruitingRowena_Simbulan_Black_Belt_Certificate - Technical IT Recruiting
Rowena_Simbulan_Black_Belt_Certificate - Technical IT Recruiting
 
Co-branding
Co-brandingCo-branding
Co-branding
 
Actividad 5 joaquin
Actividad 5 joaquinActividad 5 joaquin
Actividad 5 joaquin
 

Similar to SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching

ANISH_and_DR.DANIEL_augmented_reality_presentation
ANISH_and_DR.DANIEL_augmented_reality_presentationANISH_and_DR.DANIEL_augmented_reality_presentation
ANISH_and_DR.DANIEL_augmented_reality_presentation
Anish Patel
 
Юрій Іванов Тема: Програмні лічильники однотипних рухомих об’єктів: алгоритми...
Юрій Іванов Тема: Програмні лічильники однотипних рухомих об’єктів: алгоритми...Юрій Іванов Тема: Програмні лічильники однотипних рухомих об’єктів: алгоритми...
Юрій Іванов Тема: Програмні лічильники однотипних рухомих об’єктів: алгоритми...
Lviv Startup Club
 
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed realitySynthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
NAVER Engineering
 

Similar to SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching (20)

Visual Mapping and Collision Avoidance Dynamic Environments in Dynamic Enviro...
Visual Mapping and Collision Avoidance Dynamic Environments in Dynamic Enviro...Visual Mapping and Collision Avoidance Dynamic Environments in Dynamic Enviro...
Visual Mapping and Collision Avoidance Dynamic Environments in Dynamic Enviro...
 
Visual Environment by Semantic Segmentation Using Deep Learning: A Prototype ...
Visual Environment by Semantic Segmentation Using Deep Learning: A Prototype ...Visual Environment by Semantic Segmentation Using Deep Learning: A Prototype ...
Visual Environment by Semantic Segmentation Using Deep Learning: A Prototype ...
 
IRJET- Design the Surveillance Algorithm and Motion Detection of Objects for ...
IRJET- Design the Surveillance Algorithm and Motion Detection of Objects for ...IRJET- Design the Surveillance Algorithm and Motion Detection of Objects for ...
IRJET- Design the Surveillance Algorithm and Motion Detection of Objects for ...
 
ANISH_and_DR.DANIEL_augmented_reality_presentation
ANISH_and_DR.DANIEL_augmented_reality_presentationANISH_and_DR.DANIEL_augmented_reality_presentation
ANISH_and_DR.DANIEL_augmented_reality_presentation
 
Dataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problemsDataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problems
 
100519 ver final
100519 ver final100519 ver final
100519 ver final
 
Юрій Іванов Тема: Програмні лічильники однотипних рухомих об’єктів: алгоритми...
Юрій Іванов Тема: Програмні лічильники однотипних рухомих об’єктів: алгоритми...Юрій Іванов Тема: Програмні лічильники однотипних рухомих об’єктів: алгоритми...
Юрій Іванов Тема: Програмні лічильники однотипних рухомих об’єктів: алгоритми...
 
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed realitySynthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
 
06-08 ppt.pptx
06-08 ppt.pptx06-08 ppt.pptx
06-08 ppt.pptx
 
report
reportreport
report
 
Flow Trajectory Approach for Human Action Recognition
Flow Trajectory Approach for Human Action RecognitionFlow Trajectory Approach for Human Action Recognition
Flow Trajectory Approach for Human Action Recognition
 
AUTO LANDING PROCESS FOR AUTONOMOUS FLYING ROBOT BY USING IMAGE PROCESSING BA...
AUTO LANDING PROCESS FOR AUTONOMOUS FLYING ROBOT BY USING IMAGE PROCESSING BA...AUTO LANDING PROCESS FOR AUTONOMOUS FLYING ROBOT BY USING IMAGE PROCESSING BA...
AUTO LANDING PROCESS FOR AUTONOMOUS FLYING ROBOT BY USING IMAGE PROCESSING BA...
 
med_poster_spie
med_poster_spiemed_poster_spie
med_poster_spie
 
Inria Tech Talk : Améliorez vos applications de robotique & réalité augmentée
Inria Tech Talk : Améliorez vos applications de robotique & réalité augmentéeInria Tech Talk : Améliorez vos applications de robotique & réalité augmentée
Inria Tech Talk : Améliorez vos applications de robotique & réalité augmentée
 
Isvc08
Isvc08Isvc08
Isvc08
 
Human Action Recognition using Contour History Images and Neural Networks Cla...
Human Action Recognition using Contour History Images and Neural Networks Cla...Human Action Recognition using Contour History Images and Neural Networks Cla...
Human Action Recognition using Contour History Images and Neural Networks Cla...
 
J017377578
J017377578J017377578
J017377578
 
Real-time Moving Object Detection using SURF
Real-time Moving Object Detection using SURFReal-time Moving Object Detection using SURF
Real-time Moving Object Detection using SURF
 
DICTA 2017 poster
DICTA 2017 posterDICTA 2017 poster
DICTA 2017 poster
 
Cartoonization of images using machine Learning
Cartoonization of images using machine LearningCartoonization of images using machine Learning
Cartoonization of images using machine Learning
 

Recently uploaded

Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Lovely Professional University
 
Teachers record management system project report..pdf
Teachers record management system project report..pdfTeachers record management system project report..pdf
Teachers record management system project report..pdf
Kamal Acharya
 
ONLINE VEHICLE RENTAL SYSTEM PROJECT REPORT.pdf
ONLINE VEHICLE RENTAL SYSTEM PROJECT REPORT.pdfONLINE VEHICLE RENTAL SYSTEM PROJECT REPORT.pdf
ONLINE VEHICLE RENTAL SYSTEM PROJECT REPORT.pdf
Kamal Acharya
 
DR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdf
DR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdfDR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdf
DR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdf
DrGurudutt
 

Recently uploaded (20)

Diploma Engineering Drawing Qp-2024 Ece .pdf
Diploma Engineering Drawing Qp-2024 Ece .pdfDiploma Engineering Drawing Qp-2024 Ece .pdf
Diploma Engineering Drawing Qp-2024 Ece .pdf
 
Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1Research Methodolgy & Intellectual Property Rights Series 1
Research Methodolgy & Intellectual Property Rights Series 1
 
15-Minute City: A Completely New Horizon
15-Minute City: A Completely New Horizon15-Minute City: A Completely New Horizon
15-Minute City: A Completely New Horizon
 
Instruct Nirmaana 24-Smart and Lean Construction Through Technology.pdf
Instruct Nirmaana 24-Smart and Lean Construction Through Technology.pdfInstruct Nirmaana 24-Smart and Lean Construction Through Technology.pdf
Instruct Nirmaana 24-Smart and Lean Construction Through Technology.pdf
 
ROAD CONSTRUCTION PRESENTATION.PPTX.pptx
ROAD CONSTRUCTION PRESENTATION.PPTX.pptxROAD CONSTRUCTION PRESENTATION.PPTX.pptx
ROAD CONSTRUCTION PRESENTATION.PPTX.pptx
 
Introduction to Artificial Intelligence and History of AI
Introduction to Artificial Intelligence and History of AIIntroduction to Artificial Intelligence and History of AI
Introduction to Artificial Intelligence and History of AI
 
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...
 
Lesson no16 application of Induction Generator in Wind.ppsx
Lesson no16 application of Induction Generator in Wind.ppsxLesson no16 application of Induction Generator in Wind.ppsx
Lesson no16 application of Induction Generator in Wind.ppsx
 
The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...
The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...
The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...
 
NEWLETTER FRANCE HELICES/ SDS SURFACE DRIVES - MAY 2024
NEWLETTER FRANCE HELICES/ SDS SURFACE DRIVES - MAY 2024NEWLETTER FRANCE HELICES/ SDS SURFACE DRIVES - MAY 2024
NEWLETTER FRANCE HELICES/ SDS SURFACE DRIVES - MAY 2024
 
2024 DevOps Pro Europe - Growing at the edge
2024 DevOps Pro Europe - Growing at the edge2024 DevOps Pro Europe - Growing at the edge
2024 DevOps Pro Europe - Growing at the edge
 
BRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWING
BRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWINGBRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWING
BRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWING
 
Quiz application system project report..pdf
Quiz application system project report..pdfQuiz application system project report..pdf
Quiz application system project report..pdf
 
ChatGPT Prompt Engineering for project managers.pdf
ChatGPT Prompt Engineering for project managers.pdfChatGPT Prompt Engineering for project managers.pdf
ChatGPT Prompt Engineering for project managers.pdf
 
Artificial Intelligence Bayesian Reasoning
Artificial Intelligence Bayesian ReasoningArtificial Intelligence Bayesian Reasoning
Artificial Intelligence Bayesian Reasoning
 
Teachers record management system project report..pdf
Teachers record management system project report..pdfTeachers record management system project report..pdf
Teachers record management system project report..pdf
 
ONLINE VEHICLE RENTAL SYSTEM PROJECT REPORT.pdf
ONLINE VEHICLE RENTAL SYSTEM PROJECT REPORT.pdfONLINE VEHICLE RENTAL SYSTEM PROJECT REPORT.pdf
ONLINE VEHICLE RENTAL SYSTEM PROJECT REPORT.pdf
 
DR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdf
DR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdfDR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdf
DR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdf
 
BURGER ORDERING SYSYTEM PROJECT REPORT..pdf
BURGER ORDERING SYSYTEM PROJECT REPORT..pdfBURGER ORDERING SYSYTEM PROJECT REPORT..pdf
BURGER ORDERING SYSYTEM PROJECT REPORT..pdf
 
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5
 

SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching

  • 1. Nagoya University Parallel & Distributed Systems Lab. パターン・映像情報処理特論 (2016.5.6) SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching Yuki KITSUKAWA yuki@ertl.jp Edahiro & Kato Lab. Graduate School of Information Science Nagoya University 1
  • 2. Nagoya University Parallel & Distributed Systems Lab. About Paper p Title SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching p Authors Xiangyun Meng, Wei Wang, Ben Leong (National University of Singapore) p Conference Proceedings of the 23rd ACM international conference on Multimedia, 2015 URL - https://www.comp.nus.edu.sg/~bleong/publications/acmmm15-skystitch.pdf 2
  • 3. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 3
  • 4. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 4
  • 5. Nagoya University Parallel & Distributed Systems Lab. Introduction Applications using UAV-based surveillance system ü firefighting ü search and rescue ü Border monitoring To cover large target area… Multiple UAVs are used to capture videos and put them together (stitch) consistently. 5 SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching Xiangyun Meng, Wei Wang and Ben Leong School of Computing National University of Singapore idmmeng@nus.edu.sg, weiwang@comp.nus.edu.sg, benleong@comp.nus.edu.sg ABSTRACT Recent advances in unmanned aerial vehicle (UAV) technologies have made it possible to deploy an aerial video surveillance system to provide an unprecedented aerial perspective for ground moni- toring in real time. Multiple UAVs would be required to cover a large target area, and it is difficult for users to visualize the over- all situation if they were to receive multiple disjoint video streams. To address this problem, we designed and implemented SkyStitch, a multiple-UAV video surveillance system that provides a single and panoramic video stream to its users by stitching together mul- tiple aerial video streams. SkyStitch addresses two key design chal- lenges: (i) the high computational cost of stitching and (ii) the diffi- culty of ensuring good stitching quality under dynamic conditions. To improve the speed and quality of video stitching, we incorporate several practical techniques like distributed feature extraction to re- Ground station Video streaming Video capturing Video capturing Video stitching Figure 1: Illustration of SkyStitch. Each quadcopter transmits a live video stream to the ground station, which stitches them together to provide a single composite view of the target area. For example, a UAV could be deployed to hover over a bush- Illustration of SkyStitch UAV – Unmanned Aerial Vehicle http://www.robotshop.com/en/550mm-rtf-quadcopter-uav.html
  • 6. Nagoya University Parallel & Distributed Systems Lab. Introduction SkyStitch - a quadcopter-based HD video stitching system for surveillance Challenges 1. computational cost 2. video stitching (distortion, frame drops, moving objects…) Solutions 1. Reduce workload of the ground station by offloading pre- processing to the UAVs 2. Exploit flight information to speed up feature matching 3. GPU-based RANSAC implementation 4. Utilize optical flow and Kalman filtering 5. Closing loops when stitching multiple video sources 6
  • 7. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 7
  • 8. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 8
  • 9. Nagoya University Parallel & Distributed Systems Lab. Real-time Video Stitching Image Stitching – finding the transformation (H: homography matrix) between two images Estimate H using feature-based method i. feature extraction ii. feature matching iii. Robust homography estimation 9 𝑥" 𝑦" 1 & ~H ) 𝑥 𝑦 1 & mputational ess into sev- Vs. In this g the work- l bottleneck ation is that mation like ht controller perspective and optical term stabil- filter-based H 2. Feature matching 4. Compositing3. Estimation of H1. Feature extraction Image alignment Figure 2: Overview of image stitching. Overview of image stitching
  • 10. Nagoya University Parallel & Distributed Systems Lab. Offloading Feature Extraction Traditional stitching system – Transmits only video frames Feature extraction is performed on ground station. It would become bottleneck when the # of UAVs increase. SkyStitch – Transmits video frames and corresponding features Feature extraction is performed on UAVs*. * 1,000 features extraction from 1280×1024 on NVIDIA Jetson TK1(192 CUDA cores) - less than 35 ms 10
  • 11. Nagoya University Parallel & Distributed Systems Lab. Exploiting Flight Status Information Difficulty in achieving fast and accurate stitching 1. perspective distortion due to wind turbulence → Orthorectification of Perspective Distortion 2. time-consuming feature matching between two images ex) two images with 1,000 features - 1,000,000 combination! → Search Region Reduction for Feature Matching 11
  • 12. Nagoya University Parallel & Distributed Systems Lab. Orthorectification of Perspective Distortion Orthorectification – Transform the image so that it looks as if the camera is pointed vertically downwards. ü Transformation matrix – downwards camera direction to the tilted direction R = 1 0 0 0 cos 𝜃1 sin 𝜃1 0 −sin 𝜃1 cos 𝜃1 cos 𝜃5 0 −sin 𝜃5 0 1 0 sin 𝜃5 0 cos 𝜃5 cos 𝜃6 sin 𝜃6 0 −sin 𝜃6 cos 𝜃6 0 0 0 1 ü Orthorectification matrix M = KBR:;B:;K:; K: transform the pixel into world coodinates (camera intrinsic matrix) B: transform camera into flight controller Orthorectified coordinates can be expressed by 𝑥" 𝑦" 𝑧′ & ~M 𝑥 𝑦 1 & 12 Pitch Yaw Roll Heading Figure 3: Flight attitude of quadcopter: roll, pitch and yaw. N Flight attitude of quadcopter: roll, pitch and yaw Roll Pitch Yaw obtain from flight controller
  • 13. Nagoya University Parallel & Distributed Systems Lab. Search Region Reduction for Feature Estimation To reduce feature matching time, GPS location is exploited to find the mapping of pixel locations between two images. 1. transform the pixel locations from pixel coordinate to a world coordinate system 2. Two image in world coordinate systems can be correlated using the difference in the GPS locations. 13 P r Feature point transformed from Image 1 Bins to be searched for the matched feature Image 2 Figure 5: Search region reduction for fea- ture matching. Search region reduction for feature matching ch opter: BA P P A B P Actual scenario North Pixel coordinates (after orthorectification) (a) Mapping a pixel location from one image to another. CameraCamera GPS difference P (b) Merging coordinate systems. Figure 4: Overlap determination using flight status information. , yw, zw) can be obtained by, To further improve RANSAC performance when the number of Overlap determination using flight status information Pixel coordinate (in the picture) 𝑥" , 𝑦" , 𝑧′ World coordinate (on the ground) 𝑥?, 𝑦?, 𝑧? = 𝑠 𝑥′ 𝑧′ − 𝑊 2 , 𝑠 𝑦′ 𝑧′ − 𝐻 2 , ℎ 𝑠 = 2ℎ tan 𝜃G 2 𝑊H s: # of meters representedby the side of a pixel W×H: image resolution (in pixel) 𝜃G: angle of horizontal field of view in the camera h: quadcopter altitude
  • 14. Nagoya University Parallel & Distributed Systems Lab. Optimizing RANSAC RANSAC (Random Sample Consensus) – method of removing outliers i. Select four matched features to compute homography matrix H ii. Apply estimated H on all the matched features iii. Repeat i-ii for N iterations and find the one with the most inliers Large N improves H but also increases computation time GPU-accelerated RANSAC Ø Gauss-Jordan elimination instead of SVD (Singular Value Decomposition) Ø Maximize parallelism and minimize IO Ø Improve score calculation (use the distribution of inliers) 14 P r point transformed from Image 1 e searched for the matched feature 2 earch region reduction for fea- ng. H H H H H 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 ... ... ... ... qp qp qp qp qp qp qp qp qp Uploaded from CPU Uploaded from CPU ... Reprojection Error Pair Pair ScoresInlier maskAll matchescorrespondences 4−point homographies Candidate Gauss Jordan Elimination Pair ... Pair DownloadedDownloaded from GPU from GPU ... ... ... i1, j1 ik, jk i1, j1 ik, jk Hn−1 Hn−1 H0 H0 Figure 6: RANSAC GPU Pipeline Camera 1 Camera 2 Optical FlowStitching H1 H2 RANSAC GPU Pipeline
  • 15. Nagoya University Parallel & Distributed Systems Lab. Optimizing Video Stitching Quality Causes of stitching failures ü small overlapping area between two images ü concentration of inliers selected by RANSAC in a small region How to detect ill-formed H and prevent them from affecting the perceptual experience? Ø # of inliers produced by H If insufficient, discard the current frame Ø the angle between the two image planes - 𝜃 H = R + tn& (R: 3×3 rotation matrix between two images) cos 𝜃 = 𝑟KK (angle around z-axis) If 𝜃 > 5°, H is erronous 15 P r Feature point transformed from Image 1 Bins to be searched for the matched feature Image 2 Figure 5: Search region reduction for fea- ture matching. H H H H H 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 ... Uploaded from CPU ... correspondences 4−point homographies Candidate Gauss Jordan Elimination Figu (a) Inaccurate rotation. Notice the broken and bent touchline of the football field, below the running track. (b) Perspective distortion. Notice that the left part of the image is erroneously warped and cannot be perfectly aligned with the right im- age. Figure 7: Examples of perceptual distortion. Each blue cross rep- resents a feature point. When there are sufficient inliers, we also check whether H badly warps the images and introduces perspective distortion. Let θ be the Fram Fram Cam Figur optica forme was n ing to 3.4. Example of perceptual distortion
  • 16. Nagoya University Parallel & Distributed Systems Lab. Optimizing Video Stitching Quality – Recovering from Failed Stitching Prediction of H by utilizing the optical flow of adjacent frames to reduce frame drops due to ill-formed H HOP; = MQ,OP;FQ,OP;MQ,O :; HOM;,OF;,OP; :; M;,OP; :; F: 3×3 homography matrix of optical flow M: orthorectification matrix Hn : transformation matrix of the previous stitching 16 ge 1 d feature ion for fea- H H H H H 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 ... ... ... ... qp qp qp qp qp qp qp qp qp Uploaded from CPU Uploaded from CPU... Reprojection Error Pair Pair ScoresInlier maskAll matchescorrespondences 4−point homographies Candidate Gauss Jordan Elimination Pair ... Pair DownloadedDownloaded from GPU from GPU ... ... ... i1, j1 ik, jk i1, j1 ik, jk Hn−1 Hn−1 H0 H0 Figure 6: RANSAC GPU Pipeline (b) Perspective distortion. Notice that the left part of the image is erroneously warped and cannot be perfectly aligned with the right im- age. ual distortion. Each blue cross rep- Frame Frame Frame Frame Camera 1 Camera 2 F1,n+1 F2,n+1 Hn+1 Hn n n n+1 n+1 Figure 8: Predict Hn+1 using optical flow when Hn+1 is ill- formed. Optical FlowStitching Translation Matched features Solve Measurement Prediction MEKF H1 H2 n2t2R2R1t1n1 R′ H′ t′ Figure 9: Fusing information from both stitching and optical flow. Predict Hn+1 using optical flow when Hn+1 is ill-formed
  • 17. Nagoya University Parallel & Distributed Systems Lab. Optimizing Video Stitching Quality – Fusing Stitching and Prediction Filtering techniques to achieve a better estimate by considering both stitching and prediction results i. Two homography matrices are decomposed into rotations and translations ii. Rotation matrices(R1,R2) are converted quaternions and fed into a MEKF (Multiplicative Extended Kalman Filter) iii. Compute a least-squares solution of translation t’ using matched features based on the filtered rotation R’ iv. Form a new homography matrix H’ combining R’ and t’ 17 H H H H H 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 ... ... ... ... qp qp qp qp qp qp qp qp qp Uploaded from CPU Uploaded from CPU ... Reprojection Error Pair Pair ScoresInlier maskAll matchescorrespondences 4−point homographies Candidate Gauss Jordan Elimination Pair ... Pair DownloadedDownloaded from GPU from GPU ... ... ... i1, j1 ik, jk i1, j1 ik, jk Hn−1 Hn−1 H0 H0 Figure 6: RANSAC GPU Pipeline ve distortion. Notice part of the image is warped and cannot be ned with the right im- Frame Frame Frame Frame Camera 1 Camera 2 F1,n+1 F2,n+1 Hn+1 Hn n n n+1 n+1 Figure 8: Predict Hn+1 using optical flow when Hn+1 is ill- formed. Optical FlowStitching Translation Matched features Solve Measurement Prediction MEKF H1 H2 n2t2R2R1t1n1 R′ H′ t′ Figure 9: Fusing information from both stitching and optical flow. Fusing information from both stitching and optical flow
  • 18. Nagoya University Parallel & Distributed Systems Lab. Optimizing Video Stitching Quality – GPS Drift Compensation GPS drift error can be estimated by comparing the ground truth with the transformation → Feed this estimated GPS error to feature matching and reduce search region 18
  • 19. Nagoya University Parallel & Distributed Systems Lab. Multiple Video Sources Stitching multiple video frames can be done by iteratively performing pairwise stitching based on grid-like pattern. ex) Stitching 6 images Pairwise stitching <I1,I2>,<I2,I3>,<I3,I4>…<I6,I1> 19 I1 I2 I3 I6 I5 I4 Figure 10: A loop sequence of stitching 6 images. Figure 11: Our self-as quadcopter. (Section 3.2.2), we can estimate the GPS error. Subseq A Loop sequence of stitching 6 images.
  • 20. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 20
  • 21. Nagoya University Parallel & Distributed Systems Lab. System Implementation 21 Components Model Flight controller APM 2.6 GPS receiver uBlox LEA-6H Motor SunnySky X3108S Electric speed controller Hobby Wing 30AOPTO Propeller APC 12×3.8 Airframe Hobby KingX550 On-board computer NVIDIA Jetson TK1 Camera PointGrey BlackFly 14S2C Lens Computer H0514-MP2 WiFi adapter Compex WLE350NX uence of Figure 11: Our self-assembled quadcopter. n estimate the GPS error. Subsequently, we S error to feature matching and reduce r by Table 1: Major components used in our quadc weight (including components not listed) is 2.1 ki total cost is about USD1,200. Components Mod Flight controller APM GPS receiver uBlox LE Motor SunnySky Electronic speed controller HobbyWing 3 Propeller APC 12 Airframe HobbyKin On-board computer NVIDIA Jet Self-assembled quadcopter Video Synchronization If not synchronized… Ghosting – a mobile object on the ground appear at different locations from differnet cameras → Hardware-based video synchronization using GPS receiver Major components used in the quadcopter. Total weight: 2.1 kilograms, Total cost: USD 1,200
  • 22. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 22
  • 23. Nagoya University Parallel & Distributed Systems Lab. Evaluation Venue ü Grass field – rich in features ü Running track – not have many good features Altitude – 20m Benchmark for comparison - OpenCV (CPU and GPU) stitching algorithms 23
  • 24. Nagoya University Parallel & Distributed Systems Lab. Stitching Time – Execution Time of Individual Step 24 0 50 100 150 200 250 300 2 4 6 8 10 12 Featureextractiontime(ms) Number of video sources benchmark, CPU benchmark, GPU SkyStitch (a) Feature extraction. 0 2 4 6 8 10 12 14 16 18 20 2 4 6 8 10 12 Featurematchingtime(ms) Number of video sources benchmark, GPU SkyStitch (b) Feature matching. 0 20 40 60 80 100 120 140 160 180 2 4 6 8 10 12 RANSACtime(ms) Number of video sources benchmark, CPU benchmark, GPU SkyStitch (c) RANSAC. Figure 12: Execution time of each step of video stitching. The number of features extracted from every frame is 1,000. 5 10 15 20 25 30 35 40 Stitchingrate(fps) SkyStitch Benchmark GPU Benchmark CPU 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Featurematchingtime(ms) benchmark, GPU SkyStitch 0 0.5 1 1.5 2 Featurematchingtime(ms) benchmark, GPU SkyStitch Execution time of each step of video stitching SkyStitch:scalable to many video sources because feature extraction is offloaded to each quadcopter Behchmarks:proportional to the # of video sources ×4 (×70 over CPU) # of features: 1,000 Overlap: 80% 12 video sources (2×6 grid) N>7: slopes becomes steeper due to 2×6 grid pattern ×18 ×24 Gap CPU-GPU is not large. - SVD solver not run GPU efficiently - Many CPU-GPU memory transcation
  • 25. Nagoya University Parallel & Distributed Systems Lab. Stitching Time – Stitching Rate 0 50 100 2 4 6 8 10 12 Featureext Number of video sources (a) Feature extraction. 0 2 4 6 8 2 4 6 8 1 Featurema Number of video sourc (b) Feature matching. Figure 12: Execution time of each step of video stitching. The number 0 5 10 15 20 25 30 35 40 2 4 6 8 10 12 Stitchingrate(fps) Number of video sources SkyStitch Benchmark GPU Benchmark CPU Figure 13: Stitching rate (in terms of frames per second) with re- spect to the number of video sources. time of the remaining steps (i.e., feature matching, RANSAC and image compositing). Then the execution time per stitching is equal 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 500 Featurematchingtime(ms) (a) Imp Figure 1 executio 25 Stitching rate (frames per second) ×4 over GPU ×16 over CPU Effect of offloading feature extraction to the UAVs exeution time = W 𝑚𝑎𝑥 𝑇[]1^_], 𝑇`]a[1b ⋯SkyStitch 𝑇[]1^_] + 𝑇`]a[1b ⋯Benchmarks Bottleneck of SkyStitch shifts to the ground station
  • 26. Nagoya University Parallel & Distributed Systems Lab. Stitching Time – Impact of Feature Count and Overlap Percentage 26 6 8 10 12 mber of video sources eature matching. 0 2 4 6 8 10 12 Number of video sources (c) RANSAC. ing. The number of features extracted from every frame is 1,000. e- nd al 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 500 600 700 800 900 1000 Featurematchingtime(ms) Number of Features benchmark, GPU SkyStitch (a) Impact of feature count. 0 0.5 1 1.5 2 20 30 40 50 60 70 80 Featurematchingtime(ms) Overlap Percentage benchmark, GPU SkyStitch (b) Impact of overlap. Figure 14: Impact of feature count and overlap percentage on the execution time of feature matching. rate of increase is slower than than benchmark Benchmark – pairwise comparison all the features SkyStitch – search in a small area Because when the overlap is small, there are more stitching failures which lead to a larger search radius. GPU: Independent of the overlapping area
  • 27. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Number of Inliers 27 -3 -2 -1 0 1 2 0 50 100 150 200 250 Angle(d Frames (a) Roll -3 -2 -1 0 1 2 0 50 100 150 20 Angle(d Frames (b) Pitch Figure 15: Jerkiness of rotation components in co 0 50 100 150 200 250 300 350 400 20 30 40 50 60 70 80 90 Numberofinliers Overlap Percentage SkyStitch benchmark (a) Grass field. 0 50 100 150 200 250 300 350 400 20 30 40 50 60 70 80 90 Numberofinliers Overlap Percentage SkyStitch benchmark (b) Running track. Figure 16: Impact of overlap percentage on the number of inliers. The number of features extracted from every frame is 1,000. 100 100 (a) N Figu Impact of overlap percentage on the number of inliers SkyStitch:only searches in a small target area and thus it is less likely to have false matching Benchmark:pairwise comparison among all the features and the probability of false matching is high
  • 28. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Success Rate of Stitching 28 0 50 100 150 20 30 40 50 60 70 80 90 Nu Overlap Percentage (a) Grass field. 0 50 100 150 20 30 40 50 60 70 80 90 Nu Overlap Percentage (b) Running track. Figure 16: Impact of overlap percentage on the number of inliers. The number of features extracted from every frame is 1,000. 0 20 40 60 80 100 20 30 40 50 60 70 80 90 SuccessRate Overlap Percentage SkyStitch benchmark (a) Grass field. 0 20 40 60 80 100 20 30 40 50 60 70 80 90 SuccessRate Overlap Percentage SkyStitch benchmark (b) Running track. Figure 17: Impact of overlap percentage on success rate. make things more intuitive, we convert the rotation matrix into (a Fi (a) U Figure We fo the succ wise sti multi-im video so 97%, wh Impact of overlap percentage on succes rate SkyStitch has higher success rate than the benchmark, because failed stitching can be recovered using optical flow.
  • 29. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Frame Jerkiness 29 Jerkiness of rotation components in computed homographies -3 -2 -1 0 1 2 3 4 5 0 50 100 150 200 250 Angle(deg) Frames Only Stitching Only Optical Flow Kalman Filtered (a) Roll -3 -2 -1 0 1 2 3 4 5 0 50 100 150 200 250 Angle(deg) Frames Only Stitching Only Optical Flow Kalman Filtered (b) Pitch -3 -2 -1 0 1 2 3 4 5 0 50 100 150 200 250 Angle(deg) Frames Only Stitching Only Optical Flow Kalman Filtered (c) Yaw Figure 15: Jerkiness of rotation components in computed homographies 200 250 300 350 400 rofinliers SkyStitch benchmark 200 250 300 350 400 rofinliers SkyStitch benchmark Only stitching: produces very significant jerkiness in pitch and roll Only optical flow: much smoother but suffers from drift especially in yaw and pitch Kalman filtered: angles are not prone to either of the problems
  • 30. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Multiple Video Sources 30 Effect of loop closing for 6-image stitching0 150 200 250 rames Pitch -3 -2 -1 0 50 100 150 200 250 Frames (c) Yaw mponents in computed homographies (a) No loop closing. (b) With loop closing. Figure 18: Effect of loop closing for 6-image stitching. The misalignment is mitigated by applying loop closing method.
  • 31. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Elimination of Ghosting Artefacts 31 Effect of video synchronization on ghost artefacts (a) No loop closing. (b) With loop closing. Figure 18: Effect of loop closing for 6-image stitching. (a) Unsynchronized frames. (b) Synchronized frames. Figure 19: Effect of video synchronization on ghosting artefacts. The image is stitched from two video frames synchronized by GPS.
  • 32. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 32
  • 33. 33
  • 34. Nagoya University Parallel & Distributed Systems Lab. Conclusion SkyStitch – a multi-UAV-based video surveillance system that supports real-time stitching Ø Feature extraction offloading Ø Utilization of flight status information Ø Efficient GPU-based RANSAC implementation Ø Jerkiness mitigation by Kalman filtering 4 times faster than state-of-the-art GPU accelerated method & Good stitching quality Future Work ü Other types of features (color features, low-dimensional feature…) ü Performance in challenging environment (rough ground with buildings…) 34