Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching

1,415 views

Published on

パターン・映像情報処理特論において論文を紹介した時の発表資料です。
Xiangyun Meng, Wei Wang, and Ben Leong. 2015. SkyStitch: A Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching. In Proceedings of the 23rd ACM international conference on Multimedia (MM '15). ACM, New York, NY, USA, 261-270. DOI=http://dx.doi.org/10.1145/2733373.2806225

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching

  1. 1. Nagoya University Parallel & Distributed Systems Lab. パターン・映像情報処理特論 (2016.5.6) SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching Yuki KITSUKAWA yuki@ertl.jp Edahiro & Kato Lab. Graduate School of Information Science Nagoya University 1
  2. 2. Nagoya University Parallel & Distributed Systems Lab. About Paper p Title SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching p Authors Xiangyun Meng, Wei Wang, Ben Leong (National University of Singapore) p Conference Proceedings of the 23rd ACM international conference on Multimedia, 2015 URL - https://www.comp.nus.edu.sg/~bleong/publications/acmmm15-skystitch.pdf 2
  3. 3. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 3
  4. 4. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 4
  5. 5. Nagoya University Parallel & Distributed Systems Lab. Introduction Applications using UAV-based surveillance system ü firefighting ü search and rescue ü Border monitoring To cover large target area… Multiple UAVs are used to capture videos and put them together (stitch) consistently. 5 SkyStitch: a Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching Xiangyun Meng, Wei Wang and Ben Leong School of Computing National University of Singapore idmmeng@nus.edu.sg, weiwang@comp.nus.edu.sg, benleong@comp.nus.edu.sg ABSTRACT Recent advances in unmanned aerial vehicle (UAV) technologies have made it possible to deploy an aerial video surveillance system to provide an unprecedented aerial perspective for ground moni- toring in real time. Multiple UAVs would be required to cover a large target area, and it is difficult for users to visualize the over- all situation if they were to receive multiple disjoint video streams. To address this problem, we designed and implemented SkyStitch, a multiple-UAV video surveillance system that provides a single and panoramic video stream to its users by stitching together mul- tiple aerial video streams. SkyStitch addresses two key design chal- lenges: (i) the high computational cost of stitching and (ii) the diffi- culty of ensuring good stitching quality under dynamic conditions. To improve the speed and quality of video stitching, we incorporate several practical techniques like distributed feature extraction to re- Ground station Video streaming Video capturing Video capturing Video stitching Figure 1: Illustration of SkyStitch. Each quadcopter transmits a live video stream to the ground station, which stitches them together to provide a single composite view of the target area. For example, a UAV could be deployed to hover over a bush- Illustration of SkyStitch UAV – Unmanned Aerial Vehicle http://www.robotshop.com/en/550mm-rtf-quadcopter-uav.html
  6. 6. Nagoya University Parallel & Distributed Systems Lab. Introduction SkyStitch - a quadcopter-based HD video stitching system for surveillance Challenges 1. computational cost 2. video stitching (distortion, frame drops, moving objects…) Solutions 1. Reduce workload of the ground station by offloading pre- processing to the UAVs 2. Exploit flight information to speed up feature matching 3. GPU-based RANSAC implementation 4. Utilize optical flow and Kalman filtering 5. Closing loops when stitching multiple video sources 6
  7. 7. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 7
  8. 8. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 8
  9. 9. Nagoya University Parallel & Distributed Systems Lab. Real-time Video Stitching Image Stitching – finding the transformation (H: homography matrix) between two images Estimate H using feature-based method i. feature extraction ii. feature matching iii. Robust homography estimation 9 𝑥" 𝑦" 1 & ~H ) 𝑥 𝑦 1 & mputational ess into sev- Vs. In this g the work- l bottleneck ation is that mation like ht controller perspective and optical term stabil- filter-based H 2. Feature matching 4. Compositing3. Estimation of H1. Feature extraction Image alignment Figure 2: Overview of image stitching. Overview of image stitching
  10. 10. Nagoya University Parallel & Distributed Systems Lab. Offloading Feature Extraction Traditional stitching system – Transmits only video frames Feature extraction is performed on ground station. It would become bottleneck when the # of UAVs increase. SkyStitch – Transmits video frames and corresponding features Feature extraction is performed on UAVs*. * 1,000 features extraction from 1280×1024 on NVIDIA Jetson TK1(192 CUDA cores) - less than 35 ms 10
  11. 11. Nagoya University Parallel & Distributed Systems Lab. Exploiting Flight Status Information Difficulty in achieving fast and accurate stitching 1. perspective distortion due to wind turbulence → Orthorectification of Perspective Distortion 2. time-consuming feature matching between two images ex) two images with 1,000 features - 1,000,000 combination! → Search Region Reduction for Feature Matching 11
  12. 12. Nagoya University Parallel & Distributed Systems Lab. Orthorectification of Perspective Distortion Orthorectification – Transform the image so that it looks as if the camera is pointed vertically downwards. ü Transformation matrix – downwards camera direction to the tilted direction R = 1 0 0 0 cos 𝜃1 sin 𝜃1 0 −sin 𝜃1 cos 𝜃1 cos 𝜃5 0 −sin 𝜃5 0 1 0 sin 𝜃5 0 cos 𝜃5 cos 𝜃6 sin 𝜃6 0 −sin 𝜃6 cos 𝜃6 0 0 0 1 ü Orthorectification matrix M = KBR:;B:;K:; K: transform the pixel into world coodinates (camera intrinsic matrix) B: transform camera into flight controller Orthorectified coordinates can be expressed by 𝑥" 𝑦" 𝑧′ & ~M 𝑥 𝑦 1 & 12 Pitch Yaw Roll Heading Figure 3: Flight attitude of quadcopter: roll, pitch and yaw. N Flight attitude of quadcopter: roll, pitch and yaw Roll Pitch Yaw obtain from flight controller
  13. 13. Nagoya University Parallel & Distributed Systems Lab. Search Region Reduction for Feature Estimation To reduce feature matching time, GPS location is exploited to find the mapping of pixel locations between two images. 1. transform the pixel locations from pixel coordinate to a world coordinate system 2. Two image in world coordinate systems can be correlated using the difference in the GPS locations. 13 P r Feature point transformed from Image 1 Bins to be searched for the matched feature Image 2 Figure 5: Search region reduction for fea- ture matching. Search region reduction for feature matching ch opter: BA P P A B P Actual scenario North Pixel coordinates (after orthorectification) (a) Mapping a pixel location from one image to another. CameraCamera GPS difference P (b) Merging coordinate systems. Figure 4: Overlap determination using flight status information. , yw, zw) can be obtained by, To further improve RANSAC performance when the number of Overlap determination using flight status information Pixel coordinate (in the picture) 𝑥" , 𝑦" , 𝑧′ World coordinate (on the ground) 𝑥?, 𝑦?, 𝑧? = 𝑠 𝑥′ 𝑧′ − 𝑊 2 , 𝑠 𝑦′ 𝑧′ − 𝐻 2 , ℎ 𝑠 = 2ℎ tan 𝜃G 2 𝑊H s: # of meters representedby the side of a pixel W×H: image resolution (in pixel) 𝜃G: angle of horizontal field of view in the camera h: quadcopter altitude
  14. 14. Nagoya University Parallel & Distributed Systems Lab. Optimizing RANSAC RANSAC (Random Sample Consensus) – method of removing outliers i. Select four matched features to compute homography matrix H ii. Apply estimated H on all the matched features iii. Repeat i-ii for N iterations and find the one with the most inliers Large N improves H but also increases computation time GPU-accelerated RANSAC Ø Gauss-Jordan elimination instead of SVD (Singular Value Decomposition) Ø Maximize parallelism and minimize IO Ø Improve score calculation (use the distribution of inliers) 14 P r point transformed from Image 1 e searched for the matched feature 2 earch region reduction for fea- ng. H H H H H 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 ... ... ... ... qp qp qp qp qp qp qp qp qp Uploaded from CPU Uploaded from CPU ... Reprojection Error Pair Pair ScoresInlier maskAll matchescorrespondences 4−point homographies Candidate Gauss Jordan Elimination Pair ... Pair DownloadedDownloaded from GPU from GPU ... ... ... i1, j1 ik, jk i1, j1 ik, jk Hn−1 Hn−1 H0 H0 Figure 6: RANSAC GPU Pipeline Camera 1 Camera 2 Optical FlowStitching H1 H2 RANSAC GPU Pipeline
  15. 15. Nagoya University Parallel & Distributed Systems Lab. Optimizing Video Stitching Quality Causes of stitching failures ü small overlapping area between two images ü concentration of inliers selected by RANSAC in a small region How to detect ill-formed H and prevent them from affecting the perceptual experience? Ø # of inliers produced by H If insufficient, discard the current frame Ø the angle between the two image planes - 𝜃 H = R + tn& (R: 3×3 rotation matrix between two images) cos 𝜃 = 𝑟KK (angle around z-axis) If 𝜃 > 5°, H is erronous 15 P r Feature point transformed from Image 1 Bins to be searched for the matched feature Image 2 Figure 5: Search region reduction for fea- ture matching. H H H H H 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 ... Uploaded from CPU ... correspondences 4−point homographies Candidate Gauss Jordan Elimination Figu (a) Inaccurate rotation. Notice the broken and bent touchline of the football field, below the running track. (b) Perspective distortion. Notice that the left part of the image is erroneously warped and cannot be perfectly aligned with the right im- age. Figure 7: Examples of perceptual distortion. Each blue cross rep- resents a feature point. When there are sufficient inliers, we also check whether H badly warps the images and introduces perspective distortion. Let θ be the Fram Fram Cam Figur optica forme was n ing to 3.4. Example of perceptual distortion
  16. 16. Nagoya University Parallel & Distributed Systems Lab. Optimizing Video Stitching Quality – Recovering from Failed Stitching Prediction of H by utilizing the optical flow of adjacent frames to reduce frame drops due to ill-formed H HOP; = MQ,OP;FQ,OP;MQ,O :; HOM;,OF;,OP; :; M;,OP; :; F: 3×3 homography matrix of optical flow M: orthorectification matrix Hn : transformation matrix of the previous stitching 16 ge 1 d feature ion for fea- H H H H H 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 ... ... ... ... qp qp qp qp qp qp qp qp qp Uploaded from CPU Uploaded from CPU... Reprojection Error Pair Pair ScoresInlier maskAll matchescorrespondences 4−point homographies Candidate Gauss Jordan Elimination Pair ... Pair DownloadedDownloaded from GPU from GPU ... ... ... i1, j1 ik, jk i1, j1 ik, jk Hn−1 Hn−1 H0 H0 Figure 6: RANSAC GPU Pipeline (b) Perspective distortion. Notice that the left part of the image is erroneously warped and cannot be perfectly aligned with the right im- age. ual distortion. Each blue cross rep- Frame Frame Frame Frame Camera 1 Camera 2 F1,n+1 F2,n+1 Hn+1 Hn n n n+1 n+1 Figure 8: Predict Hn+1 using optical flow when Hn+1 is ill- formed. Optical FlowStitching Translation Matched features Solve Measurement Prediction MEKF H1 H2 n2t2R2R1t1n1 R′ H′ t′ Figure 9: Fusing information from both stitching and optical flow. Predict Hn+1 using optical flow when Hn+1 is ill-formed
  17. 17. Nagoya University Parallel & Distributed Systems Lab. Optimizing Video Stitching Quality – Fusing Stitching and Prediction Filtering techniques to achieve a better estimate by considering both stitching and prediction results i. Two homography matrices are decomposed into rotations and translations ii. Rotation matrices(R1,R2) are converted quaternions and fed into a MEKF (Multiplicative Extended Kalman Filter) iii. Compute a least-squares solution of translation t’ using matched features based on the filtered rotation R’ iv. Form a new homography matrix H’ combining R’ and t’ 17 H H H H H 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 ... ... ... ... qp qp qp qp qp qp qp qp qp Uploaded from CPU Uploaded from CPU ... Reprojection Error Pair Pair ScoresInlier maskAll matchescorrespondences 4−point homographies Candidate Gauss Jordan Elimination Pair ... Pair DownloadedDownloaded from GPU from GPU ... ... ... i1, j1 ik, jk i1, j1 ik, jk Hn−1 Hn−1 H0 H0 Figure 6: RANSAC GPU Pipeline ve distortion. Notice part of the image is warped and cannot be ned with the right im- Frame Frame Frame Frame Camera 1 Camera 2 F1,n+1 F2,n+1 Hn+1 Hn n n n+1 n+1 Figure 8: Predict Hn+1 using optical flow when Hn+1 is ill- formed. Optical FlowStitching Translation Matched features Solve Measurement Prediction MEKF H1 H2 n2t2R2R1t1n1 R′ H′ t′ Figure 9: Fusing information from both stitching and optical flow. Fusing information from both stitching and optical flow
  18. 18. Nagoya University Parallel & Distributed Systems Lab. Optimizing Video Stitching Quality – GPS Drift Compensation GPS drift error can be estimated by comparing the ground truth with the transformation → Feed this estimated GPS error to feature matching and reduce search region 18
  19. 19. Nagoya University Parallel & Distributed Systems Lab. Multiple Video Sources Stitching multiple video frames can be done by iteratively performing pairwise stitching based on grid-like pattern. ex) Stitching 6 images Pairwise stitching <I1,I2>,<I2,I3>,<I3,I4>…<I6,I1> 19 I1 I2 I3 I6 I5 I4 Figure 10: A loop sequence of stitching 6 images. Figure 11: Our self-as quadcopter. (Section 3.2.2), we can estimate the GPS error. Subseq A Loop sequence of stitching 6 images.
  20. 20. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 20
  21. 21. Nagoya University Parallel & Distributed Systems Lab. System Implementation 21 Components Model Flight controller APM 2.6 GPS receiver uBlox LEA-6H Motor SunnySky X3108S Electric speed controller Hobby Wing 30AOPTO Propeller APC 12×3.8 Airframe Hobby KingX550 On-board computer NVIDIA Jetson TK1 Camera PointGrey BlackFly 14S2C Lens Computer H0514-MP2 WiFi adapter Compex WLE350NX uence of Figure 11: Our self-assembled quadcopter. n estimate the GPS error. Subsequently, we S error to feature matching and reduce r by Table 1: Major components used in our quadc weight (including components not listed) is 2.1 ki total cost is about USD1,200. Components Mod Flight controller APM GPS receiver uBlox LE Motor SunnySky Electronic speed controller HobbyWing 3 Propeller APC 12 Airframe HobbyKin On-board computer NVIDIA Jet Self-assembled quadcopter Video Synchronization If not synchronized… Ghosting – a mobile object on the ground appear at different locations from differnet cameras → Hardware-based video synchronization using GPS receiver Major components used in the quadcopter. Total weight: 2.1 kilograms, Total cost: USD 1,200
  22. 22. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 22
  23. 23. Nagoya University Parallel & Distributed Systems Lab. Evaluation Venue ü Grass field – rich in features ü Running track – not have many good features Altitude – 20m Benchmark for comparison - OpenCV (CPU and GPU) stitching algorithms 23
  24. 24. Nagoya University Parallel & Distributed Systems Lab. Stitching Time – Execution Time of Individual Step 24 0 50 100 150 200 250 300 2 4 6 8 10 12 Featureextractiontime(ms) Number of video sources benchmark, CPU benchmark, GPU SkyStitch (a) Feature extraction. 0 2 4 6 8 10 12 14 16 18 20 2 4 6 8 10 12 Featurematchingtime(ms) Number of video sources benchmark, GPU SkyStitch (b) Feature matching. 0 20 40 60 80 100 120 140 160 180 2 4 6 8 10 12 RANSACtime(ms) Number of video sources benchmark, CPU benchmark, GPU SkyStitch (c) RANSAC. Figure 12: Execution time of each step of video stitching. The number of features extracted from every frame is 1,000. 5 10 15 20 25 30 35 40 Stitchingrate(fps) SkyStitch Benchmark GPU Benchmark CPU 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Featurematchingtime(ms) benchmark, GPU SkyStitch 0 0.5 1 1.5 2 Featurematchingtime(ms) benchmark, GPU SkyStitch Execution time of each step of video stitching SkyStitch:scalable to many video sources because feature extraction is offloaded to each quadcopter Behchmarks:proportional to the # of video sources ×4 (×70 over CPU) # of features: 1,000 Overlap: 80% 12 video sources (2×6 grid) N>7: slopes becomes steeper due to 2×6 grid pattern ×18 ×24 Gap CPU-GPU is not large. - SVD solver not run GPU efficiently - Many CPU-GPU memory transcation
  25. 25. Nagoya University Parallel & Distributed Systems Lab. Stitching Time – Stitching Rate 0 50 100 2 4 6 8 10 12 Featureext Number of video sources (a) Feature extraction. 0 2 4 6 8 2 4 6 8 1 Featurema Number of video sourc (b) Feature matching. Figure 12: Execution time of each step of video stitching. The number 0 5 10 15 20 25 30 35 40 2 4 6 8 10 12 Stitchingrate(fps) Number of video sources SkyStitch Benchmark GPU Benchmark CPU Figure 13: Stitching rate (in terms of frames per second) with re- spect to the number of video sources. time of the remaining steps (i.e., feature matching, RANSAC and image compositing). Then the execution time per stitching is equal 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 500 Featurematchingtime(ms) (a) Imp Figure 1 executio 25 Stitching rate (frames per second) ×4 over GPU ×16 over CPU Effect of offloading feature extraction to the UAVs exeution time = W 𝑚𝑎𝑥 𝑇[]1^_], 𝑇`]a[1b ⋯SkyStitch 𝑇[]1^_] + 𝑇`]a[1b ⋯Benchmarks Bottleneck of SkyStitch shifts to the ground station
  26. 26. Nagoya University Parallel & Distributed Systems Lab. Stitching Time – Impact of Feature Count and Overlap Percentage 26 6 8 10 12 mber of video sources eature matching. 0 2 4 6 8 10 12 Number of video sources (c) RANSAC. ing. The number of features extracted from every frame is 1,000. e- nd al 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 500 600 700 800 900 1000 Featurematchingtime(ms) Number of Features benchmark, GPU SkyStitch (a) Impact of feature count. 0 0.5 1 1.5 2 20 30 40 50 60 70 80 Featurematchingtime(ms) Overlap Percentage benchmark, GPU SkyStitch (b) Impact of overlap. Figure 14: Impact of feature count and overlap percentage on the execution time of feature matching. rate of increase is slower than than benchmark Benchmark – pairwise comparison all the features SkyStitch – search in a small area Because when the overlap is small, there are more stitching failures which lead to a larger search radius. GPU: Independent of the overlapping area
  27. 27. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Number of Inliers 27 -3 -2 -1 0 1 2 0 50 100 150 200 250 Angle(d Frames (a) Roll -3 -2 -1 0 1 2 0 50 100 150 20 Angle(d Frames (b) Pitch Figure 15: Jerkiness of rotation components in co 0 50 100 150 200 250 300 350 400 20 30 40 50 60 70 80 90 Numberofinliers Overlap Percentage SkyStitch benchmark (a) Grass field. 0 50 100 150 200 250 300 350 400 20 30 40 50 60 70 80 90 Numberofinliers Overlap Percentage SkyStitch benchmark (b) Running track. Figure 16: Impact of overlap percentage on the number of inliers. The number of features extracted from every frame is 1,000. 100 100 (a) N Figu Impact of overlap percentage on the number of inliers SkyStitch:only searches in a small target area and thus it is less likely to have false matching Benchmark:pairwise comparison among all the features and the probability of false matching is high
  28. 28. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Success Rate of Stitching 28 0 50 100 150 20 30 40 50 60 70 80 90 Nu Overlap Percentage (a) Grass field. 0 50 100 150 20 30 40 50 60 70 80 90 Nu Overlap Percentage (b) Running track. Figure 16: Impact of overlap percentage on the number of inliers. The number of features extracted from every frame is 1,000. 0 20 40 60 80 100 20 30 40 50 60 70 80 90 SuccessRate Overlap Percentage SkyStitch benchmark (a) Grass field. 0 20 40 60 80 100 20 30 40 50 60 70 80 90 SuccessRate Overlap Percentage SkyStitch benchmark (b) Running track. Figure 17: Impact of overlap percentage on success rate. make things more intuitive, we convert the rotation matrix into (a Fi (a) U Figure We fo the succ wise sti multi-im video so 97%, wh Impact of overlap percentage on succes rate SkyStitch has higher success rate than the benchmark, because failed stitching can be recovered using optical flow.
  29. 29. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Frame Jerkiness 29 Jerkiness of rotation components in computed homographies -3 -2 -1 0 1 2 3 4 5 0 50 100 150 200 250 Angle(deg) Frames Only Stitching Only Optical Flow Kalman Filtered (a) Roll -3 -2 -1 0 1 2 3 4 5 0 50 100 150 200 250 Angle(deg) Frames Only Stitching Only Optical Flow Kalman Filtered (b) Pitch -3 -2 -1 0 1 2 3 4 5 0 50 100 150 200 250 Angle(deg) Frames Only Stitching Only Optical Flow Kalman Filtered (c) Yaw Figure 15: Jerkiness of rotation components in computed homographies 200 250 300 350 400 rofinliers SkyStitch benchmark 200 250 300 350 400 rofinliers SkyStitch benchmark Only stitching: produces very significant jerkiness in pitch and roll Only optical flow: much smoother but suffers from drift especially in yaw and pitch Kalman filtered: angles are not prone to either of the problems
  30. 30. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Multiple Video Sources 30 Effect of loop closing for 6-image stitching0 150 200 250 rames Pitch -3 -2 -1 0 50 100 150 200 250 Frames (c) Yaw mponents in computed homographies (a) No loop closing. (b) With loop closing. Figure 18: Effect of loop closing for 6-image stitching. The misalignment is mitigated by applying loop closing method.
  31. 31. Nagoya University Parallel & Distributed Systems Lab. Stitching Quality – Elimination of Ghosting Artefacts 31 Effect of video synchronization on ghost artefacts (a) No loop closing. (b) With loop closing. Figure 18: Effect of loop closing for 6-image stitching. (a) Unsynchronized frames. (b) Synchronized frames. Figure 19: Effect of video synchronization on ghosting artefacts. The image is stitched from two video frames synchronized by GPS.
  32. 32. Nagoya University Parallel & Distributed Systems Lab. Contents 1. Introduction 2. Related Work 3. Real-time Video Stitching I. Offloading Feature Extraction II. Exploiting Flight Status Information III. Optimizing RANSAC IV. Optimizing Video Stitching Quality V. Multiple Video Sources 4. System Implementation 5. Evaluation I. Stitching Time II. Stitching Quality 6. Conclusion 32
  33. 33. 33
  34. 34. Nagoya University Parallel & Distributed Systems Lab. Conclusion SkyStitch – a multi-UAV-based video surveillance system that supports real-time stitching Ø Feature extraction offloading Ø Utilization of flight status information Ø Efficient GPU-based RANSAC implementation Ø Jerkiness mitigation by Kalman filtering 4 times faster than state-of-the-art GPU accelerated method & Good stitching quality Future Work ü Other types of features (color features, low-dimensional feature…) ü Performance in challenging environment (rough ground with buildings…) 34

×