Download to read offline
![Move along x-axis with various
attitude
X
Y
Verification Experiment of Image Algorithm Fused with IMU data
5 10 15 20 25 30
- 0.4
- 0.2
0
0.2
0.4
Time [s ]
X,Ydistance[m]](https://image.slidesharecdn.com/lrzhxxdtzioclexisd7z-signature-5e8d2c37827bd56603d99b1f6f446b66a2ef9224453fa704beeaf40d5e4bc784-poli-150722184151-lva1-app6891/85/P1131210137-18-320.jpg)
Uploaded byAshraf Aboshosha
P1131210137
1) The document proposes using an embedded stereo camera and fusing optical flow and SIFT feature matching algorithms to estimate the localization of a micro aerial vehicle (MAV) in GPS-denied environments. 2) An Extended Kalman Filter is used to estimate the MAV's translational velocity and altitude from optical flow measurements separated into rotational and translational components using IMU data. 3) Initial experiments fusing optical flow and SIFT matching for altitude estimation showed promising results compared to ground truth, with room for improvement through onboard processing and successive frame SIFT matching for horizontal position estimation.
More Related Content
![6 [progress report] for this leisurely side-project I was doing in 2016](https://cdn.slidesharecdn.com/ss_thumbnails/6progressreportforthisleisurelyside-projectiwasdoingin2016-210110122806-thumbnail.jpg?width=640&height=640&fit=bounds)
P1131210137
- 1. Localization of MAVin GPS- denied Environment Using Embedded Stereo Camera Syaril Azrad, Farid Kendoul,Fadhil Mohamad,, Kenzo Nonami Department of Mechanical Engineering,Nonami Lab, Chiba University
- 2. Research Background • Visionbased Autonomous Q-Rotor Research in Chiba University – Started in 2008 , Participate in 1st US-Asian Demonstration and Assessment of Micro Aerial and , MAV’08 Agra, India, – Single camera approach • Color-based object tracking algoritm (visual-servoing) • Feature-based/Optical Flow (OF) – Stereo camera approach • Ground stereo camera based hovering • Fully-embedded based object tracking
- 3. Current Research Background •Localization of MAV using embedded camera – Research has been conducted by Farid et. al using single embedded camera using Optical Flow to localize rotorcraft position – Height estimation improvement • Fusion with Pressure Sensor • Using stereo camera for higher-precision
- 4. Altitude Estimation ForMAVs ? • GPS • Pressure sensor • Laser Range Finder • Radar Sensors • Vision System – Single Camera – Stereo Camera
- 5. For Small UAVs(MAV) Altitude Estimation We propose… • Embedded Lightweight Stereo Camera • Fusion Optical Flow (SFM algorithm) with Image Homography • Fusion Optical Flow (SFM algorithm) with Scale Invariant Feature Transform (SIFT) –based feature matching
- 6. Our Platform Introduction •Quadrotor Air-frame • Micro-controller • Sensors • Vision System
- 7. 18th Aug. 2010 MOVIC2010 Tokyo,JAPAN 7 •Size : 54cm x 54cm x 20cm •Total weight : < 700 grams •Flight time : 10 min •Range : 1~2 km •Total cost : < 4000 USD
- 8. Proposed Vision-based MAV localizationalgorithm • Horizontal Position using Optical Flow- based algorithm • Altitude Position using Optical Flow fused with Homography/SIFT-based matching approach
- 9. 18th Aug. 2010 MOVIC2010 Tokyo,JAPAN 9 Computing Optical Flow Implementation We use Lucas-Kanade Method
- 10. 18th Aug. 2010 MOVIC2010 Tokyo,JAPAN 10 the feature point (x, y) in the next image I2 will be at (x+dx, y+dy), and the same process will be repeated in the next step (t +Δt), providing the optical flow and a simple feature tracking algorithm.
- 11. Horizontal Position Estimation(OF) Imaging Model
- 12. Or can beexpressed as below Then we can express the OF equation as below WHAT DOES THE EQUATION ABOVE MEANS? The velocity of can be expressed as Pi(xi ,yi) A perspective-central camera model maps Pi(xi ,yi)
- 13. If we havedata from IMU about the rotational velocity of the camera on the body of our MAVs we can get purely translational part Meaning = velocity and position Because OF calculated from images contains rotational and translational part Kendoul et. al (2007)
- 14. The strategy isan estimation problem with the state vector So for KF dynamics model, we can write the measurement equation plus noise With H is as below, from our optical flow expression in equation (5)
- 15. Now, after estimatingOftrans, we can use them for measurement vector for (MAV)translational velocity and structure parameter Both our cameras are mounted on the MAV, and assuming camera motion is smooth, we can write the dynamic model as below with γ is the camera acc. We use the model proposed by Kendoul et. al for depth (altitude) as below We can write the discrete dynamic system using the state vector X
- 16. Which is non-linear,we Implement Extended Kalman Filter as proposed by Kendoul et. A and estimate translational velocity and depth (altitude) While the observation of the discrete model is as below
- 17. OF raw datafrom camera, Attitude data from IMU Estimate the translational part of OF, separate the rotational part Estimate the camera velocity and depth(altitude)
- 18. Move along x-axiswith various attitude X Y Verification Experiment of Image Algorithm Fused with IMU data 5 10 15 20 25 30 - 0.4 - 0.2 0 0.2 0.4 Time [s ] X,Ydistance[m]
- 19. Localization of MAVusing Optical Flow and SIFT-matching technique • SIFT? – Detect & describe feature in local image • In our computation we use SIFTGPU (Wu,2009) to speed up the matching • Matching result is filtered by RANSAC algorithm to separate between outlier in inlier • Applying threads in computations we can get the Optical Flow based algorithm 50fps and SIFT matching 7-8fps. This inculding triangulation
- 20. Manual flying testwith embedded stereo camera
- 21.
- 22.
- 23. Implementation Strategy 1. ViconData 2. Control Program 3. Receive Image Data 4. Process Image Data Image Processing 1.Optical Flow Base 2.Stereo Feature Matching GPU Graphical Processing Unit Socket 1 Computer with two separate Process, or 2 Computers
- 24. Implementation • Implemented on1 Computer due to high capability, but have 2 share GPU • Core-i7 (4 Cores 8 Threads) – Separate Core for Image Processing – Separate Thread for Control – Separate Core for Receiving Image Data – Separate Thread for Vicon Data
- 25. Results • Process thathas been implemented with stable frequency – PID Control (30Hz) – Vicon Data Acquisition (30Hz) – Image Processing SIFT (8Hz) and Optical Flow (15Hz)
- 26. Improvement of resultstrategy
- 27.
- 28.
- 29. Future Work &Suggestions • Apply SIFTGpu for horizontal odometry estimation and fuse with OF base horizontal odometry – Successive frame feature matching • On-board camera/ image processing development, sharing technology.
- 30.
Editor's Notes
- #5 Standard GPS vertical precision between 25m to 50m Pressure sensor depends on environment will have an error between 6% and 7% moreover, pressure sensor only calculate relative height Laser altimeter is good but for our type of MAV it would be not suitable due to its energy consumption and surface requirement Radar sensor would not be suitable due to its high cost and energy consuming Thus we still think the vision system is one of the best alternative. However due to its high computing cost requirement, depending on algorithm there is an issue whether we can implement on-board or host-based.
- #8 Our platform is off-the shelf platform from Ascending Technologies, German. The size of our platform is 54 cm in diameters, and 20cm of height, with total weight of less than 700 grams, and flight time up to 12 minutes. However we state here 10 minutes for safety purposes. The flight range would be 1 to 2 kilometers. And the total cost is less than 4000 USD, which is around 350 thousand yen. The details of the price can be seen on the right corner table. In this platform we incorporated the flight computer GUMSTIX, which has a computing speed of 400MHz, equipped with wi-fi. The GUMSTIX is linux-based. We also use the Crossbow MNAV IMU, pressure, gyro sensor for avionics sensor, and mount a small analog camera on the bottom of the platform, which operates using 1.3GHz band.
- #10 The Optical Flow algorithm can be summarized by the picture here. When we select the object of interest as shown as the blue shape here. The Tomasi-Shi algorithm will automatically specify good features to track. We can limit the good features to track to a certain number of features. In this example five point is selected. Even though it is not exactly the center of the OOI, we output the center of the five features as the object center, and relay it to the controller as an input. Based on Optical Flow algorithm, the five same features will be tracked in the next frame. A process of reselection of feature is executed when the feature of interest is lost in detection. We can summarize the Optical Flow algorithm as in the next slide.
- #11 As shown in the formula above, feature point x,y will be x+dx and y+dy in the next image frame I2. dx and dy is defined when the argument of the minimum of this equation is fulfilled. The same process then will be repeated in the next step, to provide an optical flow and simple feature tracking algorithm. However there is a problem when feature detection. First it only selected limited points or features, next it will lose the object profile when the object is lost from the screen due to movement or just simply image blur due to bad video streaming reception. The feature based algorithm does not “remember” the object when it first selected. Thus we introduce a hypothetical algorithm to solve the problem, combining the color based and feature based algorithm.
- #14 The study has been done by Kendoul et al. using single camera, however we want to exploit the potential of stereo camera especially to predict correctly depth (Zi) used in the equation.