Serene 2015 Davide Scaramuzza Abstract: With drones becoming more and more popular, safety is a big concern. A critical situation occurs when a drone temporarily loses its GPS position information, which might lead it to crash. This can happen, for instance, when flying close to buildings where GPS signal is lost. In such situations, it is desirable that the drone can rely on fall-back systems and regain stable flight as soon as possible. In this talk, I will present novel methods to automatically recover and stabilize a quadrotor from any initial condition or execute emergency landing. On the one hand, this new technology will allow quadrotors to be launched by simply tossing them in the air, like a “baseball ball”. On the other hand, it will allow them to recover back into stable flight or land on a safe area after a system failure. Since this technology does not rely on any external infrastructure, such as GPS, it enables the safe use of drones in both indoor and outdoor environments. Thus, it can become relevant for commercial use of drones, such as parcel delivery. Recent videos: Automatic failure recovery without GPS: https://youtu.be/pGU1s6Y55JI Autonomous Landing-site detection and landing: https://youtu.be/phaBKFwfcJ4

1. Davide Scaramuzza
Towards Robust and Safe
Autonomous Drones
Website: http://rpg.ifi.uzh.ch/people_scaramuzza.html
Software & Datasets: http://rpg.ifi.uzh.ch/software_datasets.html
YouTube: https://www.youtube.com/user/ailabRPG/videos
Publications: http://rpg.ifi.uzh.ch/publications.html

2. http://rpg.ifi.uzh.ch
My Research Group

3. Autonomous Navigation of Flying Robots
[AURO’12, RAM’14, JFR’15]
Event-based Vision for Agile Flight
[IROS’3, ICRA’14, RSS’15]
Visual-Inertial State Estimation
[T-RO’08, IJCV’11, PAMI’13, RAM’14, JFR’15]
Current Research
Air-ground collaboration
[IROS’13, SSRR’14]

4. Unmanned Aerial Vehicles (UAVs or drones)
Mass
Micro Aerial Vehicles (MAVs)
Low cost, small size (<1m) , weight <5kg)

5. Today’s Applications

6. Toys

7. Aerial Photography and 3D Mapping
www.sensefly.com

8. Goods’ Delivery: e.g., Amazon Prime Air

9. Transportation Search and rescue Aerial photography
Law enforcementInspection Agriculture
Today’s Applications of MAVs

10. How to fly a drone
 Remote control
 Requires line of sight or communication link
 Requires skilled pilots
Drone crash during soccer
match, Brasilia, 2013Interior of an earthquake-damaged building in Japan
 GPS-based navigation
 Doesn’t work indoors
 Can be unreliable outdoors

11. Problems of GPS
 Does not work indoors
 Even outdoors it is not a reliable service
 Satellite coverage
 Multipath problem

12. Why do we need autonomy?

13. Autonomous Navigation is crucial for:
Remote InspectionSearch and Rescue

14. Fontana, Faessler, Scaramuzza
How do we Localize without GPS ?
Mellinger, Michael, Kumar

15. This robot is «blind»
How do we Localize without GPS ?

16. This robot is «blind»
How do we Localize without GPS ?
Motion capture system
Markers

17. This robot can «see»This robot is «blind»
How do we Localize without GPS ?
Motion capture system
Markers

18. AutonomousVision-based Navigation in GPS-denied Environments
[Scaramuzza, Achtelik, Weiss, Fraundorfer, et al., Vision-Controlled Micro Flying Robots: from System
Design to Autonomous Navigation and Mapping in GPS-denied Environments, IEEE RAM, 2014]

19. Problems with Vision-controlled MAVs
Quadrotors have the potential to navigate quickly through unstructured
environments, enter and rapidly explore collapsed buildings, but…
 Autonomous operation is currently restricted to controlled environments
 Autonomous maneuvers with onboard cameras are still slow and inaccurate
compared to those attainable with motion-capture systems
Why?
 Perception algorithms are mature but not robust
 Unlike lidars and Vicon, localization accuracy depends on depth & texture!
 Sparse models instead of dense environment models
 Control & perception have been mostly considered separately
 Not capable of adapting to changes

20. Outline
 Vision-based, GPS-denied Navigation
 From Sparse to Dense Models
 Active Vision and Control
 Low-latency State Estimation for agile motion

21. Vision-based, GPS-denied
Navigation

22. Image 𝐼 𝑘−1 Image 𝐼 𝑘
𝑇𝑘,𝑘−1
Visual Odometry
1. Scaramuzza, Fraundorfer. Visual Odometry, IEEE Robotics and Automation Magazine, 2011
2. D. Scaramuzza. 1-Point-RANSAC Visual Odometry, International Journal of Computer Vision, 2011
3. Forster, Pizzoli, Scaramuzza, SVO: Semi Direct Visual Odometry, IEEE ICRA’14]

23. Keyframe-based Visual Odometry
Keyframe 1 Keyframe 2
Initial pointcloud New triangulated points
Current frame
New keyframe

24. Feature-based vs. Direct Methods
Feature-based (e.g., PTAM, Klein’08)
1. Feature extraction
2. Feature matching
3. RANSAC + P3P
4. Reprojection error
minimization
𝑇𝑘,𝑘−1 = arg min
𝑇
𝒖′𝑖 − 𝜋 𝒑𝑖
2
𝑖
Direct approaches (e.g., Meilland’13)
1. Minimize photometric error
𝑇𝑘,𝑘−1
𝐼 𝑘
𝒖′𝑖
𝒑𝑖
𝒖𝑖
𝐼 𝑘−1
𝑇𝑘,𝑘−1 = ?
𝒑𝑖
𝒖′𝑖𝒖𝑖
𝑇𝑘,𝑘−1 = arg min
𝑇
𝐼 𝑘 𝒖′𝑖 − 𝐼 𝑘−1(𝒖𝑖) 2
𝑖
[Soatto’95, Meilland and Comport, IROS 2013], DVO [Kerl et al., ICRA
2013], DTAM [Newcombe et al., ICCV ‘11], ...

25. Feature-based vs. Direct Methods
1. Feature extraction
2. Feature matching
3. RANSAC + P3P
4. Reprojection error
minimization
𝑇𝑘,𝑘−1 = arg min
𝑇
𝒖′𝑖 − 𝜋 𝒑𝑖
2
𝑖
Direct approaches
1. Minimize photometric error
𝑇𝑘,𝑘−1 = arg min
𝑇
𝐼 𝑘 𝒖′𝑖 − 𝐼 𝑘−1(𝒖𝑖) 2
𝑖
 Large frame-to-frame motions
 Slow (20-30 Hz) due to costly feature
extraction and matching
 Not robust to high-frequency and
repetive texture
 Every pixel in the image can be
exploited (precision, robustness)
 Increasing camera frame-rate reduces
computational cost per frame
 Limited to small frame-to-frame
motion
Feature-based (e.g., PTAM, Klein’08)

26. Feature-based vs. Direct Methods
1. Feature extraction
2. Feature matching
3. RANSAC + P3P
4. Reprojection error
minimization
𝑇𝑘,𝑘−1 = arg min
𝑇
𝒖′𝑖 − 𝜋 𝒑𝑖
2
𝑖
Direct approaches
1. Minimize photometric error
𝑇𝑘,𝑘−1 = arg min
𝑇
𝐼 𝑘 𝒖′𝑖 − 𝐼 𝑘−1(𝒖𝑖) 2
𝑖
 Large frame-to-frame motions
 Slow (20-30 Hz) due to costly feature
extraction and matching
 Not robust to high-frequency and
repetive texture
 Every pixel in the image can be
exploited (precision, robustness)
 Increasing camera frame-rate reduces
computational cost per frame
 Limited to small frame-to-frame
motion
Feature-based (e.g., PTAM, Klein’08)
Our solution:
SVO: Semi-direct Visual Odometry [ICRA’14]
Combines feature-based and direct methods

27. SVO: Experiments in real-world environments
[Forster, Pizzoli, Scaramuzza, «SVO: Semi Direct Visual Odometry», ICRA’14]
Robust to fast and abrupt motions
Video:
https://www.youtube.com/watch?v=2YnI
Mfw6bJY

28. Processing Times of SVO
Laptop (Intel i7, 2.8 GHz)
Embedded ARM Cortex-A9, 1.7 GHz
400 frames per second
Up to 70 frames per second
 Open Source available at: github.com/uzh-rpg/rpg_svo
 Works with and without ROS
 Closed-Source professional edition available for companies
Source Code

29. Absolute Scale Estimation

30. Scale Ambiguity
 With a single camera, we only know the relative scale
 No information about the metric scale

31. Absolute Scale Estimation
 The absolute pose 𝑥 is known up to a scale 𝑠, thus
𝑥 = 𝑠𝑥
 IMU provides accelerations, thus
𝑣 = 𝑣0 + 𝑎 𝑡 𝑑𝑡
 By derivating the first one and equating them
𝑠𝑥 = 𝑣0 + 𝑎 𝑡 𝑑𝑡
 As shown in [Martinelli, TRO’12], for 6DOF, both 𝑠 and 𝑣0 can be determined in closed form
from a single feature observation and 3 views
The scale and velocity can then be tracked using
 Filter-based approaches
 Losely-coupled approaches [Lynen et al., IROS’13]
 Tightly-coupled approached (e.g., Google TANGO) [Mourikis & Roumeliotis, TRO’12]
 Optimization-based approaches [Leutenegger, RSS’13], [Forster, Scaramuzza, RSS’14]

32.  Fusion is solved as a non-linear optimization problem (no Kalman filter):
 Increased accuracy over filtering methods
IMU residuals Reprojection residuals
[Forster, Carlone, Dellaert, Scaramuzza, IMU Preintegration on Manifold for efficient Visual-Inertial
Maximum-a-Posteriori Estimation, RSS’15]
Visual-Inertial Fusion [RSS’15]

33. Comparison with Previous Works
Google TangoProposed ASLAM
Forster, Carlone, Dellaert, Scaramuzza, IMU Preintegration on Manifold for efficient Visual-Inertial Maximum-a-
Posteriori Estimation, Robotics Science and Systens’15
Accuracy: 0.1% of the travel distance
Video: https://www.youtube.com/watch?v=CsJkci5lfco

34. Integration on a
Quadrotor Platform

35. Quadrotor System
PX4 (IMU)
Global-Shutter Camera
• 752x480 pixels
• High dynamic range
• 90 fps
450 grams
Odroid U3 Computer
• Quad Core Odroid (ARM Cortex A-9) used in Samsung Galaxy S4 phones
• Runs Linux Ubuntu and ROS

36. Flight Results: Hovering
RMS error: 5 mm, height: 1.5 m – Down-looking camera
Faessler, Fontana, Forster, Mueggler, Pizzoli, Scaramuzza, Autonomous, Vision-based Flight and Live Dense 3D
Mapping with a Quadrotor Micro Aerial Vehicle, Journal of Field Robotics, 2015.

37. Flight Results: Indoor, aggressive flight
Speed: 4 m/s, height: 1.5 m – Down-looking camera
Faessler, Fontana, Forster, Mueggler, Pizzoli, Scaramuzza, Autonomous, Vision-based Flight and Live Dense 3D
Mapping with a Quadrotor Micro Aerial Vehicle, Journal of Field Robotics, 2015.
Video: https://www.youtube.com/watch?v=l3TCiCe_T3g

38. Autonomous Vision-based Flight over Mockup Disaster Zone
Firefighters’ training area, Zurich
Faessler, Fontana, Forster, Mueggler, Pizzoli, Scaramuzza, Autonomous, Vision-based Flight and Live Dense 3D
Mapping with a Quadrotor Micro Aerial Vehicle, Journal of Field Robotics, 2015.
Video:
https://www.youtube.com/watch?v
=3mNY9-DSUDk

39. Probabilistic Depth Estimation
Depth-Filter:
• Depth Filter for every feature
• Recursive Bayesian depth estimation
Mixture of Gaussian + Uniform distribution
[Forster, Pizzoli, Scaramuzza, SVO: Semi Direct Visual Odometry, IEEE ICRA’14]

40. Robustness to Dynamic Objects and Occlusions
• Depth uncertainty is crucial for safety and robustness
• Outliers are caused by wrong data association (e.g., moving objects, distortions)
• Probabilistic depth estimation models outliers
Faessler, Fontana, Forster, Mueggler, Pizzoli, Scaramuzza, Autonomous, Vision-based Flight and Live Dense 3D
Mapping with a Quadrotor Micro Aerial Vehicle, Journal of Field Robotics, 2015.
Video:
https://www.youtube.com/watch?v
=LssgKdDz5z0

41. Failure Recovery
• Loss of GPS
• From aggressive flight
•Visual tracking

42. Automatic Failure Recovery from Aggressive Flight [ICRA’15]
Faessler, Fontana, Forster, Scaramuzza, Automatic Re-Initialization and Failure Recovery for Aggressive Flight
with a Monocular Vision-Based Quadrotor, ICRA’15. Featured in IEEE Spectrum.
Video:
https://www.youtube.com/watch?v
=pGU1s6Y55JI

43. Recovery Stages
Throw
IMU

44. Recovery Stages
Attitude Control
IMU

45. Recovery Stages
Attitude + Height
Control
IMU
Distance Sensor

46. Recovery Stages
Break Velocity
IMU
Camera

47. Automatic Failure Recovery from Aggressive Flight [ICRA’15]
Faessler, Fontana, Forster, Scaramuzza, Automatic Re-Initialization and Failure Recovery for Aggressive Flight
with a Monocular Vision-Based Quadrotor, ICRA’15. Featured in IEEE Spectrum.

48. From Sparse to Dense 3D
Models
[M. Pizzoli, C. Forster, D. Scaramuzza, REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time, ICRA’14]

49. Goal: estimate depth of every pixel in real time
 Pros:
- Advantageous for environment interaction (e.g., collision avoidance,
landing, grasping, industrial inspection, etc)
- Higher position accuracy
 Cons: computationally expensive (requires GPU)
Dense Reconstruction in Real-Time
[ICRA’15] [IROS’13, SSRR’14]

50. Dense Reconstruction Pipeline
 Local methods
 Estimate depth for every pixel independently using photometric cost aggregation
 Global methods
 Refine the depth surface as a whole by enforcing smoothness constraint
(“Regularization”)
𝐸 𝑑 = 𝐸 𝑑 𝑑 + λ𝐸𝑠(𝑑)
Data term Regularization term:
penalizes non-smooth
surfaces
[Newcombe et al. 2011]

51. [M. Pizzoli, C. Forster, D. Scaramuzza, REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time, ICRA’14]
REMODE: Probabilistic Monocular Dense Reconstruction [ICRA’14]
Running at 50 Hz on GPU on a Lenovo W530, i7
Video:
https://www.youtube.com/watch?v
=QTKd5UWCG0Q

52. Try our iPhone App: 3DAround
Dacuda

53. Autonomus, Flying 3D Scanning [ JFR’15]
• Sensing, control, state estimation run onboard at 50 Hz (Odroid U3, ARM Cortex A9)
• Dense reconstruction runs live on video streamed to laptop (Lenovo W530, i7)
2x
Faessler, Fontana, Forster, Mueggler, Pizzoli, Scaramuzza, Autonomous, Vision-based Flight and Live Dense 3D
Mapping with a Quadrotor Micro Aerial Vehicle, Journal of Field Robotics, 2015.
Video:
https://www.youtube.com/watch?v
=7-kPiWaFYAc

54. • Sensing, control, state estimation run onboard at 50 Hz (Odroid U3, ARM Cortex A9)
• Dense reconstruction runs live on video streamed to laptop (Lenovo W530, i7)
Faessler, Fontana, Forster, Mueggler, Pizzoli, Scaramuzza, Autonomous, Vision-based Flight and Live Dense 3D
Mapping with a Quadrotor Micro Aerial Vehicle, Journal of Field Robotics, 2015.
Autonomus, Flying 3D Scanning [ JFR’15]

55. Applications: Industrial Inspection
Industrial collaboration with Parrot-SenseFly targets:
 Real-time dense reconstruction with 5 cameras
 Vision-based navigation
 Dense 3D mapping in real time
Faessler, Fontana, Forster, Mueggler, Pizzoli, Scaramuzza, Autonomous, Vision-based Flight and Live Dense 3D
Mapping with a Quadrotor Micro Aerial Vehicle, Journal of Field Robotics, 2015.
Video: https://www.youtube.com/watch?v=gr00Bf0AP1k

56. Inspection of CERN tunnels
 Problem: inspection of CERN tunnels currently done by technicians,
who expend much of their annual quota of safe radiation dose
 Goal: inspection of LHC tunnel with autonomous drone
 Challenge: low illumination, cluttered environment

57. Autonomous Landing-Spot Detection and Landing [ICRA’15]
Forster, Faessler, Fontana, Werlberger, Scaramuzza, Continuous On-Board Monocular-Vision-based Elevation Mapping
Applied to Autonomous Landing of Micro Aerial Vehicles, ICRA’15.
Video: https://www.youtube.com/watch?v=phaBKFwfcJ4

58. Having an autonomous landing-spot detection
can really help!
The Philae lander while approaching the comet on November 12, 2014

59. rpg.ifi.uzh.ch
Thanks! Questions?
Website: http://rpg.ifi.uzh.ch/people_scaramuzza.html
Software & Datasets: http://rpg.ifi.uzh.ch/software_datasets.html
YouTube: https://www.youtube.com/user/ailabRPG/videos
Publications: http://rpg.ifi.uzh.ch/publications.html