Updates provided to the D-STOP Business Advisory Council at the 2017 Symposium and Board Meeting: https://ctr.utexas.edu/2018/04/12/d-stop-2017-symposium-archive/

1. © 2017, Robert W. Heath Jr.
Flying with SAVES
Dr. Nuria Gonzalez-Prelcic
SAVES Assistant Director

2. Motivation
2

3. Motivation
3
3
It is the right time for SAVES to go aerial!
SAVES has addressed many aspects
of vehicular engineering for ground
vehicles during its first year of life
Problems like collaborative sensing,
vehicular communication, map making
or navigation have common aspects in
aerial and ground scenarios
Drone-based systems are
driving disruptive
applications in the market
Taking SAVES to the skies

4. Big numbers for the commercial drone industry
4*Numbers predicted by the FAA and the Association for Unmanned Vehicle Systems International (AUVSI) https://www.dartdrones.com/blog/drone-industry-impact/
2025
82.1 billion
Tax revenue 2015-2025
$482 million
Jobs by 2025
100,000
The estimated economic impact of the drone industry is enormous
Commercial
drone fleet
42,000
420,000
2016 2021
States predicted to
see the most gains
in terms of job
creation and
additional revenue

5. 5
Aerial vehicles
UAVs as imaging
sensors
UAVs for
wireless
UAVs for sensing
and monitoring
UAVs for
transportation
TECHNOLOGIES
SENSING + COMMUNICATION in drones enables revolutionary applications

6. 10
Technologies for disruptive UAV applications
Positioning/mapping/
navigation
Collaborative
sensing
MIMO
communication
SAVES faculties are well positioned on key UAV technologies

7. Autonomy levels
11
Different autonomy levels require different technologies
Fully automated
operation with high
connectivity
Automated
navigation with
moderate rate
communication
GPS available
Automated
development of a
task/control signals
through
communication
possible/distributed
computation
possible
Fully automated
operation with moderate
connectivity
Automated
navigation with
high rate
communication
GPS available
Automated
development of a
task/control signals
through
communication
possible/distributed
computation
possible
Autonomous
operation
Autonomous
development of
a task
Autonomous
navigation when
communication/GPS
not available
NO CONNECTIVITY MODERATE/HIGH CONNECTIVITY

8. On going
research
12

9. Core networkGateway
Aerial communication expertise
13
MmWave MIMO for air-to-
air communication
Capacity analysis
PHY design
Channel models for the mmWave
A2G and A2A channels
Network topologies for
A2G and A2A @
mmWave
MU millimeter wave MIMO
for A2G
Initial work supported
by LMCO

10. 14
Trajectory planning, obstacle
avoidance
14
Different drone applications
require different on-board
sensors
GPS signal may not be available
in certain environments
(canyons, forests, etc.)
How can the drone navigate
autonomously with the set of
sensors required by the
application and no
GPS/communication?
Develop and evaluate navigation algorithms
using one camera/two
cameras/camera+radar/etc.
Navigating without GPS
Example: navigating with an on board camera

11. UAV based traffic monitoring
15
Video feed over WiFi/cellular
Video Processing
Data Processing
CLOUD
the unbiased result, we avoided choosing consecutive fram
in the same testing data set. For each data set, we calculat
the ratio of number of vehicles being detected and the to
number of vehicles. Then we average the ratios we got fro
all of the data sets. Furthermore, we repeated the process t
times from generating testing sets to averaging the ratios.
An example of the output of the tracking algorithm can
seen in Fig. 7 (a) and (b). In order to observe the tracki
results, we assigned each detected vehicle a unique numb
and display it. For each real-world aerial video as an inp
data set, we observed if the assigned number of a vehic
changed from its entering to the screen to its exit. The res
shows that unique numbers assigned to vehicles do not chan
for every testing video.
(a)
Video feed over WiFiVVideo feed over
Video Processing
Data Processing
Web Application
Fig. 4: Illustration of our experimental setup.eps
network between them, and the computer can access the UAV
and the controller over the provided IP address. After deciding
on a UAV, we chose a GoPro 4 camera for the system. The
GoPro 4 camera is compatible with the 3DR Solo gimble,
and it has adjustable frame rate and resolution that makes it
possible to collect different types of data.
Fig. 6: Overview of the structure of the traffic monitori
application.
contour detection, when the color of the vehicle and the co
of background are very similar, it cannot generate good resul
The Haar cascade model can detect cars accurately even wh
the drone shifts. By training with a large number of pictur
its accuracy can be increased steadily.
We chose the OpenCV module in Python to implement t
Haar cascade model. OpenCV’s open-source library of ima
processing functions allows us to process the input vid
Video feed over WiFiVVideo feed over
Video Processing
Data Processing
Web Application
Fig. 4: Illustration of our experimental setup.eps
network between them, and the computer can access the UAV
and the controller over the provided IP address. After deciding
on a UAV, we chose a GoPro 4 camera for the system. The
GoPro 4 camera is compatible with the 3DR Solo gimble,
and it has adjustable frame rate and resolution that makes it
possible to collect different types of data.
Fig. 5: 3DR Solo quadcopter equipped with a GoPro 4 camera.
The general processing steps are illustrated in Fig. 6. The
computer module is composed of three submodules: video
processing, data processing, and web application. Users make
monitoring request via a web application. After they enter the
location information, the web application takes the request and
generates a flight script that can be sent to the drone over
Wi-Fi. Then the drone flies to the desired location and start
collecting video.
For software decisions, the methods we tested to detect
vehicles are background subtraction [14], contour detection
[15], and the Haar cascade model. Background subtraction
and contour detection are the most common methods being
applied to vehicle detection. After running the background
subtraction algorithm, we found that it is inaccurate when
the drone’s position shift during video taking process. As for
Fig. 6: Overview of the structure of the traffic monitoring
application.
contour detection, when the color of the vehicle and the color
of background are very similar, it cannot generate good results.
The Haar cascade model can detect cars accurately even when
the drone shifts. By training with a large number of pictures,
its accuracy can be increased steadily.
We chose the OpenCV module in Python to implement the
Haar cascade model. OpenCV’s open-source library of image
processing functions allows us to process the input video
frame by frame and implement vehicle detection functions.
Even though MATLAB has similar functionalities for video
processing, it operates much slower than the OpenCV and
Python combination. Furthermore, based on our experience,
MATLAB needs more RAM and delay real-time processing
compared with OpenCV. After all the hardware and software
decisions, our first step is to get access to drone video feed
through the computer. We use VLC media player to view
the live video captured by the camera. Therefore, one of
the hardware requirements for the system is a computer with
VLC installed. The computer communicates with the drone by
connecting to the drone’s Wi-Fi and building a TCP connection
with a Telnet client. To build a TCP connection, an SDP file
including the TCP parameters is needed.
V. RESULTS
Before testing the experimental system, we collected 3750
vehicle and non-vehicle images to train the Haar cascade
model. The images were captured from the aerial video filmed
in several areas in Austin, Texas. We used the built-in sample
generating functions in OpenCV to apply distortions to the
input images, and to label data. After that, we flew the UAV
and recorded different sets of aerial videos captured at different
heights and times for testing the system.
The detection accuracy of our system lies in the range 83-
90% for any given frame. To compute the detection accuracy,
we chose frame samples from the input videos. In order to get

12. Project ideas
16

13. Summary of ideas
17
Navigating in a team without
GPS and connectivity Joint positioning and
communication using 5G
signalsSLAM aided mmWave
communication
Channel variation
models for A2G/A2A
Leveraging cellular
infrastructure
for automated operation
Managing/leveraging interference
in sub 6-GHz networks
Designing mmWave
hotspots
Infrastructure to support
automated flying Integrating autonomous
and manned vehicles
3D coverage maps for
trajectory planning

14. Trajectory planning, obstacle
avoidance
Operating in a team without connectivity with the
infrastructure
18
How to implement team
navigation, which sensing data have
to be shared between UAVs? how
performance depends on data rate?
Collaborative sensing
framework for task
development

15. 19
Assume drones are equipped with
mmWave MIMO transceivers
How to use the mmWave
communication signal for positioning?
Prior work on joint positioning and
communication does not consider high
mobility conditions in aerial scenarios
Develop joint positioning and
communication algorithms at mmWave
for the aerial environment
Location of the
aerial BS is known
with some errorLocation of the
aerial MS or
distance to aerial
BS are unknown
Joint positioning and communication using 5G signals

16. SLAM is a popular solution for
localization and mapping that can be
used for drone navigation
20
Infrared imaging
Radar
4K video
Communications relay
Air-to-ground link
Aerial access point
Networked airborne users
in a rescue mission
Some applications may require high data
rate A2A/A2G communication links
Use drone navigation algorithms to aid millimeter wave beam alignment and reduce
communication overhead by mapping SLAM outputs and channel estimates
PI: Profs. Nuria Gonzalez-Prelcic and Robert Heath
SLAM aided mmwave communication

17. Developing a channel variation model for mmWave A2G/A2A
links
21
Trajectory
1 2 3 4 5 6 7 8
Distance from start point [m]
0
20
40
60
80
100
120
140
160
180
Angle[deg]
Azimuth of Departure
AoA serie
generated from
Quadriga*
Incorporate high mobility and spatial
consistency into the A2A and A2G
channel models
A channel variation model is the key to develop channel tracking
algorithms to reduce training overhead for beamformers update

18. Leveraging cellular infrastructure
for automated operation
22
Cellular infrastructure can
supplement aerial traffic
management
Cellular infrastructure can
play a roll for drone
localization and tracking
Sensing at the infrastructure
provides distributed tracking
without high power radar
Processing can be offloaded to a
centralized processor, cell edge, or cloud

19. 23
LOS conditions in the A2G channel
impact interference level at the UAV
Design MIMO strategies at the UAV to mitigate multi-BS interference
Managing/leveraging interference in sub 6-GHz networks
Interference

20. Designing mmWave hotspots
24
Air-to-ground link
Aerial access point
Air-to-air link
Develop MU MIMO strategies for mixed
aerial-ground networks
Optimize location of aerial access point
to maximize coverage

21. Integrating autonomous and manned vehicles
25
Study the ways that autonomous
airplanes may be integrated with
manned vehicles, starting withVFR, in
the next five years
Identify critical and optional
components in the aircraft and on
the ground to support such
operations
ADS-B for position location
Radio
Cameras to detect legacy aircraft
Speech processing
Sensor
fusion
Radar altimeter
Radar for collision avoidance

22. Infrastructure to support automated flying
26
How to use the communication
signals and sensor fusion to aid
positioning?
Design trajectory planning algorithms which
account for dynamic coverage maps of the
environment
How can sensing at the infrastructure
enhance situational awareness in mixed
piloted and automated environments?
Study the role of cellular
infrastructure in supporting
automated flying, including new
modes of communication at higher
rates and also sensing at the base
station

23. 27
(x,y,z)
GPS signal
Cellular coverage
Wi-Fi signal
Design trajectory planning algorithms which account for dynamic coverage
maps of the environment including communication and sensing
3D coverage maps for trajectory planning

24. Thanks!
29