Summary _ Multi-agent robotic system (MARS) for UAV-UGV path planning and automatic sensory data collection in cluttered environments .pdf

1. Multi-Agent Robotic System (MARS)
for UAV-UGV Path Planning and
Automatic Sensory Data Collection in
Cluttered Environments
Difeng Hu, Vincent J.L. Gan, Tao Wang, Ling Ma
Building and Environment _ Published: 2022
Presenter: Ricardo Hogan 顔永裕
Advisor: Jacob J. Lin , PhD, Assistant Professor
1

2. Contents
1. Introduction
2. Literature Review
3. Methodology
4. Experiment
5. Conclusion
6. Future Work
2

3. Introduction
● There has been growing interest in increasing the application of robotic and automation
technologies for building inspection.
● Automation and robotic technologies have potential to reduce the workforce and time
required to complete inspection tasks.
● Current challenges for real-life adaptation:
○ Indoor applications usually take place in cluttered environments containing various
building components and obstacles, which are different from outdoor environments
○ Robotic constraints: UAVs are more agile and have better views than Unmanned
Vehicle (UGV) but suffer from smaller payloads and shorter operational times
● Responding to the difficulties mentioned above, the paper presents a multi-agent robotic
system (MARS) for automatic UAV-UGV path planning and indoor navigation.
3

4. Literature Review
● UGV for Indoor Applications
○ Automated inspection using fiducial markers navigated UGV, which is more effective
and economical but it cannot actively avoid obstacles. (Mantha et al., 2018)
○ Automated inspection using hybrid tracks and legs robot, teleoperated by an
inspector. (Rea and Ottaviano,2018)
● UAV for Data Collection
○ Bridges defect inspection using UAV equipped with LiDAR and flight path
optimization using genetic algorithm & A* algorithm (Bolourian and Hammad, 2020)
○ Construction inspections using UAV and BIM information as a map (Freimuth and
Konig, 2018)
4

5. Literature Review
● UAV-UGV Collaborative Data Collection
○ Unified UAV-UGV framework for collecting data in the disaster-rescue scenario.
UAV took ground images and created a map while, the UGV navigated within the
indoor environment based on the map information. (Lakas et al. 2018)
○ UAV-UGV system for geometric data collection and 3D visualization, in which UAV
collected images of a construction site and created a map while UGV navigating to
collects data (Kim et al., 2019）
5

6. Methodology
1. System Architecture &
hardware Connection
2. Automated Path Planning
3. Indoor Navigation Coordination
Control Algorithm
6
Figure 1
MARS Framework

7. Methodology - System Architecture
7
Figure 2
Hardware Connection

8. Methodology - System Architecture
8
Figure 3
System Architecture

9. Methodology - System Architecture
The UAV pipeline contains:
● Built-in Camera → Micro Controller → Sent to Mediating Agent
The UGV use Robot Operating System (ROS) and contains 2 pipelines:
● UGV leverages a 2D LiDAR to scan 2D layout and room geometry information —>
Mapping Module → 2D occupancy map → Sent to Mediating Agent
● RGB-D images & Camera Poses → RGB-D synchronization module → Sent to Mediating
Agent
Both UAV and UGV movement are coordinated by Mediating Agent
9

10. Methodology - System Architecture
The mediating agent contains 3 modules:
● RTABMAP : reconstruct the 3D point clouds using the RGB-D information collected by
UGV
● Display modules : displays the image data and video streams from UAV.
● Coordinating algorithm : interoperate multiple UAV and UGV devices for data collection
to improve inspection efficiency
10

11. Methodology - Automated Path Planning
● The path then is optimized using enhanced Shunting Short-Term Memory (SSTM) model
● The enhanced SSTM model is built on the construction of neural network architecture
11
Figure 4
SSTM Schematic Diagram
Figure 5
Original VS Enhanced SSTM Schematic Diagram

12. Methodology - Indoor Navigation and Control
● The 3D Occupancy Map is converted to 2D Grid
Map, where each grid is considered as a neuron in
the SSTM Model
12
Figure 6
Indoor Navigation Procedure for UAV
Figure 7
Indoor Navigation Rules for UAV

13. Methodology - Multi-robot Coordination
13
Figure 8
Coordination Algorithm Mechanism

14. Experiment
● Experiments are conducted in a Construction Technology Laboratory
● Separately test 3 scenarios of application:
○ Single UAV data collection
○ Dual UAVs data collection
○ Combined UAV-UGV data collection
14
Figure 9
Experiment Location Details

15. Experiment - Single UAV
15
To test optimized path for
single robot
Figure 10
Single UAV Flight Path in Grid Map

16. Experiment - Single UAV
16
Figure 11
Single UAV Flight Path in Real Experiment

17. Experiment - Dual UAVs
●
17
To test multi robot communication
for navigation problem
Figure 12
Dual UAVs Flight Path in Grid Map

18. Experiment - Dual UAVs
18
Figure 13
Dual UAVs Flight Path in Real Experiment

19. Experiment - Combined UGV-UAV
19
To test multi-robot
communication for data
collection data problem
Figure 14
3D Point Cloud Model Collected by UGV

20. Experiment - Combined UGV-UAV
20
Figure 15
Combined UGV-UAV Path in Real Experiment

21. Conclusion
● The proposed MARS in the study is able to combine :
○ Environment mapping and localization → UGV (Using Lidar)
○ Optimized path planning while dodging obstacles and avoid collision
→ UGV， UAV (Using Enhanced SSTM)
○ Multi robot device communication → Coordination Algorithm
○ Data Collection → UGV (RGBD data)， UAV (Photos & Videos)
● The study is one of the early attempts to introduce MARS into indoor navigation for
automated data collection, and it shows the potential for revolutionizing data collection
and indoor inspection.
21

22. Conclusion
● The study provides new insights :
○ It is possible to construct a UAV-UGV system for automatic data collection in a
cluttered indoor environment.
○ Multiple types of sensory data can be collected using a UAV-UGV system, which is
beneficial for facility management.
○ The UAV-UGV system can process the collected data in real-time using a low
computational platform, which makes possible real-time facility inspection.
● The study is one of the early attempts to introduce MARS into indoor navigation for
automated data collection, and it shows the potential for revolutionizing data collection
and indoor inspection.
22

23. Future Work
● Develop dynamic obstacles detection algorithm
● Installing different kinds of robotic devices and sensors
○ Integrating visual SLAM, into the MARS for more accurate indoor localization
○ Installing Lidar on UAV
● Developing algorithms dealing with 3D cooperative navigation for different robots for
indoor inspection.
23