Liang Yan, Tomohiro Fukuda, Nobuyoshi Yabuki Osaka University, Japan Presented at CAADRIA 2019, Victoria University of Wellington, New Zealand

1. Integrating UAV Development
Technology with Augmented Reality
Toward Landscape Tele-simulation
Liang Yan, Tomohiro Fukuda, Nobuyoshi Yabuki

2. Contents
1. Introduction
1.1. Background
1.2. Literature Review
1.3. Objective
2. Overview
2.1. Design Flow
2.2. Implementing Flow
2.3. Logic Flow in Android Studio
3. Implementation
3.1. Test of the Performance of AR Toolkits
3.2. Outdoor Stability Analysis by Using Kudan AR
3.3. Integrating Kudan, UAV Video and Construction Model
3.4. Implementing System in Real Time
4. Conclusion & Future Work

3. Introduction
4
Building process
The entire process from planning, design to construction is carried out through a
water-fall model with very low reversibility.
If there are some errors or omissions to correct after the construction begins, it will
cause many unnecessary losses.
We need a method to find and correct more errors in the design stage.
Set the purpose and
conditions of the
construction
……
Client
Concept drawings
Developed plans
Working drawings and
specifications
……
Architect/Designer
Start building according
to the design drawing
……
Builder
Background

4. Introduction
5
Augmented Reality (AR)
Superimposing virtual models into corresponding real scene by registering
and tracking accurately
Access limitation: AR viewpoints must be set near the user
Can AR viewpoints move more freely apart from the user's position?
Present
Design proposal
CAADRIA2018
Thermal simulation after renovation
A user wearing optical see-through HMD

5. Introduction
6
Unmanned Aerial Vehicle (UAV)
An aircraft without a human pilot aboard.
Applications: Architecture, Agriculture, Commerce, Military, Recreation, Science
1. UAV aircraft
2. Ground-based controllerUnmanned Aircraft System
3. Communication system between 1 and 2
Simultaneous Localization and Mapping (SLAM)
Generating an environmental map while estimating the self-position without a maker
Realizing a marker-less AR
Map Initialization
Tracking Map Updating
Camera Calibration
CAADRIA2017

6. Literature Review
7
Construction with UAV
Siebert et al (2014) tested the performance of UAVs in civil engineering, estimated
location, etc., and identified the feasibility of UAV development in the construction
industry.
Ham et al (2016) simplifies the process of collecting visual data by using UAV for
visual monitoring of buildings, using 3D reconstruction to save time.
Insufficient: Not using UAV data in real time
① Simulation angles and heights are
limited to 2D display.
② UAV/AR is used to improve user's
interface experirence
③ Collecting data and using data is not
in the same time
Construction
AR UAV
①
②
③
?
Ham, Y., Han, K.K., Lin, J J., Mani Golparvar-Fard. (2016). Visual monitoring of civil infrastructure systems via camera-equipped Unmanned Aerial Vehicles
(UAVs): a review of related works, Visualization in Engineering 2016 4:1Novitski, B. (1994). Virtual Reality for Architects, Architecture
Siebert, S., Teizer, J. (2014). Mobile 3D mapping for surveying earthwork projects using an Unmanned Aerial Vehicle (UAV) system, Automation in Construction,
Volume 41, pages 1-14

7. Introduction
8
Objective
Design a landscape tele-simulation system that integrates AR and UAV technology
To
• free the limitation of
simulating angle
and height
• save time and
reduce potential loss
image

8. Overview
9
Marker-less AR processing
3D model creating
Portable Device Controller connection Aircraft connection
Android development platform
UAV developing tools
System Design Flow
Preprocess

9. Overview
10
Implementing Flow
Android
Studio
Unity

10. Overview
11
Logic Flow in Android Studio
Android Studio
Request Permission
Registration
Record video function
calling
Layout Jumping
Aircraft Connecting
UAV video stream
returns in real time
Stream data shows on
screen in UAV view
AR function calling
UAV video stream
returns to AR view
Detect plane and
determine where 3D
model appears
SLAM tracking starts

11. Implementation
12
Test of the Performance of AR Toolkits
Table 1. Similarities and differences between AR algorithms
NAME PLATFORM HOW TO
START SLAM
TRACKING
PLANE
DETECTION
USING
VIRTUAL
MARKER
HARDWARE
CONDITION
Kudan Unity 5.6.6 Click button Horizontal Yes Compass
Vuforia
Unity
2018.3.2 Touch screen Horizontal No
Compass,
Accelerometer
Wikitude
Unity
2018.3.2
Click button Horizontal
& Vertical
No
Compass,
Accelerometer,
Quad-Core
CPU

12. Implementation
13
Performance test of AR Toolkits
 Location: Miyazaki Muhara River Green Park (Recorded)
 Hardware: Phantom 4 from DJI Company
Angle: 30° , Max height: 11m, Moving speed: 2m/s, Aircraft route: rises vertically
Best Result: Kudan
Kudan Wikitude
Vuforia (1) Vuforia (2)
H
A

13. Implementation
14
Front view
Plan view
Hardware Android tablet Nexus 9
Software
Unity5.6.6
Kudan AR
Floor Height (H) Meter Angle (α) Degree
1.7 6.046
6.7 22.658
10.3 32.691
13.9 40.894
17.5 47.475
1H 1
2H 2
3H 3
4H 4
5H 5
L=16.05m
Outdoor Stability Analysis by Using Kudan AR
Hardware and software Data measurement and calculation results

14. Implementation
15
Outdoor Stability Analysis by Using Kudan AR
Best Result: from 5th floor
 The size of the 3D model
generated later is related
to the size of the
auxiliary marker shown
on the display.
 As the height increases,
the angle α increases, the
marker size becomes
closer to the set value.
 When the angle α is too
small, the stability will
be greatly reduced.
1F 2F
3F 4F
5F

15. Implementation
16
Integrating Kudan, UAV Video and Construction Model
Real world SketchUp Unity

16. Implementation
17
Implementing System in Real Time
Android Studio
Request Permission
Registration
Record video function
calling
Layout Jumping
Aircraft Connecting
UAV video stream
returns in real time
Stream data shows on
screen in UAV view
AR function calling
UAV video stream
returns to AR view
Detect plane and
determine where 3D
model appears
SLAM tracking starts
 Streaming data
back to device
 Markerless AR
function using
tablet’s camera

17. Conclusion & Future Work
18
Conclusion
 Complete the design of landscape simulation system by integrating UAV
developing with AR.
 A series of experiments were directly carried out by combining with the UAV
record video, and which AR Toolkit to use is confirmed.
 Real time system is functionally completed.
Future Work
 Streaming data resource replacement
 Telecommunication implementation
Acknowledgments
 This research has been partly supported by
JSPS KAKENHI Grant Number
JP16K00707.