How to better use AI and Big Data Techniques for scaling Drone Data

1. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
AI and Big Data for Drone processing
usecase of photovoltaic inspection
25-April 2022

2. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Agenda
• Big Data/AI and Drone
• Opportunities
• Challenges, Why is it Hard?
• Big Data Challenges…
• Toward a new architecture for drone Big Data
• Partitioning
• Storage
• Computing
• Some existing Big Data/AI frameworks for Drone
• Use case:
• Using AI and Drone for photovoltaic inspection

3. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Big Data / AI and Drone

4. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Why!

5. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Moreover
• The potential of drones data is often underestimated
• Archiving collected data
• Curretly, we are doing more archiving tasks than managing drone data efficiently
• Almost no existing Big Data infrastructure can handle drones efficiently,
• Even if Big data is almost mature for other domains: Finance, Banking…
• Often it is
• Hard to store
• Hard to manage
• Hard to process
• Hard to get insight
• How ???

6. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Hard to store: Volume
A very small drone project can generate more than
10 GB, sometimes more than 40Gb
15 million images of drone can make up more than 175
terabytes of data.
How to Store and Compute such growing volume?
FEDS : 13,000 flights this year

7. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Hard to store: Variety
• “Drones can now provide a wide variety
of data types, everything from a few
basic photos through to complex
measurable 3D models with annotations
and overlays.”
Visual Encylopedia of drone data
Aerial Photography and Video
Orthomosaic Map
Digital Elevation Model (DEM)
3D Pointcloud Model
Multispectral Mapping
Thermal Imagery and Mapping

8. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Hard to process:
Computing Model and Scalability
• Currently, drone image processing are done in one server: NOT SCALABLE
• Scalability is the property of a system to handle a growing amount of work by
adding resources to the system
• In Big Data, It is mostly done by distributing storage and computing
• Distributed computing can provide Scalability, but drone data friendly is Difficult
Processing/Querying drone data can take up to a few hours
 Objective : real time (few seconds)

9. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
 Going beyond traditional algorithms
 Why not use Neural networks that have made great success with image:
▪ Semantic segmentation
▪ Object recognition, Classification..
▪ Description Generation for Drone Images Using Attribute Attention Mechanism
 But theses new algorithms require more storage capacities and computing
power
Hard to get Insights

10. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Current approaches are obsolete
we need to reinvent everything
Storage
Access Availability
Computing
Fast Accurate
Analytics
Machine
Learning
Deep
Learning
Search
By
semantic
By Spatial
Queries
…

11. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
1. New architecture to be redefined
Analytical Queries
Structured Storage
Cluster
Computing Cluster
…
Large Scale
Time series NDVI
•Distributing both STORAGE
•AND COMPUTING

12. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
2. Need to correlate drone data with external
datasets
More Insights
Census Data
Economic Data
Weather
…

13. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
3. Toward a declarative language (SQL-Like) over drone
data
Change in NDVI over the spring and early summer of 2018
Select normalized_difference(nir, red) as ndvi
From Feds_droneDataset
Where
date between ‘10-10-2017’ and ‘10-10-2019’
Examples from
‘10-10-2017’ to ‘10-10-2019’
Best option for Data Scientists

14. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Recall that storage should be distributed across a cluster
• Before detailing storage techniques, let’s talk about Partitioning
Structured Storage
Cluster
…
Node A
Node B
Node F
Node G

15. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Challenge for going distributed:
Data Partitioning
 Partitioning means the process of physically dividing data into separate data
stores
 Data is divided into partitions that can be managed and accessed separately.
Node 1
Node 2
Node 3
Node 4

16. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Node 1
Node 2
Node 3
By Band
RGB
Red Band
Green Band
Blue Band
First simple approach is to partition by band

17. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Node 1
Node 2
Node 3
By Time
Spring
Summer
Autumn
Other simple approach is to partition by time
(season)

18. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Node 1
Node 2
Node 3
Decompose into NxN regular grids
But the Most efficient approach is to combine Tiling and Distribution
Tiling allows large raster datasets to be broken-up into manageable pieces  higher level raster I/O interface.

19. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Which Partition strategy to choose?
• Not in the scope of this presentation
• Check with your main objective:
• If for Scalability,
• If for Query Performance,
• If for Availability
• Many Best practices are available
• Sometimes we make use of Global Index for Optimizing Queries

20. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
The computing part
Computing
Model
HADOOP/MapReduce Spark/Spark SQL
We have at least two interesting computing models

21. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Spark vs Hadoop MapReduce
Source: Data Flair
We will focus Next on Apache Spark
According to benchmarks studies, Spark is much better than Hadoop
MapReduce

22. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Frameworks for Raster Big Data

23. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Frameworks for Raster Big Data
Apache Spark / Spark SQL
• Rasterframes (My favorite)
Earth AI (To follow)
Google Earth Engine
Rasdaman
SciDB

24. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
• Spark project for Raster Data
• Spark Dataframe like abstraction for handling Raster Data : Provides ability to work with
Raster imagery in a convenient yet scalable format
• You can use Spark ML for building ML Models
B1
B2
B3
B4
tile or tile_n (where n is a band number)

25. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Standard Tile Operations
• Many raster operations are ready to be executed in a distributed manner : can be
executed over Spark Cluster
• Ready to use

26. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
RasterFrames: SQL Query
• Can I Use spatial predicate in my query: intersection query?

27. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
SQL query in Rasterframes
SELECT month, ndvi_stats.*
FROM ("
SELECT month, rf_agg_stats(rf_normalized_difference(nir, red)) as ndvi_stats
FROM red_nir_tiles_monthly_2017
WHERE st_intersects(st_reproject(rf_geometry(red), rf_crs(red), 'EPSG:4326'),
st_makePoint(34.870605, -4.729727))
GROUP BY month
ORDER BY month )
"")
 Compute the average NDVI per month for a single tile in an Area of
Interest

28. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Earth AI
• is a Cloud-native software that enables you to apply advanced machine
learning algorithms to EO data at scale
• Both a non-code-based visual interface and pre-built workflows
• Ready-To-Use Datasets
• data archive includes more years of historical imagery and scientific datasets
• Elastic Compute
• Designed for scalability from the beginning, Earth AI platform scales seamlessly, so
you can think more about insights than Dev Ops

29. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Earth AI

30. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Earth AI
• Classifying an ecoregion using Decision Tree Classifier

31. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
SciDB
• Array-based data management and analytical system
• Arrays are divided into equally sized chunks
• Chunks are distributed over many SciDB instances
• Size and shape of chunks are defined by users per array and
have strong effects on computation times
• Storage is nearly sparse
• Relies on shared nothing architectures
• Open-source version available, extensible by UDFs

32. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
INTELLIGENT INSPECTION OF LARGE-SCALE
PHOTOVOLTAIC INSTALLATIONS THROUGH
RGB AND THERMAL INFRARED IMAGERY
ACQUIRED BY UNMANNED AERIAL VEHICLES
Imane SEBARI (a), Yahya ZEFRI (a), Hicham
HAJJI (a), Ghassane ANIBA (b)
(a) Photogrammetry-Cartography Department, School of Geomatics and Surveying
Engineering, IAV Hassan II, Rabat, Morocco
(b) Electrical Engineering Department, Mohammadia School of Engineers, Mohammed V
University in Rabat, Morocco
www.smartdrone4pv.com

33. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
CONTEXT & PROBLEMATIC
Solar
Photovoltaic
Modules
Contrasted
Temperatures
Humidity
Intrusion
Fierce
Winds
Rain/Snow/Hail
Handling
& Installation
Multiple
Defects
that develop over
time and penalize
the electricity
production Remotely Sensed
RGB and infrared
Imagery by UAVs
Contactless characterization
Faster image acquisition
Large-scale coverage
Increased accessibility
Hotspots
Delaminations
Discolorations
Cracks

34. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
THE SMARTDRONE4PV PROJECT
1. ADVANCED UAV
PHOTOGRAMMETRY
for RGB and long-wave
thermal infrared image
acquisition
2. DEEP LEARNING
SOLUTIONS
for defect detection and
classification on the used
imagery types
3. BIG DATA
ANALYTICS
to handle the huge
datasets that are generated
by large-scale PV plants
School of Geomatics and Surveying Engineering, IAV Hassan II, Rabat, Morocco
Mohammadia School of Engineers, Mohammed V University in Rabat, Morocco
Research Institute for Solar Energy and New Energies, Rabat, Morocco.
ETAFAT, Casablanca, Morocco.
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES

35. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
DATA ACQUISITION
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
RGB and thermal infrared on-
field image acquisition
SfM-MVS
photogrammetric
post-processing

36. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
DEEP LEARNING-BASED DEFECT DETECTION
IMAGE CLASSIFICATION

37. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
DEEP LEARNING-BASED DEFECT DETECTION
SEMANTIC SEGMENTATION

38. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
DEEP LEARNING-BASED DEFECT DETECTION
OBJECT DETECTION

39. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
THANK
YOU.