SlideShare a Scribd company logo
1 of 39
Download to read offline
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
AI and Big Data for Drone processing
usecase of photovoltaic inspection
25-April 2022
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Agenda
• Big Data/AI and Drone
• Opportunities
• Challenges, Why is it Hard?
• Big Data Challenges…
• Toward a new architecture for drone Big Data
• Partitioning
• Storage
• Computing
• Some existing Big Data/AI frameworks for Drone
• Use case:
• Using AI and Drone for photovoltaic inspection
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Big Data / AI and Drone
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Why!
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Moreover
• The potential of drones data is often underestimated
• Archiving collected data
• Curretly, we are doing more archiving tasks than managing drone data efficiently
• Almost no existing Big Data infrastructure can handle drones efficiently,
• Even if Big data is almost mature for other domains: Finance, Banking…
• Often it is
• Hard to store
• Hard to manage
• Hard to process
• Hard to get insight
• How ???
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Hard to store: Volume
A very small drone project can generate more than
10 GB, sometimes more than 40Gb
15 million images of drone can make up more than 175
terabytes of data.
How to Store and Compute such growing volume?
FEDS : 13,000 flights this year
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Hard to store: Variety
• “Drones can now provide a wide variety
of data types, everything from a few
basic photos through to complex
measurable 3D models with annotations
and overlays.”
Visual Encylopedia of drone data
Aerial Photography and Video
Orthomosaic Map
Digital Elevation Model (DEM)
3D Pointcloud Model
Multispectral Mapping
Thermal Imagery and Mapping
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Hard to process:
Computing Model and Scalability
• Currently, drone image processing are done in one server: NOT SCALABLE
• Scalability is the property of a system to handle a growing amount of work by
adding resources to the system
• In Big Data, It is mostly done by distributing storage and computing
• Distributed computing can provide Scalability, but drone data friendly is Difficult
Processing/Querying drone data can take up to a few hours
 Objective : real time (few seconds)
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
 Going beyond traditional algorithms
 Why not use Neural networks that have made great success with image:
▪ Semantic segmentation
▪ Object recognition, Classification..
▪ Description Generation for Drone Images Using Attribute Attention Mechanism
 But theses new algorithms require more storage capacities and computing
power
Hard to get Insights
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Current approaches are obsolete
we need to reinvent everything
Storage
Access Availability
Computing
Fast Accurate
Analytics
Machine
Learning
Deep
Learning
Search
By
semantic
By Spatial
Queries
…
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
1. New architecture to be redefined
Analytical Queries
Structured Storage
Cluster
Computing Cluster
…
Large Scale
Time series NDVI
•Distributing both STORAGE
•AND COMPUTING
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
2. Need to correlate drone data with external
datasets
More Insights
Census Data
Economic Data
Weather
…
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
3. Toward a declarative language (SQL-Like) over drone
data
Change in NDVI over the spring and early summer of 2018
Select normalized_difference(nir, red) as ndvi
From Feds_droneDataset
Where
date between ‘10-10-2017’ and ‘10-10-2019’
Examples from
‘10-10-2017’ to ‘10-10-2019’
Best option for Data Scientists
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Recall that storage should be distributed across a cluster
• Before detailing storage techniques, let’s talk about Partitioning
Structured Storage
Cluster
…
Node A
Node B
Node F
Node G
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Challenge for going distributed:
Data Partitioning
 Partitioning means the process of physically dividing data into separate data
stores
 Data is divided into partitions that can be managed and accessed separately.
Node 1
Node 2
Node 3
Node 4
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Node 1
Node 2
Node 3
By Band
RGB
Red Band
Green Band
Blue Band
First simple approach is to partition by band
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Node 1
Node 2
Node 3
By Time
Spring
Summer
Autumn
Other simple approach is to partition by time
(season)
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Node 1
Node 2
Node 3
Decompose into NxN regular grids
But the Most efficient approach is to combine Tiling and Distribution
Tiling allows large raster datasets to be broken-up into manageable pieces  higher level raster I/O interface.
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Which Partition strategy to choose?
• Not in the scope of this presentation
• Check with your main objective:
• If for Scalability,
• If for Query Performance,
• If for Availability
• Many Best practices are available
• Sometimes we make use of Global Index for Optimizing Queries
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
The computing part
Computing
Model
HADOOP/MapReduce Spark/Spark SQL
We have at least two interesting computing models
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Spark vs Hadoop MapReduce
Source: Data Flair
We will focus Next on Apache Spark
According to benchmarks studies, Spark is much better than Hadoop
MapReduce
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Frameworks for Raster Big Data
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Frameworks for Raster Big Data
Apache Spark / Spark SQL
• Rasterframes (My favorite)
Earth AI (To follow)
Google Earth Engine
Rasdaman
SciDB
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
• Spark project for Raster Data
• Spark Dataframe like abstraction for handling Raster Data : Provides ability to work with
Raster imagery in a convenient yet scalable format
• You can use Spark ML for building ML Models
B1
B2
B3
B4
tile or tile_n (where n is a band number)
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Standard Tile Operations
• Many raster operations are ready to be executed in a distributed manner : can be
executed over Spark Cluster
• Ready to use
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
RasterFrames: SQL Query
• Can I Use spatial predicate in my query: intersection query?
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
SQL query in Rasterframes
SELECT month, ndvi_stats.*
FROM ("
SELECT month, rf_agg_stats(rf_normalized_difference(nir, red)) as ndvi_stats
FROM red_nir_tiles_monthly_2017
WHERE st_intersects(st_reproject(rf_geometry(red), rf_crs(red), 'EPSG:4326'),
st_makePoint(34.870605, -4.729727))
GROUP BY month
ORDER BY month )
"")
 Compute the average NDVI per month for a single tile in an Area of
Interest
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Earth AI
• is a Cloud-native software that enables you to apply advanced machine
learning algorithms to EO data at scale
• Both a non-code-based visual interface and pre-built workflows
• Ready-To-Use Datasets
• data archive includes more years of historical imagery and scientific datasets
• Elastic Compute
• Designed for scalability from the beginning, Earth AI platform scales seamlessly, so
you can think more about insights than Dev Ops
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Earth AI
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
Earth AI
• Classifying an ecoregion using Decision Tree Classifier
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
SciDB
• Array-based data management and analytical system
• Arrays are divided into equally sized chunks
• Chunks are distributed over many SciDB instances
• Size and shape of chunks are defined by users per array and
have strong effects on computation times
• Storage is nearly sparse
• Relies on shared nothing architectures
• Open-source version available, extensible by UDFs
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
INTELLIGENT INSPECTION OF LARGE-SCALE
PHOTOVOLTAIC INSTALLATIONS THROUGH
RGB AND THERMAL INFRARED IMAGERY
ACQUIRED BY UNMANNED AERIAL VEHICLES
Imane SEBARI (a), Yahya ZEFRI (a), Hicham
HAJJI (a), Ghassane ANIBA (b)
(a) Photogrammetry-Cartography Department, School of Geomatics and Surveying
Engineering, IAV Hassan II, Rabat, Morocco
(b) Electrical Engineering Department, Mohammadia School of Engineers, Mohammed V
University in Rabat, Morocco
www.smartdrone4pv.com
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
CONTEXT & PROBLEMATIC
Solar
Photovoltaic
Modules
Contrasted
Temperatures
Humidity
Intrusion
Fierce
Winds
Rain/Snow/Hail
Handling
& Installation
Multiple
Defects
that develop over
time and penalize
the electricity
production Remotely Sensed
RGB and infrared
Imagery by UAVs
Contactless characterization
Faster image acquisition
Large-scale coverage
Increased accessibility
Hotspots
Delaminations
Discolorations
Cracks
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
THE SMARTDRONE4PV PROJECT
1. ADVANCED UAV
PHOTOGRAMMETRY
for RGB and long-wave
thermal infrared image
acquisition
2. DEEP LEARNING
SOLUTIONS
for defect detection and
classification on the used
imagery types
3. BIG DATA
ANALYTICS
to handle the huge
datasets that are generated
by large-scale PV plants
School of Geomatics and Surveying Engineering, IAV Hassan II, Rabat, Morocco
Mohammadia School of Engineers, Mohammed V University in Rabat, Morocco
Research Institute for Solar Energy and New Energies, Rabat, Morocco.
ETAFAT, Casablanca, Morocco.
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
DATA ACQUISITION
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
RGB and thermal infrared on-
field image acquisition
SfM-MVS
photogrammetric
post-processing
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
DEEP LEARNING-BASED DEFECT DETECTION
IMAGE CLASSIFICATION
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
DEEP LEARNING-BASED DEFECT DETECTION
SEMANTIC SEGMENTATION
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4
INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND
THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
DEEP LEARNING-BASED DEFECT DETECTION
OBJECT DETECTION
Spatial Data Infrastructure and Earth Observation Education and Training for North Africa
THANK
YOU.

More Related Content

What's hot

Building Data Quality Audit Framework using Delta Lake at Cerner
Building Data Quality Audit Framework using Delta Lake at CernerBuilding Data Quality Audit Framework using Delta Lake at Cerner
Building Data Quality Audit Framework using Delta Lake at CernerDatabricks
 
An Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseAn Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseDataWorks Summit
 
“Accelerating Newer ML Models Using the Qualcomm AI Stack,” a Presentation fr...
“Accelerating Newer ML Models Using the Qualcomm AI Stack,” a Presentation fr...“Accelerating Newer ML Models Using the Qualcomm AI Stack,” a Presentation fr...
“Accelerating Newer ML Models Using the Qualcomm AI Stack,” a Presentation fr...Edge AI and Vision Alliance
 
Giving MongoDB a Way to Play with the GIS Community
Giving MongoDB a Way to Play with the GIS CommunityGiving MongoDB a Way to Play with the GIS Community
Giving MongoDB a Way to Play with the GIS CommunityMongoDB
 
Data training tips and tricks
Data training tips and tricksData training tips and tricks
Data training tips and tricksSarah Jones
 
MongoDB + GeoServer
MongoDB + GeoServerMongoDB + GeoServer
MongoDB + GeoServerMongoDB
 
Tutorial on Object Detection (Faster R-CNN)
Tutorial on Object Detection (Faster R-CNN)Tutorial on Object Detection (Faster R-CNN)
Tutorial on Object Detection (Faster R-CNN)Hwa Pyung Kim
 
Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...Universitat Politècnica de Catalunya
 
Human Pose Estimation by Deep Learning
Human Pose Estimation by Deep LearningHuman Pose Estimation by Deep Learning
Human Pose Estimation by Deep LearningWei Yang
 
HIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on HadoopHIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on HadoopZheng Shao
 
Parallel/Distributed Deep Learning and CDSW
Parallel/Distributed Deep Learning and CDSWParallel/Distributed Deep Learning and CDSW
Parallel/Distributed Deep Learning and CDSWDataWorks Summit
 
YOLOv4: optimal speed and accuracy of object detection review
YOLOv4: optimal speed and accuracy of object detection reviewYOLOv4: optimal speed and accuracy of object detection review
YOLOv4: optimal speed and accuracy of object detection reviewLEE HOSEONG
 
Data Visualization Project Presentation
Data Visualization Project PresentationData Visualization Project Presentation
Data Visualization Project PresentationShubham Shrivastava
 
Occlusion and Abandoned Object Detection for Surveillance Applications
Occlusion and Abandoned Object Detection for Surveillance ApplicationsOcclusion and Abandoned Object Detection for Surveillance Applications
Occlusion and Abandoned Object Detection for Surveillance ApplicationsEditor IJCATR
 
Machine Learning at Scale with MLflow and Apache Spark
Machine Learning at Scale with MLflow and Apache SparkMachine Learning at Scale with MLflow and Apache Spark
Machine Learning at Scale with MLflow and Apache SparkDatabricks
 
“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentat...
“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentat...“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentat...
“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentat...Edge AI and Vision Alliance
 
Plotly dash and data visualisation in Python
Plotly dash and data visualisation in PythonPlotly dash and data visualisation in Python
Plotly dash and data visualisation in PythonVolodymyr Kazantsev
 
3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -Kazuyuki Miyazawa
 
Transform Your Data Integration Platform From Informatica To ODI
Transform Your Data Integration Platform From Informatica To ODI Transform Your Data Integration Platform From Informatica To ODI
Transform Your Data Integration Platform From Informatica To ODI Jade Global
 
공간정보아카데미 - Day1 오픈소스개발 일반
공간정보아카데미 - Day1 오픈소스개발 일반공간정보아카데미 - Day1 오픈소스개발 일반
공간정보아카데미 - Day1 오픈소스개발 일반BJ Jang
 

What's hot (20)

Building Data Quality Audit Framework using Delta Lake at Cerner
Building Data Quality Audit Framework using Delta Lake at CernerBuilding Data Quality Audit Framework using Delta Lake at Cerner
Building Data Quality Audit Framework using Delta Lake at Cerner
 
An Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseAn Apache Hive Based Data Warehouse
An Apache Hive Based Data Warehouse
 
“Accelerating Newer ML Models Using the Qualcomm AI Stack,” a Presentation fr...
“Accelerating Newer ML Models Using the Qualcomm AI Stack,” a Presentation fr...“Accelerating Newer ML Models Using the Qualcomm AI Stack,” a Presentation fr...
“Accelerating Newer ML Models Using the Qualcomm AI Stack,” a Presentation fr...
 
Giving MongoDB a Way to Play with the GIS Community
Giving MongoDB a Way to Play with the GIS CommunityGiving MongoDB a Way to Play with the GIS Community
Giving MongoDB a Way to Play with the GIS Community
 
Data training tips and tricks
Data training tips and tricksData training tips and tricks
Data training tips and tricks
 
MongoDB + GeoServer
MongoDB + GeoServerMongoDB + GeoServer
MongoDB + GeoServer
 
Tutorial on Object Detection (Faster R-CNN)
Tutorial on Object Detection (Faster R-CNN)Tutorial on Object Detection (Faster R-CNN)
Tutorial on Object Detection (Faster R-CNN)
 
Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...
 
Human Pose Estimation by Deep Learning
Human Pose Estimation by Deep LearningHuman Pose Estimation by Deep Learning
Human Pose Estimation by Deep Learning
 
HIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on HadoopHIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on Hadoop
 
Parallel/Distributed Deep Learning and CDSW
Parallel/Distributed Deep Learning and CDSWParallel/Distributed Deep Learning and CDSW
Parallel/Distributed Deep Learning and CDSW
 
YOLOv4: optimal speed and accuracy of object detection review
YOLOv4: optimal speed and accuracy of object detection reviewYOLOv4: optimal speed and accuracy of object detection review
YOLOv4: optimal speed and accuracy of object detection review
 
Data Visualization Project Presentation
Data Visualization Project PresentationData Visualization Project Presentation
Data Visualization Project Presentation
 
Occlusion and Abandoned Object Detection for Surveillance Applications
Occlusion and Abandoned Object Detection for Surveillance ApplicationsOcclusion and Abandoned Object Detection for Surveillance Applications
Occlusion and Abandoned Object Detection for Surveillance Applications
 
Machine Learning at Scale with MLflow and Apache Spark
Machine Learning at Scale with MLflow and Apache SparkMachine Learning at Scale with MLflow and Apache Spark
Machine Learning at Scale with MLflow and Apache Spark
 
“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentat...
“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentat...“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentat...
“Tools for Creating Next-Gen Computer Vision Apps on Snapdragon,” a Presentat...
 
Plotly dash and data visualisation in Python
Plotly dash and data visualisation in PythonPlotly dash and data visualisation in Python
Plotly dash and data visualisation in Python
 
3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -
 
Transform Your Data Integration Platform From Informatica To ODI
Transform Your Data Integration Platform From Informatica To ODI Transform Your Data Integration Platform From Informatica To ODI
Transform Your Data Integration Platform From Informatica To ODI
 
공간정보아카데미 - Day1 오픈소스개발 일반
공간정보아카데미 - Day1 오픈소스개발 일반공간정보아카데미 - Day1 오픈소스개발 일반
공간정보아카데미 - Day1 오픈소스개발 일반
 

Similar to SEED4NA _AI4DRONE.pdf

Processing Drone data @Scale
Processing Drone data @ScaleProcessing Drone data @Scale
Processing Drone data @ScaleDr Hajji Hicham
 
AGIT 2015 - Hans Viehmann: "Big Data and Smart Cities"
AGIT 2015  - Hans Viehmann: "Big Data and Smart Cities"AGIT 2015  - Hans Viehmann: "Big Data and Smart Cities"
AGIT 2015 - Hans Viehmann: "Big Data and Smart Cities"jstrobl
 
Scaling up with Cisco Big Data: Data + Science = Data Science
Scaling up with Cisco Big Data: Data + Science = Data ScienceScaling up with Cisco Big Data: Data + Science = Data Science
Scaling up with Cisco Big Data: Data + Science = Data ScienceeRic Choo
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;Larry Smarr
 
Big Data with IOT approach and trends with case study
Big Data with IOT approach and trends with case studyBig Data with IOT approach and trends with case study
Big Data with IOT approach and trends with case studySharjeel Imtiaz
 
Hadoop Infrastructure @Uber Past, Present and Future
Hadoop Infrastructure @Uber Past, Present and FutureHadoop Infrastructure @Uber Past, Present and Future
Hadoop Infrastructure @Uber Past, Present and FutureDataWorks Summit
 
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksUsing Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksMapR Technologies
 
Data & Analytics - Session 1 - Big Data Analytics
Data & Analytics - Session 1 -  Big Data AnalyticsData & Analytics - Session 1 -  Big Data Analytics
Data & Analytics - Session 1 - Big Data AnalyticsAmazon Web Services
 
It's All About the Data - Tia Dubuisson
It's All About the Data - Tia DubuissonIt's All About the Data - Tia Dubuisson
It's All About the Data - Tia DubuissonCatalina Arango
 
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksUsing Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksDataWorks Summit
 
First in Class: Optimizing the Data Lake for Tighter Integration
First in Class: Optimizing the Data Lake for Tighter IntegrationFirst in Class: Optimizing the Data Lake for Tighter Integration
First in Class: Optimizing the Data Lake for Tighter IntegrationInside Analysis
 
IoT and Big Data - Iot Asia 2014
IoT and Big Data - Iot Asia 2014IoT and Big Data - Iot Asia 2014
IoT and Big Data - Iot Asia 2014John Berns
 
ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture
ADV Slides: When and How Data Lakes Fit into a Modern Data ArchitectureADV Slides: When and How Data Lakes Fit into a Modern Data Architecture
ADV Slides: When and How Data Lakes Fit into a Modern Data ArchitectureDATAVERSITY
 
Big data analytics and machine intelligence v5.0
Big data analytics and machine intelligence   v5.0Big data analytics and machine intelligence   v5.0
Big data analytics and machine intelligence v5.0Amr Kamel Deklel
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 

Similar to SEED4NA _AI4DRONE.pdf (20)

Processing Drone data @Scale
Processing Drone data @ScaleProcessing Drone data @Scale
Processing Drone data @Scale
 
AGIT 2015 - Hans Viehmann: "Big Data and Smart Cities"
AGIT 2015  - Hans Viehmann: "Big Data and Smart Cities"AGIT 2015  - Hans Viehmann: "Big Data and Smart Cities"
AGIT 2015 - Hans Viehmann: "Big Data and Smart Cities"
 
Scaling up with Cisco Big Data: Data + Science = Data Science
Scaling up with Cisco Big Data: Data + Science = Data ScienceScaling up with Cisco Big Data: Data + Science = Data Science
Scaling up with Cisco Big Data: Data + Science = Data Science
 
Hawaii Pacific GIS Conference 2012: LiDAR for Intrastructure and Terrian Mapp...
Hawaii Pacific GIS Conference 2012: LiDAR for Intrastructure and Terrian Mapp...Hawaii Pacific GIS Conference 2012: LiDAR for Intrastructure and Terrian Mapp...
Hawaii Pacific GIS Conference 2012: LiDAR for Intrastructure and Terrian Mapp...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Ies brazil dm 2013june25 ecopetrol
Ies brazil dm 2013june25 ecopetrolIes brazil dm 2013june25 ecopetrol
Ies brazil dm 2013june25 ecopetrol
 
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;
Panel: Building the NRP Ecosystem with the Regional Networks on their Campuses;
 
Big Data with IOT approach and trends with case study
Big Data with IOT approach and trends with case studyBig Data with IOT approach and trends with case study
Big Data with IOT approach and trends with case study
 
Hadoop Infrastructure @Uber Past, Present and Future
Hadoop Infrastructure @Uber Past, Present and FutureHadoop Infrastructure @Uber Past, Present and Future
Hadoop Infrastructure @Uber Past, Present and Future
 
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksUsing Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
 
Data & Analytics - Session 1 - Big Data Analytics
Data & Analytics - Session 1 -  Big Data AnalyticsData & Analytics - Session 1 -  Big Data Analytics
Data & Analytics - Session 1 - Big Data Analytics
 
It's All About the Data - Tia Dubuisson
It's All About the Data - Tia DubuissonIt's All About the Data - Tia Dubuisson
It's All About the Data - Tia Dubuisson
 
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise NetworksUsing Familiar BI Tools and Hadoop to Analyze Enterprise Networks
Using Familiar BI Tools and Hadoop to Analyze Enterprise Networks
 
DGterzo
DGterzoDGterzo
DGterzo
 
First in Class: Optimizing the Data Lake for Tighter Integration
First in Class: Optimizing the Data Lake for Tighter IntegrationFirst in Class: Optimizing the Data Lake for Tighter Integration
First in Class: Optimizing the Data Lake for Tighter Integration
 
IoT and Big Data - Iot Asia 2014
IoT and Big Data - Iot Asia 2014IoT and Big Data - Iot Asia 2014
IoT and Big Data - Iot Asia 2014
 
ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture
ADV Slides: When and How Data Lakes Fit into a Modern Data ArchitectureADV Slides: When and How Data Lakes Fit into a Modern Data Architecture
ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture
 
Big data analytics and machine intelligence v5.0
Big data analytics and machine intelligence   v5.0Big data analytics and machine intelligence   v5.0
Big data analytics and machine intelligence v5.0
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 

More from Dr Hajji Hicham

Slides Edataday2021_V2.pdf
Slides Edataday2021_V2.pdfSlides Edataday2021_V2.pdf
Slides Edataday2021_V2.pdfDr Hajji Hicham
 
Visual Transformer Overview
Visual Transformer OverviewVisual Transformer Overview
Visual Transformer OverviewDr Hajji Hicham
 
Distributed computing with Spark 2.x
Distributed computing with Spark 2.xDistributed computing with Spark 2.x
Distributed computing with Spark 2.xDr Hajji Hicham
 
Overview of Interpretability Approaches in Deep learning: Focus on Convnet ar...
Overview of Interpretability Approaches in Deep learning: Focus on Convnet ar...Overview of Interpretability Approaches in Deep learning: Focus on Convnet ar...
Overview of Interpretability Approaches in Deep learning: Focus on Convnet ar...Dr Hajji Hicham
 
Presentation intis 2017 version27112017
Presentation intis 2017 version27112017Presentation intis 2017 version27112017
Presentation intis 2017 version27112017Dr Hajji Hicham
 
Syllabus advanced big data with spark
Syllabus advanced big data with sparkSyllabus advanced big data with spark
Syllabus advanced big data with sparkDr Hajji Hicham
 

More from Dr Hajji Hicham (7)

Urban Big Data .pdf
Urban Big Data .pdfUrban Big Data .pdf
Urban Big Data .pdf
 
Slides Edataday2021_V2.pdf
Slides Edataday2021_V2.pdfSlides Edataday2021_V2.pdf
Slides Edataday2021_V2.pdf
 
Visual Transformer Overview
Visual Transformer OverviewVisual Transformer Overview
Visual Transformer Overview
 
Distributed computing with Spark 2.x
Distributed computing with Spark 2.xDistributed computing with Spark 2.x
Distributed computing with Spark 2.x
 
Overview of Interpretability Approaches in Deep learning: Focus on Convnet ar...
Overview of Interpretability Approaches in Deep learning: Focus on Convnet ar...Overview of Interpretability Approaches in Deep learning: Focus on Convnet ar...
Overview of Interpretability Approaches in Deep learning: Focus on Convnet ar...
 
Presentation intis 2017 version27112017
Presentation intis 2017 version27112017Presentation intis 2017 version27112017
Presentation intis 2017 version27112017
 
Syllabus advanced big data with spark
Syllabus advanced big data with sparkSyllabus advanced big data with spark
Syllabus advanced big data with spark
 

Recently uploaded

Bios of leading Astrologers & Researchers
Bios of leading Astrologers & ResearchersBios of leading Astrologers & Researchers
Bios of leading Astrologers & Researchersdarmandersingh4580
 
Predictive Precipitation: Advanced Rain Forecasting Techniques
Predictive Precipitation: Advanced Rain Forecasting TechniquesPredictive Precipitation: Advanced Rain Forecasting Techniques
Predictive Precipitation: Advanced Rain Forecasting TechniquesBoston Institute of Analytics
 
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarjSCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarjadimosmejiaslendon
 
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证pwgnohujw
 
obat aborsi Bontang wa 082135199655 jual obat aborsi cytotec asli di Bontang
obat aborsi Bontang wa 082135199655 jual obat aborsi cytotec asli di  Bontangobat aborsi Bontang wa 082135199655 jual obat aborsi cytotec asli di  Bontang
obat aborsi Bontang wa 082135199655 jual obat aborsi cytotec asli di Bontangsiskavia95
 
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...Klinik Aborsi
 
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi ArabiaIn Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabiaahmedjiabur940
 
如何办理澳洲拉筹伯大学毕业证(LaTrobe毕业证书)成绩单原件一模一样
如何办理澳洲拉筹伯大学毕业证(LaTrobe毕业证书)成绩单原件一模一样如何办理澳洲拉筹伯大学毕业证(LaTrobe毕业证书)成绩单原件一模一样
如何办理澳洲拉筹伯大学毕业证(LaTrobe毕业证书)成绩单原件一模一样wsppdmt
 
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24  Building Real-Time Pipelines With FLaNKDATA SUMMIT 24  Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNKTimothy Spann
 
Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024patrickdtherriault
 
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...ThinkInnovation
 
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...mikehavy0
 
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...Elaine Werffeli
 
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证acoha1
 
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...ThinkInnovation
 
RESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptx
RESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptxRESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptx
RESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptxronsairoathenadugay
 
社内勉強会資料_Object Recognition as Next Token Prediction
社内勉強会資料_Object Recognition as Next Token Prediction社内勉強会資料_Object Recognition as Next Token Prediction
社内勉強会資料_Object Recognition as Next Token PredictionNABLAS株式会社
 
Unsatisfied Bhabhi ℂall Girls Vadodara Book Esha 7427069034 Top Class ℂall Gi...
Unsatisfied Bhabhi ℂall Girls Vadodara Book Esha 7427069034 Top Class ℂall Gi...Unsatisfied Bhabhi ℂall Girls Vadodara Book Esha 7427069034 Top Class ℂall Gi...
Unsatisfied Bhabhi ℂall Girls Vadodara Book Esha 7427069034 Top Class ℂall Gi...Payal Garg #K09
 

Recently uploaded (20)

Bios of leading Astrologers & Researchers
Bios of leading Astrologers & ResearchersBios of leading Astrologers & Researchers
Bios of leading Astrologers & Researchers
 
Predictive Precipitation: Advanced Rain Forecasting Techniques
Predictive Precipitation: Advanced Rain Forecasting TechniquesPredictive Precipitation: Advanced Rain Forecasting Techniques
Predictive Precipitation: Advanced Rain Forecasting Techniques
 
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarjSCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
SCI8-Q4-MOD11.pdfwrwujrrjfaajerjrajrrarj
 
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
 
obat aborsi Bontang wa 082135199655 jual obat aborsi cytotec asli di Bontang
obat aborsi Bontang wa 082135199655 jual obat aborsi cytotec asli di  Bontangobat aborsi Bontang wa 082135199655 jual obat aborsi cytotec asli di  Bontang
obat aborsi Bontang wa 082135199655 jual obat aborsi cytotec asli di Bontang
 
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...
 
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi ArabiaIn Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia
 
如何办理澳洲拉筹伯大学毕业证(LaTrobe毕业证书)成绩单原件一模一样
如何办理澳洲拉筹伯大学毕业证(LaTrobe毕业证书)成绩单原件一模一样如何办理澳洲拉筹伯大学毕业证(LaTrobe毕业证书)成绩单原件一模一样
如何办理澳洲拉筹伯大学毕业证(LaTrobe毕业证书)成绩单原件一模一样
 
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24  Building Real-Time Pipelines With FLaNKDATA SUMMIT 24  Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
 
Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024
 
Abortion pills in Riyadh Saudi Arabia| +966572737505 | Get Cytotec, Unwanted Kit
Abortion pills in Riyadh Saudi Arabia| +966572737505 | Get Cytotec, Unwanted KitAbortion pills in Riyadh Saudi Arabia| +966572737505 | Get Cytotec, Unwanted Kit
Abortion pills in Riyadh Saudi Arabia| +966572737505 | Get Cytotec, Unwanted Kit
 
Abortion pills in Doha {{ QATAR }} +966572737505) Get Cytotec
Abortion pills in Doha {{ QATAR }} +966572737505) Get CytotecAbortion pills in Doha {{ QATAR }} +966572737505) Get Cytotec
Abortion pills in Doha {{ QATAR }} +966572737505) Get Cytotec
 
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
 
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
Abortion Clinic in Kempton Park +27791653574 WhatsApp Abortion Clinic Service...
 
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...
 
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UPenn毕业证书)宾夕法尼亚大学毕业证成绩单本科硕士学位证留信学历认证
 
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
 
RESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptx
RESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptxRESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptx
RESEARCH-FINAL-DEFENSE-PPT-TEMPLATE.pptx
 
社内勉強会資料_Object Recognition as Next Token Prediction
社内勉強会資料_Object Recognition as Next Token Prediction社内勉強会資料_Object Recognition as Next Token Prediction
社内勉強会資料_Object Recognition as Next Token Prediction
 
Unsatisfied Bhabhi ℂall Girls Vadodara Book Esha 7427069034 Top Class ℂall Gi...
Unsatisfied Bhabhi ℂall Girls Vadodara Book Esha 7427069034 Top Class ℂall Gi...Unsatisfied Bhabhi ℂall Girls Vadodara Book Esha 7427069034 Top Class ℂall Gi...
Unsatisfied Bhabhi ℂall Girls Vadodara Book Esha 7427069034 Top Class ℂall Gi...
 

SEED4NA _AI4DRONE.pdf

  • 1. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa AI and Big Data for Drone processing usecase of photovoltaic inspection 25-April 2022
  • 2. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Agenda • Big Data/AI and Drone • Opportunities • Challenges, Why is it Hard? • Big Data Challenges… • Toward a new architecture for drone Big Data • Partitioning • Storage • Computing • Some existing Big Data/AI frameworks for Drone • Use case: • Using AI and Drone for photovoltaic inspection
  • 3. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Big Data / AI and Drone
  • 4. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Why!
  • 5. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Moreover • The potential of drones data is often underestimated • Archiving collected data • Curretly, we are doing more archiving tasks than managing drone data efficiently • Almost no existing Big Data infrastructure can handle drones efficiently, • Even if Big data is almost mature for other domains: Finance, Banking… • Often it is • Hard to store • Hard to manage • Hard to process • Hard to get insight • How ???
  • 6. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Hard to store: Volume A very small drone project can generate more than 10 GB, sometimes more than 40Gb 15 million images of drone can make up more than 175 terabytes of data. How to Store and Compute such growing volume? FEDS : 13,000 flights this year
  • 7. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Hard to store: Variety • “Drones can now provide a wide variety of data types, everything from a few basic photos through to complex measurable 3D models with annotations and overlays.” Visual Encylopedia of drone data Aerial Photography and Video Orthomosaic Map Digital Elevation Model (DEM) 3D Pointcloud Model Multispectral Mapping Thermal Imagery and Mapping
  • 8. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Hard to process: Computing Model and Scalability • Currently, drone image processing are done in one server: NOT SCALABLE • Scalability is the property of a system to handle a growing amount of work by adding resources to the system • In Big Data, It is mostly done by distributing storage and computing • Distributed computing can provide Scalability, but drone data friendly is Difficult Processing/Querying drone data can take up to a few hours  Objective : real time (few seconds)
  • 9. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa  Going beyond traditional algorithms  Why not use Neural networks that have made great success with image: ▪ Semantic segmentation ▪ Object recognition, Classification.. ▪ Description Generation for Drone Images Using Attribute Attention Mechanism  But theses new algorithms require more storage capacities and computing power Hard to get Insights
  • 10. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Current approaches are obsolete we need to reinvent everything Storage Access Availability Computing Fast Accurate Analytics Machine Learning Deep Learning Search By semantic By Spatial Queries …
  • 11. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 1. New architecture to be redefined Analytical Queries Structured Storage Cluster Computing Cluster … Large Scale Time series NDVI •Distributing both STORAGE •AND COMPUTING
  • 12. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 2. Need to correlate drone data with external datasets More Insights Census Data Economic Data Weather …
  • 13. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 3. Toward a declarative language (SQL-Like) over drone data Change in NDVI over the spring and early summer of 2018 Select normalized_difference(nir, red) as ndvi From Feds_droneDataset Where date between ‘10-10-2017’ and ‘10-10-2019’ Examples from ‘10-10-2017’ to ‘10-10-2019’ Best option for Data Scientists
  • 14. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Recall that storage should be distributed across a cluster • Before detailing storage techniques, let’s talk about Partitioning Structured Storage Cluster … Node A Node B Node F Node G
  • 15. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Challenge for going distributed: Data Partitioning  Partitioning means the process of physically dividing data into separate data stores  Data is divided into partitions that can be managed and accessed separately. Node 1 Node 2 Node 3 Node 4
  • 16. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Node 1 Node 2 Node 3 By Band RGB Red Band Green Band Blue Band First simple approach is to partition by band
  • 17. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Node 1 Node 2 Node 3 By Time Spring Summer Autumn Other simple approach is to partition by time (season)
  • 18. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Node 1 Node 2 Node 3 Decompose into NxN regular grids But the Most efficient approach is to combine Tiling and Distribution Tiling allows large raster datasets to be broken-up into manageable pieces  higher level raster I/O interface.
  • 19. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Which Partition strategy to choose? • Not in the scope of this presentation • Check with your main objective: • If for Scalability, • If for Query Performance, • If for Availability • Many Best practices are available • Sometimes we make use of Global Index for Optimizing Queries
  • 20. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa The computing part Computing Model HADOOP/MapReduce Spark/Spark SQL We have at least two interesting computing models
  • 21. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Spark vs Hadoop MapReduce Source: Data Flair We will focus Next on Apache Spark According to benchmarks studies, Spark is much better than Hadoop MapReduce
  • 22. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Frameworks for Raster Big Data
  • 23. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Frameworks for Raster Big Data Apache Spark / Spark SQL • Rasterframes (My favorite) Earth AI (To follow) Google Earth Engine Rasdaman SciDB
  • 24. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa • Spark project for Raster Data • Spark Dataframe like abstraction for handling Raster Data : Provides ability to work with Raster imagery in a convenient yet scalable format • You can use Spark ML for building ML Models B1 B2 B3 B4 tile or tile_n (where n is a band number)
  • 25. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Standard Tile Operations • Many raster operations are ready to be executed in a distributed manner : can be executed over Spark Cluster • Ready to use
  • 26. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa RasterFrames: SQL Query • Can I Use spatial predicate in my query: intersection query?
  • 27. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa SQL query in Rasterframes SELECT month, ndvi_stats.* FROM (" SELECT month, rf_agg_stats(rf_normalized_difference(nir, red)) as ndvi_stats FROM red_nir_tiles_monthly_2017 WHERE st_intersects(st_reproject(rf_geometry(red), rf_crs(red), 'EPSG:4326'), st_makePoint(34.870605, -4.729727)) GROUP BY month ORDER BY month ) "")  Compute the average NDVI per month for a single tile in an Area of Interest
  • 28. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Earth AI • is a Cloud-native software that enables you to apply advanced machine learning algorithms to EO data at scale • Both a non-code-based visual interface and pre-built workflows • Ready-To-Use Datasets • data archive includes more years of historical imagery and scientific datasets • Elastic Compute • Designed for scalability from the beginning, Earth AI platform scales seamlessly, so you can think more about insights than Dev Ops
  • 29. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Earth AI
  • 30. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa Earth AI • Classifying an ecoregion using Decision Tree Classifier
  • 31. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa SciDB • Array-based data management and analytical system • Arrays are divided into equally sized chunks • Chunks are distributed over many SciDB instances • Size and shape of chunks are defined by users per array and have strong effects on computation times • Storage is nearly sparse • Relies on shared nothing architectures • Open-source version available, extensible by UDFs
  • 32. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES Imane SEBARI (a), Yahya ZEFRI (a), Hicham HAJJI (a), Ghassane ANIBA (b) (a) Photogrammetry-Cartography Department, School of Geomatics and Surveying Engineering, IAV Hassan II, Rabat, Morocco (b) Electrical Engineering Department, Mohammadia School of Engineers, Mohammed V University in Rabat, Morocco www.smartdrone4pv.com
  • 33. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES CONTEXT & PROBLEMATIC Solar Photovoltaic Modules Contrasted Temperatures Humidity Intrusion Fierce Winds Rain/Snow/Hail Handling & Installation Multiple Defects that develop over time and penalize the electricity production Remotely Sensed RGB and infrared Imagery by UAVs Contactless characterization Faster image acquisition Large-scale coverage Increased accessibility Hotspots Delaminations Discolorations Cracks
  • 34. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa THE SMARTDRONE4PV PROJECT 1. ADVANCED UAV PHOTOGRAMMETRY for RGB and long-wave thermal infrared image acquisition 2. DEEP LEARNING SOLUTIONS for defect detection and classification on the used imagery types 3. BIG DATA ANALYTICS to handle the huge datasets that are generated by large-scale PV plants School of Geomatics and Surveying Engineering, IAV Hassan II, Rabat, Morocco Mohammadia School of Engineers, Mohammed V University in Rabat, Morocco Research Institute for Solar Energy and New Energies, Rabat, Morocco. ETAFAT, Casablanca, Morocco. INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES
  • 35. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa DATA ACQUISITION INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES RGB and thermal infrared on- field image acquisition SfM-MVS photogrammetric post-processing
  • 36. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4 INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES DEEP LEARNING-BASED DEFECT DETECTION IMAGE CLASSIFICATION
  • 37. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4 INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES DEEP LEARNING-BASED DEFECT DETECTION SEMANTIC SEGMENTATION
  • 38. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa 4 INTELLIGENT INSPECTION OF LARGE-SCALE PHOTOVOLTAIC INSTALLATIONS THROUGH RGB AND THERMAL INFRARED IMAGERY ACQUIRED BY UNMANNED AERIAL VEHICLES DEEP LEARNING-BASED DEFECT DETECTION OBJECT DETECTION
  • 39. Spatial Data Infrastructure and Earth Observation Education and Training for North Africa THANK YOU.