For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit-kraft For more information about embedded vision, please visit: http://www.embedded-vision.com Adam Kraft, Deep Learning Engineer at Orbital Insight, presents the "Challenges and Approaches for Extracting Meaning from Satellite Imagery" tutorial at the May 2019 Embedded Vision Summit. Orbital Insight is a geospatial big data company leveraging the rapidly growing availability of satellite, UAV and other geospatial data sources to understand and characterize socioeconomic trends at global, regional and hyper-local scales. The company uses recent advances in deep learning and cloud computing to process and understand the information available in millions of images. In this talk, Kraft explores three sets of technical challenges the company had to overcome to develop a robust solution. First, he discusses challenges associated with fusing data that can come from different types of image sensors as well as different ground truth measurement sources. Next, he explores challenges related to detecting trends and analyzing changes in imagery over time. Finally, he examines challenges involved in choosing the right machine learning methods for the task at hand.

1. © 2019 Orbital Insight
Challenges and Approaches for
Extracting Meaning from Satellite
Imagery
Adam Kraft
Orbital Insight
May 2019

2. © 2019 Orbital Insight
Overview
• Introduction to Orbital Insight
• Practical Machine Learning Methods for Satellite Imagery
• Working with Multiple Data Sources
• Analyzing Trends and Change Over Time
2

3. © 2019 Orbital Insight
Introduction to Orbital Insight

4. © 2019 Orbital Insight
Helping Clients Make Better Decisions Today
Financial data
Ship location
(AIS) data
EO/SAR Satellite
Imagery
Drone / Aerial
Imagery
Manufacturing
data
Identification of
areas of interest
Weather data
Mobile Device
data
Business data
IOT data
Area of Interest
(AOI) data
Image ingestion
and processing
Normalization /
pattern detection
GPU-based
CV / ML
RESTful API
Web App
SaaS
Orbital Insight
Orbital Insight sources, processes, and
transforms geospatial datasets at scale

5. © 2019 Orbital Insight
TRUCKS
LAND USE
TANK SHADOWS
SHIPS
NEW HOUSING DEVELOPMENT
AIRPLANES
BUILDINGS
RAILCARS
Computer Vision is Critical for Large Scale Processing of Satellite Imagery

6. © 2019 Orbital Insight
Practical Machine Learning
Methods for Satellite Imagery

7. © 2019 Orbital Insight
Common ML Pipeline
Data ML Model
Loss
Function(s)
7
Optimization

8. © 2019 Orbital Insight
Data Handling for Satellite Imagery
• Augmentations
• Full 360 degree rotations
• Shear transformations
• Adding artificial clouds and haze
8
Data ML Model
Loss
Function(s)

9. © 2019 Orbital Insight
Data Handling for Satellite Imagery
• Sampling
• Class imbalance
• Rare cases
• Active learning
9
Data ML Model
Loss
Function(s)
Source:
http://burrsettles.com/pub/settles.activelearning.pdf

10. © 2019 Orbital Insight
ML Model for satellite imagery
• Architecture
• Aim to retain full resolution
• Pretraining with more/less than
3 channels
• Can sample 3 channels
• Add zeros for extra
channels, freeze part of
network for a few epochs
10
Data ML Model
Loss
Function(s)
ImageNet car

11. © 2019 Orbital Insight
Loss Functions for satellite imagery
• Handle class imbalance
• Enforce temporal consistency
• Can predict “free” information
• Satellite metadata
• Distance or neighbor
information
11
Data ML Model
Loss
Function(s)
https://earthobservatory.nasa.gov/features/ColorImage/page2.php

12. © 2019 Orbital Insight
Working with Multiple Data
Sources

13. © 2019 Orbital Insight
Variety of Data / Inputs from Multiple Sources
13

14. © 2019 Orbital Insight
Variance in Observing Same Location
14

15. © 2019 Orbital Insight
Domain Transfer and Adaptation
Domains can differ across:
• Seasons
• Geographies
• Sensors
15
M. Wulfmeier, A. Bewley, and I. Posner, “Addressing Appearance Change in Outdoor Robotics with
Adversarial Domain Adaptation,” in IEEE/RSJ International Conference on Intelligent Robots and
Systems, 2017.

16. © 2019 Orbital Insight
How to Best Combine Sources?
• Fuse inputs
• Easier to integrate
• Fuse outputs
• Easier to interpret
• Intermediate fusion
• Difficult to
integrate/interpret,
yet may yield best
results
16
CNN Output
CNN Output

17. © 2019 Orbital Insight
Analyzing Trends and Change
Over Time

18. © 2019 Orbital Insight
More Data is Usually Better
• Signal increases as data increases
• Using more sources increases signal
• Errors can wash out
• Less dependent on accuracy of individual measurements
18

19. © 2019 Orbital Insight
Temporal Data
• Not just analyzing snapshots
• Non-uniform samples
• Unlike other temporal data: audio, video
• Noise from clouds and haze
19

20. © 2019 Orbital Insight
Methods for Learning on Temporal Data
• LSTMs / 3D convolutions
• Better uncertainty outputs from ML models
• Results in even better trend analysis
20
Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer
vision?." Advances in neural information processing systems. 2017.

21. © 2019 Orbital Insight
Conclusions
• Know your data. Satellite imagery has different characteristics than
consumer camera images. We can use this to our advantage.
• Satellite data contains many different sources and there are different
ways to combine information from those sources
• The temporal component to satellite imagery can add challenges, which
you should account for in your ML models.
21

22. © 2019 Orbital Insight
Additional Resources
22
Orbital Insight Links
Company Website
https://orbitalinsight.com/
New York Times Article
https://www.wsj.com/articles/startups-
mine-market-moving-data-from-fields-
parking-lotseven-shadows-1416502993
Other Satellite Imagery Links
Functional Map of the World Challenge
https://www.iarpa.gov/challenges/fmow.ht
ml

23. © 2019 Orbital Insight
Back-up Slides

24. © 2019 Orbital Insight
Orbital Insight uses computer vision
and data science to turn millions of
images into a big-picture
understanding of the world.
Port of Rotterdam. Image Source: Astrium

25. Image Source: PBS Image, SpaceX Falcon Heavy Launch
Defining the New Geospatial Analytics Category
Commercialization of Space, Artificial Intelligence, Cloud & GPUs
Commercialization
of Space
Cloud Computing
& GPUs
25
Artificial
Intelligence
(Computer Vision
& Data Science)
Launch Systems
Satellite Operations
Analytics
© Orbital Insight

26. © Orbital Insight 13
Image Source: Orbital Insight Data Overlaid on a Satellite Image
Consumer Traffic
Parked Car Counting; Retailers & Malls
UNITED STATES