SpaceNet: Accelerating Machine Learning for Foundational Mapping Challenges

P U B L I C S E C T O R
S U M M I T
Washington DC

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
SpaceNet: Accelerating Machine
Learning for Foundational Mapping
Challenges
Ryan Lewis
SVP
IQT CosmiQ Works
S e s s i o n I D
Joe Flasher
Open Geospatial Data Lead
AWS
Adam Van Etten
Research Director
IQT CosmiQ Works
Todd Bacastow
Senior Director
Maxar

S U M M I T
Agenda
SpaceNet Introduction
Previous Challenge Results
Upcoming Challenges
Information Channels

S U M M I T

S U M M I T
5
© SpaceNet LLC 2019.
(1) Machine learning algorithms & (2) increased overhead data
collection will fundamentally disrupt geospatial analytics
Convergence of Two Tech Trends
5

S U M M I T
6
Solutions Are Required
Source: DigitalGlobe/Maxar, CNN, and Humanitarian Open Street Map (HOT).
Required to Map Puerto Rico After Hurricane Maria
70+
Days to Completely
Map
5,300+
Volunteer Mappers
950,000
Building Labels
30,000
Kms of Road Labels
6

S U M M I T
3 Market Challenges
7
Lack of Curated, Labeled Data Sets
for Geospatial Applications
Open Source, AI Models Designed
for Different Problems
Open Software Tools for Geospatial
Analysis Are Limited

S U M M I T
8
SpaceNet’s Mission
SpaceNet is a nonprofit LLC focused on:
1. Data Developing Open Source Data Sets
2. Algorithms Fostering Applied Research for AI Software
3. Evaluation Benchmarking Performance for Applications

S U M M I T
SpaceNet: 4 Pillars
9
Labeled Data Sets Competitions Algorithms Software Tools
• Images of 6 Cities
• 800,000+ Building
Footprints
• 10,000 km2 Road Labels
• 4 Competitions on TopCoder
• $200,000 in Total Prizes
• 1,000+ Submissions
Worldwide
• 18 Algorithms
o 13 Building Detection
o 5 Road Detection &
Routing
• Ease Use of Imagery
• Simplify Evaluation
• Speed Up Model
Deployment

S U M M I T
SpaceNet: Opening the Floodgates for
GEOINT R&D 2016
SpaceNet 1: Building Footprint Extraction
Cars Overhead With Context (COWC)
IARPA Multi-View Stereo 3D Mapping
2017
SpaceNet 2: Multi-City Building Footprints
IARPA Functional Map of the World
USSOCOM Urban 3D Challenge
2018
SpaceNet 3: Road Network Extraction
SpaceNet 4: Off-Nadir Building Footprints
CrowdAI Mapping Challenge
DIUx xView Object Detection Challenge
2019
Microsoft U.S. & Canadian Building
Footprints
Upcoming: SpaceNet 5: Roads with travel
time 10

S U M M I T
Competitions to Date
11
SpaceNet 1
11/2016 – 1/2017
SpaceNet 2
6/2017 – 8/2018
SpaceNet 3
11/2017 – 2/2018
SpaceNet 4
10/2018 – 1/2019
Building Footprint
Detection
Rio De Janeiro
Building Footprint
Detection
Las Vegas, Paris,
Khartoum, & Shanghai
Road Extraction &
Routing
Las Vegas, Paris,
Khartoum, & Shanghai
Building Footprint
Detection (Off-
Nadir)
Atlanta

S U M M I T
Automated Overhead Imagery Analysis is
Improving
12

S U M M I T
13
Public Data Sets: 1 Open Source
Software: 1
Open Source Software:
2
Competition
Submissions
347M
Total Repository Hits
18
Algorithms
22
CosmiQ
Repositories
1,000+
Across All 4
Challenges
Public Data Sets: 2
268TB
Total Downloads
Serving an Unmet Need
International
Participation
79
Countries with
Downloads

S U M M I T
Impact: Public Data Sets
14
Top Country Hits (2018)
1. USA
2. Canada
3. China
4. India
5. UK

S U M M I T
15
Fei Fei Li’s, founder of ImageNet, presentation at CVPR
2017
Community Acknowledgement

S U M M I T
SpaceNet 4 Overview
17
Imagery Data Set
27 Collects Over Atlanta
7O to 54O Off-Nadir
655 km2 Covered
0.5 m Resolution
Labels
126,747 Building Footprints
From 20 m2 to >2,000 m2
Urban, Industrial, & Suburban
~3,000 km Road Network
Labels
Algorithms
5 Open Sourced Solutions
15 Computer Vision Models
w/
Solution Explanations
> 250 Competition
Submissions

S U M M I T
Why Off-Nadir Imagery
18
Urgent Collections are Often Off-Nadir (below)
State-of-the-art algorithms were untested on off-nadir imagery
Daiichi Power Plant | Fukushima, Japan
Look Angle: About 45º
Imagery Courtesy of DigitalGlobe

S U M M I T
Barriers to Off-Nadir Imagery Analysis
19
Variable Shadows
Occluded Structures
Footprint Displacement
Resolution Degradation

S U M M I T
Scoring: Building Footprint Extraction
20
1. Find predicted buildings with Intersection over Union (IoU) > 0.5
Truth Pred.
IoU = 0.75
Success
IoU = 0.15
Failure
2. Aggregate successes/failures across all collections in three look angle bands:
Ground truth
A. Nadir: 0-25
B. Off-nadir: 26-40
C. Very off-nadir: > 40

S U M M I T
Algorithmic Challenges: Nadir Angle
21
40% drop in score for all
algorithms from 7º to
54º

S U M M I T
22
Algorithmic Challenges?: Shadows
Image from the South
(facing north)
Image from the North
(facing south)

S U M M I T
Top buildings:
Not occluded
Bottom buildings:
Occluded by trees
Algorithmic Challenges?: Occluded
Buildings

S U M M I T
Algorithmic Challenges: Building Size

S U M M I T
Footprint Quality Threshold Matters
25
Truth Pred.
IoU = 0.75
IoU = 0.15

S U M M I T
Research Publication from SpaceNet 4
26
Link: https://arxiv.org/abs/1903.12239

S U M M I T
What’s Next: Returning to Road Networks
Scoring based on pixel masks does not always incentivize the
desired outcomes

S U M M I T
• Segmentation efforts have demonstrated some success in identifying road pixels from
overhead imagery but do not always incentivize the desired outcome
• Evaluation metrics are pixel-based: (1) completeness, correctness, quality, and
(2) relaxed F1 (correct value within 3 pixels
Wang et al 2016 (Qaulity = 0.86)
http://www.mdpi.com/2220-9964/5/7/114
Zhang et al 2017 (relaxed F1 = 0.92)
https://arxiv.org/pdf/1711.10684.pdf
Mhih and Hinton 2010 (relaxed F1 = 0.90)
http://www.cs.toronto.edu/~fritz/absps/road_detection.p
df
Limitations of Current Segmentation
Techniques

S U M M I T
Scoring Routing Information: APLS
•Average Path Length Similarity
(APLS) was developed for SN3
•Both Logical and Physical
Topology Are Important for Road
Detection
•Sum the Difference in Paths
between Ground Truth &
Proposals
•Betweenness Centrality is
Fundamental to the APLS Metric

S U M M I T
Ground
Truth
1. Parse road labels into 400m sections concurrent with SpaceNet imagery
2. Create ground truth masks by drawing a 2m buffer around road centerlines in road labels
3. Augment the training dataset by a factor of 3 via HSV rescaling and rotations [increases
performance by 8% (Vegas) to 13% (Khartoum)]
4. Train a deep learning segmentation model (PSPNet, U-Net) using SpaceNet imagery & road
masks
5. Perform post-processing to eradicate short segments and close small gaps
Predictions: Segmentation Model

S U M M I T
A. Extract skeleton from proposal mask
B. Build a proposal graph from the skeleton
C. Simplify and smooth proposal graph
A B C
Predictions: Mask to Graph

S U M M I T
Entrant Country Avg Score Las Vegas Paris Shanghai Khartoum
albu Russia 0.6663 0.7977 0.6040 0.6543 0.6093
cannab Russia 0.6661 0.7804 0.6446 0.6398 0.5996
pfr France 0.6660 0.8009 0.6008 0.6646 0.5975
selim_sef Germany 0.6567 0.7884 0.5991 0.6472 0.5922
fabastani USA 0.6284 0.7710 0.5474 0.6326 0.5628
ipraznik Germany 0.6215 0.7578 0.5668 0.6078 0.5537
tcghanareddy India 0.6182 0.7591 0.5710 0.6014 0.5415
hasan.asyari Norway 0.6097 0.7407 0.5557 0.5952 0.5472
aveysov Russia 0.5943 0.7426 0.5805 0.5751 0.4789
SpaceNet 3 Results
Winning entrant, albu, submitted a generalized model across all four
cities

S U M M I T
AOI_2_Vegas_img1011 – APLS =
0.512
AOI_2_Vegas_Img1045 – APLS =
0.988
Dave Lindenbaum, GTC 2018
Las Vegas Results

S U M M I TDave Lindenbaum, GTC 2018
Khartoum Results
AOI_5_Khartoum_img404 - APLS =
0.385
AOI_5_Khartoum_img398 – APLS =
0.897
35

S U M M I T
Ground Truth
Raw Proposal Mask
Extract Entire Khartoum Road Network
•Combined BASISS & Albu’s Implementation (w/
Extra Post-Processing from CosmiQ Works)
o Image Size (Pixels) 55,420 x 161,258 (9
terapixels)
o Image Size (Km) 16 x 48
o File Size (GB) 89
o Nodes 195,938
o Edges 258,655
Total Processing Time = 6.3 Hours (Single
GPU/CPU)

S U M M I T
SpaceNet 5: Expanding Upon Routing
37
Challenge Participants Will Be Asked to Infer:
… From a Single Satellite Image
Road
Networks
Routing
Information
Travel
Times
The SpaceNet 5 Challenge is Scheduled to Launch in September 2019

S U M M I T
APLS = 0.81
Test Image: RGB-PanSharpen_AOI_2_Vegas_img727.tif
Multiclass Baseline
38
•Trained resnet34 + unet segmentation model
o Use 7– channel training masks
RGB Image Labels Projection

S U M M I T
SpaceNet 6: Preliminary Planning
39
Model Deployability /
Generalizability
New Applications Beyond
Foundational Mapping
Sensor & Data
Fusion

S U M M I T
Information Channels
41
CosmiQ’s Repo:
https://github.com/CosmiQ
SpaceNet’s Repo:
https://github.com/SpaceNetChalle
nge
SpaceNet Competition
Hosting Site
https://www.topcoder.com/space
net
SpaceNet Data on AWS
https://registry.opendata.aws/spac
enet/
CosmiQ’s Blog:
The DownLinQ
https://medium.com/the-
downlinq
CosmiQ’s
Twitter:
@CosmiQWorks
Title:
Training_Data
Found on: Apple
Podcasts, Spotfiy,
Stitcher, & SoundCloud

S U M M I T
Thank you!
S U M M I T
Ryan Lewis
rlewis@iqt.org

S U M M I T
S U M M I T

SpaceNet: Accelerating Machine Learning for Foundational Mapping Challenges

Recommended

Recommended

More Related Content

Similar to SpaceNet: Accelerating Machine Learning for Foundational Mapping Challenges

Similar to SpaceNet: Accelerating Machine Learning for Foundational Mapping Challenges (20)

More from Amazon Web Services

More from Amazon Web Services (20)

SpaceNet: Accelerating Machine Learning for Foundational Mapping Challenges