Exploring RGB+DSM Convolutional Neural Networks Fusion Strategies for Satellite Imagery Segmentation

1. Exploring RGB+DSM Convolutional Neural
Networks Fusion Strategies for
Satellite Imagery Segmentation
Mendrika Ramarlina
www.haizaha.com

2. Dataset
● 38 tiles of 6000x6000px Orthorectified RGB Imagery
● DSM: ground height removed for each pixel => height above the ground
● Ground Sampling Distance of 5cm for both DSM and TOP
● Two classes:
○ Background
○ Building
● Pre-processing:
○ Each tile is split into patches of 2048x2048 pixels
○ Resized down to 256x256

3. Baseline Architecture: Unet-ResNet34
● Backbone: Unet - https://arxiv.org/abs/1505.04597
● Encoder: 34 Layers Residual Network - https://arxiv.org/abs/1512.03385
● Layer fusion strategy: Conv2d Fusion
○ First stack feature maps
○ Then run a 2d convolution with 1x1 kernel filter

4. Model: RGB Unet ResNet-34
● Input: RGB images
● Weights initialization: ImageNet
● IOU: 0.836

5. Model: 1-Channel DSM Only
● Input: DSM
● Weight Initialization: average of RGB channels from ImageNet pre-trained weights
● IOU: 0.832

6. Model: Elevation Detection Pre-training
● Pre-training phase: train an elevation detection model
● Re-training the model on ground truth
● IOU: 0.829

7. Model: Fused Segmentation-Heads
● Practically a model ensemble
● Fusion mechanism: Conv2d Fusion
● IOU: 0.865

8. Model: Fused Decoders
● Fusion mechanism: Conv2d Fusion
● IOU: 0.863

9. Model: Fused Encoders
● Fusion mechanism: Conv2d fusion and summation fusion
● IOU: 0.87

10. Model: 4-Channels RGB-D
● Fusion mechanism: Conv2d Fusion
● IOU: 0.888

11. Results: Final IOU vs. Fusion Location

12. Results
Input Data Initialization Fusion Location Epochs 0 - 200 Epochs 200 - 400
RGB ImageNet 0.833 0.836
RGB Elevation Detection 0.819 0.829
DSM ImageNet 0.813 0.832
RGB-DSM ImageNet Input 0.877 0.888
RGB-DSM ImageNet Encoder 0.876 0.87
RGB-DSM ImageNet Decoder 0.207 0.863
RGB-DSM ImageNet
Segmentation
Heads
0.164 0.865

13. Ways to Get in Touch
Mendrika Ramarlina
mendrika@haizaha.com
linkedin.com/in/mendrikaramarlina
github: @ramarlina