(1) Deep learning algorithms show potential for sea ice classification from SAR images but face challenges from scarce and inaccurate training data. (2) Researchers generated training datasets by manually labeling SAR image patches with ice types, assisted by optical images. (3) A modified VGG-16 network trained on augmented SAR patch data achieved 97.3% accuracy classifying ice vs water.

1. Deep Learning Algorithms for Sea Ice
Classification from Sentinel-1 SAR images:
Capabilities and Challenges
Torbjørn Eltoft, Salman Khaleghian,
Qiang Wang
The Arctic University of Norway & CIRFA

2. Acknowledgement
With contributions from:
Salman Khaleghian
Thomas Kramer
Anthony Doulgeris
Qiang Wang
Adrea Marinoni

3. Outline
• Introduction
• Some background
• Training data generation
• Examples
• Supervised DL for Sea Ice Classification
• Pixel-wise classification
• Semi-Supervised Deep Learning
• Summary and conclusions

4. ExtremeEarth (H2020-project)
• Develop and validate automatic ice mapping methods and expand them to the
wider Arctic and Antarctic
• Integrate new automated ice mapping algorithms into the Polar Thematic
Exploitation Platform and utilise network of data access and processing resources.
• Prove operational delivery of new ice information products.
The objective of the EE project is to
develop deep learning techniques for
the computation of extreme analytics
over big Copernicus data.

5. • Based on
• SAR images from S1, RS-2,
COSMO SkyMed.
• Optical satellite images
• Visual observations
• Weather forecasts
• Manually interpreted by ice
experts
• Monday - Friday
• Area: European waters
• Low resolution
• 7 days a week demand
Satellite images
Operational sea ice charts (Met.no)

6. Sea ice classification: Ice types
Sheets of nilas ice Pancake ice
Thin first-year ice Rough first year ice
Leads
Multi-first year ice

7. Challenges: Sea ice classification from SAR
• Sea ice is dynamic and complex target surface
• Signal characteristics depend on incidence angle, frequency, polarization and surface
target
• State-of-the-art methods are largely based on statistical methods
• DL architectures for sea ice classification are in an early stage of development, not
much experience to build on
• Training data scarce, inaccurate, and costly
• Resolution: Each pixel represents large area of Earth surface
• Additive Noise is an issue, specifically for S1
• Large amount of images are needed for training and validation
7

8. SAR (Sentinel-1) vs. Optical (Sentinel-2)
S1-pseudo colour: HH&HV S2 Optical
8
Courtesy: Thomas Kræmer, CIRFA

9. C & L-band
9
C-band HH
C-band VV
3 ice types
L-band HH
C-band VV
Shokr et al., 1995

10. Seasonal variability
10
Collected pre-melt sea ice, 17 April 2018 . Collected during the melt season, 16 June 2018.
Courtesy: Nick Hughes

11. SAR specific effects – systematic power decay
II. Incidence Angle Modelli
Systematic power decay
Background
Basic concepts of incidence angle effects
flight
path
azimuth
range
1
q
2
q
high backscatter
in the near range
Swath width
low backscatter in
the far range
IGARSS 2017, July 23-28, Fort Worth, Texas
20 30 40 50 60
15 20 25 30 35 40 45 50 55 60 65
Variable
ExtremeEarth KickOff meeting, Athens, January 2019 7 / 16

12. Variable noise floor with sub-swath
II. Incidence Angle Modelling
Systematic power decay
Background
Basic concepts of incidence angle effects
flight
path
azimuth
range
1
q
2
q
high backscatter
in the near range
Swath width
low backscatter in
the far range
IGARSS 2017, July 23-28, Fort Worth, Texas
20 30 40 50 60
Variable noise-floor, stitching
12
De Zan & Monti Guarnieri, 2006
Courtesy: Anthony Doulgeris, CIRFA

13. DL of sea ice classification in ExtremeEarth
DL Architecture design for sea ice
classification
• Supervised learning
• Semi-supervised learning
• Hybrid methods
• Data augmentation strategies to increase data

14. • Data extension
• Add more data (possible)
• Data augmentation
• Data generation (GANs)
• Active learning
Scarce Training Data for Big Data Analysis
• Training methods
• Unsupervised
• Semi-supervised
• Transfer Learning
Alleviating scarce training data
14
Data can be increased 10s of times
Data is increased to millions of samples
One image (10000 x 10000) contains
250 K of 20 x 20 patches without overlap

15. Areas: Training data
European Arctic: 83 Scenes, S1 - EW GRDM
Training data: Manually, supported by S2, S3
Used by: UiT
Danmarkshavn: 12 scenes, S1 - EW GRDM
Training data: Manually, from ice charts
Used by: Met.no
Belgica Bank: 24 scenes, S1 – IW GRDH
Training data: Interactively: active learning
Used by: DLR
Danmarkshavn
Belgica Bank

16. UiT: Training dataset generation for sea ice
83 SAR (S-1 EW) Images from North of Svalbard with 40 x 40 m pixel-spacing
Procedure:
• Manually label polygons, assisted by coincident, co-registered optical images (S2, S3,
Landsat-8)
• Extract patches from labeled polygons with stride 10 from all images
• The size of the patches (10x10, 20x20, 32x32, 36x36, 46x46) (i.e. patches are
overlapping)
• Each patch contains 3 quantities (HH-intensity, HV-intensity, Incidence Angle)
patch
Courtesy: Johannes Lohse, CIRFA

17. Sea ice and water classes
• Ice/Water Classification: Grouping sea and ice classes from original labeled
data
• 80% training & 20% validation
Open Water
Leads with Water
Newly formed ice
Brash/Pancake Ice
Thin Ice
Thick Ice, Flat
Thick Ice, Ridged
Sea
Ice

18. Supervised Deep Learning
Deep Learning in Remote sensing
Feature
Extractors
Transfer
Learning
Re-Training Pre-trained
New Deep Learning Arch. Existing Architecture
Weights are fixed Update the weights
Large training sets
Hyperparameter
optimization
Update the weights
Large training sets
Hyperparameter
optimization

19. Supervised approaches
Ad hoc CNN: Self-design CNN
• Determine layer structure
• Determine all hyperparameters
Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for
large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
Ad hoc CNN
VGG 16
VGG 16 from Visual Geometry Group from Oxford
• Trained from scratch
• Random initialization
• Trained by water/ice data
• Transfer learning
• Pre-trained: Initialized by ImageNet data
• Weights of layers tuned by the sea/ice data
The fully connected layers are the same for all
VGG models

20. Modified VGG 16
Max-pooling ensures translation invariance, but reduces information of local image
structure
Our modification: Reduce max-pooling
– Reduces the Translation invariance
– Keeps more of the local image structure (i.e. texture)
– This allows us to feed smaller patches to the network

21. Augmented dataset
• Augmentation (Random)
– Brightness
– Contrast
– Blur
– Rotation/Flip
• Force network to
experience more diverse properties
Patch Size Total Ice Sea
20x20 263,541 146,913 116,628
32x32 238,059 126,243 111,816
Number of samples in augmented dataset
Rotation
Blurred
Contrast/brightness

22. Results: VGG-16 training evaluation
VGG-16 trained from scratch
• Overfitting
• Validation loss
• Training loss
Num. steps
VGG-16 with transfer learning
• Overfitting
• Converge faster
Num. steps
• Ice/Water classification
• Patch size 32x32

23. VGG-16 Results with augmentation
• Ice/Water classification
• Patch size 32x32
• VGG trained from scratch
• Validation loss
• Training loss
No overfitting

24. Results: Accuracies of Ice/Water classification

25. Results: Inference
Purpose: To evaluate the classification performance of the trained networks
on new images
– Patch-wise
• Extract patches from image, side by side (without overlap)
• Feed the patches to the network and get predictions
• Label the whole patch with the predicted label

26. Inference Results

27. Ice Types Extension
Inference Results for Ice Types
Validation accuracy: 97.32 %

28. Pixel-wise sea-ice classification (training dataset generation)
28
• Pre-processing for input dataset
– Data searching (EW mode, January- March,2021)
– Boundary noise removal, noise floor removal by using Nansen
Environmental and Remote Sensing Center algorithm
(https://github.com/nansencenter/sentinel1denoised )
– Subset
Features: HH,HV,IA
• Label generation
– Baseline generation with clustering algorithm
– Manual modification to amend uncertainty

29. Pixel-wise sea-ice classification (training dataset generation)
29
S1B_EW_GRDM_1SDH_20210105T071250_20210105T071350_025016_02FA36_6592
Training: 5,713
Validation: 1,423
color legend
0
1
2
3
4
5
subset-image clustering sea-ice labeling
original image

30. Pixel-wise sea-ice classification (model architecture)
30
U-Net based model architecture

31. Pixel-wise sea-ice classification (training and validation results)
31
training and validation accuracy over epochs training and validation loss over epochs

32. Inference results
32

33. Inference results
33

34. Inference results
34
Young ice
Large smooth ice

35. Pixel-wise classification (further improvement)
35
• Better labeling
- advanced algorithm for labeling
- experts' involvement with peer-review
• Further improve the data-preprocessing for noise removal algorithms
• Solid evaluation procedure with referenced ground truth dataset
- in-situ dataset
- experts' involvement
- auxiliary data (optical image)

36. Semi-supervised learning (SSL)
• SSL alleviates the need for more labeled data by
– Combining labeled data with a large amount of unlabeled data
– Predicting labels of the unlabeled data, the amount of ‘labeled’ training data
significantly increases
• SSL generally improves performance of a deep learning model
36

37. Semi-supervised deep learning
Teacher-Student Model based on Label Propagation (TSLP-SSL)
34
S Khaleghian, H Ullah, T Krmer, T Eltoft, A Marinoni, “Deep Semi-Supervised Teacher-Student Model based on Label Propagation for Sea Ice Classification”, IEEE Journal of
Selected Topics in Applied Earth Observations and Remote Sensing, 2021

38. TSLP-SSL for Water/Ice Classification
S Khaleghian, H Ullah, T Kræmer, T Eltoft, A Marinoni, “Deep Semi-Supervised Teacher-Student Model based on Label Propagation for Sea Ice Classification”, IEEE Journal of
Selected Topics in Applied Earth Observations and Remote Sensing, 2021
Our method outperforms other STOA
methods. Especially when limited
number of samples are available
The algorithms extract possible
information from unlabeled data.
Supervised model using all training data
reached to 91.57

39. Inference Results
39
S Khaleghian, H Ullah, T Krmer, T Eltoft, A Marinoni, “Deep Semi-Supervised Teacher-Student Model based on Label Propagation for Sea Ice Classification”, IEEE Journal of
Selected Topics in Applied Earth Observations and Remote Sensing, 2021

40. Semi-supervised deep learning – Extended unlabeled data
• Use all available data in
dataset as labeled data
• Use random extracted
patches as unlabeled
data
S Khaleghian, H Ullah, T Krmer, T Eltoft, A Marinoni, “Deep Semi-Supervised Teacher-Student Model based on Label Propagation for Sea Ice Classification”, IEEE Journal of
Selected Topics in Applied Earth Observations and Remote Sensing, 2021
Supervised TSLP-SSL

41. Lessons learnt from ExtremeEarth & CIRFA
• They have excellent classification performance, but need large training data sets.
• DL offers methodologies that can remedy scarce training data
– Data augmentation
– Semi-supervised learning
– Transfer learning
• DL offers methods for handling changes in data properties
– Domain adaptation
13.12.2021 41
DL algorithms have many appreciated capacities:
• Better understanding of SAR imaging of sea ice (incidence angle, noise)
• Improved understanding of SAR scattering from sea ice and polarimetry
• Improved methods for removal of the SAR system noise
• Multi-modal data input
• Algorithms that are in-variant or adaptive to changes in the data properties
Operational sea ice charting would benefit from:

42. 43
Thank you!
Salman.khaleghian@uit.no