Data augmentation plays an important role in deep learning. Recently, RandAugment and Augmix have been proposed as effective augmentation methods. In aquaculture, echo sounder image analysis is used for catching specific kind of fish to improve annual catches. Therefore, it remains significant to research for effects of different methods and find suitable augmentation method for echo sounder image analysis. This research aims at finding effective settings for RandAugment and Augmix by comparing effects of different transformation methods. The experiment results show improvement on recall rate and f1 score of distinguishing tuna in echo sounder images with suitable transformation methods.

1. Study on Data Augmentation Methods for
Sonar Image Analysis
Hokkaido University
Graduate School of Information Science and Technology
Harmonious Systems Engineering Laboratory
MIN JIE
0

2. Background 2
• For aquatic resources management, catching
restricted fish species should be avoided.
• Electromagnetic wave is interfered severely by
environment noise, so sonar image is commonly
used for solving fishing problems.
• Neural network has been used in sonar image to
find specific kind of fish.[1]
• Data augmentation can improve performance of
neural network.
[1]平間 友大, 横山 想一郎, 山下 倫央, 川村 秀憲，鈴木 恵二，和田 雅昭 : CNNを用いた音響画像に基づく
定置網内の魚種推定の精度向上, 第18回情報科学技術フォーラム(FIT2019), CF-009, 岡山(2019)

3. ①
②
③
Echo sonder
Introduction of prediction system 3
Convolutional
neural
networkSegmentation
Determine whether cast nets
Output
percentage of
target fish
existance
Threshold
• System of avoiding restricted fish species for the
management of aquatic resources
Sonar images
from echo
sounder Input
Prediction

4. Motivation
• Sonar images have different composition compared
with real world images.
• Different tricks for neural network.
– Data augmentations (random flip, translate)
– Change the structure of the network
– Add Dropout into the network
– Fine tuning and add weight decay
• Selecting and combining basic methods in different
ways such as AutoAugment[2] and RandAugment[3]
from google research achieved best performance.
3
[3]RandAugment: Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, Quoc V.
Le.’’Practical automated data augmentation with a reduced search space’’
[2]AutoAugment: Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay
Vasudevan, Quoc V. Le. “Learning Augmentation Policies from Data”

5. Purpose
• Avoid catching restricted fish species for aquatic
resources management.
• Make a proposal of data augmentation methods
for sonar image to improve the performance of
prediction system.
– RandAugment perform well for natural image.
– Could be used for sonar image.
• Figure out fitness of basic methods used in
RandAugment for sonar image according to their
variations with original images.
5

6. Introduction of sonar image
• How to collect sonar image data
– Echo sounder
• Emits a 50kHz sonic wave every three seconds.
• The intensity image of the reflected wave is output.
– Sonar image
• The stronger the reflected wave, the brighter the image.
• Shape feature comes from sonic wave reflected form fish.
50.23m/px
3sec/px
165px
(38.65m)
25px(75s)
fishseabed
Intensity time series image
Segmentation for sonar image

7. Make label for images
• Positive samples: collected from 10 days with tuna catches in
the image.
• Negative samples: collected from 211 days without tuna
catches.
7
Numbers of total dataset
Positive samples Negative samples
positive negative
Train dataset 10,308 148,154
Valid dataset 2,756 37,039
Test dataset 3,321 46,299

8. Train data
augmentated
in same way
Data Augmentation
• Single method to process all the images.
– Commonly used augmentation methods
• SMOTE
• Mixup
• Cutout
• Random Erasing
– Basic single augmentation methods.
• Uncommon usage for bad performance
• Combination for basic methods perform better in image
classification compared with single method.
– State of art methods : AutoAugment，RandAugment
7
Single
method
Original
images

9. Combined method
1. How to select basic method?
2. How much magnitude apply on image?
– AutoAugment
• 16 methods
• Range of magnitudes :10
• Probability of applying :11 values
• 5 sub-policies : 10 methods in total
– RandAugment
• 14 methods
• Same magnitude M :sample 4 to 5 values from (1, 30)
• Randomly select N(2,3) methods from all 14 method
16 × 10 × 11 10
search space
2 × 4, 5 search space
faster and same accuracy
8

10. Basic augmentation methods 10
• Basic methods of RandAugment
– ShearX(Y): shear the image along the horizontal(vertical) axis.
– TranslateX(Y): translate the image in horizontal(vertical) axis.
– Rotate: rotate the image.
– Invert: invert the pixels of image.
– Equalize: equalize the image histogram.
– AutoContrast: maximize the image contrast.
– Brightness: adjust the brightness balance of the image.
– Solarize: reduce the value of pixels for each pixel.
– Posterize: invert the pixels above the threshold.
– Contrast: control the contrast of image.
– Color: adjust the color balance of the image.
– Sharpness: adjust the sharpness balance of the image.
• All these methods have different effects on different datasets.
• Important to figure out their fitness for sonar image datasets.
Position
Brightness
No extend

11. RandAugment 11
• Improvement method of AutoAugment, it can get high
accuracy quickly by narrowing the search space.
• Search for 2*4 hyperparameters combinations.
・N{2, 3} : number of randomly selected methods.
・M{5, 10, 15, 20} : magnitudes sampled from (1,30)
for every methods .
• Find the match of N and M with best accuracy.
Identity, AutoContrast, Rotate,
Solarize, Posterize, Contrast,
Brightness, Sharpness, ShearX(Y),
Translate(Y), Invert , Equalize
Contrast, ShearX,
Rotate
Randomly select 3 methods
from 14 methods above
apply CNN
Train process
ShearXAutoContrast Rotateoriginal
Example N=3, M=15
Method
Bigger M bigger change
ShearX
Brightness
M=15 M=30

12. Enlarged image

13. RandAugment 11
• Improvement method of AutoAugment, that can find best
methods quickly by narrowing the search space
• Search for 3*4 hyperparameters combinations.
・N{2, 3, 4} : number of randomly selected methods.
・M{5, 10, 15, 20} : magnitude sampled from (0,30)
for every methods .
• Find the match of N and M with best accuracy.
Identity, AutoContrast, Rotate,
Solarize, Posterize, Contrast,
Brightness, Sharpness, ShearX(Y),
Translate(Y), Invert , Equalize
ShearX,
AutoContrast, Rotate
Randomly select 3 methods
from 14 methods above
apply CNN
Train process
ShearXAutoContrast Rotateoriginal
Example N=3, M=15
Method
Bigger M bigger change
ShearX
Brightness
M=15 M=30
M=15

14. Enlarged image
Augmented image

15. RandAugment 11
• Improvement method of AutoAugment, that can achieve
same accuracy quickly by narrowing the search space
• Search for 3*4 hyperparameters combinations.
・N{2, 3, 4} : number of randomly selected methods.
・M{5, 10, 15, 20} : magnitude sampled from (0,30)
for every methods .
• Find the match of N and M with best accuracy.
Identity, AutoContrast, Rotate,
Solarize, Posterize, Contrast,
Brightness, Sharpness, ShearX(Y),
Translate(Y), Invert , Equalize
ShearX,
AutoContrast, Rotate
Randomly select 3 methods
from 14 methods above
apply CNN
Train process
ShearXAutoContrast Rotateoriginal
Example N=3, M=15
Method
Bigger M bigger change
ShearX
Brightness
M=15 M=30

16. Criterion for fitness of methods
• Variation of methods for dataset
– Judge by appearance of images
– Need a numerical criterion for variation.
– Mean square error(MSE) to evaluate the variation of
augmented images and original images.
12
Equalize InvertShearXOriginal

17. MSE for all methods with different M 13
For different M, define MSE in range(100,2000) as proper, and
train the model with these methods by RandAugment.
Equalize
Invert
ShearX
Original
Method M=5 M=15 M=25
Invert 55,164 55,146 55,146
Equalize 15,867 15,867 15,867
AutoContrast 1,013 1,013 1,013
TranslateY 200 525 590
Rotate 135 322 414
ShearY 112 305 341
TranslateX 29 212 259
Brightness 8.18 106 237
ShearX 11 56 61.8
Contrast 6.54 50 250
Solarize 0.53 4.42 1,107
Posterize 0.16 2.59 10
Sharpness 0.002 0.22 0.65
Color 0 0 0
56
15878
55526
Too large MSE
cause bad
performance
Conjecture1:
Too low MSE
cause bad
performance
Conjecture2:
MSE show same
tendency with
appearance .

18. Model in experiment 14
• Convolutional neural network for image classification
• Resnet50[4]
– Easier to optimize the residual mapping than to optimize the original
unreferenced mapping.
• Optimizer : Momentum SGD
• Loss function : softmax cross entropy
• Input image size(224,224) : resize sonar image to same size for
pretrained model.
[4]He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of
the IEEE conference on computer vision and pattern recognition. 2016.
weight layer
weight layer
relu
F(x)
H(x)=F(x)+x relu
x
x
identity
Resize to
(224,224)
Residual block

19. Imbalanced dataset
• Train dataset
– Number of positive samples is P
– Number of negative samples is N
– P:N≈1:14 in this experiment
• Weighted loss function
– Weight of positive samples :
– Weight of negative samples:
– Weighted cross entropy:
15
𝑁
𝑃 + 𝑁
P
𝑃 + 𝑁
𝑦 means true label and 𝑦′ means predicted label
𝑤 𝑃 =
𝑤 𝑁 =
𝐿 𝑤 = −𝑤 𝑃 ⋅ 𝑦 log 𝑦′
− 𝑤 𝑁 1 − 𝑦 log 1 − 𝑦′
(1)

20. Performance evaluation 16
• Accuracy
– In binary classification, when the dataset is extremely imbalanced(1:14 in
this experiment), accuracy is not an effective criterion for performance
evaluation.
• Confusion matrix
• Precision ,recall and f1 score
– Precision=TP/(TP+FP): efficiency of targeting tuna
– Recall=TP/(TP+FN): rate of targeting tuna
– f1 score=2/(1/precision+1/recall): harmonic value of precision and recall
True class
Positive Negative
Prediction
class
0
1 TP
TNFN
PF

21. Experiment 17
Baseline of
sonar image
dataset
Result with
RandAugment
Result with
RandAugment
RandAugment
with proper
basic methods
1. If RandAugment perform well on sonar image?
2. If RandAugment with proper basic methods
perform better than with all methods on sonar image?
Compare
Compare

22. Experiment 1
• Baselines without RandAugment and RandAugment
with all methods.
– Condition:
• Train for the baseline of sonar image dataset.
• Train for RandAugment including all basic methods with
N=3, M=(5, 15, 25)
18
Numbers of total dataset
positive negative
Train dataset 10,308 148,154
Valid dataset 2,756 37,039
Test dataset 3,321 46,299

23. Results of experiment 1
• Baseline train without RandAugment.
• Results of RandAugment with all methods
19
Baseline F1 Precision Recall
Rate 0.8640 0.8971 0.7985
• Overall 2% improvement compared with baseline for sonar image
M(N=3) F1 Precision Recall
5 0.8846 0.8947 0.8749
15 0.8779 0.9076 0.8640
25 0.8823 0.8907 0.8815

24. Results of experiment 1
• Train process figures with and without RandAugment.
20
• Train process become more stable with RandAugment
Without RandAugment With RandAugment

25. Experiment 2
• RandAugment with proper MSE methods
-(100, 2000) in table
21
Method M=5 M=15 M=25
Invert 55,164 55,146 55,146
Equalize 15,867 15,867 15,867
AutoContrast 1,013 1,013 1,013
TranslateY 200 525 590
Rotate 135 322 414
ShearY 112 305 341
TranslateX 29 212 259
Brightness 8.18 106 237
ShearX 11 56 61.8
Contrast 6.54 50 250
Solarize 0.53 4.42 1,107
Posterize 0.16 2.59 10
Sharpness 0.002 0.22 0.65
4 methods when M=5
6 methods when M=15
8 methods when M=25

26. Result of experiment 2
• RandAugment with appropriate MSE methods
22
M(N=3) F1 Precision Recall
5 0.8938 0.9041 0.8861
15 0.8985 0.8800 0.9104
25 0.9006 0.8935 0.9088
M(N=3) F1 Precision Recall
5 0.8846 0.8947 0.8749
15 0.8779 0.9076 0.8640
25 0.8823 0.8907 0.8815
• Overall improvement about 2% except M=5.
Result of all methods in RandAugment
Result of proper methods in RandAugment
4 methods when M=5

27. Conclusion
In this sonar image dataset collected from 2 years
• RandAugment can improve performance of network.
– F1 score and recall
– Stability of train process
• RandAugment with proper MSE basic methods
performs better than that with all methods for sonar
image dataset.
• Still need to be tested in more sonar image datasets
and complicated conditions before putting into use.
23