Machine vision technology is increasingly being used for fish classification as it provides fast, automated, and non-invasive classification compared to traditional manual methods. There are three main stages to machine vision-based fish classification: image acquisition, image preprocessing, and classification. Recent advances include the use of deep learning techniques which integrate feature extraction and classification. Several popular datasets are used to train models for fish classification. Machine vision holds potential for applications in fisheries management, aquaculture, and marine ecology.

1. Machine vision technology for
Fish classification
By
B. Bhaskar
Fisheries Resource Management

2. Introduction
• In 2018, global fisheries production reached another record high of a staggering
179 million tons. Of this amount, a record 156 million tons were also used for
human food consumption (FAO, 2020).
• There are more than 33000 species of fish in the world, which is the most diverse
vertebrate fauna (Eschmeyer et al., 2017).
• Whether in natural waters or aquaculture, different fish species usually live in the
same area (Xu et al., 2021).
• Classifying different species of fish is important for understanding marine ecology
and fish behaviour (Li et al., 2020), investigation of endangered species
populations, management of aquaculture (Tharwat et al., 2018), monitoring of
fish community status and health (Siddiqui et al., 2018), quality testing of
different fish (He et al., 2015), and studies of aquatic ecology (Hasija et al., 2017)
are of great importance.
• Traditional methods of fish classification are not only labour-intensive & time-
consuming but also somewhat destructive, such as the physical collection of fish
samples by divers (Cappo et al., 2003; Mallet and Pelletier, 2014).
• In contrast, marine biologists and researchers are increasingly keen to use more
efficient, automated, non-invasive, and non-destructive methods for fish sampling
and classification to avoid the problems caused by traditional manual methods
(Mclaren et al., 2015; Whitmarsh et al., 2016).
• Automatic fish classification is one of the important aspects of smart aquaculture.

3. Cont..
• Automatic classification of different species of fish is important
for the comprehension of marine ecology, fish behaviour
analysis, aquaculture management, and fish health monitoring.
• Recently many automatic classification methods have been
developed, among machine vision-based classification
methods are widely used with the advantages of being fast and
non-destructive.
• The successful application of rapidly emerging deep learning
techniques in machine vision has brought new opportunities
for fish classification.
• Machine vision technology is an important branch of artificial
intelligence (AI) that can replace the human eye and brain to
achieve fast, automatic, and non-invasive detection.

4. • Machine vision models have been widely applied in
fisheries (Mathiassen et al., 2012), which feed
information from the collected data to a computer
for analysis and decision-making, and the detection
results can be used to analyse fish behaviour,
calculate a weight (Zhang et al., 2020a), measure
lengths (Fernandes et al., 2020), estimate numbers
(Zhang et al., 2020b), and identify fish species (Xu et
al., 2021).
• In the past two decades, many models based on
machine vision technology for fish classification
have been proposed
Cont..

5. Fish classification in Machine vision

6. • Machine vision models for fish classification can be divided into three
stages: image acquisition, image preprocessing, and classification methods.
• In addition, classification methods are divided into traditional machine
learning and deep learning (Xu et al., 2021).
• Traditional machine learning algorithms such as artificial neural networks
(ANN; Hassoun et al., 1996), support vector machine (SVM; Cortes and
Vapnik, 1995), and principal component analysis (PCA; Wold et al., 1987)
have achieved excellent performance in fish classification.
• However, conventional machine learning algorithms strongly rely on
manually extracted features (LeCun et al., 2015).
• Deep learning as a new research direction in the field of machine learning
has made great progress in this area (Jiang and Learned-Miller, 2017), and it
has been widely used in the field of fish classification (Banan et al., 2020).
• In deep neural networks, the original image is fed without prior knowledge
to optimize the parameters and feature design (Nasiri et al., 2019).
• Further, deep learning combines the basic steps of image processing by
integrating the feature extraction phase and the classification phase

7. Image Acquisition
To achieve accurate fish classification, high-quality images of fish in complex environments need
to be acquired. Many researchers collect images via digital cameras (Joo et al., 2013; Li and
Hong, 2014) or smartphones (Sharmin et al., 2019) in a controlled laboratory environment
and then create their datasets (Alsmadi et al., 2010).
• The performance of a fish classification system depends on the quality of the acquired
images, but the water turbidity has been a major factor affecting the quality of the acquired
images.
• To ensure the quality of the images, Kutlu et al. (2017) placed each fish in the same position
on the white background floor before taking the pictures, but the applicability of this method
is low.
• Some researchers had also shown experimentally that relatively good results can be obtained
using frontal illumination with backlit shots is effective when the turbidity of the water
reduces the visibility of the fish (Zhang et al., 2016).
• Furthermore, to address the issue that the quality of images is affected to some extent by the
positioning of the acquisition equipment, Tharwat et al. (2018) used a Fujifilm X-T10 Camera
to capture images from different angles, distances, and lighting conditions as a dataset.

8. • To adapt to the studied scene, Rauf et al. (2019) set the mode of the automatic camera
(Canon EOS 1300D) to scene and selected the subcategory as snow scene for capturing
images.
• For fine-grained classification of fish, Jäger et al. (2015) used a stationary “baited remote
underwater video” (BRUV; Langlois et al., 2010) and a moving “diver operated video” system
(DOV) to record the Croatian fish dataset.
• Also, GoPro Hero3+ Black and GoPro Hero4+ black cameras achieved good results in
shooting underwater videos (Villon et al., 2018).
• In addition to the aforementioned dataset consisting of single-view images, 360◦ panoramic
images that obtain information from multiple views have gained more attention (Wang et
al., 2020).
• Meng et al. (2018) designed fisheye lenses and a 360◦ panoramic camera functional
underwater drone that can generate 360◦ images by using image generation algorithms.
• with the rapid development of deep learning, especially the application of Convolutional
Neural Network (CNN) in fish classification, relatively small datasets are no longer sufficient
for training high-performance models.
• As a result, more and more researchers have adopted public open-source datasets obtained
from image acquisition devices such as underwater cameras, underwater drones, digital
cameras, and Autonomous Underwater Vehicle (AUV; Cutter et al., 2015).
• One of the most widely used datasets, the Fish4-knowledge dataset, consists of videos and
images of 23 species of live underwater fish, which has been extensively used for model
training and performance testing of fish classification systems (Sun et al., 2018)
• The NCFM dataset generated from the global competition “Nature Conservancy Fisheries
Monitoring” by Kaggle has been used by many researchers to train classification models
(AliGombe et al., 2017)

9. • Important to note that the NCFM dataset is captured by
cameras mounted on fishing boats from different angles.
• Some of the images in the NCFM dataset are of low quality
due to the ubiquitous presence of workers, containers, and
fishing gear, as well as weather and lighting effects.
• Few well known datasets as follows

10. Image preprocessing
• Image preprocessing is one of the essential steps for fish
classification by using machine vision models (Deep and Dash,
2019), including various processes such as image grayscale,
image denoising, image enhancement, image segmentation,
and image augmentation.
• Since images acquired from real environments contain
inappropriate information such as shadows, complex
backgrounds, and noise (Wang et al., 2020), preprocessing
the images before the feature extraction process can improve
the image quality (Alsmadi et al., 2012; Hnin and Lynn, 2016),
which is very helpful for improving the classification accuracy
and efficiency of the models.
• Several of the most effective and widely used preprocessing
techniques applied to fish classification tasks nowadays are
detailed in the following.

11. Image grayscale
• Image grayscale When processing colour images of fish,
machine models often need to process all three channels in
sequence, and the time overhead will be high.
• Therefore, to achieve the goal of increasing the processing
speed of the whole application system, it is necessary for
image grayscale to reduce the amount of data to be
processed (Li and Hong, 2014).
• Image graying has been applied by many researchers as the
most commonly used image preprocessing technique for fish
images (White et al. , 2006; Hernández-Serna and Jiménez-
Segura, 2014; Li and Hong, 2014; Sengar et al. , 2017; Sharmin
et al. , 2019).
• Moreover, image grayscale can preserve gradient information
and highlight contour features in fish recognition. 1)Image
grayscale 2) Image denoising 3) Image enhancement 4) Image
segmentation 5) Image augmentation

12. Image denoising
• Due to the challenging real underwater environment, the fish images or videos
captured by underwater cameras are often affected by underwater noise, among
which Gaussian noise and impulse noise is typical.
• Currently, several image denoising methods have been practically applied, among
which Gaussian filtering (Salman et al. , 2016), mean filtering, and median filtering
are relatively basic and mature.
• Median filtering is a commonly used nonlinear smoothing filter, which substitutes
the value of a point in a digital image with the median value of the points sorted in
the neighborhood of that point.
• Although this method is not suitable for dealing with Gaussian noise, its
effectiveness in dealing with discrete point noise is obvious, so researchers
commonly use it to deal only with pixels contaminated by impulse noise
(Hernández-Serna and Jiménez-Segura, 2014; Jin and Liang, 2017).
• As for Gaussian noise, it is generally filtered using a linear smoothing filter such as
Gaussian filtering, which scans each pixel in the image with a template and then
replaces the value of the pixel point at the center of the template with the weighted
average gray value of the pixels in the neighborhood determined by the template.
• Besides, discriminant learning based on deep learning has also achieved good
results in solving the Gaussian noise problem (Tian et al. , 2020).
• Also, wavelet image denoising based on frequency domain has become one of the
main methods for image denoising at present (Xie et al. , 2002).

13. Image enhancement
• Image enhancement technique is a common technique in image preprocessing, which aims to enhance image
features by enhancing globally or locally useful information in the image, thereby improving the accuracy of
object recognition.
• Many image enhancement methods are limited to general images, and relatively few methods are developed
specifically for underwater images (Schettini and Corchs, 2010).
• Image enhancement for poor visibility is very challenging mainly due to the complexity of the underwater
environment.
• Histogram equalization is one of the most common enhancement methods, which transforms the distribution
of the histogram of the original image into a uniform distribution by mapping, and this increases the dynamic
range of pixel gray values, thus achieving the effect of enhancing the overall contrast of the image (Mokti
and Salam, 2009).
• Hitam et al. (2013) proposed a mixture of Contrast Limited Adaptive Histogram Equalization colour models,
which can significantly improve the visibility of underwater images by enhancing contrast and reducing
noise.
• Aiming at the problem of low illumination underwater, the image enhancement method based on a grayscale
nonlinear transformation and the Multi-Scale Retinex algorithm has achieved good results (Zhou et al.,
2017b).
• Constrained by objective conditions, most of the fish images are captured from turbid water, and the de-
scattering and colour correction of highly turbid water has a significant impact on the image classification
results.
• Li et al. (2016b) obtained satisfactory results for the enhancement of the high turbidity underwater images by
joint guidance image de-scattering and colour correction of physical spectral features.
• Further, preprocessing methods such as Gaussian Blurring, Morphological Operations, Otsu’s Thresholding,
and Pyramid Mean Shifting have been employed to enhance fish images (Rathi et al., 2018).

14. Image segmentation
• Image segmentation can divide an image into several independent sub-
regions to find the region of interest.
• In fish classification studies, it is only the part of the fish body in the image
that is being focused on, so this part of the region is separated and extracted.
• This simplified representation of fish images is of great help for feature
extraction, and thus a correct segmentation process will bring higher value to
the classification results (Alsmadi et al., 2011; Li and Hong, 2014; Kartika
and Herumurti, 2017).
• Edge segmentation is a more mature and commonly used class of algorithms
in image segmentation, which mainly uses abrupt changes in the gray level
of the image at the edges to segment the image.
• The Sobel edge operator is often used to detect fish edges and their
orientation, but it is relatively sensitive to noise (Baloch et al., 2017).
• Therefore, some researchers employed the Grabcut (Rother, 2004) algorithm
to segment fish bodies from the image background and then combined with
some morphological operations to remove unwanted shapes from the image
(Houssein et al., 2018).
• However, multi-objective segmentation in a single image remains a key
focus and difficulty in image processing research (Ren et al., 2017).

15. Image augmentation
• Image augmentation increases the size of the training
data set by altering the training images to produce
similar but different training samples.
• There are various methods of image augmentation,
such as mirror transformation, cropping, panning,
brightness modification, clipping, noise addition,
rotation, scaling, and colour transformation.
• In fish classification tasks, researchers often choose to
use multiple augmentation methods to augment the
dataset (Allken et al., 2019; Wang et al., 2019).
• The increase of training samples can reduce the
dependence of the model on certain attributes, and
thus improve the generalization ability of the model
(Shorten and Khoshgoftaar, 2019).

16. The flowchart of traditional machine
learning.

17. Fish images under controlled conditions. (D. Li et al.)

18. Traditional machine learning methods for fish
Classification

19. Flow chart of Deep learning Techniques
Deep learning methods for fish
classification

20. AlexNet architecture

21. A. The flow of the item-based soft attention module; B. .
VGG- architecture.

22. Inception module with dimensionality reduction

23. Residual learning: a building block

24. A typical example of transfer learning

25. References
• Li, , Qi Wang, Xin Li, Meilin Niu, He Wang, and Chunhong Liu, 2022. Review Article Recent advances of machine vision technology in fish
classification Daoliang. ICES Journal of Marine Science (2022), 79(2), 263–284. https://doi.org/10.1093/icesjms/fsab264