The document describes a method for classifying fish species using visual vocabulary. It involves a two stage process of training and testing. In the training stage, visual vocabularies are learned for each species by finding salient points on images and modeling parts, locations, and features. In the testing stage, features are extracted from test images and compared to visual vocabularies to classify the fish by shortest distance. The method achieves 85.2% accuracy but has problems with species appearing different from sides and curved bodies. Future work to improve classification is suggested.

1. Fish Species Classification
Using Visual Vocabulary
Wenhuan Wei - June 7, 2016
Ⅰ. Introduction
Fisheries research on fish populations and fish behavior are more and more relied on
image-based methods. Based on automatic image processing technique, scientists don’t need
to extract meaningful information manually, thus reducing a large amount of time during
research. Those technique include fish image segmentation ,fish length estimation ,fish1 2
underwater tracking ,fish species classification and so on.3 4
This project focuses on fish species classification.It is based on previous work , which5
requiring automatic detection,sizing and classification of fish targets from sterol-video
imagery of fish passing on a conveyor belt or sliding on a chute. The aim of this project is to
improve the fish species classification process of the previous project.The major problem of
the previous project is high data-sensitivity,which is the common problem in fish species
detection. High data-sensitivity means that the performance of the classifier will decrease
dramatically if the proportion of the bad training data is above some threshold.Bad training
data is caused by some factors like curved fish body, illumination changes,bad quality of
images and so on. One way to deal with this problem is discarding some of those bad
training data, but it will require some extra work. Another way is to collect large enough data
set and use deep learning to train the model so that the model can learn those bad data and
do the correct classification. It should be a good way, but unfortunately, the size of the data
set we have get is approximately 20 thousands fish images, which is not large enough for
deep learning to train the classifier.
FISH SPECIES CLASSIFICATION 1

2. In this project, we try to deal with the high data-sensitivity problem by improving the
classification algorithm.The key technique in this project is called visual vocabulary . Visual6
vocabulary is based on bag of words model to discriminate documents on topic. In image
processing and computer vision, visual vocabulary(bag of words model) are often used to do
object recognition and classification. Visual vocabulary, in this project, is composed of most
important parts of fish body such as eye,tail,fin and so on. After learning visual vocabulary to
all the species(6 species in dataset), we can classify the test fish image based on those learned
visual vocabularies.
The rest of this report is organized as follows. Section Ⅱ introduces method of learning
visual vocabulary and classifying test image. Section Ⅲ shows several problems in this
project and the future work.
Ⅱ. Visual Vocabulary Learning And Classification
There are six species in the data set which contain approximately 20 thousand fish
images. Most of the images have the similar format of one fish lying on a blue board like the
picture on the last page. Labels(fish species) are already given by fish scientists.To train visual
vocabulary for certain species, we split the training data based on species and learning the
visual vocabulary separately.
There are two stage involved in the whole classification process.
The first stage is training stage.In training stage, we first find several salient points on
each training picture based on openCV library, and initialize the visual vocabulary by
drawing box around those points. After initialization, we use a part model learning
algorithm to learn the visual vocabulary. Visual vocabulary model consists three components7
including P, X and S. P is feature vectors in each part,X is location of each part and S is the
size of each part. Learning visual vocabulary can be modeled as an optimization problem
based on fitness, separation and discrimination. Fitness means minimizing the distance
between each part of the model and the corresponding part on training data. Separation
means maximizing the locational distance between each part in the model. Discrimination
means maximizing the feature distance between each part in the model. We can achieve an
optimal model by updating one of the variables with the other two fixed and following the
same step alternatively until convergence.
FISH SPECIES CLASSIFICATION 2

3. The second stage is testing stage. In testing stage, we first extract features from test
image by initializing the test image on location X and size S with feature P equal to visual
vocabulary learned in training stage, after that we extract the visual “words” of this test
image based on the same alternative optimization problem in training stage. In the end,we
compute the distance between extracted features and the visual vocabulary and we use the
distance to do classification. We compare the distance between test image’s features and the
six species’ visual vocabulary. We classify the test fish image to the species with the nearest
distance.
The above picture shows the main steps in the whole classification process. We test our
method in the testing data and we get a 85.2% F1 score.The performance is not good enough
due to some problems in this project. Next Section will cover them.
FISH SPECIES CLASSIFICATION 3
Key technique: visual vocabulary
• Training Stage:
• initialization learning VC
• P, X, S
minimize:
• fitness
• separation
• discrimination
Do the same training for other species, in the end we get “visual vocabulary” for each species
Testing Stage:
extract features compute distance between
initialization test image's features and
X, S, fixed P(6) 6 species' “visual vocabulary”
minimize:
separation classify the test image to its
discrimination nearest “visual vocabulary”
F1-score:
85.2%

4. Ⅲ. Problems And Future Work
There are two main problems in this project so far.
The first one is caused by a special fish species. This species looks totally different from
two body sides.
As the above pictures shows, the fish color is black from one side but white from the
other. To deal with this problem, we add some supervised feature like fish body length and
aspect ratio to the training data. However, though addressing the problem to some extent, it
makes our unsupervised method supervised.
The second problem is caused by curved fish body.
As the above pictures shows, curved fish body will lose some important features on the
fish image and the classifier will have a bad performance when detecting those images. One
way to deal with it is to do image swapping before testing. However, one problem of this
solution is that it will cause extra time, another problem is that if the image is ill-structured to
a large extent image swapping still cannot reconstruct some missing part of the original fish.
So far we have not come up with a good unsupervised method to deal with this problem.
In future work, we plan to further improve our classifier training method by using
boosting, which is another common and useful training algorithm. We might use deep
learning method to change the classification process so that we can totally address the above
problems.
FISH SPECIES CLASSIFICATION 4

5. References
David Meng-Che Chuang, Jenq-Neng Hwang, Kresimir Wi-based lliams,“ Automatic Fish Segmentation and1
Recognition for Trawl-based Cameras, ” in Computer Vision and Pattern Recognition in Environmental
Informatics, ed., Jun Zhou, et al. (ed.), Ch.5, pp. 79-106, Oct. 2015.
Tsung-Wei Huang, Jenq-Neng Hwang, Craig Rose,“ Chute-based Automated fish Length Measurement and2
Water Drop Detection,” IEEE Int'l Conference on Acoustics, Speech and Signal Processing (ICASSP),
Shanghai, China, March 20-25, 2016,pp.1906-1910.
Meng-Che Chuang, Jenq-Neng Hwang, Jian-Hui Ye, Shih-Chia Huang, Kresimir Williams, “ Underwater Fish3
Tracking Underwater Fish Tracking based on Deformable Multiple Kernels, ” accepted for publication in IEEE
Trans. on Systems, Man, Cybernetics: SYstems, November 2015.
Meng-Che Chuang ,Jenq-Neng Hwang ; Fang-Fei Kuo ; Man-Kwan Shan ; Kresimir Williams,“Recognizing4
live fish species by hierarchical partial classification based on the exponential benefit”,2014 IEEE International
Conference on Image Processing (ICIP),pp.5232 - 5236.
Meng-Che Chuang, Jenq-Neng Hwang, Kresimir Williams, “ A Feature Learning and Object Recognition5
Framework for Underwater Fish Images, ” IEEE Trans. on Image Processing, 25(4):1862-1872,April 2016.
M. -E. Nilsback,A. Zisserman,”A Visual Vocabulary for Flower Classification “,2006 IEEE Computer Society6
Conference on Computer Vision and Pattern Recognition (CVPR’06),vol.2,pp.1447 - 1454.
Meng-Che Chuang, Jenq-Neng Hwang, Kresimir Williams, “ A Feature Learning and Object Recognition7
Framework for Underwater Fish Images, ” IEEE Trans. on Image Processing, 25(4):1862-1872,April 2016.
FISH SPECIES CLASSIFICATION 5