Slides of my presentation at the Self-supervised Learning for Next-Generation Industry-level Autonomous Driving workshop at ICCV 2021 for the first prize talk of the competition on continual object recognition.

1. Report of the Continual Learning
Supervised Classification challenge (Track 3A)
Gabriele Graffieti, Guido Borghi, Davide Maltoni, Matteo Ferrara
{name.surname}@unibo.it
Department of Computer Science and Engineering,
University of Bologna
Italy
ICCV 2021 Workshop: Self-supervised Learning for Next-Generation Industry-level Autonomous Driving
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 1 / 9

2. The Team
Gabriele Graffieti
Ph.D. student
Guido Borghi
Assistant prof.
Davide Maltoni
Full prof.
Matteo Ferrara
Associate prof.
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 2 / 9

3. Model and Hyperparamenters
I ResNet-50 [1] pretrained on ImageNet [2].
Last layer substituted with a fully-connected layer with 7 output neurons + biases.
I Stochastic Gradient Descent (SGD) optimizer.
Learning rate = 10−2
, weight decay and momentum = 0.
I Weighted Cross Entropy (CE) loss:
L(y, l) = −αl log
exp(yl )
P
j exp{(yj )}
!
, α0 = 0, α1,..,6 = 1
I Batch size = 10.
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 3 / 9

4. Training Procedure
Every experience
I Tensors resized from 64 × 64 → 224 × 224 to resemble the original training image size
• Using a Nearest Neighbor interpolation.
• Even though the patches are now 12× larger, network’s filters respond better after the resize.
I On-the-fly data augmentation, flipping horizontally all the images.
• Both the original and the flipped patches are fed to the network.
Only in the first experience
I Current batch put temporally inside the memory and passed twice through the model.
• Loss and optimization computed after each single batch.
• Boost the learning of the model in the first experience (especially for underrepresented
classes).
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 4 / 9

5. Classification Head Protection
Motivation
I “Learning in isolation” problem when few classes are present in an experience or some
classes are underrepresented.
I Forgetting of underrepresented classes, especially in the classification head.
Solution
I We use the CWR algorithm [3] to control forgetting in the classification head.
I Two set of weights (of the head) are maintained (7k more weights than ResNet-50, inside
the 105% limit)
• cw: Weights from the previous experience used in the consolidation phase.
• tw: Weight used to train the model in the current experince, initialized to 0 with only the
weight of the classes in the current experience loaded from cw.
I We do not freeze the feature extractor, as proposed in [4].
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 5 / 9

6. Replay Memory
I We divide the memory in 6 buffers (one per class) of 100 samples each (total 600
samples).
• Important to limit the number of samples per class in memory, due to unbalance in the data.
• The goal here is to have a memory balanced per class.
• Buffer size bounded by the number of the least represented class (tricycle, 82 samples).
• We empirically found that 100 sample per class is a good compromise.
I Batch is composed of 5 sample from the current experience and 5 samples sampled
randomply from memory
• The sampling is without replacement.
• Once all the patterns in memory are sampled the sampling start again.
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 6 / 9

7. Memory Management
I We use Reservoir Sampling [5] to insert pattern in memory with the same probability.
I Memory should not be altered when used!
• We use the 400 remaining slot to store the pattern we want to insert from the current
experience
• We want to insert a maximum of 100/i pattern per class from the i-th experience, thus
maximum allocation of memory is 900 patterns (600 from the replay memory + 50 per class
to insert in the 2nd experience).
• We randomly inserted the new patterns inside the memory at the start of a new experience.
Current used memory Sampled and to be inserted
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 7 / 9

8. Contribution of the Components
Component Contribution
Model **
Learning rate *
Optimizer ***
Loss weights *
Image resizing ***
Data augmentation **
Double batch (1st exp.) *
CWR **
Not freezing the feature extractor ***
Replay memory ***
Balanced memory ***
Reservoir sampling **
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 8 / 9

9. Bibliography
[1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
Proceedings of the IEEE conference on computer vision and pattern recognition, 2016,
pp. 770–778.
[2] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale
hierarchical image database,” in 2009 IEEE conference on computer vision and pattern
recognition, IEEE, 2009, pp. 248–255.
[3] D. Maltoni and V. Lomonaco, “Continuous learning in single-incremental-task scenarios,”
Neural Networks, vol. 116, pp. 56–73, 2019.
[4] L. Pellegrini, G. Graffieti, V. Lomonaco, and D. Maltoni, “Latent replay for real-time
continual learning,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS), IEEE, 2020, pp. 10 203–10 209.
[5] J. S. Vitter, “Random sampling with a reservoir,” ACM Trans. Math. Softw., vol. 11,
no. 1, pp. 37–57, Mar. 1985, issn: 0098-3500. doi: 10.1145/3147.3165. [Online].
Available: https://doi.org/10.1145/3147.3165.
Choco Leibniz team (Unibo team) Continual Learning Supervised Classification challenge Report 9 / 9