Learning continually from sequential data is a difficult task, especially if the stream is divided into many small batches that contains non-i.i.d. data. Stochastic gradient descent usually fails in this settings, and the models experience high forgetting of past acquired knowledge. In this talk I'll try to address the problem of continually acquire new knowledge from many (almost 400) small batches (300 highly correlated frames each) of images from natural video streams.

1. Continual Learning over Small non-i.i.d. Batches
of Natural Video Streams
ContinualAI May Meetup
Gabriele Graffieti
gabriele.graffieti@unibo.it
PhD Student
Computer Science and Engineering Department
University of Bologna
Italy
May 29, 2020
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 1 / 18

2. Sequential data
A CL ideal application
• A new object is shown to a CL agent.
• The agent acquire a short video of the object.
• Frame extracted from the video constitute one or more small mini-batches, containing a
highly correlated patterns from single class.
Problems
• standard SGD-based optimization does not work well on this setting.
• High forgetting, since a single batch contains only one object.
• In popular benchmarks (CIFAR, ImageNet) instances of the same class are independent.
Its unlikely that an application experiences at time t a set of independent images of the same
class. It’s more realistic to encounter a single object and observe it under different poses.
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 2 / 18

3. Our setting
CORe50
• 50 classes, 10 categories 5 different objects per category.
• ∼165,000 128×128 RGB-D images.
• for each object there are 11 different video sessions (∼300 frames recorded with a Kinect
2 at 20 fps).
NICv2-391
• New Instances and Classes scenario.
• 391 batches of 300 images each (first batch 10 times larger).
• Only one object per incremental batch.
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 3 / 18

4. NICv2-391
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 4 / 18

5. AR1*
Convolutionallayers
Head
Data layer
For each batch:
• weights in the online head are zero-init for new
classes and reload from off-line head for known
classes.
• at the end of the batch weights are consolidated in
the off-line memory by weighted averaging.
cw[j] =
cw[j] · wpastj + (tw[j] − avg(tw))
wpastj + 1
wpastj =
pastj
curj
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 5 / 18

6. AR1*
Convolutionallayers
Head
Data layer
• To control forgetting in the lower layers we use
Synaptic Intelligence (limit update of important
weights).
• SI exploits information made available by SGD,
does not require any further gradient propagation.
• Weight update does not require to store the old
weights.
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 6 / 18

7. Results on CORe50 NICv2
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 7 / 18

8. Results on CORe50 NICv2
Strat. Run time (m) Data (MB) Params. (MB)
CWR* 21.4 0 0.2
Naive 25.6 0 0
LWF 27.8 0 0
EWC 31.2 0 24.4
AR1* 39.9 0 12.2
DSLDA 79.1 0 0.2
Cumul. 2826.2 5,898.3 0
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 8 / 18

9. Improve the results with replay
Convolutionallayers
Head
Data layer Replay memory
• Straightforward technique: just store past
data and repeat them through the
network.
• No need to weight constraints in the
convolutional layers (AR1*free).
• If few images per class are taken, not so
critical impact on memory overhead.
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 9 / 18

10. Improve the results with replay
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 10 / 18

11. Replay problems
Convolutionallayers
Head
Data layer Replay memory
• Requires extra storage (e.g. for ImageNet,
if we store 20 patterns per class, the total
storage is about 3.8 GB)
• Requires extra forward/backward steps
when mixing new and old patterns more
iterations for epoch.
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 11 / 18

12. Latent replay
Advantages:
• Efficiency: extra forward and backward
steps take place only in the upper layers.
• Less storage required.
• Activations can be quantized/compressed
with negligible accuracy loss.
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 12 / 18

13. Latent replay
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 13 / 18

14. Strategy Run time (m) Mem. overhead (MB) Final Accuracy
Naive 25.6 0 + 0 7.13%
CWR* 21.4 0 + 0.2 56.99%
DSLDA 79.1 0 + 0.2 48.02%
AR1* 39.9 0 + 12.4 56.32%
AR1*free (Image) 133.3 75 + 0 77.30%
AR1*free (conv5 4/dw) 41.2 48 + 0 72.23%
AR1*free (pool6) 23.7 5.8 + 0 59.75%
Cumulative 2826.2 ∼6,000 + 0 85.26%
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 14 / 18

15. Future works
• Latent Generative Replay.
• Self-training by exploiting temporal coherence.
• Openset classification (automatic discovery of new classes).
• Sparse human supervision (active learning).
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 15 / 18

16. Thank You!
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 16 / 18

17. Bibliography I
Vincenzo Lomonaco, Davide Maltoni
CORe50: a new Dataset and Benchmark for continual Object Recognition
CoRL 2017
Vincenzo Lomonaco, Davide Maltoni, Lorenzo Pellegrini
Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches
CVPR 2020 (Workshop)
Li Zhizhong, Derek Hoiem
Learning without Forgetting
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 17 / 18

18. Bibliography II
James Kirkpatrick et al.
Overcoming catastrophic forgetting in neural networks
Proceedings of the National Academy of Sciences, 2017
Tyler L. Hayes, Christopher Kanan
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
arXiv Preprint, 2019
Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, Davide Maltoni
Latent Replay for Real-Time Continual Learning
arXiv Preprint, 2020
Gabriele Graffieti CL over Small non-i.i.d. Batches of Video Streams May 29, 2020 18 / 18