[CVPR2021] Rainbow Memory: Continual Learning with a Memory of Diverse Samples
•Download as PPTX, PDF•
0 likes•215 views
This document presents Rainbow Memory, a new approach for continual learning that uses a memory of diverse samples to preserve knowledge from previous tasks. The key aspects are: 1) It addresses the realistic but challenging "blurry-online" continual learning setting where the data stream changes gradually over time and the model must learn continuously. 2) It proposes a memory management technique that uses data uncertainty to store a diverse set of samples from previous tasks to regularize training on new tasks. 3) Experimental results show the approach outperforms existing continual learning methods, especially on the blurry-online setting, demonstrating its effectiveness at preserving knowledge over multiple tasks in a continuously learning system.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Recently uploaded
Recently uploaded (20)
[CVPR2021] Rainbow Memory: Continual Learning with a Memory of Diverse Samples
- 1. Rainbow Memory: Continual Learning with a Memory of Diverse Samples Jihwan Bang* Heesu Kim* YoungJoonYoo Jung-Woo Ha Jonghyun Choi * Equal Contribution Search Solution Inc. CVPR 2021
- 2. What is the Continual Learning? 2 • A Realistic Continual Learning Scenario Popularity changes of four items during one year in a real-world e-commerce service
- 3. What is the Continual Learning? 3 • A Continual Learning Setup
- 4. Literatures about Continual Learning 4 • CL understudied on blurry-online setting Method CIL Settings Knowledge Preserv. Online Disjoint Blurry Memory Regularize Distill EWC X O X X O X iCaRL X O X O X O Rwalk X O X O O X GSS O X O O X X BiC X O X O X O Gdumb O O O O X X RM (ours) O O O O X X
- 5. Approach: Diversify Samples 5 • Memory management using uncertainty
- 6. Approach: Diversify Samples 6 • Memory management using uncertainty • Data Augmentation
- 7. Results: Accuracy on Blurry-Online 7 • Outperform existing methods with large margin ∗ indicates the reproduction of BiC with only using classification loss without distilling loss to be better suited for Blurry10 setup.
- 8. Results: Accuracy on various blurry level 8 • Outperform existing methods with large margin on online-blurry{0, 10, 30} setup
- 9. Conclusions 9 • Tackle disjoint setting on continual learning, and analyze various setting • Propose memory management using data uncertainty • Show DA significantly improves performance in online blurry CL Codes, data, and set-ups are available online https://github.com/clovaai/rainbow-memory Thank you!
- 10. Please visit our poster (#4189) for more experiments and analysis! Search Solution Inc.
Editor's Notes
- Hi. My name is Jihwan and I’m working at NAVER. This work was conducted through joint research between Heesu Kim, YoungJoon Yoo, and Jung-Woo Ha who are members of NAVER AI LABS and professor Jonghyun Choi at GIST. This presentation is presented by NAVER Clova Dubbing model, called Matthew. We propose to diversify the visual appearances of samples in episodic memory for realistic continual learning setup, and name it as rainbow memory.
- Continual learning has been recently rehighlighted as a realistic learning scenario. However, the disjoint configuration of classes across the task is rather artificial, and very recently, the blurry task configuration has been proposed for a more realistic continual learning setup. The figure is the graph for item search frequency by time in our e-commerce service, and it shows blurry is more realistic than disjoint configuration.
- There are some scenario setups for continual learning. The disjoint setup consists of tasks which do not share the classes. Whereas blurry can share the classes across the tasks. In addition, online and offline continual learning is whether continual learning allows the temporary buffer for storing incoming samples of a current task or not during model training, respectively. We address arguably the most practical setup, which is the blurry and online continual learning.
- Most of the literature addresses the disjoint-offline setup and there are very few works on blurry and online setup. We are the first to conduct extensive experimental validations on blurry online setup.
- We approach continual learning by diversifying the visual appearance of the samples in the previous tasks. We argue that the exemplars which are selected to be stored in the memory should be not only representative of their corresponding class but also discriminative to the other classes. Specifically, we choose the samples that are near the classification boundary are the most discriminative and the samples that are close to the center of the distribution are the most representative. As a measure for discriminability and representativity, we compute the uncertainty of a sample by the variance of model outputs of perturbed samples by various transformation methods for data augmentation, and select data from the lowest uncertainty to the highest with a uniform interval for each class.
- To further enhance the diversity of exemplars from the memory, we employ data augmentation. We found that the efficacy of the data augmentation’s has not been well investigated in the literature. The data augmentations diversify a given set of samples by image-level or feature level perturbations, which correspond to the philosophy of updating memory by securing the diversity. We consider various perturbation types including simple single-image-based data augmentation perturbing the original input image, and mixed-labeled data augmentation which integrates multiple images and automated data augmentations.
- In extensive empirical validation, we show that the proposed method consistently outperforms prior works in various datasets, and the gain becomes larger when the number of classes increases, which is more challenging.
- Even though blurry-continual learning is the main task of our interests, we also investigate the performance of the our proposed method on disjoint-Continual learning setup and in various blurry levels. Our method outperforms various online and blurry including disjoint setups. Although the proposed method is designed for blurry and online setup, it comparably performs to the other methods in offline setup.
- In sum, we address the continual learning by diversifying the visual appearance of the sample by a memory management scheme with uncertainty via invariance. In addition, we also show data augmentation significantly improves performance in online blurry. Please visit our github repository for the codes, data, and detailed experiment setups.
- Please visit our poster for more experiments and analysis! Thank you.