Download to read offline
More Related Content
Similar to Introduction to self-Supervised learning - kuliah machine learning STEI ITB
Introduction to self-Supervised learning - kuliah machine learning STEI ITB
- 1. Self-Supervised Learning (SSL) SetiaJuli Irzal Ismail 5 Nov 2024
- 2. Taxonomy DL – Learning Paradigm 2 Supervised Learning •Self-Supervised Learning Unsupervised Learning Reinforcement Learning
- 3. AI today ismostly supervised learning • Training a machine by showing examples instead of programming it • When the output is wrong, tweak the parameters of the machine leCun, 2019
- 4. Weakness 1. Supervised Learning •Labeling is difficult and expensive • ImageNet with 14 Million images, 1000 category took 22 human years • Require expert: medicine 2. Unsupervised Learning • Clustering/dimensionality reduction • Representations?
- 5. Reinforcement learning • RLworks well for games • Too many trial and error • Its ok in a game • Its not ok in the real world • 100 hours to reach the performance that a human can reach in 15 minutes on Atari games (Hessel ArXiv:1710.02298) • Real world don't have explicit reward
- 6. SSL 1. Create labelfrom the data 2. Generate useful representations from the data 3. Adaptability to multiple data & modality (images, text, audio, video)
- 7. Analogy – Howbaby learn
- 8.
- 9. How do SSLworks Raw Data Learn from unlabeled data Transfer learning
- 10. SSL Pretext task • Learnfrom unlabeled data • Captured pattern & feature • Trained model Downstream task • Supervised learning • Fine-tuning • Classification
- 11. SSL Pretext task • Representhow data relate to one another • Big Dataset • Often different dataset than downstram task Downstream task • Train linear classifier on Frozen weight/features • Full finetuning
- 12. SSL Taxonomy 1. Contrastivelearning 2. Masked modeling/Predictive learning 3. Clustering
- 13. 1. Contrastive Learning • Modellearn to distinguish between similar (positive) and dissimilar (negative) data samples • Pushing similar pairs together • Pushing dissimilar pairs farther apart • Model learns feature that are invariant to transformations Chen, SimCLR, 2020
- 14.
- 15. Positive vs Negative(similarity?) Cosine similarity; Dot product Model calculate similarity score between positive pair and negative pairs Isra, PIRL, 2019
- 16. Contrastive Learning components 1.Siamese Network 2. Augmentation • Random cropping • Rotation • Color jittering 3. Loss Function https://ubiai.tools/what-are-the-advantages-anddisadvantages-of-data-augmentation-2023-update/
- 17. Contrastive loss function •Maximize the similarity of positive pairs • Minimize the similarity of negative pairs • SimCLR, MoCo, SimSiam, BYOL Chen, SimCLR, 2020
- 18. 2. Masked Modelling/Predictive • Model learns to predict missing parts of data • Model learn complex relationship https://medium.com/@shaikhrayyan123/a-comprehensive-guide-to-understanding-bert-from-beginners-to-advanced-2379699e2b51
- 19.
- 20.
- 21. Masked modelling Components •Data masking • Predicting the masked parts • Training objective • Reconstruction loss, cross entropy loss
- 22. example • BERT • RoBERTa •MAE • BEiT
- 23. 3. Clustering • Groupingsimilar data into clusters Misra, 2019
- 24. Clustering steps Feature Extraction Clusteringthe embbeddings • K-means, KNN, sinkhorn-knopp Self-training with pseudo labels Update cluster assignments Model representations
- 26.
- 27. Morgado, Avid CMA,https://github.com/facebookresearch/AVID-CMA
- 28. Clustering example DeepCluster SwAV (SwappingAssignments between Views SCAN (Semantic Clustering by adopting Nearest Neighbors)