The effects of noisy labels on deep convolutional neural networks for music tagging
•
2 likes•331 views
발표자: 최근우(Queen Mary University of London 박사과정) 발표일: 2017.11. 부제: 내 거친 생각과 불안한 라벨링 그걸 학습하는 너 그건 아마도 뉴런같은 넷퉉ㅋ 개요: 태깅 데이터셋은 보통 +라벨은 정확한 편이지만 -라벨은 ‘False’가 아니라 라벨링이 안 된 ‘Unknown’에 가까운 경우가 많습니다. 음악 태깅 데이터셋도 마찬가지입니다. 이 발표에서는 이 잡음을 정량화하고 잡음이 학습과 평가에 미치는 영향을 알아봅니다.
Recommended
Recommended
More Related Content
More from NAVER Engineering
More from NAVER Engineering (20)
Recently uploaded
Recently uploaded (20)
The effects of noisy labels on deep convolutional neural networks for music tagging
- 1. The Effects of Noisy Labels Keunwoo.Choi @qmul.ac.uk on deep convolutional neural networks for music tagging arXiv:1706.02361
- 3. @KeunwooChoi 2014--present: PhD, Queen Mary University of London 2016--present: Buzzmusiq lnc. 2016/ 06--12: Visiting PhD, NYU 2015/ 06--09: Intern, Naver Labs 2011--2014: Audio research team, ETRI 2009--2011: Applied Acoustic Lab, EECS, SNU 2005--2009: EECS, SNU Papers on ISMIR/ICASSP/IEEE Trans./Etc. Python/Keras/Pytorch
- 4. Contents • Intro to music x deep learning [15min] • Tagging dataset and noise [30min] • Label vector analysis [10min]
- 5. The Effects of Noisy Labels Keunwoo.Choi @qmul.ac.uk on deep convolutional neural networks for music tagging György Fazekas, Kyunghyun Cho, Mark Sandler arXiv:1706.02361 0. INTRODUCTION TO MUSIC X DEEP LEARNING
- 6. Find out the SoTA-coolness method eg) ResNets, GANs, VGG, Attention, Seq2seq, VAE.. Google if someone applied it on music “domain adaptation” i.e., words → notes image → STFT Profit!
- 7. Examples
- 8. Issues
- 9. Data efficiency := required amount of data for the same perception
- 10. 8-bit x 28x28 vs. 16-bit x 1000000 45 MB 50,000 training items 10,000 testing items 10 categories quick decoding (.jpg) vs. 1200 MB 1,000 training + test items 10 categories slow decoding (.mp3)
- 11. “it takes approx. 5x times with the same performance in practice” End-to-end is not always desirable
- 12. Preprocessing audio files Mp3’s (30s) x 1M 650GB Npy’s (decoded) x 1M 300GB Npy’s (specgram) x 1M 300GB num_FFT? window_length? hop_length? Npy’s (specgram) x 1M 300GB Npy’s (specgram) x 1M 300GB • It takes time • and storage • while all except one will be deleted • therefore no one cares about it
- 13. Kapre; for quicker audio preprocessing
- 15. Copyright
- 16. How to get away? Solutions Notes Examples Non-copyrighted music Free Music Archieve FMA-dataset BSS Eval Releasing YouTube URL “You break the rule, not us” AudioSet Crawling the previews Dangerous Million Song Dataset Sharing the previews Even more dangerous
- 17. The Effects of Noisy Labels Keunwoo.Choi @qmul.ac.uk on deep convolutional neural networks for music tagging György Fazekas, Kyunghyun Cho, Mark Sandler arXiv:1706.02361 (A short Q&A? +) 1. INTRODUCTION
- 18. Tagging • Anyone can tag any words (or non-words) to any song • The quality is ****. • Poor, innocent, (financially) poor researchers need to use it
- 19. Tagging (Tag, count) rock 101071 pop 69159 alternative 55777 indie 48175 electronic 46270 female vocalists 42565 favorites 39921 00s 31432 Awesome 26248 american 22694 seen live 20705 cool 19581 Favorite18864 Favourites 17722 female vocalist 17328 guitar 17302 loved 12483 favorite songs 12392 heard on Pandora 10470 USA 8725 2000s 8671 Favourite Songs 8661 drjazzmrfunkmusic 8364 77davez-all-tracks7278 fav 6155 bass 3364 songs I absolutely love 3293 vocals 2369 drums2281
- 20. 🤔 Female vocalists Male vocalist Guitar Bass Vocals Drums 0% 25% 50% 75% 100% True False
- 21. Questions How noisy? Is training alright? How about evaluation? What are they learning?
- 22. The Effects of Noisy Labels Keunwoo.Choi @qmul.ac.uk on deep convolutional neural networks for music tagging György Fazekas, Kyunghyun Cho, Mark Sandler arXiv:1706.02361 2. HOW NOISY? IS TRAINING OK?
- 23. Measuring the noise • We need strongly-labelled re-annotations • Instrumentation labels are (sort of) objective (instrumental, female vocal, male vocal, guitar) • 242K songs are still a lot → select a subset (or two)! I can do it! ..but not all of them
- 24. Strongly labelling: Subset100 • Subset100: random 50 from ‘True’ + random 50 from ‘False’ (for each label) Instrumental Female vocalists Male vocalist Guitar True False 50songs 50songs 50 50 50 50 50 50
- 25. Strongly labelling: Subset400 • Subset400: Just random 400 items 242K songs × 50 tags 400 songs 4 tagsSubset400
- 27. Evaluating groundtruth on Subset100 0 25 50 75 100 + Error rate Precision Instrumental female voc male vocal guitar 0 25 50 75 100 - Error rate Recall Instrumental female voc male vocal guitar
- 28. #Occurrences estimation 0 20 40 60 80 In all, by GT My estimation using S100 My re-annotation on S400 Instrumental female voc male vocal guitar
- 29. Again, with box plots {Instrumental, female vocalists} vs. {male vocalists, guitar}
- 30. Group A vs B, but why? • Tagging ‘vocals’, ‘drums’, ‘bass’ is like.. → They’re not tag-worthy → Let’s call it ‘taggability’ Female vocalists Male vocalist Guitar Bass Vocals Drums 0% 25% 50% 75% 100 True False ***? What’s on the desk?
- 31. The hypothesis If unusual → high taggability. Instrumental, female vocal : high taggability Male vocal, guitar: low taggability
- 32. The hypothesis If unusual → high taggability. If high taggability → less false negative = higher recall (of GT) Instrumental, female vocal : high taggability, less false neg, higher recall Male vocal, guitar: low taggability, more false neg, lower recall
- 33. The hypothesis If unusual → high taggability. If high taggability → less false negative = higher recall (of GT) If higher recall (=more reliable GT), → ?
- 34. [33] Choi et al. 2017, Convolutional recu... Hypothesis If unusual → high taggability. If high taggability → less false negative = higher recall (of GT) If higher recall (=more reliable GT), → ? Performance(AUC) !!!
- 35. The hypothesis If unusual → high taggability. If high taggability → less false negative = higher recall (of GT) Instrumental, female vocal : high taggability, less false neg, higher recall, better classification Male vocal, guitar: low taggability, more false neg, lower recall, worse classification If higher recall (=more reliable GT), → better classification
- 37. The Effects of Noisy Labels Keunwoo.Choi @qmul.ac.uk on deep convolutional neural networks for music tagging György Fazekas, Kyunghyun Cho, Mark Sandler arXiv:1706.02361 3. IS EVALUATION OK?
- 38. Really? So, we evaluate the classifier based on.. 🤔 I need a noise-free groundtruth...
- 39. Evaluate the evaluation 242K songs × 50 tags 400 songs 4 tagsSubset400 HAHAHAH!Subset400!
- 40. Results
- 41. Evaluate the evaluation Interesting! With such noise, the results are still okay. It’s not perfect though. HAHAHA!
- 42. The Effects of Noisy Labels Keunwoo.Choi @qmul.ac.uk on deep convolutional neural networks for music tagging György Fazekas, Kyunghyun Cho, Mark Sandler arXiv:1706.02361 (A short Q&A? +) 4. LABEL VECTOR ANALYSIS
- 44. Label vector similarity • Similarity between labels according to the trained convnet.
- 45. Label vector
- 46. Label vector vs co-occurrence (GT)
- 47. Label vector vs co-occurrence (GT) • Mostly, LV reproduces the groundtruth. • Except: similar pairs only by label vector: (sad, beautiful), (happy, catchy), (rnb, sexy) ‘Sad songs are beautiful.’ ‘Catchy songs are often happy songs.’ ‘R&B claims to be sexy.’ 🤔 Makes sense..
- 48. The Effects of Noisy Labels Keunwoo.Choi @qmul.ac.uk on deep convolutional neural networks for music tagging György Fazekas, Kyunghyun Cho, Mark Sandler arXiv:1706.02361 5. CONCLUSIONS
- 49. Conclusions • We quantified how noisy weakly-labelled groundtruth is. • We conjectured why some labels are noisier. • We showed what happens to the noisier labels on training and evaluation. • We investigated what a convnet learns.
- 50. The Effects of Noisy Labels Keunwoo.Choi @qmul.ac.uk on deep convolutional neural networks for music tagging György Fazekas, Kyunghyun Cho, Mark Sandler arXiv:1706.02361
- 51. Links My blog | blog post 1, blog post 2 | Paper!