Video from PhD thesis defence: Marius Miron - Source Separation Methods for Orchestral Music: Timbre-Informed and Score-Informed Strategies
More info here: http://mariusmiron.com/research/thesis/
http://mtg.upf.edu/node/3863
Video here: https://youtu.be/FDrFTTVOtr0
PhD Thesis Marius Miron - Source Separation Methods for Orchestral Music
1. Source separation methods for
orchestral music: timbre-informed
and score-informed strategies
Universitat Pompeu Fabra, Barcelona, Music Technology Group
Thesis supervisors:
Dr. Jordi Janer
Dr. Emilia Gómez
Marius Miron
Thesis defence committee:
Dr. Emmanuel Vincent
Dr. Xavier Serra
Dr. Maximo Cobos
Doctoral thesis presentation
33. Opportunities
33
Widmer, G., & Goebl, W. (2004). Computational models of expressive music performance:
The state of the art. JNMR, 33(3), 203-216.
Part IV
Tempo
Dynamics
Timbre
Local timing
35. Contributions
• Note refinement using image processing to fix local misalignments
• Deep learning framework for classical music source separation
• Deep learning score-informed source separation
35
36. Contributions
• Reproducible research (datasets, frameworks, code, results)
• Orchestral dataset with score annotations
• Cloud-based source separation framework on Repovizz.
• Deep learning repository for audio applications.
36
41. Other dataset
Bach10 Dataset
10 Bach chorales, 20-40 seconds each
Perfectly and automatically aligned scores
41
Duan, Z. and Pardo B.,(2011), Bach10 dataset
46. Note refinement for source separation
46
Logfrequency
(1/4semitoneres)
Time (s)
Trained timbre bases Time gains initialised with score
Magnitude
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
47. Note refinement using NMF gains
47
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Time gains after NMF
Time (s)
Blob detection for a single note
48. Note refinement using NMF gains
48
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Time gains after NMF
Time (s)
Blob detection for a single note
49. Note refinement using pitch salience
49
Miron et al (2014). Audio-to-score alignment at the note level for orchestral recordings, ISMIR 2014
frequencyrelativetonote'sf0(centbins)
time (seconds)
➩
A B
time (seconds)
50. Note refinement using NMF gains
50Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Evaluation with Bach10 dataset: alignment
NMF gains
score follower
NMF gains
pitch salience
pitch salience
score follower
Error threshold (seconds)
Alignmentrate
51. Note refinement using NMF gains
51
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Source separation evaluation metrics - BSS EVAL :
• Signal to Distortion Ratio (SDR)
• Signal to Interference Ratio (SIR)
• Signal to Artefacts Ratio (SAR)
• Image to Spatial Distortion Ratio (ISR)
52. Note refinement using NMF gains
52
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Time gains after NMF
Time (s)
Blob detection for a single note
53. Note refinement using NMF gains
53
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
A
B
C
D
E
F
ground truth
misaligned
refined pitch salience
refined NMF gains
implicit
refined NMF gains
submatrix
time & frequency
Evaluation with Bach10 dataset: source separation
54. Multi-channel source separation
54
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
55. Multi-channel source separation
55
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
Ref1
56. Multi-channel source separation
56
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
Ref2
57. PARAFAC gains estimation
57
[ ]…
Ref2
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
58. Multi-channel source separation
58
GT
Ali
Ext
Ref1
Ref2
ground truth
refined gains
extended
refined gains multi
submatrix
aligned
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
59. Multi-channel source separation
59
SDR(dB)
Mozart Beethoven Mahler Bruckner
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
60. Multi-channel source separation
60
SDR(dB)
Mozart Beethoven Mahler Bruckner
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
61. Multi-channel source separation
61
SIR(dB)
Mozart Beethoven Mahler Bruckner
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
62. Multi-channel source separation
62
SAR(dB)
Mozart Beethoven Mahler Bruckner
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
70. Data generation method
70
Miron et al,(2017). Generating data to train neural networks for classical music source separation
SMC 2017
Score
Augmented
Synthesis
Audio
Augmentation
Multi-track
Renditions
71. Data generation method
71
Miron et al,(2017). Generating data to train neural networks for classical music source separation
SMC 2017
STFT
Data
Processing
Multi-track
Renditions
CNN
Training
Trained model
72. Data generation method
72
Miron et al,(2017). Generating data to train neural networks for classical music source separation
SMC 2017
STFT
Data
Processing
Multi-track
Renditions
CNN
Training
Trained model
STFT
Data
Processing
Target
piece
CNN
Separation
Separated
Sources
73. Score-informed separation with CNN
73
Score-based binary matrices
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017
77. Score-informed separation with CNN
77
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017
conv1
fshape(1,30)
stride(1,4)
conv2
fshape(20,1)
stride(1,1)
dense1
256
inverse
conv2
inverse
conv1
(J,T,F)
(J,T,F)
(30,T,F)
11
(30,T,F)
2 2
=
with
(J,T,F) (J,T,F)
(T,F)
(30,T,F)
11
(30,T,F)
2 2
dense2
78. Experiments
CNN autoencoder NMF
78
Multi-source filter model
Score informed
Trained on RWC
vs
Score-informed
Trained on renditions
synthesised with RWC
Duan, Z. and Pardo B.,(2011), Bach10 dataset
80. Results
80
Mean(dB)
SDR SIR SAR
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017
81. Results
81
Mean(dB)
SDR SIR SAR
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017
82. Results
82
Mean(dB)
SDR SIR SAR
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017
83. Results
How much data we need?
83
No. of instances
SDR(dB)
Sampling:
Bootstrapping
with replacement
Fixed samples
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017
100. Future work
• Instrument recognition using deep learning
• Deepconvsep is already a starting point and a baseline
• Better databases : NSynth to generate training data
• Explore other music traditions, genres, and a more flexible context
100
101. Publications by the author
101
Peer-reviewed journals
• Miron, M., Carabias-Orti, J.J., Bosch, J.J., Gomez, & Janer, J. (2016). Score-Informed Source Separation
for Multichannel Orchestral Recordings. Journal of Electrical and Computer Engineering
Full articles in peer-reviewed conferences
• Miron, M., Gomez, & Janer, J. (2017). Monaural score-informed source separation for classical music using
convolutional neural networks. In Proceedings of the 18th International Society for Music Information
Retrieval Conference (ISMIR)
• Miron, M., Gomez,E., & Janer, J. (2017). Generating data to train convolutional neural networks for
classical music source separation. In Sound and Music Computing Conference (SMC)
• Martel, H., Miron, M. (2017). Data augmentation for deep learning source separation of HipHop songs. In
10th International Workshop on Machine Learning and Music (MML)
• Chandna, P.,Miron, M., Janer, J., & Gomez, E. (2017). Monoaural Audio Source Separation Using Deep
Convolutional Neural Networks. In 13th International Conference on Latent Variable Analysis and Signal
Separation (LVA/ICA)
• Miron, M., Carabias-Orti, J.J., & Janer, J. (2015). Improving score-informed source separation for classical
music through note refinement. In 16th International Society for Music Information Retrieval Conference
(ISMIR)
• Miron, M., Carabias-Orti, J.J., & Janer, J. (2014). Audio-to-score alignment at the note level for orchestral
recordings. In 15th International Society for Music Information Retrieval Conference (ISMIR)
102. Source separation methods for
orchestral music: timbre-informed
and score-informed strategies
Universitat Pompeu Fabra, Barcelona, Music Technology Group
Thesis supervisors:
Dr. Jordi Janer
Dr. Emilia Gómez
Marius Miron
Thesis defence committee:
Dr. Emmanuel Vincent
Dr. Xavier Serra
Dr. Maximo Cobos
Doctoral thesis presentation
