Recommended
More Related Content
What's hot
What's hot (16)
Similar to Sander Dieleman, Research Scientist at DeepMind - Generating Music in the Raw Audio Domain
Similar to Sander Dieleman, Research Scientist at DeepMind - Generating Music in the Raw Audio Domain (20)
More from Codiax
More from Codiax (20)
Recently uploaded
Recently uploaded (20)
Sander Dieleman, Research Scientist at DeepMind - Generating Music in the Raw Audio Domain
- 1. Generating music in the raw audio domain Sander Dieleman - November 16-17, 2018
- 2. PhD student, Ghent University, Belgium (2010 - 2016) “Learning feature hierarchies for musical audio signals” Machine learning intern, Spotify, NYC (summer 2014) Scaling up content-based music recommendation Research Scientist, DeepMind, London UK (since July 2015) AlphaGo, WaveNet, ... http://benanne.github.io sanderdieleman@gmail.com @sedielem
- 3. Overview Why model music in the raw audio domain? WaveNet: an autoregressive model of raw audio Generating music with WaveNets Modelling long-range correlations with WaveNets Generating music with long-range consistency 3
- 4. Why model music in the raw audio domain?
- 5. Why raw audio? Music generation is typically studied in the symbolic domain 5
- 6. 6
- 7. 7
- 8. Why raw audio? 8 Many instruments have complex action spaces ⇒ Rich palette of sounds and timbral variations Guitar ● pick vs. finger ● picking position ● frets ● harmonics ● ...
- 9. WaveNet: an autoregressive model of raw audio “WaveNet: A Generative Model for Raw Audio”, van den Oord et al. (2016)
- 10. 10
- 11. Modelling raw audio: μ-law encoding / quantisation 11 uniform quantisation μ-law quantisation WaveNet models audio as a discrete time series
- 12. WaveNet: an autoregressive model of raw audio 12 input hidden layer hidden layer hidden layer output “WaveNet: A Generative Model for Raw Audio”, van den Oord et al. (2016)
- 13. 13 WaveNet: an autoregressive model of raw audio input hidden layer hidden layer hidden layer output
- 14. 14 Dilated convolutions to enlarge receptive fields input hidden layer hidden layer hidden layer output dilation 20 = 1 dilation 21 = 2 dilation 22 = 4 dilation 23 = 8
- 15. 15 Dilated convolutions to enlarge receptive fields input hidden layer hidden layer hidden layer output dilation 20 = 1 dilation 21 = 2 dilation 22 = 4 dilation 23 = 8
- 16. Generating music with WaveNets
- 17. 17
- 18. 18
- 19. 19
- 20. 20
- 21. Aside: NSynth
- 22. NSynth: WaveNet autoencoders for timbre modelling 22 Collaboration with “Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders”, Engel et al. (ICML 2017)
- 23. NSynth: WaveNet autoencoders for timbre modelling 23Blog post: https://magenta.tensorflow.org/nsynth
- 24. Modelling long-range correlations with WaveNets “The challenge of realistic music generation: modelling raw audio at scale”, Dieleman et al. (2018)
- 26. 26 Dilation: #layers ~ log(receptive field length) input hidden layer hidden layer hidden layer output dilation 20 = 1 dilation 21 = 2 dilation 22 = 4 dilation 23 = 8
- 27. 27 … but memory usage ~ receptive field length Required model depth is logarithmic in the desired receptive field length Required memory usage during training is still linear in the desired receptive field length! ⇒ We cannot scale indefinitely using dilation
- 28. Autoregressive discrete autoencoders (ADAs) 28x input q = f(x) query q’ = VQ(q) quantised query encoder local model quantisation modulator y code … 5 207 131 18 168 … decoder
- 29. Autoregressive discrete autoencoders (ADAs) 29 “Neural Discrete Representation Learning”, van den Oord et al. (2017) “The challenge of realistic music generation: modelling raw audio at scale”, Dieleman et al. (2018) Two variants: ● VQ-VAE: vector quantisation variational autoencoder learn a codebook, quantise the queries to elements of this codebook ● AMAE: argmax autoencoder encourage the queries to be close to one-hot (e.g. use softmax), quantise them on the simplex
- 30. VQ-VAE 30 encoder decoder vector quantisation x input x’ = g(q’) reconstruction q = f(x) query q’ = VQ(q) quantised query y integer sequence
- 32. Vector quantisation 32 D-dimensional Euclidean space K codes 0 1 2 4 5 6 7 3
- 33. Vector quantisation 33 D-dimensional Euclidean space encoder q
- 34. Vector quantisation 34 D-dimensional Euclidean space encoder q q’
- 36. Straight-through estimation for VQ 36 encoder q q’ decoder backward pass
- 37. Training VQ autoencoders 37 ℒ = -log p(x|q’) Reconstruction loss (NLL)
- 38. Training VQ autoencoders 38 ℒ = -log p(x|q’) + α · (q’ - [q])2 Reconstruction loss (NLL) Codebook loss [...] = tf.stop_gradient(...)
- 39. Training VQ autoencoders 39 ℒ = -log p(x|q’) + α · (q’ - [q])2 + β · ([q’] - q)2 Reconstruction loss (NLL) Codebook loss Commitment loss [...] = tf.stop_gradient(...)
- 40. Argmax autoencoder (AMAE) 40 encoder decoder Encoder outputs real values
- 42. Argmax autoencoder (AMAE) 42 encoder decoder Normalise so that activations sum to 1
- 43. Argmax autoencoder (AMAE) 43 encoder decoder Perform argmax and pass to decoder
- 44. AMAE loss function 44 ℒ = -log p(x|q’) + β · (q’ - q)2 + ν · (Et [q] - C-1 )2 Reconstruction loss (NLL) Commitment loss Diversity loss
- 45. AMAE: alternative interpretation 45 AMAE = VQ-VAE with a fixed codebook, consisting of one-hot vectors
- 46. Stacking ADAs 46 … 17 95 88 145 ... … 210 37 121 8 ... level 1 ADA level 2 ADA level 3 unconditional model
- 47. Stacking ADAs 47 … 17 95 88 145 ... … 210 37 121 8 ... level 1 ADA level 2 ADA level 3 unconditional model 16 kHz 2 kHz 250 Hz
- 48. Code sequences are harder to model than audio 48 Training second level ADAs is much harder ● Train VQ-VAE with population-based training (PBT) ● Train AMAE, which is less unstable Predictability(NLL) Receptive field length Audio Code sequence “Population Based Training of Neural Networks”, Jaderberg et al. (2017)
- 49. Generating music with long-range consistency
- 50. Some results from a human evaluation 50
- 52. 52 Keynote speakers Kenneth Stanley, University of Central Florida David Ha, Google Brain Allison Parrish, NYU ITP Yaroslav Ganin, DeepMind Yaniv Taigman, Facebook AI Research Submission deadline (papers & art): October 28th https://nips2018creativity.github.io/