These slides were presented at PyCon India, 2016. The slide deck mainly talks about adopting the appropriate Neural Network in order to make computers generate it's own music and focuses on LSTM (Long Short Time Memory - a special type of Recurrent Neural Network) architecture. It also talks about three prime phases involved in algorithmic music generation, which are as follows: 1. Preprocessing the input i.e. converting the MP3 files into np-tensors. 2. Training the model. 3. Generation phase. It also focusses on various challenges involved in doing so i.e. composing the music algorithmically. The slides contain the visualization of the input files given to the model as well as music generated by the neural network model. You can find the entire python code used for developing this model on GitHub: https://github.com/unnati-xyz/music-generation

1. ALGORITHMIC
MUSIC
GENERATION
PadmajaV Bhagwat
3rd year, B.Tech, IT, NITK Surathkal. 01
25th Sept, 2016PyCon India 2016

2. Unnati Data
Labswww.unnati.xyz
Acknowledgement:
My team members
1. Chandana NT
2. Anirudh Sriram
3. Subramaniam Thirunavakkarasu
Under the guidance of
1. Nischal HP
2. Bargava Subramanian
3. Amit Kapoor
4. Raghotam Sripadraj 02

3. What are
Artificial
Neural
Networks?
 It is biologically-inspired programming paradigm
which enables a computer to learn from
observational data.
 It resembles human brains mainly these two ways
1. A neural network acquires knowledge through
learning.
2. A neural network’s knowledge is stored within
the interconnection strengths known as synaptic
weight.
03

4. sample #2
sample #1
Using AI to
generate
Music.
Can you
tell the
difference?
04

5. How to
make
computer to
generate
music?
Step 1: Converting Mp3 files to np-tensors
Step 2:Traing the model
Step 3: Generating the music
05

6. Uncompress
the music:
mp3 to
monaural
WAV using
LAME cmd = 'lame -a -m m {0} {1}'.format(
quote(filename), quote(filename_tmp))
os.system(cmd)
06

7. Converting
to Numpy
array.
These are divided into blocks of equal size
by zero padding it.
07

8. Convert
from time to
frequency
domain
Convert from the time domain to its
corresponding frequency domain using a
Discrete Fourier Transform.
08

9. Can we
use
traditional
RNN?
Drawback: They cannot retain information for
long periods of time.
RNNs have loops.
 Allows persistence of information.
 Each neuron accepts the input from
previous layer as well as from previous
neuron in the same layer.
09

10. Can we
make it
remember
for longer
period of
time?
Yes we can! By using
LSTM networks.
10

11. What are
LSTM
Networks?
Long Short Term Memory
Special type of RNN.
Capable of learning long term dependencies.
It is well-suited to learn from experience to
classify, process and predict time series when
there are very long time lags of unknown size
between important events.
Explicitly designed to avoid the long-term
dependency problem
11

12. Let’s
understand
its
architecture.
12

13. This is how it looks!
A separate vector called
cell state is dedicated to
remember information.
Advantage: Number of
parameters that it needs to
learn is less compared to
traditional networks.
13

14. Step 1:  The first step in LSTM is to decide what
information we’re going to throw away from the cell
state.
 This decision is made by a sigmoid layer called the
“forget gate layer”.
14

15. Step 2:  Decide what new information we’re going to store
in the cell state.
 This has two parts:
1. A sigmoid layer called the “input gate layer”
decides which values we’ll update.
2. A tanh layer creates a vector of new candidate
values, Ct that could be added to the state
15

16. Step 3:
we update the old cell state Ct-1 to the new cell state Ct
16

17. Step 4:  we need to decide what we’re going to output.
 Sigmoid layer decides what parts of the cell states
we are going to output.
 The updated cell state vector is passed through a
tanh layer and multiply it by the output of the
sigmoid gate.
17

18. Training
the
model:
 Input data is given to the input of the network,
which then fires all the corresponding neurons.
 The LSTM generates a sequence of notes which is
compared against the expected output and the
errors are back-propagated, thus adjusting the
parameters learnt by the LSTM.
After shifting
18
The vector used for computing loss function is same as the input layers but shifted by 1 block.

19. Training
the
model:
 The optimizer used is 'rms-prop' and the loss
function used is 'mean squared error‘.
 The learnt parameters are stored in a file, that is
further used in generation phase.
19
Model =
network_utils.create_lstm_network(num_freque
ncy_dimensions=freq_space_dims,
num_hidden_dimensions=hidden_dims)
#hidden_dims=1024

20. Generation
phase:
 Step 1 - Given A = [X0, X1, ... Xn], generate Xn + 1
 Step 2 - Append Xn + 1 to A.
 Step 3 - Repeat Steps 1 and 2 again and again depending on
the length of the music piece user wish to generate.
#We take first chunk of the training data as a seed sequence
seed_len = 1
seed_seq = seed_generator.generate_copy_seed_sequence(seed_length=seed_len, training_data=X_train)
#This defines how long the final song is. Total song length in samples = max_seq_len * example_len
max_seq_len = 10
output = sequence_generator.generate_from_seed(model=model, seed=seed_seq,
sequence_length=max_seq_len, data_variance=X_var, data_mean=X_mean)
20

21. Challenges!  The music that is generated has to be unique without
infringing the copyrights.
 It has to sound good, and what sounds good is very
subjective.
 Unlike traditional math problem, this cannot be solved
by set of formulae.
 It takes lot of time to train the model in order to make it
generate good music.
21

22. How long
does it
take?
Well it requires time…
 After about 100 iterations over 20
different music files.
 After 2000 iterations over 20
different music files.
22

23. Python
Libraries
used:
• Keras version 0.1.0 with Theano as the backend.
• NumPy and SciPy for various mathematical
computation on tensors.
• Matplotlib for visualizing the input.
• LAME and SoX to convert mp3 files into other
formats such as wav.
23

24. The entire code is on GitHub.
https://github.com/unnati-xyz/music-generation
24

25. Key notes
for using
this code to
generate
your own
music:
Step 1: Converting the given mp3 files into np
tensors.
python convert_directory.py // creates
YourMusicLibraryNP_x.npy, YourMusicLibraryNP_y.npy
Step 2: Training the model.
python train.py // creates LSTM model
Step 3: Generating the music.
python generate.py // final generated music which is stored
in a file named generated_song.wav.
25

26. References:
 http://colah.github.io/posts/2015-08-Understanding-LSTMs/
 http://karpathy.github.io/2015/05/21/rnn-effectiveness/
 https://cs224d.stanford.edu/reports/NayebiAran.pdf
 https://www.quora.com/What-is-machine-learning-in-
laymans-terms-1
 http://www.hexahedria.com/2015/08/03/composing-music-
with-recurrent-neural-networks/

27. Fun at UNNATI! 

28. THANK YOU! 
CC-Attribution-ShareAlike
https://creativecommons.org/licenses/by-sa/4.0/
www.linkedin.com/in/padmajavb