The following slides were presented at PyCon 2017, Portland, USA. It is the improved version of the project presented at PyCon India 2016. It mainly highlights the ability of machines to generate something as beautiful as music using a neural network model called as LSTM. It also covers some of the challenges, trade-offs that we faced during the project.

1. Algorithmic
Music Generation
PADMAJA V BHAGWAT
www.linkedin.com/in/padmajavb 01

2. 02
#Junior
#Bachelors
#Art
#Music
#Dance
#Technology

3. 03

4. sample #1
sample #2
Can you
tell the
difference?
04

5. How to
make
computer
generate
music?
Step 1: Convert Mp3 files to np-tensors
Step 2:Train the model
Step 3: Generate the music
05

6. Uncompress
the music:
mp3 to
monaural
WAV using
LAME cmd = 'lame -a -m m {0} {1}'.format(
quote(filename), quote(filename_tmp))
06

7. Convert to
Numpy
array
These are divided into blocks of equal
size by zero padding it.
07

8. Convert
from time to
frequency
domain
Convert from the time domain
to its corresponding frequency
domain using a Discrete
FourierTransform.
08
fft_block = np.fft.fft(block)

9. What model to use?
09

10. Can we use
traditional
Machine
Learning
algorithm?
10

11. Neural networks…
11

12. We’ll go with RNN...
12
* http://colah.github.io/posts/2015-08-Understanding-LSTMs/

13. Can we
make it
remember
for longer
period of
time?
Yes we can! By using
LSTM networks.
13
* http://colah.github.io/posts/2015-08-Understanding-LSTMs/

14. What are
LSTM
Networks?
14
Long
Short
Term
Memory

15. Let’s
understand
its
architecture
.
15
* http://deeplearning.net/tutorial/lstm.html

16. This is how it looks!
Who carries the
legacy??
-cell state
16
* http://colah.github.io/posts/2015-08-Understanding-LSTMs/

17. Step 1 Deciding what to throw
away;
Who does that?
- forget gate layer
17
* http://colah.github.io/posts/2015-08-Understanding-LSTMs/

18. Step 2
Deciding what to
update;
Who does that?
-Input gate layer
&
-tanh layer
18
* http://colah.github.io/posts/2015-08-Understanding-LSTMs/

19. Step 3
old cell state Ct-1 new cell state Ct
19
* http://colah.github.io/posts/2015-08-Understanding-LSTMs/

20. Step 4 Deciding the output;
Who does that?
-Sigmoid layer
20
* http://colah.github.io/posts/2015-08-Understanding-LSTMs/

21. How to code
all this??
21

22. How to do it with Python?
from keras.layers.recurrent import LSTM # import LSTM network
from keras.models import Sequential # import Sequential model
model = Sequential() # instantiate model
# add LSTM layer to the model
model.add(LSTM(input_dim = num_hidden_dimensions,
output_dim=num_hidden_dimensions, return_sequences=True))
22

23. Training
the
model
Optimizer – ‘rms-prop’
Loss function – ‘mean squared error’
23
* http://sebastianruder.com/optimizing-gradient-descent/

24. Train the model
while cur_iter < num_iters:
# Iterate over the training data in batches
history = model.fit(X_train, y_train,
batch_size=batch_size, nb_epoch=epochs_per_iter,
verbose=1, validation_split=0.0)
cur_iter += epochs_per_iter
24

25. Generation
phase
A = [X0, X1, ... Xn]
Predict Xn + 1Append Xn + 1 to A
25

26. # We take first chunk of the training data as a seed sequence
seed_seq = seed_generator.generate_copy_seed_sequence(seed_length=seed_len,
training_data=X_train)
# This defines how long the final song is. Total song length in samples = max_seq_len * example_len
output = sequence_generator.generate_from_seed(model=model, seed=seed_seq,
sequence_length=max_seq_len)
26
Generate the music

27. Challenges! Challenges! Challenges!
27

28. Data
intensive
28

29. Memory
intensive
29
0
0.5
1
1.5
2
2.5
3
3.5
Tensors
Memory Consumed in GB
10 20 50
AWS instance:
t2.2xlarge:
32 GiB of memory
8 vCPUs
EBS-only
64 bit platform

30. Time intensive
4
18
72
0.7
3.5
14
0
10
20
30
40
50
60
70
80
100 500 2000
TIIME
ITERATIONS
CPU GPU
30

31. How long does it take?
After about 100 iterations
over 20 different music files.
2231
# Parameters
num_iters = 100
epochs_per_iter = 25
batch_size = 5

32. After 2000 iterations over 20
different music files.
Well it takes time…
# Parameters
num_iters = 2000
epochs_per_iter = 25
batch_size = 5
32

33. Python
Libraries
used
• LAME and SoX to convert mp3 files
into other formats such as wav.
• NumPy and SciPy for various
mathematical computation on
tensors.
• Matplotlib for visualizing the input.
• Keras version 0.1.0 with Theano as
the backend.
33

34. The entire code is on GitHub
https://github.com/unnati-xyz/music-generation
34

35. Key notes
for using
this code to
generate
your own
music
Step 1: Convert given mp3 files into np
tensors.
python convert_directory.py
# converts mp3 files to numpy tensors
Step 2: Train the model.
python train.py # creates LSTM model
Step 3: Generate the music.
python generate.py # final generated music which is
stored in a file named generated_song.wav.
35

36. Unnati Data
Labswww.unnati.xyz
Mentors: Nischal HP, Bargava Subramanian, Amit Kapoor, Raghotam Sripadraj
Team: Chandana NT, Anirudh Sriram, Subramaniam Thirunavakkarasu
Thank you!
https://github.com/PadmajaVB