Machine learning work on creating harmony to a given melody. The model was trained on Nottingham Music Database. Modified Markov decision process algorithm was used to compose chord progressions to harmonize melody. Measure of dissonance was introduced to verify the model and choose the best result.

1. Melody Harmonizer
Andrew Zatserkovny

2. Contents
• Collecting training data
• Pre-processing and cleaning data
• Extracting features
• Composing list of states
• Populating transition matrix
• Generating chord progressions
• Verifying the model
• Summary
• References

3. Collecting training data
http://abc.sourceforge.net/NMD
The Nottingham Music Database maintained by Eric Foxley
contains over 1000 folk tunes

4. Collecting training data
http://abcnotation.com/wiki/abc:standard:v2.1
The abc music standard 2.1 (Dec 2011)
Abc is a text-based music notation system designed to be
comprehensible by both people and computers
X: 40
T:The Castle
% Nottingham Music Database
S:Trad, arr Phil Rowe
M:6/8
K:G
D | "G" G3 GAB | "Am" A3 "D7" Abc | "G" edd gdd |
"C" edB "D7" dBA | "G" G3 GAB| "Am" A3 "D7" Abc |
"G" edd gdB | "D7" AGF "G" G2::A | "G" BAB dBd | "C"
e/2f/2ge "D" dBA | "G" BAB dBG | "D7" ABA A2A | "G"
BAB dBd | "C" e/2f/2ge "D" dBA | "G" gfg "D7" aga |
"G" bgg g2: |

5. Pre-processing data
• Exclude files without harmony
• Edit incorrect harmony symbols
• Convert abc to musicxml format
• Detect mode (major/minor) (Krumhansl-
Schmuckler key-finding algorithm)
• Transpose scores to C major and relative
A minor modes
http://rnhart.net/articles/key-finding

6. Extracting features
• Return the
metrical accent of
this object in a
measure (0 to 1)
[Ref. 1]
• Highlight strong
beats (most
audible and
prominent to a
listener)
strong beats

7. Composing list of states
Measure k Measure k+1
)
)
states
transition between states

8. Populating frequency matrix
(+1)
occurrences of
for state
( , ,
)

9. Populating probability matrix
probability of
harmonizing
given

10. Generating chord progression
Markov Decision Process is applied to the task and is
based on input vector of notes and transition matrix

11. Generating chord progression
Major/minor chord sets are comprised of chords encountered
throughout training data
Major chord set (40 unique chords) Minor chord set (21 unique chords)
'A', 'A-', 'A7', 'Am', 'Am7' 'A', 'A7', 'Am',
'B', 'B-', 'B-7', 'B-m', 'B7', 'Bm', 'B', 'B-', 'B7', 'Bm'
'C', 'C#dim', 'C6/F', 'C7', 'Cdim', 'Cm' 'C', 'C#m', 'C7',
'D', 'D6/C', 'D7', 'Dm', 'Dm7' 'D', 'D7', 'Dm',
'E', 'E-', 'E-7', 'E-dim', 'E7', 'Em', 'Em7' 'E', 'E7', 'Em', 'Em7'
'F', 'F#', 'F#dim', 'F7', 'Fdim', 'Fm', 'F', 'Fm',
'G', 'G+', 'G7', 'Gdim', 'Gm' 'G', 'G7'

12. Generating chord progression
Probability matrix has major and minor versions as well
40 chords 21 chords
5760
states
3024
states

13. Generating chord progression
Given these two measures of a sample to be harmonized, we only
extract first note on strong beats

14. Generating chord progression
Input melody is processed by semi-measure strides (on strong beats)
Step 0: Identify mode and start with tonic chord
State 0: (Measure 1, (C chord, E note, E note))
Step 1: Move window one stride to the right along the score
(1, (C, E, E))
C (1, (C, E, D))
C

15. Output chord progression
Result of processing all measures in the score (random progression)
(5, ('C', 'E', 1.0))
(5, ('C', 'E', 0.5))
(6, ('G7', 'D', 1.0))
(6, ('G7', 'F', 0.5))
(7, ('C', 'D', 1.0))
(7, ('F', 'F', 0.5))
(8, ('Am', 'C', 1.0))
(8, ('G7', 'G', 0.5))
(1, ('C', 'E', 1.0))
(1, ('C', 'E', 0.5))
(2, ('D7', 'D', 1.0))
(2, ('D7', 'F', 0.5))
(3, ('G7', 'D', 1.0))
(3, ('G7', 'F', 0.5))
(4, ('C', 'C', 1.0))
(4, ('G', 'B', 0.5))
chordmeasure note beat strength

16. Verifying the model
Musical accent may be defined as an increased prominence,
noticeability, or salience ascribed to a given sound event. Cooper and
Meyer characterized accent as “a stimulus which is marked for
consciousness” [Ref. 2].
Let us introduce the measure of dissonance based on a
notion of musical accent of each note in a score:
i-th note in a score
i-th harmonic context

17. Verifying the model
The most optimal solution implies that the loss function should be
minimized:
Thomassen melodic accent m adds a numeric attribute to each note
according to the method postulated in [Refs. 3, 4]
Dissonance function returns a value depending on the
intervals between a note and chord tones. Semitones, tritones
and whole tones with their respective weights ( ) contribute
to the overall value:
w

18. Verifying the model
The minimum of the loss function (L = 3.027) is achieved with following
limitations and assumptions:
• number of generated progressions is limited to 2000 samples
• dissonance function weights are set as below:
,
The final chord progression that yields the lowest loss function
value is presented on this score:

19. Summary
• The method for extracting melodic and harmonic
features with context to enable chord progression
continuity was proposed
• Markov decision process was modified and applied to
fit the case with no obviously right solution
• Measure of dissonance was introduced to calculate
harmonic proximity of a note to a chord
• Loss function was proposed to evaluate harmonic
adequacy of chord progressions generated by the
model

20. References
1. Fred Lerdahl and Ray Jackendoff. On the Theory of Grouping
and Meter. The Musical Quarterly, Vol. 67, No. 4 (Oct., 1981),
pp. 479-506
2. Cooper, G. & Meyer, L. B. (1960). The rhythmic structure of
music. Chicago & London: University of Chicago Press.
3. Joseph M. Thomassen, Melodic accent: Experiments and a
tentative model. Journal of the Acoustical Society of America,
Vol. 71, No. 6 (1982) pp. 1598-1605
4. David Huron and Matthew Royal, What is melodic accent?
Converging evidence from musical practice. Music Perception,
Vol. 13, No. 4 (1996) pp. 489-516