MediaEval 2016 - the C@merata task: Natural Language Queries Derived from Exam Papers, Articles and Other Sources against Classical Music Scores in MusicXML
Presenters: Richard Sutcliffe and Cynthia Liem
The C@merata task at MediaEval 2016: Natural Language Queries Derived from Exam Papers, Articles and Other Sources against Classical Music Scores in MusicXML In Working Notes Proceedings of the MediaEval 2016 Workshop, Hilversum, Netherlands, October 20-21, CEUR-WS.org (2016) by Richard Sutcliffe, Tom Collins, Eduard Hovy, Richard Lewis, Chris Fox, and Deane Root
Paper: http://ceur-ws.org/Vol-1739/MediaEval_2016_paper_55.pdf
Video: https://youtu.be/dL6ouVaM5eo
Abstract: Cl@ssical Music Extraction of Relevant Aspects by Text Analysis
(C@merata) is a shared evaluation held at MediaEval and this is the third time the task has run. The input is a natural language query (‘F# in the cello’) and the output is a passage in a MusicXML score which contains this note played on the instrument in question. There are 200 such questions each year and evaluation is via modified versions of Precision, Recall and FMeasure. In 2014 and 2015 the best Beat F (BF) scores were 0.797 and 0.620, both attained by CLAS. This year, queries were more difficult and in addition the most experienced groups from previous years were unable to take part. In consequence, the best BF was 0.070. This year, there was progress concerning the development of the queries, many of these being derived from real sources such as exam papers, books and scholarly articles. We are thus converging on our goal of relating musical references in complex natural language texts to passages in music scores.
MediaEval 2016 - Multimodal Person Discovery in TV Broadcast
Similar to MediaEval 2016 - the C@merata task: Natural Language Queries Derived from Exam Papers, Articles and Other Sources against Classical Music Scores in MusicXML
Similar to MediaEval 2016 - the C@merata task: Natural Language Queries Derived from Exam Papers, Articles and Other Sources against Classical Music Scores in MusicXML (20)
Dubai Call Girls Beauty Face Teen O525547819 Call Girls Dubai Young
MediaEval 2016 - the C@merata task: Natural Language Queries Derived from Exam Papers, Articles and Other Sources against Classical Music Scores in MusicXML
1. The C@merata task at MediaEval 2016: Natural Language Queries Derived
from Exam Papers, Articles and Other Sources against Classical Music
Scores in MusicXML
Richard Sutcliffe, University of Essex
Cynthia Liem, Delft University of Technology
plus...
Tom Collins, Lehigh University
Eduard Hovy, Carnegie-Mellon University
Richard Lewis, Goldsmiths, University of London
Chris Fox, University of Essex
Deane Root, University of Pittsburgh
3. 3
C@merata Task
There is a series of 200 questions each year:
• A short noun phrase in English referring to musical features in a score,
• A short classical music score in MusicXML.
Required Answer:
• The location(s) in the score of the requested musical feature. We call each such location a
passage.
4. 4
Applications
Linking musical references to scores:
[ah25] repeated crotchet chords
[dc364lh] tonic triad of E major
[ah26] giant unison G from the entire orchestra
Allowing experts to search scores interactively using phrases:
triplet quavers against a chord in the bass
Helping students to learn music theory:
perfect cadence brings up an example in a score
ah: Hopkins, A. (1982). The Nine Symphonies of Beethoven. London: Pan Books.
dc: Cooke, D. (1995). Bruckner, (Joseph) Anton. In S. Sadie (ed), New Grove Dictionary of
Music and Musicians, Volume 3, Section 7. Music (p362-366). London, UK: Macmillan.
5. 5
Task Design
There are 200 queries.
We saw earlier that we have:
Provided Question:
• A short noun phrase in English referring to musical features in a score,
• A short classical music score in MusicXML.
Required Answer:
• The location(s) in the score of the requested musical feature. We call each such location a
passage.
6. 6
Passages and Divisions
We specify a passage as follows:
• a time signature,
• a divisions value
• a start bar and beat
• an end bar and beat
Example:
[4/4,1,1:1-2:4]
time signature is 4/4.
divisions value is 1 so we count in crotchets (quarter notes)
Passage starts in bar 1 before the first crotchet (i.e. 1:1)
Passage ends in bar two after the fourth crotchet (i.e. 2:4)
Thus passage consists of the two complete bars (measures) numbered one and two.
7. 7
What is a Divisions Value?
The number of beats into which we divide a crotchet.
A suitable value depends on what we wish to demarcate.
Working in whole crotchets: Divisions = 1
Working in quavers: Divisions = 2
Working in quaver triplets: Divisions = 3
Working in semiquavers or quavers or triplet quavers: Divisions = 12 !!
With Divisions = 12:
one crotchet = 12 beats
one quaver = 6 beats
one quaver triplet = 4 beats
one semiquaver = 3 beats
Note: MusicXML uses the divisions concept and we got the idea from there.
8. 8
Q: C D E F D E C in semiquavers repeated after a semiquaver
A: [ 4/4, 4, 1:2-1:16 ]
Q: semiquaver F# in the treble against a crotchet C in the bass
A: [ 4/4, 4, 4:13-4:13 ]
9. 9
Query Types
In 2014 there were twelve types of query:
Type No Example
simple_pitch 30 G5
simple_length 30 dotted quarter note
pitch_and_length 30 D# crotchet
perf_spec 10 D sharp trill
stave_spec 20 D4 in the right hand
word_spec 5 word "Se" on an A flat
followed_by 30 crotchet followed by semibreve
melodic_interval 19 melodic octave
harmonic_interval 11 harmonic major sixth
cadence_spec 5 perfect cadence
triad_spec 5 tonic triad
texture_spec 5 polyphony
All 200
10. 10
In 2015 there were eight types of query, each modified in various ways:
Type No Example
1_melod 40
D4 minim
eighth note in measure 9
1_melod
qualified by
perf, instr, clef,
time, key
40
trill on a quaver A
G# in the Cello part in measures 29-39
sixteenth note C# in the left hand
half note E3 in 2/2
sixteenth note G in G minor in measures 1-5
n_melod 20
F# E G F# A
Do Mi Do Sol Do Mi Sol Do in bars 1-20
twenty semiquavers
five note melody in bars 1-10
n_melod
qualified by
perf, instr, clef,
time, key
20
two staccato quarter notes in the Violin 1
crotchet, crotchet rest, crotchet rest, crotchet, crotchet rest, crotchet,
crotchet, crotchet, crotchet, crotchet in the Timpani
melodic octave leap in the bass clef in measures 70-80
G4 B4 E5 in 3/4
rising G minor arpeggio
11. 11
1_harm poss
qualified by
perf, instr, clef,
time, key
20
eighth note chord Bb, C, E
chord of D minor in measures 109-110
harmonic minor sixth in the Violas
dotted minim chord in the left hand
texture 6
monophonic passage
homophony in measures 1-14
polyphony in measures 10-14
Alberti bass in measures 0-4
follow possibly
qualified on
either or both
sides by perf,
instr, clef, time,
key
40
quavers F4 E4 in the oboe followed by quavers E2 G#2 in the bass clef
quarter note minor third followed by eighth note unison
C followed by mordent Bb
chord C4 G4 C5 E5 then a quaver
three eighth notes in the Violin I followed by twelve sixteenth notes in
the Violin II in measures 87-92
synch possibly
qualified in
either or both
parts by perf,
instr, clef, time,
key
14
four eighth notes against a half note
crotchet D3 on the word “je” against a minim D2
four staccato quavers in the Violoncello against a minim chord Ab3 C4
F4 in the Harpsichord
All 200
12. 12
In 2016 there are five types of query: 1_melod, n_melod, 1_harm, n_harm and texture
1_melod to n_harm can be qualified by perf, instr, clef, time, key
All can be linked by follow or synch
Type No Example
1_melod 4
A#1 in bars 44-59
quarter-note rest in measures 1-5
1_melod qualified by
perf, instr, clef, time,
key
36
dotted quarter note D6 in the first violin
solo C5 in the oboe in measures 32 onwards
flute dotted half note only against strings
half note on the tonic in the bass clef
A4 sung to the word 'bow'
n_melod 15
two-note dotted rhythm in measures 1-24
eight note rising passage in quarter notes
repeated Bb4 whole note
D4 D5 A5 D6 in sixteenth notes repeated twice
two tied dotted minims in bars 72-88
13. 13
n_melod qualified by
perf, instr, clef, time,
key
45
dotted minims C B A in the Bass clef in bars 70-90
melodic interval of a minor 7th in the voice
rising arpeggio in the left hand in measures 1-10
five-note melody in the cello in measures 20-28
whole note rest, quarter note in the Violin 4 in measures 1-103
1_harm 17
7th triad in measures 1-3
Ia chord in bars 1-10
chord of C
whole-note unison E2 E3 E4
chord III in bars 44-59
1_harm possibly
qualified by perf,
instr, clef, time, key
23
chord of F#3, D4 and A4 in the lower three parts
harmonic fifth in the oboe
harmonic octave in the bass clef
harmonic perfect fourth between the Soprano and Alto in bars 1-9
cello and viola playing dotted minims an octave apart in bars 40-70
14. 14
n_harm 25
interrupted cadence
A5 pedal in bars 116-138
authentic cadence in measures 14-18
plagal cadence in bars 134-138
three consecutive thirds in bars 1-43
n_harm possibly
qualified by perf,
instr, clef, time, key
15
consecutive sixths between the Altos and Basses in measures 73-80
flute, oboe and bassoon in unison in measures 1-56
consecutive descending sixths in the left hand
alternating fourths and fifths in the Oboe in bars 1-100
Soprano and Alto moving one step down together in measures 1-12
texture 20
all three violn parts in unison in measures 1-59
polyphony in measures 5-12
homophonic texture in measures 125-138
imitative texture in bars 1-18
counterpoint in bars 1-14
All 200
15. 15
Type No Example
follow possibly
qualified on either or
both sides by perf,
instr, clef, time, key
20
C D E F D E C in semiquavers repeated after a semiquaver
eighth-note twelfth followed by whole-note minor tenth between
Cello and Viola
D C# in the right hand, then F A G Bb in semiquavers in the left
hand
B flat in the cbass followed a quarter note later by B natural in the
cbass
5 B4s followed by a C5
synch possibly
qualified in either or
both parts by perf,
instr, clef, time, key
13
quarter note E5 against a quarter note C#3
C#3 minim and E4 semibreve simultaneously
D3 in the bass at the same time as C5 in soprano 1
three-note chord in the harpsichord right hand against a two-note
chord in the harpsichord left hand in measures 45-52
A#3 in the piano and F#5 in the voice simultaneously
16. 16
Q: three-note chord in the harpsichord right hand against a two-note chord in the harpsichord left
hand in measures 45-52
A: [ 3/4, 1, 45:3-45:3 ], [ 3/4, 1, 51:1-51:3 ]
17. 17
Q: imitation in the piano left hand and voice in measures 1-6
A: [ 4/4, 1, 1:2-3:4 ]
20. 20
handel_water_music_suite_air 4 2 vn, va, vc Amer.
palestrina_alma_redemptoris_mater 4 SATB Amer.
schubert_str_quartet_no10_op125_d87_m3 4 2 vn, va, vc Eng.
morley_now_is_the_month_of_maying 5 SATTB Eng.
weelkes_hark_all_ye_lovely_saints 5 SSATB Eng.
vivaldi_conc_4_vns_op3_no10_rv580 8 4 vn, 2 va, vc, db Amer.
vivaldi_conc_vn_op6_no6_rv239_m1 8 3 vn, va, vc, db, hpd Amer.
mozart_symphony_no40_m4 10
fl, 2 ob, 2 bn, 2 hn, 2 vn, va,
vc, db
Eng.
beethoven_symphony_no3_m3 13
2 fl, 2 ob, 2 cl, 2 bs, 2 hn, 2
tpt, timp, 2 vn, va, vc, db,
Amer.
handel_messiah_and_the_glory 18
fl, 2 ob, cl, bs, hn, tbn, tuba,
SATB, hpd, 2 vn, va, vc, db
Amer.
21. 21
Evaluation of a Returned Passage
Passage is beat correct if it starts at exactly the correct beat (as specified by the divisions value)
in the correct start bar and also ends at the correct beat in the end bar.
- Useful for applications of results which are themselves automatic.
Passage is measure correct if it starts in the bar where the requested feature starts and ends in the
bar where the requested feature ends.
- Sufficient in many cases for humans to use.
22. 22
Beat Precision (BP) is the number of beat correct passages returned by a system divided by the
number of passages (correct or incorrect) returned.
Beat Recall (BR) is the number of beat correct passages returned by a system divided by the
total number of answer passages known to exist.
Measure Precision (MP) is the number of measure correct passages returned by a system
divided by the number of passages (correct or incorrect) returned.
Measure Recall (MR) is the number of measure correct passages returned by a system divided
by the total number of answer passages known to exist.
23. 23
Participants
Runtag Leader Affiliation Country
DMUN Andreas Katsiavalos De Montfort University England
KIAM Marina Mytrova
Keldysh Institute of Applied
Mathematics
Russia
OMDN Donncha Ó Maidín University of Limerick Ireland
UMFC Paweł Cyrta
Fryderyk Chopin University of
Music
Poland
30. 30
Overall Conclusions
We refined our ideas
Some questions were chosen from real sources such as exam papers
Questions got harder and hence more realistic
This was difficult for participants!
31. 31
We finish with examples of three interesting and complex question types:
against
follow
texture
32. 32
Q: dotted crotchet against quavers in the bass in bars 1-43
A: [ 3/4, 2, 9:3-9:5 ], [ 3/4, 2, 10:3-10:5 ], [ 3/4, 2, 11:3-11:5 ], [ 3/4, 2, 12:3-12:5 ]
33. 33
Q: dotted crotchet followed by D5
A: [ 3/4, 4, 9:5-9:12 ], [ 3/4, 4, 11:5-11:12 ]
37. 37
Work: J.S. Bach, Das Wohltemperierte Klavier, Book 1, Prelude No. 2 in C minor BWV 847
Extract:
38. 38
Q: passage in common time
A: [4/4,1,1:1-4:4]
Q: interval of a melodic sixth
A: [4/4,4,1:1-1:2], [4/4,4,1:8-1:9] [4/4,4,1:9-1:10], [4/4,4,2:1-2:2], [4/4,4,2:8-2:9],
[4/4,4,2:9-2:10], [4/4,4,3:1-3:2], [4/4,4,3:8-3:9], [4/4,4,3:9-3:10]
Q: second
A: None, because a second is harmonic by default
Q: C followed by Eb
A: [4/4,4,1:5-1:6], [4/4,4,1:13-1:14], [4/4,4,4:1-4:2], [4/4,4,4:9-4:10]
Q: C followed by Eb in the bass clef
A: [4/4,4,4:1-4:2], [4/4,4,4:9-4:10]
Q: semiquaver E natural / sixteenth note E natural
A: [4/4,4,2:3-2:4]
39. 39
Work: J.S. Bach, Suite No. 3 in C Major for Cello, BWV 1009, Sarabande
Extract:
Q: harmonic interval of a minor third
A: [3/4,2,208:1-208:1]
Q: minor third
A: [3/4,2,208:1-208:1] (thirds are harmonic by default)
Q: dotted quaver / dotted eighth note
A: [3/4,4,206:5-206:7], [3/4,4,207:5-207:7], [3/4,4,208:5-208:7]
Q: harmonic perfect fifth
A: [3/4,1,206:1-206:1], [3/4,1,207:1-207:1], [3/4,4,208:5-208:7]
(in last passage, C and G should be dotted but were not in the original)
Q: simultaneous harmonic perfect fifth and harmonic eleventh
A: [3/4,4,208:5-208:7]
(same point, C and G are assumed dotted in this example)
40. 40
Work: J. Dowland, King of Denmark's Galliard, P 40
Extract:
Q: perfect cadence
A: [3/4,1,3:3-4:3]
(We are assuming the cadence continues until the end of the bar)
Q: four consecutive quavers / four consecutive quarter notes
A: [3/4,2,2:3-2:6]
Q: dotted minim in the bass / dotted half note in the bass
A: [3/4,1,2:1-2:3]
Q: harmonic fourth
A: [3/4,1,1:1-1:1], [3/4,1,1:2-1:3], [3/4,1,4:1-4:3]
(Note that there are two instances in bar (measure) 1 because the chord is played twice)
41. 41
Work: J. Dowland, Pauana Dulandi, P 86
Extract:
Q: consecutive 5ths in the bass
A: [4/4,1,2:1-2:4]
Q: semibreve Bb in the treble clef / whole note Bb in the treble clef
A: [4/4,1,1:1-1:4], [4/4,1,4:1-4:4]
Q: octave followed two bars later by another octave
A: [4/4,1,1:1-3:4] (assumed harmonic)
Q: Vc triad
A: [4/4,1,2:3-2:3]
42. 42
Work: G.F. Handel, Messiah, HG xlv, "And the Glory of the Lord"
Extract:
Q: crotchet rest / quarter note rest
A: [3/4,1,1:1-1:1]
Q: dotted crotchet followed by three quavers / dotted quarter note followed by three eighth notes
A: [3/4,1,5:1-5:3], [3/4,1,6:1-6:3]
Q: four hemidemisemiquavers / four sixty-fourth notes
A: [3/4,4,3:9-3:9]
Q: three quavers in a row / three eighth notes in a row
A: [3/4,2,5:4-5:6], [3/4,2,6:4-6:6]
43. 43
Work: G.F. Handel, Messiah, HG xlv, Overture
Extract:
Q: D sharp crotchet / D sharp quarter note
A: [4/4,1,22:1-22:1], [4/4,1,25:1-25:1]
Q: D natural quaver / D natural eighth note
A: [4/4,2,23:3-23:3], [4/4,2,24:4-24:4]
Q: open VII triad in the first inversion
A: [4/4,1,23:3-23:3]
Q: melodic fourth in the bass clef
A: [4/4,1,24:1-24:2]
45. 45
Q: eight staccato notes in succession
A: [4/4,1,17:1-17:4]
Q: augmented melodic fourth
A: [4/4,2,19:3-19:4], [4/4,2,20:4-20:5]
Q: third
A: [4/4,4,17:13-17:13], [4/4,2,19:2-19:2], [4/4,2,19:4-19:4], [4/4,4,19:9-19:9], [4/4,4,20:9-20:9]
(these are assumed to be harmonic and can thus be across parts)
Q: a quaver, then a major third / an eighth note, then major third
A: [4/4,2,19:1-19:2], [4/4,2,19:3-19:4]
(third assumed to be harmonic and to follow the quaver immediately)
Q: change from bass to treble clef
A: [4/4,1,17:2p]
(recall that this means the clef change is immediately after 17:2)
46. 46
Work: D. Scarlatti, Keyboard Sonata in D major, K 430
Extract:
Q: three fourths
A: [3/8,2,56:1-56:3]
(assumed consecutive)
Q: treble clef F natural
A: [3/8,2,62:2-62:2]
Q: melodic octave
A: [3/8,4,57:1-57:2], [3/8,4,57:2-57:6]
(melodic must be stated, otherwise it is harmonic)
Q: harmonic 5th followed by harmonic 4th
A: [3/8,2,58:1-58:2], [3/8,2,62:1-62:2]
47. 47
Architecture of Baseline System
We have built a very basic system to perform the task:
1. Analyse the input xml file and extract questions and score files
2. For each question:
• Parse question using Stanford Parser
• Determine question type (similar to normal QA)
• Parse the score file using Music21
• Based on question type, search for answer passages in score
3. Write out answers to output xml file
48. 48
Linguistic Observations
Queries are noun phrases
Head noun group is the main feature (e.g. F#)
PP modifiers qualify this (e.g. in the bass clef)
A lot of terminology is used (e.g. semiquaver, soprano part, forte bar)
Search based on query classification will work at least for simple examples.
49. 49
Summary
We are trying to find ways to link natural language descriptions of music to musical scores.
Starting with the C@merata task, we are working with very simple tasks to develop the
technology.
Then we will progress to more complex tasks which are of genuine interest to musicologists.
In due course, we aim to tie passages in musicological texts such as in Grove Online to the
corresponding music scores.
50. 50
Music Extracts
symphony opening with a horn call over shimering strings
Bruckner 4th Symphony
http://www.youtube.com/watch?v=J8t1TzN0RRY
Start: 0:00
End: 0:44
symphony closing with six unison chords
Sibelius 5th Symphony
http://www.youtube.com/watch?v=nkzrSZKA4cM
Start: 9:36
End: 9:56
Nielsen 4th Symphony, Finale (shows two timpanists well)
http://www.youtube.com/watch?v=yXDe1hj4HBo
Start: 32:10
End: 32:48
51. 51
Extracts Not Used
symphony opening with two octave descending arpeggio
Beethoven 9th Symphony
string quartet opening including double stopping on all four instruments
Beethoven op 127
symphony featuring a battle between two timpanists
Nielsen 4th Symphony, Finale (not so good)
http://www.youtube.com/watch?v=sD9I-UiYfW8
Start: 6:30
End: 7:08
52. 52
Grove’s Dictionary of Music and Musicians
• 1879-1889 George Grove, civil engineer, music administrator, writer and then Director of
Royal College of Music, wrote A Dictionary of Music and Musicians in four volumes.
• 1904-910 Fuller Maitland edited the second edition - Grove's Dictionary of Music and
Musicians - in five volumes.
• 1927 Henry Colles edited the third edition - in five volumes.
• 1940 Henry Colles edited the fourth edition - in seven volumes.
• 1954 Eric Blom edited the fifth edition - in nine volumes.
53. 53
• 1980 Stanlie Sadie edited the sixth edition - The New Grove Dictionary of Music and Musicians
- in twenty volumes. Contained 22,500 articles and 16,500 biographies.
• 2001 Stanlie Sadie edited the seventh edition - in 29 volumes. This was also available online.
• 2009 Deane Root was appointed editor of Grove Music Online. By this time it contained more
than 50,000 articles.
Grove is now considered to be the most comprehensive and scholarly source of information on
Western Classical Art Music which exists.
It is used daily by musicians and musicologists worldwide.
54. 54
What might our queries be like in future?
We will show some examples:
• A natural language phrase describes a musical feature.
• A regular expression or other pattern could not capture it.
56. 56
Some Examples
Q: 'horn call over shimmering strings'
A: Anton Bruckner, Symphony No. 4 "Romantic"
57. 57
Some Examples
Q: 'horn call over shimmering strings'
A: Anton Bruckner, Symphony No. 4 "Romantic"
Q: 'movement ending with six stacatto chords for full orchestra'
58. 58
Some Examples
Q: 'horn call over shimmering strings'
A: Anton Bruckner, Symphony No. 4 "Romantic"
Q: 'movement ending with six stacatto chords for full orchestra'
A: Jean Sibelius, Symphony No. 5
59. 59
Some Examples
Q: 'horn call over shimmering strings'
A: Anton Bruckner, Symphony No. 4 "Romantic"
Q: 'movement ending with six stacatto chords for full orchestra'
A: Jean Sibelius, Symphony No. 5
Q: 'symphony featuring a battle between two timpanists'
60. 60
Some Examples
Q: 'horn call over shimmering strings'
A: Anton Bruckner, Symphony No. 4 "Romantic"
Q: 'movement ending with six stacatto chords for full orchestra'
A: Jean Sibelius, Symphony No. 5
Q: 'symphony featuring a battle between two timpanists'
A: Carl Nielsen, Symphony No. 4 "The Inextinguishable"
61. 61
What are the Characteristics of these Queries?
Not very long
Not that specific ('over', 'featuring')
Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani')
Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle')
62. 62
What are the Characteristics of these Queries?
Not very long
Not that specific ('over', 'featuring')
Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani')
Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle')
Nevertheless, experts can answer them!
