The document discusses an improvisation support system called JamSketch that allows musical novices to enjoy creating music. JamSketch uses a simple drawing interface where users draw melodic outlines, and a genetic algorithm then generates a complete melody in real-time based on the outline. An experiment found the system-generated melodies were comparable to inexperienced human players but lacked human-likeness. Ongoing work aims to improve rhythm variety and allow input on smartphones or with eye tracking. The author believes systems should act as ghostwriters to support but not replace human musical ideas.

1. 音楽の非専門家が創作・演奏を通じて
音楽を楽しめる世界を目指して
北原 鉄朗 （日本大学文理学部）
Twitter: @tetsurokitahara
Feel free to take & distribute photos on Twitter etc!
Towards the World in which Musical Novices Enjoy Music
through Creation and Performance
Tetsuro Kitahara (Nihon University)

2. Introduction to myself

3. Introduction to myself
● Name:
きたはら てつろう
北原 鉄朗 (Tetsuro Kitahara)
● Born in Sapporo, Japan
● Living in Tokyo, Japan
● Age: 39 (born in 1979)
● Position: Associate professor at Nihon University
● Favorites: Music, Drinking alcohol, etc.
● Studied music at Yamaha Junior Music School

4. My academic history
● 2002: BSc from Tokyo University of Science
– Musical instrument recognition
– (Virtually) supervised by Prof. Okuno and Dr. Goto
● 2007: PhD from Kyoto University
– Musical instument recognition and its appl. to MIR
– Supervised by Prof. Okuno and Dr. Goto
● 2007-2010: PostDoc at Kwansei Gakuin University
– Project head: Prof. Katayose
● 2010-2014: Assistant professor at Nihon University
● 2014-present: Associate professor at Nihon Univ.
– 2016-2017: Visiting reseacher at UPF (about 5 months)

5. born here
Tokyo Univ. of Sci.
(Chiba branch)
Nihon Univ.
Kyoto Univ.
Universitat Pompeu Fabra
Kwansei Gakuin Univ.
(super country-side)

6. Kitahara Lab. (Sound Computing Lab.)
Music-related topics
Automatic music generation
HCIs for assisting
non-musicians’ composition
C
G
E
C
Am
A
E
C
Music recommendation
Karaoke enhancement
Guitar transcription
Bass analysis etc.
Other topics
Analysis of cuteness of female voices
Human-agent interaction via sound etc.

7. Spectrum of topics in music computing
What to support
Appreciation Performance Composition
Whotosupport
ExpertsNon-experts

8. JamSketch: Improvisation support system
with GA-based melody creation
from user’s drawing
Special thanks to Rafael Ramirez and Sergio Giraldo
(Universitat Pompeu Fabra)

9. Our goal
Enable non-musicians to enjoy improvisation
Why it’s difficult
Players have to create melodies and
play an instrument at the same time
Technical issues
Human Interface
Non-musician-friendly,
easy-to-input, intuitive
& quickly inputable
Melody generation
Automatic melody generation
based on the user’s abstract,
incomplete input in real time

10. Our idea: JamSketch
Human Interface
User draws a melodic outline on the pianoroll
Music generation
Once the melodic outline is drawn, a melody is
immediately generated with a genetic algorithm

11. System overview
time
During the playback of the accompaniment
0th
bar 1st
bar 2nd
bar
Draws a melodic outline at least one bar in advance
Determines the rhythm of the melody
User
If the mouse cursor moves to the next bar
or the mouse button is released,
Determines the pitch of each note
Determines expression parameters
Plays the melody

12. Determine the rhythm
● High gradient → short notes
● Low gradient → long notes
Basic idea
3
Details of algorithm
0 Candidates
3
33
3 3
33
1 Tentative rhythm
Threshold
the gradient
2 Best-fit rhythm
Search the
closest one
from 0
3

13. Determine the pitches
Basic idea
closeness to
the outline
Search a sequence of notes satisfying
both
3
C A F
and musical
likeliness
using a genetic algorithm (GA)
Details of formulation
Chromosome N = (n0
, n1
, …, nL-1
) (ni
: note number)
Fitness function
F(N) = w0
sim(N) + w1
seq1
(N) + w2
seq2
(N)
+ w3
harm(N) + w4
ent(N)
Similarity
to outline
Bigram
P(ni
| ni-1
)
Delta bigram
P(ni
-ni-1
| ni-1
-ni-2
)
P(notes | chord) Entropy

14. Details of fitness function
Similarity to outline
sim(N) = - Σ(ni
- y(ti
))2
Pitch bigram probability
seq1
(N) = Σ log P(ni
| ni-1
)
(y(ti
): pitch at time ti
in outline)
Delta (pitch-motion) bigram probability
seq2
(N) = Σ log P(ni
– ni-1
| ni-1
– ni-2
)
Conditional probability for given chords
harm(N) = Σ log P(ni
| ci
, bi
)
(ci
: chord at time ti
)
(bi
: metric position at ti
,
{head, on-beat, off-beat})Entropy
ent(N) = -(H(N) – Hmean
– ε)2

15. Determine expression parameters
Use a model presented at [Giraldo & Ramirez 2016]
C
Feature extraction
for each note
k-NN, multilayer perceptron, ...
onset deviation
duration ratio
energy ratio
(Duration, onset, prev. duration,
next duration, prev. interval,
next interval, namour, chord, etc.)

16. Implementation
Java
Groovy
Octave
MIDI Toolbox
WekaJamSketch
CrestMuse
Toolkit
Apache
Commons Math
Java
Octave
Estimation of
expression parameters
GA
Feature extraction
Main program
Training data
53 melodies with the Tonality-type of Blues from
Weimar Jazz Database

17. Experiment
Purpose
Confirm the quality of generated melodies
Method
1. Participants play improvisation with our system and
a normal MIDI keyboard
2. The performances with our system and the keyboard
are listed without distinction
3. Other participants evaluate them
If there are no clear differences between evaluations
for our system and the keyboard, our system can be
considered to have human-like composition capability

18. Participants & performances
Participants
Experience Use
Play Impr. KB Syst.
P1 Yes Yes Yes No
P2 Yes No Yes Yes
P3 Yes No No Yes
P4 No No No Yes
P5 Yes No Yes Yes
P6 Yes Yes Yes No
Performances
?
?
?
?
?
?
?
?
STOP
System
System
KB (P2)
KB (P1)
System
KB (P5)
System
KB (P6)
s2
s1
u2
u1
s4
u3
s3
u4

19. Listening test
● We asked 12 researchers / students in musical
fields to evaluate them on a scale of 0 to 10
● Criteria
– Overall quality
– Pitch appropriateness
– Rhythmic appropriateness
– Melodic variation
● http://www.kthrlab.jp/proj/jamsketch/listen/
– Blues-likeness
– Skillfulness
– Human-likeness
Please participate in the listening test!

20. Result: Overall quality
{u4} > {u1, u2} > {s1, s2, s3, s4} ≈ {u3}

21. Result: Pitch appropriateness
{u1, u2, u4} > {s1, s2, s3, s4} > {u3}

22. Result: Rhythmic appropriateness
{u2, u4} > {u1, u3} > {s1, s2, s3, s4}

23. Result: Melodic variation
{u4} >> {s1, s2, s4} ≈ {u1, u2, u3} > {s3} ≈ {u3}

24. Result: Blues-likeness
{u4} > {u2} > {s1, s2, s3, s4} ≈ {u1} >> {u3}

25. Result: Skillfulness
{u4} >> {u2} > {s1, s2, s3, s4} ≈ {u1, u3}

26. Result: Human-likeness
{u4} > {u1, u2} > {u3} >> {s1, s2, s3, s4}

27. Discussions
● Our system is inferior to experienced players
● But it is superior (or close) to inexperienced players
● Human-likeness is lower than both-class players
● Possible reasons:
– Out-of-scale pitches appear at the ends of phrases
● The system does not consider boundaries of phrases
– Out-of-scale pitches appear at long notes
● Rhythm and pitches are independently determined
– Rhythm is too monotonous
● Rhythm needs more variations
● We obtained positive opinions from participants

28. On-going next plans
● Improvement of rhythm determination
– Data-driven
● Input with eye tracking
– Focusing on disabled people
● Input with smartphones
– Approach 1: moving them up and down
– Approach 2: capture the hand’s motion with the camera
(collaboration with Shiramatsu Lab., NITech)
● Performance by multiple players
(collaboration with Shiramatsu Lab., NITech)
● Incremental melody generation

29. General discussions
(just my personal opinions)

30. What is the state-of-the-art of
automatic music composition?
● Recent systems are capable to create musically
appropriate pieces
● In particular, some systems imitate existing
composers’ features well
– e.g. DeepBach (F. Pachet)
https://www.youtube.com/watch?v=QiBM7-5hA6o
● Systems using deep neural nets (CNN, RNN) are
increasing
● Large-scale DBs are still lacking

31. Are there better human interfaces for
non-experts to express musical ideas?
● Melodic outlines are non-musician-friendly,
easy-to-input, intuitive, and quickly inputable
● Basic concept: “Macro structure by the user,
micro structure by the system”
● Other possibilities:
– verbal expressions (impression words)
– melodic examples (existing melodies)
● With melodic outlines, users cannot specify
rhythmic features

32. What is the best relationship
between humans and systems?
● Relation of a producer and a composer?
● Ideally, systems should be ghostwriters?
– Human ghostwriters can compose but cannot
improvise according to another person’s instructions
● Truly original part should be created by humans
● When humans give musically low-quality ideas,
how can systems suggest better ideas?
● What can systems do if the user’s final goal is to
be able to compose music without any support?

33. Conclusion

34. Conclusion
● To enable non-experts to create music, systems
need to have
– an intuitive user interface
– a technique for generating music
● JamSketch: an improvisation support system with
– melodic outlines
– GA-based melody generator
https://github.com/kthrlab/JamSketch/tree/jamsketch-0.1.0-lite
Let’s try it! (on Windows or other Java-compatible platforms)
Please participate in listening test!
http://www.kthrlab.jp/proj/jamsketch/listen/