oral defense slides

1. A System to Generate Rhythms
Automatically for Songs in Rhythm Game
Kuan-Ting Chen 陳冠廷
Advisor : Yi-Shin Chen
Institute of Information Systems and Applications
National Tsing Hua University
1

2. Outline
• Introduction
• Related Work
• Methodology
• Experiment
• Conclusion and Future Work
• Demo
2

3. Introduction
• There are many different music genres
• People have their preferred and not preferred
music genres
• Making people to engage in the music genres
which they are not preferred is our purpose
3

4. Introduction
• Rhythm Game
– Widely be played
So we use rhythm game to achieve our goal
4

5. Related Work
• Conducting systems
– iConductor-[Liu et al. 2009]
– An interactive conducting system using kinect-[Toh
et al. 2012]
– Music interaction on mobile phones-[Chao et al.
2014]
5

6. Related Work
• Rhythm games
– Related to melody
• Guitar Hero
• Magic Piano
• Sing! Karaoke
– Only related to tempo
• 太鼓の達人
• ドラムマニア(DrumMania)
6

7. Related Work
• But in existing rhythm game, every time
players play, the tempo hints will always be
the same
• Replayability-[B. Shelley. Guidelines for Developing
Successful Games. 2001]
it motivates us to generate the dynamic tempo hints
for the rhythm game
7

8. Objective
• Static Mode
– For people who want to make good score in
rhythm game
• Dynamic Mode
– For people who want freshness in rhythm game
– make people stick in rhythm game more
– make game more flexible(just input a MIDI file,
then can directly play, no need to compose
rhythm sheet by players themselves)
8

9. Data
Collection
Number
Or
Character
Format
Data
Preprocessing
Patterns
Pattern
Extraction
Longest
Common
Subsequence
Algorithm
Aggregate
result
from all
songs
Delete
duplicate
result
actually in
rhythm game
Keep patterns
whose count is
greater than
threshold,
Split to
patterns
whose length
is one
measure
Difficulty
Score
Calculation
Metrical
Complexity
Numbers of
Note
Longest
Repetition
method
Patterns
with
difficulty
score
Rhythm game
Easy difficulty level
Medium difficulty level
Hard difficulty level
Differentiate
Methodology
Framework
Pattern
Replacement
9

10. Data
Collection
Number
Or
Character
Format
Data
Preprocessing
Patterns
Pattern
Extraction
Longest
Common
Subsequence
Algorithm
Aggregate
result
from all
songs
Delete
duplicate
result
actually in
rhythm game
Keep patterns
whose count is
greater than
threshold,
Split to
patterns
whose length
is one
measure
Difficulty
Score
Calculation
Metrical
Complexity
Numbers of
Note
Patterns
with
difficulty
score
Rhythm game
Easy difficulty level
Medium difficulty level
Hard difficulty level
Differentiate
Methodology
10
Framework
Pattern
Replacement
Longest
Repetition
method

11. Methodology
Data Collection
• Static Mode
– Downloaded specific MIDI file from
“https://freemidi.org/”
• Dynamic Mode
– We want to extract common rhythm patterns in
many songs
– Downloaded 24836 MIDI files from
“https://freemidi.org/”
11

12. Data
Collection
Number
Or
Character
Format
Data
Preprocessing
Patterns
Pattern
Extraction
Longest
Common
Subsequence
Algorithm
Aggregate
result
from all
songs
Delete
duplicate
result
actually in
rhythm game
Keep patterns
whose count is
greater than
threshold,
Split to
patterns
whose length
is one
measure
Difficulty
Score
Calculation
Metrical
Complexity
Numbers of
Note
Patterns
with
difficulty
score
Rhythm game
Easy difficulty level
Medium difficulty level
Hard difficulty level
Differentiate
Methodology
12
Framework
Pattern
Replacement
Longest
Repetition
method

13. Methodology
Data Preprocessing
• Static Mode
– Covert that specific MIDI file to XML file
• Dynamic Mode
– Only process with 15493 4/4 beat MIDI files
– Covert them to XML files
13

14. • Instrument number
• Measure number
• Number Or Character Format
– 32nd note → 1
– Fourth note → 8
– Half note → G
– Whole note → W
– 32nd rest → (1)
– Fourth rest → (8)
– Half rest → (G)
– Whole rest → (W)
Methodology
Data Preprocessing
14

15. Methodology
Data Preprocessing
15

16. 16
Data
Collection
Number
Or
Character
Format
Data
Preprocessing
Patterns
Pattern
Extraction
Longest
Common
Subsequence
Algorithm
Aggregate
result
from all
songs
Delete
duplicate
result
actually in
rhythm game
Keep patterns
whose count is
greater than
threshold,
Split to
patterns
whose length
is one
measure
Difficulty
Score
Calculation
Metrical
Complexity
Numbers of
Note
Patterns
with
difficulty
score
Rhythm game
Easy difficulty level
Medium difficulty level
Hard difficulty level
Differentiate
Methodology
Framework
Pattern
Replacement
Longest
Repetition
method

17. Methodology
Pattern Extraction
• Longest Common Subsequence Algorithm
17
(G)8(8)(4)4(8)8(8) (4)4(8)8(8) (4)4(8)8(4)4 (4)4(8)8(8)(4)4(8)8(8)(4)4(8)8(4)4 (4)4(8)44(4)4
Longest common subsequence algorithm
(4)4(8)8(8)(4)4(8)8(8)(4)4(8)8(4)4

18. 18
Aggregate LCS result from all songs
Combine duplicate result actually in rhythm game
Keep patterns whose count is greater than threshold,
split to patterns whose length is one measure
Using LCS algorithm and then divide the count depending on the length of the song
Methodology
Pattern Extraction

19. 19
Data
Collection
Number
Or
Character
Format
Data
Preprocessing
Patterns
Pattern
Extraction
Longest
Common
Subsequence
Algorithm
Aggregate
result
from all
songs
Delete
duplicate
result
actually in
rhythm game
Keep patterns
whose count is
greater than
threshold,
Split to
patterns
whose length
is one
measure
Difficulty
Score
Calculation
Metrical
Complexity
Numbers of
Note
Patterns
with
difficulty
score
Rhythm game
Easy difficulty level
Medium difficulty level
Hard difficulty level
Differentiate
Methodology
Framework
Pattern
Replacement
Longest
Repetition
method

20. • Static Mode
– Ex. Easy difficulty level:
→ Medium difficulty level:
• Dynamic Mode
– Metrical Complexity
– Numbers of Note
– Longest Repetition method
Methodology
Difficulty Score Calculation
20

21. Methodology
Difficulty Score Calculation
• Metrical Complexity [G. T. Toussaint,2002]
• Formula: 𝑖=1
𝑛
𝑚𝑎𝑥 𝑤𝑖 − 𝑖=1
𝑛
𝑤𝑖
(6+5+4+4) - (6+4+5+4) = 0
1
0 2 5
3 6
4 7 12
9
8 14
10 11 13 17
16
15 18 19 20 22
21 31
29
28
27
26
25
24
23 30
x x x x
21

22. Methodology
Difficulty Score Calculation
• Numbers of Note
8
2
22

23. • Longest Repetition method
– “SuccessiveSubPattern” function
• Ex. Input: “44442(2)2(2)44” Output: “4444”, “2(2)2(2)” and “44”
– “FindUnitPattern” function
• Ex. Input: “4444” Output: “4”
– “LongestRepeatedSubPattern” function
• Ex. Input: “84(8)84” Output: “84”, because “84” occurs twice in this pattern
– “Difficulty Score” function
• rt: rhythm notation
• Note:
1
𝑟𝑡
Ex. quarter note → 1/8 32nd note → 1/1
• Rest: -
1
𝑟𝑡∗2
Ex. quarter rest → -1/16
• Because one quarter note plus one quarter rest equals one half note in rhythm
game, so we set the difficulty score for rest this way (1/8 +(-1/16) = 1/16)
23
Methodology
Difficulty Score Calculation

24. • Output of “SuccessiveSubPattern”function is
not null:
– Ex. Pattern: “44442(2)2(2)44”
24
Methodology
Difficulty Score Calculation
4444 2(2)2(2) 44
(1/4)
repeat four times
*(1/4) + [(1/2-1/4)]
repeat two times
*(1/2) +(1/4)
repeat two times
*(1/2)

25. • Output of “SuccessiveSubPattern”function is null, but
output of “LongestRepeatedSubPattern” function is
not null:
– Ex. Pattern: “84(8)84”
25
Methodology
Difficulty Score Calculation
84 84 (8)
(1/8+1/4) *(1/2)
repeat two times
but not adjacent
*1.2 +(-1/16)

26. • Output of “SuccessiveSubPattern”function and
output of “LongestRepeatedSubPattern” function are
all null:
– Ex. Pattern: “(2)BC7”
26
Methodology
Difficulty Score Calculation
(-1/4) +(1/11) +(1/12) +(1/7)

27. 27
Methodology
Difficulty Score Calculation
Metrical
Complexity
Numbers of
Note
Longest
Repetition
method
Pattern
mcScore ntScore lrScore
FinalScore
*1/3 *1/3 *1/3

28. Methodology
• Sort patterns by difficulty scores
• Split these patterns into three parts
28

29. 29
Data
Collection
Number
Or
Character
Format
Data
Preprocessing
Patterns
Pattern
Extraction
Longest
Common
Subsequence
Algorithm
Aggregate
result
from all
songs
Delete
duplicate
result
actually in
rhythm game
Keep patterns
whose count is
greater than
threshold,
Split to
patterns
whose length
is one
measure
Difficulty
Score
Calculation
Metrical
Complexity
Numbers of
Note
Patterns
with
difficulty
score
Rhythm game
Easy difficulty level
Medium difficulty level
Hard difficulty level
Differentiate
Methodology
Framework
Pattern
Replacement
Longest
Repetition
method

30. • Static Mode
– Ex. Easy difficulty level:
→ Medium difficulty level:
• Dynamic Mode
– We want to match the specific song, do not want
that generated rhythm is totally unrelated to that
specific song
– AAAB → CCCD
(length of A,B,C,D are all one measure)
Methodology
Pattern Replacement
30

31. Experiment
Experimental Setup
• Nexus 5 smartphone and a rhythm game open
source(Beats)
• Two set of experiment:
– In Quiet environment
• 22 people(11 people started from their preferred music
genre, other 11 people started from their not preferred
music genre)
– In noisy environment
• 6 people(all started from their not preferred music
genre)
31

32. • Classical music and Popular music
• Provide 5 songs in each music genre for
selection
• Listening → Playing
• Open triangle sound for classical music, bass
drum sound for popular music
• Let them rate the interesting level by 5 point
Likert scale
32
Experiment
Experimental Setup

33. Experiment
Experimental Setup
32

34. Experiment
Experimental Setup
33

35. Experiment
Experiment Result
35
Preferred music genre
Not preferred music genre
The hypothesis passed the t-test with
the p-value 2.41941*10−9
,
which indicates that the results is
significant at the p < 0.001 level
(99% confidence level).
The hypothesis passed the t-test with
the p-value 6.5886*10−8,
which indicates that the results is
significant at the p < 0.001 level
(99% confidence level).
0
100
200
300
400
500
600
700
800
900
1000
1100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
seconds
user no.
listening duration playing duration
0
100
200
300
400
500
600
700
800
900
1000
1100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
seconds
user no.
listening duration playing duration
In Quiet environment

36. 36
Experiment
Experiment Result
0
1
2
3
4
5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
5
points
Likert
scale
user no.
preferred music genre not preferred music genre
Preferred music genre average points: 3.77
Not preferred music genre average points: 3.63
Feel open triangle sound does not match the song
In dynamic mode, it happens that rhythm does not match the song,
so bass drum sound makes it strange
The quality of music is not good(because of using MIDI file)
In Quiet environment

37. 37
Experiment
Experiment Result
0
20
40
60
80
100
120
140
160
180
1 2 3 4 5 6
seconds
user no.
listening duration playing duration
Preferred music genre
Not preferred music genre
0
100
200
300
400
500
1 2 3 4 5 6
seconds
user no.
listening duration playing duration
The hypothesis passed the t-test with
the p-value 0.03606,
which indicates that the results is
significant at the p < 0.05 level
(95% confidence level).
The hypothesis passed the t-test with
the p-value 0.00156,
which indicates that the results is
significant at the p < 0.05 level
(95% confidence level).
In Noisy environment

38. 38
Experiment
Experiment Result
0
1
2
3
4
5
1 2 3 4 5 6
5
points
Likert
scale
user no.
preferred music genre not preferred music genre
Preferred music genre average points: 4.83
Not preferred music genre average points: 4
Tempo hints in that popular song is not exciting
In Noisy environment

39. Conclusion and Future Work
• we propose a system which can generate rhythms
for songs in rhythm game automatically
• The experiment result reveals that in either
preferred or not preferred music genre, the
playing duration is longer significantly than
listening duration of users
• we expect that our rhythm game can engage
people in music genre that is uncommon in their
daily lives
39

40. Conclusion and Future Work
• In our rhythm game, we use MIDI files to let
players to play. If could use MP3 files instead,
the game experience will be better
• If could make further adjustment about
patterns to make it more suitable for specific
song, the game experience will be better
36

41. Demo
37

42. Thanks for your listening
38

43. References
• Beats portable. https://github.com/Keripo/Beats.
• DrumMania (ドラムマニア). https://en.wikipedia.org/wiki/Drum_Mania.
• Freemidi.org. https://freemidi.org/
• Guitar Hero. https://en.wikipedia.org/wiki/Guitar_Hero.
• Longest common subsequence.
https://en.wikipedia.org/wiki/Longest_common_subsequence_problem.
• Magic Piano.
https://play.google.com/store/apps/details?id=com.smule.magicpiano&hl=zh_TW.
• Musescore. https://musescore.org/.
• Sing! Karaoke. https://en.wikipedia.org/wiki/Sing!_Karaoke.
• Taiko no Tatsujin (太鼓の達人) https://en.wikipedia.org/wiki/Taiko_no_Tatsujin.
• [Chao et al. 2014] W. Chao, K.-T. Chen, and Y.-S. Chen. Music interaction on mobile
phones. In Advances in Multimedia Information Processing - PCM 2014, pages
153–162. Springer, 2014.
43

44. References
• [Lerdahl et al. 1983] F. Lerdahl and R. Jackendoff. A Generative Theory of Tonal Music.
MIT Press, Cambridge, 1983.
• [Liu et al. 2009] Y.-L. Liu and Y.-S. Chen. iconduct:music control in the interactive
conducting system. Technical report, 2009.
• B. Shelley. Guidelines for Developing Successful Games.
http://www.gamasutra.com/view/feature/3041/guidelines_for_developing_.php,
2001. [Online; accessed 15-August-2001].
• [Toh et al. 2012] L.-W. Toh, W. Chao, and Y.-S. Chen. An interactive conducting
system using kinect. In Proceedings of The 2013 IEEE International Conference on
Multimedia and Expo, pages 1–6. IEEE, 2012.
• [Toussaint 2002] G. T. Toussaint. A mathematical analysis of african, brazilian and
cuban clave rhythms. In Proceedings of BRIDGES: Mathematical Connections
in Art, Music and Science, pages 157–168, 2002.
44