This document describes a system for supporting improvisational music ensembles using smartphone sensors. The system aims to allow beginners to play improvised melodies that are consonant with a background tune by tracking the vertical motion of a user's hand. It uses Bayesian networks trained on motion data to estimate pitch notation, attack timing, and the vertical motion of the hand based on smartphone sensor values. Evaluation experiments show the system can estimate pitch notation with over 70% accuracy but has low precision for attack timing estimation. Future work includes improving motion tracking, adding ensemble support for multiple users, and collecting supervised training data using a Kinect camera.

1. Support System of Improvisational
Ensemble Based on User’s Motion
Using Smartphone Sensors
Souta MIZUNO1 Shugo ICHINOSE1
Shun SHIRAMATSU1 Tetsuro KITAHARA2
1. Nagoya Institute of Technology 2. Nihon University

2. Purpose
Developing an improvisational ensemble support system
– Even beginners are possible to play the improvisational ensemble with a
background tune
– Difficult element：tonality（chord progression）
→Our system corrects tonality of melody by outputting consonant tones
– Easy element：rhythm, up-and-down of melody（pitch contour）
→ Input using the physical gesture
– The previous study [Ichinose 17] developed an improvisational ensemble
support system with user’s body motion detected by a motion sensor camera
• Rhythm and pitch contour can be specified the body motion
– In this study, we aim to develop using more widely-used devices, i.e.,
smartphone sensors
[Ichinose 17] Ichinose et al., "Improvisation Ensemble Support Systems for Music Beginners Based on Body Motion
Tracking," in Proceedings of the 2017 6th IIAI International Congress on Advanced Applied Informatics, pp. 794-798
523 Hz
494 Hz
440 Hz
392 Hz

3. Issue
1. Estimating the vertical motion of user’s hand to determine pitch notation
- We need to develop an accurate position tracking method to estimate the
vertical motion of user’s hand using smartphone sensors
2. Deciding to input method of attack (output) timing
- By the intuitive motion
3. Outputting consonant tones satisfying tonality with a background tune
- By restricting output sounds
drop hand ⇒ down pitch
raise hand ⇒ up pitch
Outputting the consonant tones
(correct tonality)

4. 1. Method of Position Tracking and the
configuration of our system
– Propose approach to determine pitch
– Description method
– Evaluation experiment of Position Tracking
– About the system configuration
2. Input method of attack timing by intuitive motion
3. Estimating pitch notation by machine learning
4. Evaluation experiment
5. Unsolved issue
Outline

5. Two approaches to determine pitch notation
We try the following two approaches to determine the pitch notation
based on the vertical motion of user’s hand:
Approach1 Cumulative sum of acceleration:
– The pitch notation is determined by estimating the hand position of the vertical
direction from values of acceleration sensor, gyro sensor, etc
Approach2 Cumulative sum of acceleration + Machine learning:
– The pitch notation is determined by machine learning
– Values of smartphone sensors and cumulative sum of acceleration are used as
input values for machine learning
– Method of Bayesian Network

6. Position tracking
Used smartphone sensors
• gyro
• Acceleration
• Gravity
• Magnetic field
Acceleration of the vertical direction × Time
Moving distance ( the vertical motion of user’s hand)
The vertical motion of user’s hand is estimated based on value of
smartphone sensors
Moving distance：p

7. Evaluating accuracy of position tracking
• Comparing accuracy of position tracking
when users did a predefined motion (up ⇒ down ⇒ down ⇒ up …)
- The Blue line is tracking result by smartphone sensors
- The Orange line is tracking result by Kinect
• Estimation error gradually increase
- Error depend on the user
We use the approach2: machine learning

8. Background tune
System configuration
Machine learning
Estimate attack timing and pitch
notation based on user’s motion and
background tune
Position tracking
Detect the feature value of user’s
motion using smartphone sensors.
Improvisational ensemble
support system
User’s physical gesture
(input rhythm and pitch contour)
Output
a background tune
Output melodies based on estimated
attack timing and pitch notation
Input a chord progression of
background tune to machine learning
Input values of smartphone sensors

9. 1. Method of Position Tracking
2. Input method of attack timing by intuitive motion
3. Estimating pitch notation by machine learning
4. Evaluation experiment
5. Unsolved issue
Outline

10. Input method of attack timing
Output by shaking smart phone
 Play by more intuitive motion
 Detection is not easy by
sensors value
Output by tapping button on the
screen
 Since simple method, possible
to accurately output
 Lack of intuitive feel in motion

11. 1. Method of Position Tracking
2. Input method of attack timing by intuitive motion
3. Estimating pitch notation by machine learning
– Create training data
– Description about Bayesian networks
– Definition of Input value (moving distance)
4. Evaluation experiment
5. Unsolved issue
Outline

12. Create training data
To create training data, recording data samples of smartphone sensors when test
subjects move a smartphone along with a particular melody, i.e., edelweiss
(data is sampled per 5ms )
By learning a training data, three Bayesian network models were created
1. For estimating attack timing
2. For estimating pitch notation
3. For estimating the vertical motion of user’s hand

13. Output
h: the vertical motion
of user’s hand
ni: pitch notation
t: attack timing
Bayesian Network (user’s vertical motion)
C D A
• Raise motion
• Drop motion
• No change
• Attack timing
• No attack
Input
• From motion values
ay: acceleration (y-axis)
v: velocity
vc: velocity change
p: moving distance
g: gravity
t: attack timing
rm: the most number of
times estimated result
• From musical context
c : chord chart
(background tune)
ni-1: last pitch notation

14. Define of moving distance
Change the define of input value: p (moving distance)
1. Absolute distance： moving distance from start position
2. Relative distance： moving distance from last output position
- Aim: In case of using relative ldistance, error of position tracking is smaller
than absolute distance
First
attack timing
Second
attack timing
height
time
Absolute distance : 10cm
Relative distance : 10 -5 = 5cm
Start position

15. 1. Method of Position Tracking
2. Input method of attack timing by intuitive motion
3. Estimating pitch notation by machine learning
4. Evaluation experiment
– experiment1:melody generation using training data
– experiment2:evaluation of match rate of user’s
motion and change of pitch
5. Unsolved issue
Outline

16. Evaluation experiment
Experiment1. generate melody using training data
- Accuracy rate of the pitch notation estimation
- Accuracy rate of the vertical motion estimation
- The Recall and the precision of attack timing estimation
Experiment2. generate change of pitch matching user’s motion
- Match rate between user’s motion and estimated change of pitch

17. Evaluation of estimation accuracy
• Evaluation of estimation accuracy of pitch notation and the vertical user’s motion
- Using training data as test data
- Background tune：edelweiss
• Particle size set
- training data sample units
- midi note units
1sample (about every 5ms)
estimation result using first sample
estimation result of note
C
D
E
1note (midi note)
pitch
time
number of note:3
Number of sumple:24

18. Evaluation estimation accuracy
recall =
𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑡𝑡𝑎𝑐𝑘 𝑡𝑖𝑚𝑖𝑛𝑔 𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 𝑏𝑦
𝑜𝑢𝑟 𝑠𝑦𝑠𝑡𝑒𝑚 𝑓𝑜𝑙𝑙𝑜𝑤 𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑 𝑡𝑢𝑛𝑒
𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑡𝑡𝑎𝑐𝑘 𝑡𝑖𝑚𝑖𝑛𝑔
𝑜𝑓 𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑 𝑡𝑢𝑛𝑒
precision =
𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑡𝑡𝑎𝑐𝑘 𝑡𝑖𝑚𝑖𝑛𝑔 𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 𝑏𝑦
𝑜𝑢𝑟 𝑠𝑦𝑠𝑡𝑒𝑚 𝑓𝑜𝑙𝑙𝑜𝑤 𝑏𝑎𝑐𝑘𝑔𝑜𝑟𝑢𝑛𝑑 𝑡𝑢𝑛𝑒
𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑡𝑡𝑎𝑐𝑘 𝑡𝑖𝑚𝑖𝑛𝑔
𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 𝑏𝑦 𝑜𝑢𝑟 𝑠𝑦𝑠𝑡𝑒𝑚
・・・
・・・
Estimated attack timings by our system
・・・
・・・
• Evaluation of estimation accuracy of attack timing
- Particle size is note unit
- The recall and the precision to the background tune were
examined
Attack timings of background tune
・・・
・・・
・・・
・・・
Estimated attack timings by our system
Estimated attack timings by our system

19. absolute distance relative distance absolute distance relative distance
shake motion 0.45 (121/270)
not conducted
(future work)
0.56 (152/270)
not conducted
(future work)
touch motion 0.56 (152/270) 0.71 (192/270) 0.68 (184/270) 0.91 (247/270)
note unit
estimation accuracy of pitch notation
(correct notes / all notes)
estimation accuracy of vertical motion
(correct samples / all samples)
sample unit
estimation accuracy of pitch notation
(correct samples / all samples)
estimation accuracy of vertical motion
(correct samples / all samples)
shake motion 0.70 (45888/65278) 0.60 (42975/65278)
touch motion 0.82 (53764/65251) 0.78 (50909/65251)
Result of experiment1
recall precision
shake motion 0.63 (171/270) 0.26 (171/661)
estimation accuracy of attack timing
• Both shake and touch, accuracy
rate of sample units is higher
than that of note units
• The value in case of using relative
distance is higher than that in case
of absolute distance
• The Precision is very low because even small
motion are recognized shake motion

20. Evaluation match rate between user’s motion
and estimated change of pitch
- Comparing case of using absolute distance and relative distance
- Using musical context: case of using last pitch notation ni-1
- Not using musical context: case of not using last pitch notation ni-1
- Case of using absolute distance: not much difference
- Case of using relative distance: improving accuracy rate
shake motion absolute distance relative distance
using musical context 0.48 (31/64)
not using musical context 0.53 (34/64)
not conducted
(future work)
touch motion absolute distance relative distance
using musical context 0.55 (35/64) 0.53 (34/64)
not using musical context 0.51 (33/64) 0.75 (48/64)
It is the reason of this result
• Estimated melody transition is affected by melody in training data
• Training data has only one tune: edelwess

21. 1. Method of Position Tracking
2. Input method of attack timing by intuitive motion
3. Estimating pitch notation by machine learning
4. Evaluation experiment
5. Unsolved issue
Outline

22. Unsolved issue
• We could not support ensemble case of multiple users
– Currently, improvisational ensemble with a user and background
tune
– The system must correct consonance relation of each user’s
output sound
• There was not accurate training data of the vertical motion
of user’s hand
– Currently, we cannot confirm correctness of the estimated
movement of user’s hand
• because we assume that the vertical motion of user’s hand is the same
vertical motion as sound of background tune (without observation)
– Using data Recorded by Kinect camera as supervised data

23. Recording supervised data by Kinect
We are currently making a training dataset by using Kinect
- As reference data of the vertical motion of user ‘s hand
- This part is not included in our paper yet
Recognized hand position by kinect

24. Data recording experiment for
machine learning
1. The subject did “shake” and “tap” in accordance with the
quadruple rhythm
–the test subjects vertically move smartphone in a predefind motion
“high→high→middle→low” or “high→middle→low→low“
–smart sensor data and tap timing were recorded
high
middle
low
high
middle
low
Shake
Shake
Shake
Tap
Tap
Tap
(This part is not included in our paper)

25. Data collection experiment for
machine learning
2. The hand of test subjects was recognized by
Kinect and record the height of hand as the
accurate training data
(This part is not included in our paper)

26. Characteristics of recorded data
The difference of smartphone’s angle according
to the height of user’s hand
high lowmiddle
(This part is not included in our paper)

27. Discussion: importance of hand angle
Found the height of user’s hand might relate to the hand angle
- We need to importance smartphone’s angle for the more accurate
estimation of the vertical motion
(This part is not included in our paper)

28. Conclusion
• We developed the system estimates the vertical motion of user’s hand
using smartphone sensors
• We created Bayesian network estimates attack timing and pitch notation
by values of smartphone sensors as input values
– Developed case of using shake motion and case of using touch motion
– Estimation accuracy of attack timing is unstable case of shake motion
Future works
• We will improve position tracking method using the accurate
training data that is currently collected
• We need to implement an estimation method suitable for time
series data
– HMM,LSTM,etc
• We aim to implement an ensemble function by multiple users
Conclusion and future work

29. Bayesian Network (attack timing)
Input
ax: acceleration (x-axis)
ay: acceleration (y-axis)
v: velocity
vc: velocity change
p: moving distance
g: gravity
Output
t: attack timing (0 or 1)
Attack timing
or
No attack
Input value

30. Bayesian Network (pitch notation)
Output
ni: pitch notation
Input
• From motion values
ay: acceleration (y-axis)
v: velocity
vc: velocity change
p: moving distance
g: gravity
t: attack timing
rm: the most number of times estimated result
• From musical context
c : chord chart
(background tune)
ni-1: last pitch notation
Input values
C
D
pitch notation

31. Bayesian Network (user’s vertical motion)
Input
ay: acceleration (y-axis)
v: velocity
vc: velocity change
p: moving distance
g: gravity
Output
h: the vertical motion of
user’s hand
Raise motion
or
Drop motion
or
No change

32. Multi users ensemble by multi users using
inter-device communication
Planning to implementation ensemble system by multi users
• The system support ensemble by inter-device communication
- Send and receive information of musical performance (chord progression)
• Planning to implementation communication method by Bluetooth
- The system must take into account delay time by sending data
Send and receive information
of musical performance
(chord progression)
Ensemble support system
(user 1)
Input values of
smartphone sensors
Ensemble support system
(user 2)
Input values of
smartphone sensors

33. Recording supervised data by Kinect
• We made an experiment recording supervised data by Kinect
- recorded the height value of user’s hand recognized by Kinect as supervised
data when users did a predefined motion
• As a result of experiment
- Observed unexpected difference of smartphone’s angle according to the height
of user’s hand
- Smartphone’s angle is important for the more accurate estimation of the
vertical motion
high
middle
low
Predefined motion:
high→high→middle→low