SII2024

1. Tactile Presentation of
Orchestral Conductor's Motion Trajectory
1 ) Ke i o U n i v e r s i t y ， 2 ) T h e U n i v e r s i t y o f S y d n e y
Yuto Ueda1)，Anusha Withana2), Yuta Sugiura1)
2024 IEEE/SICE International Symposium on System Integration

2. • Promotion of Diversity & Inclusion in musical activities
• Visually impaired individuals may experience difficulty perceiving conducting
movements in certain situations.
• Scenarios where participating in an ensemble can be challenging [1]
2
[1]Luca Turchet et al. Musical Haptic Wearables for Synchronisation of Visually-impaired Performers: a Co-design Approach. In ACM IMX '21. 20–27.
Background: Diversity of Participants in Musical Activities

3. 3
Research Presentation Methods
Asakawa et al.[2]
Kawarazaki et al.[3]
Vibration on the wrist in sync with the beat timing
Turchet et al.[1]
Indicating start and other commands with vibration
patterns
Galea et al.[4] Feedback using both tactile and auditory cues
Related Work: Sharing Conducting Using Tactile Feedback
Asakawa et al.[2]
[2]Takashi Asakawa et al. An electric music baton system using a haptic interface for visually disabled persons. SICE '12. IEEE, 602–607.
[3]Noriyuki Kawarazaki et al. A Supporting System of Choral Singing for Visually Impaired Persons Using Depth Image Sensor. JRM '14, 735–742.
[4]Claire Galea et al. Accessible Choral Ensembles for Visually Impaired Singers. HCI '18. Article 37, 10 pages.
Turchet et al.[1]
Challenges of previous work
• Difficult to present the trajectory of conducting movements
• Musicians predict the beat timing by observing the conductor's
trajectory
• Expression is conveyed through variations in the scale of the
conductor's movements

4. • Motion capture sensing of conducting movements
• Extraction of conducting motions to 2D coordinates
• Presentation of conducting trajectories on the user's back using 16 vibrators
arranged in a plane
4
Presentation of Conducting Trajectories Using Tactile Apparent Movement
Tactile Display
Vibration Motors

5. • The phenomenon of perceiving continuous
movement on the skin from time-delayed
stimuli at two points.[5]
5
[5]東山篤規, 宮岡徹, 谷口俊治, 佐藤愛子：触覚と痛み, ブレーン出版（2000）
[6]小谷賢太郎, et al. 触覚の仮現運動における知覚速度の定量的評価. ヒューマンインタフェース学会論文誌, 2010, 12.2: 167-175.
Principle: Tactile Apparent Movement
[6]
• Implementation of tactile apparent movement based on conducting movements
• Normalization of the x, y coordinate values
• Divide into 4×4=16 areas
• Activate the corresponding vibrators
time
Stimulus
Sensation

6. • Trajectory Presentation (proposed) vs Point Presentation
• Point presentation: Vibrating a single vibrator in time with the beat
• Response Time: The time discrepancy (smaller is better)
6
Experiments Using Response Time
Response Time
time
correct predicted
16 participants aged 19-26 (average 21.8, variance
1.87), including 9 with musical experience
Participants’ Information
Condition Variables
• Tempo changes
• Timing of the start of the performance

7. • Accel (gradually faster) and Rit (gradually slower)
• Three types of tempo transitions: Linear, Exponential [7], and Mean of linear and
exponential
7
Experiments: Tempo Changes (Accelerando, Ritardando)
Tempo changes BPM
accel 60→80, 80→100, 100→120
rit 60→40, 80→60, 100→80
[7]Masuzo Yanagida et al. Effects of the mode of tempo change on perception of tempo change. Proc. Meet. Acoust. 28 (2016).
BPM (Beats Per Minute) Settings

8. • For some BPMs in Linear and Exponential, trajectory presentation resulted in
significantly shorter response times.
8
Results: Accelerando
*: Significant Difference(p<0.05)
*
𝑝𝑝 = 0.031
*
𝑝𝑝 = 0.017

9. 9
Results: Ritardando
*
𝑝𝑝 = 6.06 × 10−4
*
𝑝𝑝 = 0.002
*
𝑝𝑝 = 1.50 × 10−6
*
𝑝𝑝 = 0.004
*
𝑝𝑝 = 1.95 × 10−4
*: Significant Difference(p<0.05)
• For all tempo transitions, there were several instances where trajectory
presentation resulted in significantly shorter response times.

10. • Response time for the initial beat following a
preparatory motion
• Preparatory Motion: Conductor's conducting gestures
to signal the commencement of the performance
• Preparatory Beats: 1 beat, 2 beats
10
Experiments & Results: Start Timing of Performance
• For 1-beat preparatory motions, the trajectory
presentation resulted in a significantly shorter
response time.
*
𝑝𝑝 = 9.77 × 10−4
*: Significant Difference(p<0.05)

11. Tempo Changes
11
Discussion: Trajectory presentation is effective.
It's possible to express the transition of movements leading up to the beat.
 Trajectory presentation may make it easier to predict the beat even with tempo changes.
At least two beats of vibration are necessary to accurately capture the tempo.
Conducting trajectory is continuously presented between beats.
 Even with one preparatory beat, trajectory presentation helps beat timing prediction.
Trajectory
Start Timing of the Performance
Trajectory
Point

12. Limitations
• The environment with the capability for motion capture is required.
Future Work
• Conduct workshops with performers of various instruments to create a device
that can be used universally across instruments.
12
Limitations and Future Work

13. 13
Summary
Background
Visually impaired individuals may find it difficult to perceive
conducting movements
Proposed
Method
Presentation of conducting trajectory using tactile apparent
movement
Implementation
Sensing conducting movements through motion capture.
Trajectory presentation using 16 vibrators.
Results
Trajectory presentation is effective for understanding tempo
changes and start timings.
Future Work Develop a universal device for all instruments