The document describes a "Pinch & Play" vocoder project that allows users to modify the sound of their voice by pinching different objects. The system would analyze how pinching an object modifies a user's voice and generate a "voice modulation method" to mimic that effect. The goals are to allow fine-tuning of accent, texture, and tone, and to potentially visualize the sound modifications. Milestones include developing a mimicking voice by October, adding generative modeling by November, and fine-tuning the system and adding visualization by December. Challenges include fine-tuning the modifications and enabling synesthesia through visualization.

1. Pinch & Play
– Vocoder based on Resonance of Object
Special Topics in CDE 6 : Computational interaction design
CDE 20208002
Byounghern, Kim.
2020. 10. 06.
“콕 집어서 말하기”

2. Motivation
Electronic Kazoo-like Vocoder

3. Experience

4. Experience
Pinch,
Play,
Feel,
Collect what you love
With any object

5. Modeling : System
Device for Pinch Laptop
Play

6. Modeling : Data Collection
Device for Pinch
Laptop
Analyze and Generate
‘Voice Modulation Method’
Transmitter
Receiver
voice gesture (accent, texture, tone)

7. Modeling : Data Collection (Ideal Result)
Laptop
Analyze and Generate
‘Voice Modulation Method’
RNN
Invertible System

8. Modeling : Data Collection (First step)
Laptop
Analyze and Generate
‘Voice Modulation Method’
Mimicking voice
(sampled with a mouth and object)

9. Modeling :
Transformation
Rendering
(Under 100ms-delay)
Microphone Input
(Continuous Audio Data)
Voice gesture (accent, texture, tone)
Audio Output
(with noise elimination)

10. Challenges
Fine tuning
(Accent, texture, tone)
Synesthesia
(Visualize)

11. Milestones
Dec
Fine tuning
Oct
Mimicking
Voice
Generative
model
validation
Nov
Visualization
Implementation

12. Milestones
Oct
Mimicking
Voice
Generative
model
validation
Voice
Conversion

13. Thanks
Special Topics in CDE 6 : Computational interaction design
CDE 20208002
Byounghern, Kim.
2020. 10. 06.
Voice Controlled Interface 4 Digital Musical Instruments (Fasciani, 2014)
WaveNet (Google, 2016)
ADAPTIVE WAVENET VOCODER FOR RESIDUAL COMPENSATION IN GAN-BASED VOICE
CONVERSION (Sisman, 2018)