Version of the PSST challenge presentation from my 30% seminar

1. Speech data augmentation for improving phoneme
transcriptions of aphasic speech using wav2vec 2.0
for the PSST Challenge
Birger Moëll, Jim O’Regan, Shivam Mehta, Ambika Kirkland, Harm Lameris,
Joakim Gustafson, Jonas Beskow

2. Experiments
2022-05-30 2
• Data augmentation:
– Augmenting the original
– Matching out-of-domain speech (TIMIT)
• (Phonetic) language models
• Voice conversion
• Synthetic voices

3. Data augmentations
2022-05-30 3
• Approach
• We propose data augmentations and training on non-
aphasiatic datasets to increase robustness and
accuracy of phoneme transcription models for aphasia.
• Steps:
• Data augmentations using pitch shift, gaussian
noise, time stretch, voice conversion and room
impulse response
• Joint training on non aphasiatic dataset (Common
Voice, Timit) and aphasiatic dataset (PSST)

4. Datasets
2022-05-30 4
• Augmenting with datasets with
manual transcriptions (TIMIT) was
successful
• Augmenting with automatically
phonetized dataset (common voice)
was unsuccessful.
• Acoustically aligning the TIMIT data to
PSST using Room Impulse
Response (RIR) improved
performance
Key findings
● PSST Dataset
○ Boston Naming Test - Short
Form (BNT-SF) and the Verb
Naming Test (VNT)
● TIMIT Dataset
○ Acoustic-Phonetic Speech
Corpus
● Common voice
○ Volunteer-based open-
source dataset for ML

5. Models
2022-05-30 5
We compared base and large
wav2vec2 models
• The bigger model performed better
• Training with the base model was still
useful for faster experimentations

6. Voice conversion
2022-05-30 6
VOICE CONVERSION WAS USED TO
AUGMENT DATA BY NEURAL VOICE
CLONING
EXPERIMENTS WITH VOICE CLONING
PROVED UNSUCCESSFUL, LIKELY
BECAUSE OF PSST DATA QUALITY
ISSUES

7. Pitch shift
2022-05-30 7
PITCH SHIFT WAS THE MOST
SUCCESSFUL PSST AUGMENTATION
IMPROVING RESULTS COMPARED TO
BASELINE
THE PITCH WAS BOTH LOWERED AND
RAISED RANDOMLY

8. Gaussian Noise
2022-05-30 8
GAUSSIAN NOISE WAS
USEFUL AS A
REGULARIZATION
TECHNIQUE
A PURELY ADDITIVE
TECHNIQUE FOR DATA
AUGMENTATION

9. Time stretch
2022-05-30 9
TIME STRETCH IS USED TO
REGULARISE THE DURATION
OF THE SPEECH
LIMITED IMPROVEMENT OVER
THE BASELINE

10. Room impulse response
2022-05-30 10
TO MIMIC THE REVERBERANT
CONDITIONS OF THE PSST
DATA
MOST SUCCESSFUL
AUGMENTATION OF THE TIMIT
DATA

11. Augmentations on augmentations
2022-05-30 11
We experimented adding
Pitch Shift + Time Stretch +
Gaussian Noise + Room
Impulse Response to
TIMIT.

12. Augmentations on augmentations
2022-05-30 12
We experimented adding
Pitch Shift + Time Stretch +
Gaussian Noise + Room
Impulse Response to
TIMIT.
Because that’s how we roll.

13. Language models
2022-05-30 13
WE EXPERIMENTED WITH
LANGUAGE MODELS
THEY LED TO LITTLE OR NO
IMPROVEMENTS OVER
BASELINE

14. Results
2022-05-30 14

15. Result
2022-05-30 15
• Data augmentations improve model performance.
• Increasing the size of the model decreases FER and PER.
• Manually-transcribed speech from non-aphasic speakers (TIMIT) improves
performance
– when Room Impulse Response is used to augment the data.
• The best performing model combines aphasic and non-aphasic data
– 21.0% PER
– 9.2% FER
– relative improvement of 9.8%
• Data augmentation, larger model size, and additional non-aphasic data
sources can be helpful

16. Thank you!
2022-05-30 16

17. Challenge Results
2022-05-30 17
We came second! 😊

18. Challenge Results
2022-05-30 18
We came second! 😊
Of two teams 😭

19. Challenge Results
2022-05-30 19
We came second! 😊
Of two teams 😭
The other team was Baidu 🙄

20. Challenge Results
2022-05-30 20

21. Challenge Results
2022-05-30 21
Gale, R. C., Fleegle, M., Fergadiotis, G., & Bedrick, S. (2022, June). The Post-
Stroke Speech Transcription (PSST) Challenge. Proceedings of The RaPID4
Workshop - Resources and ProcessIng of Linguistic, Para-Linguistic and Extra-
Linguistic Data from People with Various Forms of
Cognitive/Psychiatric/Developmental Impairments - within the 13th Language
Resources and Evaluation Conference, 41–55. Retrieved from
https://aclanthology.org/2022.rapid4-1.6