Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 23rd Abstract. Speaker classification is an essential task in the machine learning domain, with many practical applications in identification and natural language processing. This work concentrates on speaker classification as a subtask of general speaker diarization for real-world conversation scenarios. We research the domain of modern speech processing and present the original speaker classification approach based on the recent developments in convolutional neural networks. Our method uses a spectrogram as input to the CNN classifier model, allowing it to capture spatial information about voice frequencies distribution. Presented results show beyond human ability performance and give strong prospects for future development.

2. 2

3. General Speaker Diarization - a “Who spoke when?” problem:
- Crucial for speech understanding in a multi-speaker environment
- Many practical applications
- Challenging, the general solution does not exist
3

4. Problem: Speaker Classification
Research goals:
- Provide overview of the related work in the scientific field of the
speaker classification / diarization problem
- Explore existing solution with regard to the research hypothesis
- Propose and evaluate original solution to the speaker classification
problem
Industry related goals:
- First step towards product level diarization system
4

5. Sound is multidimensional unstructured data.
Therefore modern neural networks models should
perform comparatively well for the classification task.
5

6. ●
●
●
6

7. CNNs in sound and voice processing
●
●
●
7

8. ●
●
●
●
●
8

9. - LibriSpeech ASR corpus - derived from
audiobooks, 251 speakers, 100 hours of speech.
- ICSI Meeting Speech corpus - derived from real
meetings, 53 speakers, 72 hours of speech.
9

10. Stage 1 Front-end:
● MFCC 16
● MFCC 32
● Mel-Spectrogram 256
● Mel-Spectrogram 512
● Spectrogram 128
● Spectrogram 256
● Spectrogram 512
10
Stage 1 Back-end:
● CNN with 6 layers
Stage 1 Samples:
● 0.5 sec
● 1 sec
● 2 sec

11. Back-end example for 128x128 input
11

12. Mel Frequency Cepstral Coefficients (MFCC) process.
12

13. MFCC with 16 coefficients for 2 seconds audio.
13

14. Mel sprectrogram with window size = 512 and cepstral coefficients =
128 for 1 sec audio.
14

15. Spectrogram of 1 sec audio fragment, window length = 256.
15

16. 16

17. ● Spectrograms perform better then MFCC
● Longer fragments perform better
● Inputs with equal dimensions perform better
17

18. 18

19. 19

20. ● Accuracy of 0.94 gives strong promise for future
development
● Model is context independent
● Data preprocessing is more important than
classifier architecture
20

21. Extended Research
● LSTMs
● TDNN
● D-vector
Bridging the Diarization Gap
● Ensemble models
● End-2-End models
21

22. ● In-depth overview of Speaker diarization
scientific field
● Performance comparison of different sound
representation with CNN classifier
● Proposed original approach for speaker
classification task
22

23. ● Model was trained for each dataset
● Only activation sizes are depicted on Figures.
● The model can not be directly compared to other
models because they use different proprietary or
costly datasets
● Split ratio for test and train data was 0.2 and 0.8
23