This document summarizes a research paper on efficient Transformer-based speech enhancement using long frames and STFT magnitudes. It introduces the task of speech enhancement and issues with learned-domain approaches. The proposed method uses STFT magnitudes as input to an encoder-decoder architecture with a Transformer masker. Experiments show the method achieves equivalent quality and intelligibility scores compared to learned features, while reducing computations by 8x for 10-second utterances. The conclusions are that using STFT magnitudes enables efficient, high-quality speech enhancement on embedded devices.

1. Efficient Transformer-based
Speech Enhancement
Using Long Frames and STFT
Magnitudes
Presenter : 何冠勳 61047017s
Date : 2022/09/29
Danilo de Oliveira et al.,
“Efficient Transformer-based Speech Enhancement Using Long Frames and STFT Magnitudes,”
in INTERSPEECH 2022

2. Outline
2
1 3 5
Introduction Experiments Conclusion
4
2
Architecture Results &
Discussion

3. Introduction
What’s the task?
1
3

4. Introduction
◉ A neural network-based approach that has shown good results and flourished with the increased
capacity and computational power of modern processors is learned-domain masking-based
speech processing.
(learn latent representations of the audio inputs and perform processing steps on them)
◉ In <GTF>, equivalent results were obtained by replacing the learned encoder with a multi-phase
gammatone analysis filterbank. In <Demy>, the authors show that gains from ConvTasNet can be
attributed to the high time resolution and the time-domain loss.
◉ Learned-domain methods usually work on short frames (2ms), which implies having to deal with
a larger number of frames if compared to traditional STFT frame sizes (∼32ms).
4

5. Introduction
◉ Dual-path methods have managed to alleviate some issues related to the modeling of long
sequences for speech applications.
◉ Another drawback of learned-domain approaches is that the models usually work with 8kHz
audio data, which is a considerable disadvantage against wideband methods.
◉ Additionally, the learned-encoder features have reduced interpretability compared to well-
established, fixed filters such as the STFT.
◉ Therefore, using larger frame time-frequency representations still presents itself as a desirable
feature, though working with complex representations brings additional challenges.
5

6. Introduction
◉ From multiple studies, one phenomenon has shown:
➢ the loss of spectral resolution renders the magnitude less relevant
at very short frames (≤ 2ms).
➢ the magnitude is more important than phase
for larger frames (~ 32ms).
◉ In this paper, we investigate what compromises and benefits can be attained when working with
magnitudes of longer frames.
6

7. Architecture
- Masking-based approach
- Encoder/Decoder
- Masker
2
7

8. Masking-based Approach
8

9. Encoder/Decoder
◉ Learned en/decoder pair:
9
◉ STFT/iSTFT:
where f , l, and n are the indices of frequency bin,
frame, and local time, respectively, W is the window
length, and H is the hop size.

10. Masker
10

11. Masker
11

12. Experiments
- Dataset
- Model
3
12

13. Experiments
◉ Dataset:
○ The models were trained on the DNS-Challenge dataset. We generated 100 hours of 4 second
long noisy mixtures sampled at 16kHz, with 20% reserved for validation.
○ Testing was performed on clean samples from a subset of the WSJ0 corpus mixed with noise
from the CHiME3 Challenge dataset, at SNRs ranging from -10dB to 15dB, at 5dB intervals.
◉ Model:
○ Learned En/Decoder basis 256 ; Kernel size 32 (2 ms for 16 kHz); Stride 16 ;
○ STFT window function Hann ; Learned En/Decoder basis 512 (32 ms for 16 kHz);
Overlap 50 / 75% ;
○ Chunk size 250 ; Transformers 4 ; SepFormer blocks 2 ; FFW-Dimension 256 ;
○ Optimizer Adam ; Loss SI-SNR ; Learning rate 1e-3 ;
13

14. Results &
Discussion
- Enhancement
- Executing profiling
3
14

15. Enhancement
◉ The estimated utterances were evaluated on instrumental perceptual metrics:
- POLQA for speech quality
- ESTOI for intelligibility.
15

16. Enhancement
◉ In the learned-domain case, the chunk size of 250 as in [16] performs best against a setup with
shorter chunks, hinting at the importance of modeling short-term relations in the sequence.
◉ The configuration with chunks size 50 seems to find a balance between short- and long-term, if
compared to the models with 25 and 100.
◉ The learned-domain method estimates the signal containing a buzzing sound that is absent from
the magnitude STFT outputs.
<Audio Samples>
16

17. Executing profiling
17

18. Conclusions
5
18

19. Conclusions
◉ We obtained equivalent quality and intelligibility evaluation scores while reducing the number of
operations by a factor of approximately 8 for a 10-second utterance.
◉ Motivated by previous contributions on learned and traditional filterbanks and on the relation
between frame size and magnitude/phase processing, we show that by replacing the learned
features with STFT magnitudes, we can obtain equivalent performance in terms of perceptually-
motivated metrics while considerably reducing resource allocation and processing time.
◉ These findings are a big step towards making the implementation of state-of-the-art
transformer-based speech enhancement systems possible in real-life applications, especially on
embedded devices.
19

20. Bi-Sep:
A Multi-Resolution Cross-Domain
Monaural Speech Separation Framework
Presenter : 何冠勳 61047017s
Date : 2022/09/29
Kuan-Hsun Ho et al.,
“Bi-Sep: A Multi-Resolution Cross-Domain Monaural Speech Separation Framework,”
in TAAI 2022

21. Brief
21

22. Brief
22

23. Brief
23

24. Any questions ?
You can find me at
◉ jasonho610@gmail.com
◉ NTNU-SMIL
Thanks!
24