The individual and combined impacts of various front-end approaches on the performance of deep neural network (DNN) based speech recognition systems are examined in distant talking situations. The contents were published in: Takuya Yoshioka and Mark J. F. Gales, "Environmentally robust ASR front-end for deep neural network acoustic models," Computer Speech and Language, vol. 31, no. 1, pp. 65-86, May 2015.

2. • Do not compare results across different tables!
– Configurations may differ
• Most results shown here can be found in:
Takuya Yoshioka and Mark J. F. Gales, “Environmentally
robust ASR front-end for deep neural network acoustic
models,” Computer Speech and Language, vol. 31, no. 1, pp.
65-86, May 2015

3. 1. Motivation
2. Corpus
• AMI meeting corpus
3. Baseline systems
• SI and SAT set-ups
4. Assessment of environmental robustness of
DNN acoustic models
5. Front-end techniques
6. Combined effects

6. Little investigation done

7. • Multi-party interaction
– 4 participants in each meeting
• Multi-channel recordings
– Distant microphones – only first channel used
– Head-set & lapel microphones
• 2 recording set-ups
– 70h scenario-based meetings
– 30h real meetings

8. • Different rooms
• Multiple sources of distortion
– Reverberation
– Additive noise
– Overlapping speech
• Moving speakers
• Many non-natives

9. • SI : speaker independent
– For online transcription
– DNN-HMM hybrid
• SAT: speaker adaptive training
– For offline transcription
– MLP tandem

11. • Manual segmentations used
• Overlapping segments ignored

13. State output distributions modelled with
– GMM or
– DNN
¦ –
/
Q
T
t
tttt qpqqPqPp
q
xX
1
10 )|()|()()|(
¦
M
m
jmjm
mjmt Ncjp
1
)()(
),;()|( Σμxx
)(
)|(
)|(
jp
jp
jp t
t
x
x

14. Æ
• Discriminative pre-training
• Cross entropy fine-tuning
• Discriminative pre-training

15. • Trained on Telta K20
• cuBLAS 5.5 used
• Mini-batch size: 800 frames
• Learning rate: “newbob” scheduling
• 10% held-out data for CV

16. System
Parame-
terisation
%WER
Dev Eval Avg
MPE GMM-HMM HLDA 54.7 55.6 55.2
DNN-HMM hybrid FBANK 43.5 42.6 43.1
This work 40.0 39.3 39.7

18. Data Set
Parame-
terisation
%WER
Dev Eval Avg
SDM FBANK 43.5 42.6 43.1
IHM FBANK 28.2 24.6 26.4
• 39.2% of the errors caused by acoustic distortion
• DNN-HMMs not so robust

20. Æ
• Discriminative pre-training
• Cross entropy fine-tuning
• Discriminative pre-training

22. Align-
ment
DNN
input
%WER
Dev Eval Avg
SDM IHM 30.6 27.0 28.8
IHM SDM 41.8 40.8 41.3
IHM SDM 41.7 40.6 41.2
Using 648-2,000 5-4,000 DNN:
DNN training more sensitive to noise than state
alignment

24. Speech enhancement
Feature transformation
Multi-stream features

25. Speech enhancement
Feature transformation
Multi-stream features

26. Previous work
– Beamforming yields gains
– No investigation on single-microphone algorithms

27. • Based on linear time (almost) invariant filters
• Applied to complex-valued STFT coefficients
• The filters automatically adjusted using observations
– WPE for 1ch dereverberation (NTT’s work)
– BeamformIt for denoising (ICSI’s work)
• 8 microphones used, dedicated to meetings
• Unlikely to produce irregular transitions
¦
1
0
,,,,
T
Tk
ktfkftftf xgxy

28. Align-
ment
Dev Eval
SDM +Derev
BFIt
(8mics)
SDM +Derev
BFIt
(8mics)
MPE 43.8 41.8 38.6 43.0 41.3 36.6
Hybrid 43.5 41.7 38.8 43.3 41.4 36.7
• Dereveberation helps even with single microphone
• Multi-microphone beamforming works well

29. DNN size
Context
frames
Dev Eval
SDM +Derev SDM +Derev
1,000 5 9 43.8 41.8 43.0 41.3
1,500 5
9 43.5 42.0 42.6 41.1
13 42.8 41.8 42.9 41.2
19 43.0 41.7 42.9 41.2
2,000 5 9 43.8 41.3 42.9 40.4
4.7% gain from 1ch dereverberation (relative)

30. Speech enhancement
Feature transformation
Multi-stream features

31. No positive results reported previously

32. • Applied to magnitude spectra
• Cross terms (often) ignored
• Frame-by-frame modification
– Harmful for DNN?
• Noise estimated using long-term statistics
– IMCRA (used here), minimum statistics, etc
• Deltas from un-enhanced speech
– Essential for obtaining gains
2
,
2
,
2
, tftftf nxy

34. • Applied to FBANK features
• The following mismatch function used
• Frame-by-frame modification
• Noise model estimated with EM
• Deltas from un-enhanced speech
))exp(1log( hynhxy tttt

35. Enhancement target %WER
Spectrum Feature Dev Eval Avg
N N 42.0 41.1 41.6
Y N 41.3 40.9 41.1
N Y 41.4 40.5 41.0
Y Y 42.0 41.0 41.5
• Small consistent gains
• Different methods should not be connected

36. Enhancement target %WER
Spectrum Feature Dev Eval Avg
N N 42.0 41.1 41.6
Y N 41.3 40.9 41.1
N Y 41.4 40.5 41.0
Y Y 42.0 41.0 41.5
Y Y 41.4 40.4 40.9
Using multi-stream approach:

37. Speech enhancement
Feature transformation
Multi-stream features

38. • Frame level
– FMPE, RDT, FE-CMLLR
– Seems to be subsumed by DNN
• Speaker (or environment) level
– Global CMLLR, LIN, fDLR, VTLN
– Multiple decoding passes required Æ SAT
• Utterance level
– Single-pass decoding Æ SI

39. • Seems robust against supervision errors
• STC transform used to deal with correlations:
»
»
»
¼
º
«
«
«
¬
ª
tx
)()()()( ss
t
ss
t bLxAy

42. Form of speaker
transform
%WER
Dev Eval Avg
None (SI) 42.6 40.2 41.4
Full 37.4 37.4 37.4
Block diagonal 37.3 36.6 37.0
• ~10% relative gains obtained
• “Block diagonal” outperforms “full”

43. Form of speaker
transform
%WER
Dev Eval Avg
None (SI) 42.6 40.2 41.4
Full 37.4 37.4 37.4
Block diagonal 37.3 36.6 37.0
None (SI) 27.8 24.2 26.0
Full 23.8 21.6 22.7
On IHM data set

44. ))(()())(()( ucu
t
ucu
t bLxAy
uuc :)(
Clustering performed using:
– utterance-specific iVectors
– Kmeans (GMM yielded similar performance figures)

45. )()0()( uu
Twmm
m(0)
T
w(u)
Subspace representation of the deviation from
UBM
m(0)
m(1)
m(2)
m(3)
Variability
subspace

49. #Clusters
%WER
Dev Eval Avg
No QCMLLR 41.9 40.9 41.4
64 41.0 40.4 40.7
32 41.0 40.0 40.5
16 41.5 40.5 41.0
No QCMLLR 27.8 24.2 26.0
32 26.9 23.5 25.2
On IHM data set

50. • Using 32 clusters yielded best performance
• Similar gains on both SDM and IHM

51. Speech enhancement
Feature transformation
Multi-stream features

52. • Originally proposed by Aachen for shallow MLP
tandem configurations
• Exploits DNN’s insensitivity to the increase in input
dimensionality
• (Hopefully) complement features masked by noise
• Allows multiple enhancement results to be
combined

53. • Four types of auxiliary features investigated:
– MFCC (Δ/Δ2)
– PLP
– Gammatone cepstra
• Different frequency warping
• STFT not used
– Intra-frame delta ( )
• Emphasises spectral peaks/dips

54. Feature set #features
%WER
Dev Eval Avg
FBANK+Δ+Δ2 (baseline) 72 41.9 40.9 41.4
+PLP 85 40.7 40.3 40.5
+Gammatone 88 40.8 40.0 40.4
+MFCC 85 41.1 39.7 40.4
+MFCC+Δ+Δ2 111 40.6 40.2 40.4
+ + 2 120 40.9 39.8 40.4
+MFCC+ + 2 133 40.4 39.8 40.1

55. • Speech enhancement
– Linear filtering
– Spectral/feature enhancement
• Feature transformation
– Quantised CMLLR
– (Global CMLLR for SAT)
• Multi-stream features

56. ' '
Baseline

57. Front-end
%WER
Dev Eval Avg
FBANK baseline 43.1 42.4 42.8
+WPE 41.8 40.7 41.3
+MFCC+ + 2 40.5 40.1 40.3
+IMCRA+FE-VTS 40.0 39.3 39.7
+QCMLLR 40.9 39.5 40.2
• Effects additive except for QCMLLR
• QCMLLR may work if applied to the entire feature set

58. ÆÆ

59. System
Parame-
terisation
%WER
Dev Eval Avg
SAT GMM-HMM
MPE trained
HLDA 48.8 50.2 49.5
SAT tandem
MPE trained
FBANK 40.7 40.9 40.8
SI hybrid FBANK 43.5 42.6 43.1
• Outperforms SAT GMM-HMM
• Outperforms SI hybrid

60. ' '
Baseline

61. Front-end
%WER
Dev Eval Avg
FBANK baseline 40.1 41.3 40.7
+WPE 38.9 39.3 39.1
+MFCC 38.5 38.5 38.5
+IMCRA+FE-VTS 38.4 38.7 38.6
+CMLLR 36.6 36.7 36.7
+CMLLR 36.9 37.0 37.0
+CMLLR 38.4 38.6 38.5
• Effects of WPE and CMLLR are additive
• Using auxiliary features yields small gains over CMLLR
features
• Denoising subsumed by CMLLR (as expected)

62. • Front-end processing approaches yield gains
over state-of-the-art DNN-based AMs
– Linear filtering (WPE, BeamformIt)
– Spectral/feature enhancement (IMCRA, FE-VTS)
– Feature transformation (QCMLLR, CMLLR)
– Multi-stream features
• Possible to combine different classes of
approaches