Tetsuji Ogawa, Kenshiro Ueda, Kouichi Katsurada, Tetsunori Kobayashi, Tsuneo Nitta, ``Bilinear map of filter-bank outputs for DNN-based speech recognition,’’ Proc. INTERSPEECH2015, pp.16-20, Sept. 2015.

1. Bilinear map of filter‐bank outputs
for DNN‐based speech recognition
Tetsuji Ogawa1, Kenshiro Ueda1, Kouichi Katsurada2,
Tetsunori Kobayashi1, Tsuneo Nitta1,2
1Waseda University
2Toyohashi University of Technology
1

2. Abstract
Aim
• Accurate representation of DNN inputs
Approach
• Quadratic expansion (QE) of acoustic features
Effectiveness
2
FBANK FBANK‐QE FBANK + FBANK‐QE
18.8 17.3 15.8
Phoneme Error Rate (%) on TIMIT (dev)

3. Quadratic Expansion of Features
3
x1x1 x1x2 x1x3 x1x4
x2x1 x2x2 x2x3 x2x4
x3x1 x3x2 x3x3 x3x4
x4x1 x4x2 x4x3 x4x4
x1
x2
x3
x4
x1x1
x2x1
x3x1
x3x3
x4x3
x4x4
D
x 
   xxxg  tri
vec
tri
vec
vectorization
  2/)1( 
 DD
xgx  x

quadratic
expansion

4. Quadratic expansion makes it possible to extract
a more precise structure of TF pattern.
4
xx 
QE of FBANK
x
FBANK
xt xt+1xt-1

5. /b/
/d/
/g/
/b/
/d/
/g/
(a) FBANK (b) QE of FBANK
xt xt+1xt-1
xt xt+1xt-1
xt xt+1xt-1

6. 6
QE can improve class separability.

7. Tensor Feature Extraction
7
Context
window
(+5/‐5)
LDA QE
(2)
ty
FBANK
feature
(3)
tz
Compressed
FBANK feature
(4)
tv
Tensor
feature
 tt zz tri
vec
bases:
tensor space
bases:
 ia  ji aa 
ti ya 
(1)
FBANK
sequence
tx
23 253 40 820

8. DNN System Architecture
8
0
5
10
‐5
‐10
context
x21
(‐10/+10)
BN Feature Extractor Classifier
# of hidden units: 1024
# of hidden layers: 5
# of BN units: 40
# of outputs: 1951 (states)
# of hidden units: 1024
# of hidden layers: 5
# of outputs: 1951 (states)
[M. Karafiat et al., ``BUT ASR system for Babel surprise evaluation 2014,’’ Proc. SLT2014, pp.501‐506, 2014.]

9. 9
FBANK
x11
FBx11
0
5
10
‐5
‐10
context
x21
(‐10/+10)
BN Feature Extractor Classifier
FBANK
xN
QELDA
FBxN‐QE
0
5
10
‐5
‐10
context
x21
(‐10/+10)
BN Feature Extractor Classifier

10. Phoneme Recognition on TIMIT
• Speech materials:
• Training: 3696 utterances
• Development: 400 utterances
• Test: 192 utterances
• Language model: Phoneme bigram
10
D
FBANK
xN
QELDA 0
5
10
‐5
‐10
context
x21
(‐10/+10)
BN Feature Extractor Classifier
• N: context window size
• D: Dimensionality reduced by LDA

11. 1. Effect of Dimension Reduction for
Quadratic Expansion
11
D
FBANK
x5
QELDA
0
5
10
‐5
‐10
context
x21
(‐10/+10)
BN Feature Extractor Classifier
FBANK
x5
QE

12. 12
TIMIT dev

13. 13
Compression of FBANK can help in
FBANK‐QE‐based DNN system.
TIMIT dev

14. 14
TIMIT dev

15. 2. Effectiveness of Quadratic Expansion
15
40
FBANK
xN
QELDA
0
5
10
‐5
‐10
context
x21
(‐10/+10)
BN Feature Extractor Classifier
FBANK
x5

16. 16
TIMIT dev

17. 17
TIMIT dev

18. 18
TIMIT dev

19. 19
TIMIT test

20. 20
3. Effect of System Combination
40
FBANK
x5
QELDA
0
5
10
‐5
‐10
context
x21
(‐10/+10)
BN Feature Extractor Classifier
FBANK
x5
0
5
10
‐5
‐10
context
x21
(‐10/+10)
BN Feature Extractor Classifier
+ lattice‐level combination
using MBR decoding

21. 21
Individual System
System Combination
TIMIT test
[FBANK] [MFCC][FBANK] [MFCC]
+QE
+QE
[FBANK‐QE]
[MFCC‐QE]

22. 22
Individual System
System Combination[FBANK] [MFCC]
TIMIT test
[FBANK] + [MFCC]
[FBANK] [MFCC]
+QE
+QE
[FBANK‐QE]
[MFCC‐QE]

23. 23
[FBANK] + [FBANK‐QE]
[FBANK‐QE] + [MFCC‐QE]
[MFCC] + [MFCC‐QE]
Individual System
System Combination
[FBANK] + [MFCC]
[FBANK] [MFCC]
+QE
TIMIT test
+QE
+QE
[FBANK‐QE]
[MFCC‐QE]

24. 24
Use of Speaker Adapted Features
Individual System
System Combination
TIMIT test
[MFCC+fMLLR]
[MFCC+fMLLR‐QE]
[MFCC+fMLLR] + [MFCC+fMLLR‐QE]
+QE

25. 25
[MFCC+fMLLR]
[MFCC+fMLLR‐QE]
[MFCC+fMLLR] + [MFCC+fMLLR‐QE]
[MFCC+fMLLR‐QE] + [FBANK‐QE] + [MFCC‐QE]
[FBANK‐QE]
[MFCC‐QE]
+QE
Individual System
System Combination
Use of Speaker Adapted Features
TIMIT test

26. Conclusion
26[FBANK‐QE] + [MFCC‐QE]
[FBANK] [MFCC]
[MFCC‐QE]
+QE
+QE
• QE of acoustic features is taken as input of DNN.
• QE yields improvement over original acoustic features.
+QE
[FBANK‐QE]
[FBANK] + [MFCC]