1. EXPLOITING CONTEXTUAL INFORMATION
FOR IMPROVED PHONEME RECOGNITION
Joel Pinto, B. Yegnanarayana, H. Hermansky, Mathew Magimai.-Doss
presented by
Sebastian T. Hafner

2. OVERVIEW
• Introduction
• Basic Phoneme Recognizer
• Contextual Information
• at the feature level
• at the posterior level
2

3. BASICS
3

4. TIMIT DATABASE
• read speech
• american english
• 630 speakers
•8 main dialects
4

5. TIMIT DATABASE
5

6. TIMIT DATABASE
• training set:
• 3000 utterances
• 375 speakers
5

7. TIMIT DATABASE
• training set:
• 3000 utterances
• 375 speakers
• test set:
• 1344 utterances
• 168 speakers
5

8. PHONEME RECOGNITION
6

9. PHONEME RECOGNITION
25ms
6

10. PHONEME RECOGNITION
25ms
step size: 10ms
6

11. FEATURE EXTRACTION
7

12. FEATURE EXTRACTION
• 13 PLP coefficients
7

13. FEATURE EXTRACTION
• 13 PLP coefficients
• delta values
7

14. FEATURE EXTRACTION
• 13 PLP coefficients
• delta values
• delta-delta values
7

15. FEATURE EXTRACTION
• 13 PLP coefficients
• delta values
• delta-delta values
39 features per frame
7

16. HIDDEN MARKOV MODEL
MLP % Phoneme p
8

17. HIDDEN MARKOV MODEL
MLP % Phoneme p
MLP estimates posterior probability
8

18. MULTI LAYERED PERCEPTRON
...
... ...
1000
39 39
hidden
features phonemes
units
.... ....
....
9

19. PRIOR PROBABILITY
P (qt = i | xt )
qt Phoneme index at t
xt feature vector at t
10

20. BAYES THEOREM
p (xt | qt = i) P (qt = i | xt )
p (xt ) P (qt = i)
11

21. POSTERIOR PROBABILITY
p (xt | qt = i) P (qt = i | xt )
p (xt ) P (qt = i)
P (qt = i) = P (qt = j) ∀i, j ∈ {1, 2, . . . , 39}
equal probability for each phoneme
11

22. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
12

23. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
classified: /c/ /a/ /t/
12

24. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
classified: /c/ /a/ /t/
errors:
12

25. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
classified: /c/ /a/ /t/
errors:
12

26. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
classified: /c/ /a/ /t/
errors:
12

27. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
classified: /c/ /a/ /t/
errors:
12

28. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
classified: /c/ /a/ /t/
errors:
12

29. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
classified: /c/ /a/ /t/
errors:
error rate:
12

30. PHONEME ERROR RATE
spoken: /k/ /a/ /t/ /e/
classified: /c/ /a/ /t/
errors:
error rate: 2 : 4 = 50 %
12

31. STANDARD APPROACH
68.12 %
13

32. AT THE FEATURE LEVEL
14

33. FEATURE LEVEL
15

34. FEATURE LEVEL
phoneme
15

35. FEATURE LEVEL
phoneme phoneme
influenced influenced
by earlier by next
phoneme phoneme
phoneme
15

36. FEATURE LEVEL
phoneme phoneme
influenced influenced
by earlier by next
phoneme phoneme
phoneme
15

37. 3 SINGLE MLPS
MLP %
MLP %
MLP %
16

38. 3 SINGLE MLPS
MLP %
MLP %
MLP %
each MLP with 39 classes
16

39. 3 SINGLE MLPS
MLP P (qt = i | xt , st = 1)
MLP P (qt = i | xt , st = 2)
MLP P (qt = i | xt , st = 3)
st ∈ {1, 2, 3}
17

40. 3 SINGLE MLPS
MLP P (qt = i | xt , st = 1)
MLP P (qt = i | xt , st = 2)
MLP P (qt = i | xt , st = 3)
state index st ∈ {1, 2, 3}
17

41. 1 LARGE MLP
MLP
%
18

42. 1 LARGE MLP
MLP
%
MLP with 117 classes
18

43. 1 LARGE MLP
MLP P (qt = i, st = j | xt )
19

44. 1 LARGE MLP
MLP P (qt = i, st = j | xt )
39 phonemes x 3 states
19

45. LARGE MLP VS. 3 SMALLER
labels for training MLP
classifier
uniform force aligned
one MLP
with 117 classes 69.87 71.67
three MLPs
earch 39 classes 70.13 69.70
20

46. AT THE POSTERIOR LEVEL
21

47. ESTIMATE
POSTERIOR PROBABILITY
%
state
posterior MLP
probabilities
22

48. ESTIMATE
POSTERIOR PROBABILITY
%
state
posterior MLP
probabilities
22

49. ESTIMATE
POSTERIOR PROBABILITY
state
posterior MLP P (qt = i | Qt )
probabilities
Qt
23

50. ESTIMATE
POSTERIOR PROBABILITY
state
posterior MLP P (qt = i | Qt )
probabilities
Qt trajectory of state posterior probabilities
23

51. PARAMETERS
• 3000 hidden layers
• 23 frames for windowing
24

52. PARAMETERS
• 3000 hidden layers
• 23 frames for windowing
73.4 %
24

53. ANALYSIS
25

54. INFORMATION ACROSS
STATE POSTERIORS
26

55. INFORMATION ACROSS
STATE POSTERIORS
1 state
(classic) 68.12
1 state better modeling
(sum) 70.17 by states
3 state 71.67 better decoding
26

56. INFORMATION ACROSS
STATE POSTERIORS
1 state
(classic) 68.12
1 state better modeling
(sum) 70.17 by states
3 state 71.67 better decoding
26

57. INFORMATION ACROSS
STATE POSTERIORS
1 state
(classic) 68.12
1 state better modeling
(sum) 70.17 by states
3 state 71.67 better decoding
26

58. EXPERIMENT A
27

59. EXPERIMENT A
original data modified data
0.04 0.09
0.64 0.9
0.32 0.01
27

60. EXPERIMENT A
original data modified data
0.04 0.09
0.64 0.9
0.32 0.01
27

61. EXPERIMENT A
original data modified data
0.04 0.09
0.64 0.9
0.32 0.01
27

62. EXPERIMENT A
original data modified data
0.04 0.09
0.64 0.9
0.32 0.01
remaining values: randomly =1
modified data
27

63. EXPERIMENT B
28

64. EXPERIMENT B
original data modified data
0.04 0.25
0.64 0.64
0.32 0.11
28

65. EXPERIMENT B
original data modified data
0.04 0.25
0.64 0.64
0.32 0.11
28

66. EXPERIMENT B
original data modified data
0.04 0.25
0.64 0.64
0.32 0.11
28

67. EXPERIMENT B
original data modified data
0.04 0.25
0.64 0.64
0.32 0.11
remaining values: randomly =1
modified data
28

68. RECOGNITION ACCURACY
experiment 1 state MLP 3 state MLP
baseline 68.12 71.55
experiment A 62.77 70.27
experiment B 64.24 70.75
29

69. RECOGNITION ACCURACY
experiment 1 state MLP 3 state MLP
baseline 68.12 71.55
experiment A 62.77 70.27
experiment B 64.24 70.75
Information in phoneme posteriors !
29

70. SUMMARY
• contextual information in features
• contextual information in probabilities
30