DOCTORAL DISSERTATIONSIMULTANEOUS MODELING OF PHONETIC   AND PROSODIC PARAMETERS, AND  CHARACTERISTIC CONVERSION FORHMM-BA...
AcknowledgementFirstly, I would like to express my sincere gratitude to Professor Tadashi Kitamuraand Associate Professor ...
ContentsAcknowledgement                                                                           iContents               ...
3.2.2   Multi-space distribution HMM       . . . . . . . . . . . . . . . . . 23       3.2.3   Reestimation algorithm for M...
6.5.3   Speaking rate . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 Improvement of Synthesized Speech Quality ...
MDL principle.       92                 v
List of Tables 2.1   Examples of α for approximating auditory frequency scales. . . . . . . 10 2.2   Coefficients of R4 (ω)....
List of Figures 1.1 The scheme of the HMM-based TTS system. . . . . . . . . . . . . . .           6 2.1 Source-filter model...
5.4   Decision trees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455.5   Examples of decision trees. . ...
8.3 Comparison between method (a), (b) and (c) with regard to interpo-    lation between two multi-dimensional Gaussian di...
x
AbstractA text-to-speech(TTS) system is one of the human-machine interfaces using speech.In recent years, TTS system is de...
such as speaker individualities and emotions, the TTS system based on speakerinterpolation is presented.                  ...
Abstract in Japanese       (TTS)                             HMMHMM               (1)                   (2)HMM            ...
Chapter 1Introduction1.1     Text-to-speech(TTS) systemSpeech is the natural form of human communication, and it can enhan...
3. Run-time selection of multiple instances of speech segments [4], [6].  4. Speech synthesis from HMMs themselves [7]–[10...
Speech signal   SPEECH  DATABASE                               Excitation         Spectral                               p...
quality of synthesized speech. Chapter 8 presents how to synthesize speech withvarious voice characteristics, appling spea...
Chapter 2Mel-cepstral Analysis andSynthesis TechniqueThe proposed TTS system is based on source-filter model. In order to c...
Fundamental frequency                                                    Vocal Tract                                      ...
Table 2.1: Examples of α for approximating auditory frequency scales.                    Sampling frequency           Mel ...
Table 2.2: Coefficients of R4 (ω).                                 l          A4,l                                 1    4.99...
the exponential transfer function is approximated in a cascade form                          D(z) = exp F (z)             ...
α                            α                    α   1 − α2Input                Z −1                  Z −1               ...
Chapter 3Speech Parameter ModelingBased on HMMThe most successful and widely used acoustic models in recent years have bee...
a11                  a22               a33                  a44              a55                                   a12    ...
so that the pdf is properly normalized, i.e.,                                 ∞                                     bj (o)...
2. Induction                                       N                                                                      ...
N        Si                 3         State                 2                 1                     1        2          3 ...
Using ξt (i, j), the probability of being in state i at time t, given the entire observationand the model, is represented ...
where                                 π = [π1 , π2 , · · · , πN ]                                (3.23)                   ...
voiced region (continuous value)                 200Frequency (Hz)                 150                 100                ...
3.2.1     Multi-Space Probability DistributionWe consider a sample space Ω shown in Fig. 3.4, which consists of G spaces: ...
Sample Space Ω    Space    Index          Space                                  pdf of x          Observation            ...
1                                2                            3                             w11                           ...
The forward and backward variables are also used for calculating the reestimationformulas derived in the the next section3...
Maximization of Q-functionFor given observation sequence O and model λ , we derive parameters of λ whichmaximize Q(λ , λ)....
γt (i, g)(V (ot ) − µig )(V (ot ) − µig )T                           t∈T (O,g)                 Σig =                      ...
“unvoiced” symbol occurs from the zero-dimensional space defined in Section 3.2.1,that is, by setting ng = 1 (g = 1, 2, . ....
Chapter 4Speech parameter generation fromHMMThe performance of speech recognition based on HMMs has been improved by incor...
vectors ∆ct , ∆2 ct (e.g., delta and delta-delta cepstral coefficients, respectively), thatis, ot = [ct , ∆ct , ∆2 ct ] , wh...
It is obvious that P (O|Q, λ) is maximized when O = M without the conditions(4.4), (4.5), that is, the speech parameter ve...
Q. We have developed a fast algorithm for searching for the optimal or sub-optimalstate sequence keeping C optimal in the ...
and the constant K is independent of O . The occupancy probability γt (q, i) defined by                            γt (q, i...
Frequency (kHz)                   0                   2                   4                   6                   8       ...
7.2  c(0)            4.9           0.38 ∆c(0)          -0.31∆2 c(0)          0.070      -0.068                            ...
4.2.2    Parameter generation using multi-mixuture HMMIn the algorithm of the case 3, we assumed that the state sequence (...
0        2(kHz)        4        6        8                                                             (a) 1-mixture      ...
1-mixture                                 40.88-mixtures                                          59.2             0      ...
Chapter 5Construction of HMM-basedText-to-Speech SystemPhonetic parameter and prosodic parameter are modeled simultaneousl...
represents a continous value ( voiced ).                                            represents a discrete symbol ( unvoice...
c   Stream 1                    ∆c          Continuous probability distribution(Spectrum part)                        2   ...
5.3        Duration modeling5.3.1        OverviewThere have been proposed techniques for training HMMs and their state dur...
State duration densities are estimated on the trellis which is obtained in the em-bedded training stage. The mean ξ(i) and...
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Thesis yossie
Upcoming SlideShare
Loading in …5
×

Thesis yossie

679 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
679
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
29
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Thesis yossie

  1. 1. DOCTORAL DISSERTATIONSIMULTANEOUS MODELING OF PHONETIC AND PROSODIC PARAMETERS, AND CHARACTERISTIC CONVERSION FORHMM-BASED TEXT-TO-SPEECH SYSTEMS DOCTOR OF ENGINEERING JANUARY 2002 TAKAYOSHI YOSHIMURA Supervisors : Professor Tadashi Kitamura Associate Professor Keiichi Tokuda Department of Electrical and Computer Engineering Nagoya Institute of Technology
  2. 2. AcknowledgementFirstly, I would like to express my sincere gratitude to Professor Tadashi Kitamuraand Associate Professor Keiichi Tokuda, Nagoya Institute of Technology, my thesisadvisor, for their support, encouragement, and guidance. Also, I would like toexpress my gratitude to Professor Takao Kobayashi and Research Associate TakashiMasuko, Tokyo Institute of Technology, for their kind suggestion.I would also like to thank all the members of Kitamura Laboratory and for theirtechnical support, encouragement. If somebody was missed among them, my workwould not be completed.I would be remiss if I would fail to thank Rie Matae, the secretary to KitamuraLaboratory and Fumiko Fujii, the secretary to the Department of Computer Science,for their kind assistance.Finally, I would sincerely like to thank my family for their encouragement. i
  3. 3. ContentsAcknowledgement iContents iiList of Tables viList of Figures vii1 Introduction 4 1.1 Text-to-speech(TTS) system . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Proposition of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Original contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Mel-cepstral Analysis and Synthesis Technique 8 2.1 Source-filter model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Mel-cepstral analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Synthesis filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Speech Parameter Modeling Based on HMM 14 3.1 Spectral parameter modeling . . . . . . . . . . . . . . . . . . . . . . . 14 3.1.1 Continuous density HMM . . . . . . . . . . . . . . . . . . . . 14 3.1.2 Probability calculation . . . . . . . . . . . . . . . . . . . . . . 16 3.1.3 Parameter estimation of continuous density HMM . . . . . . 18 3.2 F0 parameter modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.2.1 Multi-Space Probability Distribution . . . . . . . . . . . . . . 22 ii
  4. 4. 3.2.2 Multi-space distribution HMM . . . . . . . . . . . . . . . . . 23 3.2.3 Reestimation algorithm for MSD-HMM training . . . . . . . . 25 3.2.4 Application to F0 pattern modeling . . . . . . . . . . . . . . . 274 Speech parameter generation from HMM 29 4.1 Speech parameter generation based on maximum likelihood criterion . 29 4.1.1 Case 1: Maximizing P (O|Q, λ) with respect to O . . . . . . . 30 4.1.2 Case 2: Maximizing P (O, Q|λ) with respect to O and Q . . . 31 4.1.3 Case 3: Maximizing P (O|λ) with respect to O . . . . . . . . . 32 4.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.2.1 Effect of dynamic feature . . . . . . . . . . . . . . . . . . . . . 34 4.2.2 Parameter generation using multi-mixuture HMM . . . . . . . 365 Construction of HMM-based Text-to-Speech System 39 5.1 Calculation of dynamic feature . . . . . . . . . . . . . . . . . . . . . . 39 5.2 Spectrum and F0 modeling . . . . . . . . . . . . . . . . . . . . . . . . 40 5.3 Duration modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.3.2 Training of state duration models . . . . . . . . . . . . . . . . 42 5.4 Context dependent model . . . . . . . . . . . . . . . . . . . . . . . . 43 5.4.1 Contextual factors . . . . . . . . . . . . . . . . . . . . . . . . 43 5.4.2 Decision-tree based context clustering . . . . . . . . . . . . . . 44 5.4.3 Context clustering using MDL principle . . . . . . . . . . . . 476 HMM-based Text-to-Speech Synthesis 49 6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6.2 Text analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6.3 Duration determination . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6.4 Speech parameter generation . . . . . . . . . . . . . . . . . . . . . . . 51 6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 6.5.1 Effect of dynamic feature . . . . . . . . . . . . . . . . . . . . . 52 6.5.2 Automatically system training . . . . . . . . . . . . . . . . . . 55 iii
  5. 5. 6.5.3 Speaking rate . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 Improvement of Synthesized Speech Quality 58 7.1 Introduction of Mixed Excitation Model . . . . . . . . . . . . . . . . 58 7.1.1 Mixed excitation . . . . . . . . . . . . . . . . . . . . . . . . . 59 7.1.2 Excitation parameter modeling . . . . . . . . . . . . . . . . . 62 7.1.3 Excitation parameter generation . . . . . . . . . . . . . . . . . 64 7.2 Incorporation of postfilter . . . . . . . . . . . . . . . . . . . . . . . . 64 7.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 7.3.1 Excitation generation . . . . . . . . . . . . . . . . . . . . . . . 65 7.3.2 Effect of mixed excitation . . . . . . . . . . . . . . . . . . . . 65 7.3.3 Effect of postfiltering . . . . . . . . . . . . . . . . . . . . . . . 678 Voice Conversion Technique: Speaker Interpolation 69 8.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 8.2 Speaker Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 8.3 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 8.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 8.4.1 Generated Spectra . . . . . . . . . . . . . . . . . . . . . . . . 77 8.4.2 Experiment of Similarity . . . . . . . . . . . . . . . . . . . . . 77 8.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 789 Conclusions 80 9.1 Original contribution revisited . . . . . . . . . . . . . . . . . . . . . . 80 9.2 Future works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82List of Publications 87 Journal Papers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 International Conference Proceedings . . . . . . . . . . . . . . . . . . . . . 87 Technical Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Domestic Conference Proceedings . . . . . . . . . . . . . . . . . . . . . . . 89Appendix A Examples of decision trees constructed by using the iv
  6. 6. MDL principle. 92 v
  7. 7. List of Tables 2.1 Examples of α for approximating auditory frequency scales. . . . . . . 10 2.2 Coefficients of R4 (ω). . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 6.1 Number of distribution. . . . . . . . . . . . . . . . . . . . . . . . . . 56 8.1 Setting for simulating Japanese vowel “a” . . . . . . . . . . . . . . . 73 vi
  8. 8. List of Figures 1.1 The scheme of the HMM-based TTS system. . . . . . . . . . . . . . . 6 2.1 Source-filter model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Implementation of Synthesis filter D(z). . . . . . . . . . . . . . . . . 13 3.1 Representation of a speech utterance using a five-state HMM. . . . . 15 3.2 Implementation of the computation using forward-backward algo- rithm in terms of a trellis of observation t and state i. . . . . . . . . . 18 3.3 Example of F0 pattern. . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.4 Multi-space probability distribution and observations. . . . . . . . . . 23 3.5 An HMM based on multi-space probability distribution. . . . . . . . . 24 4.1 Spectra generated with dynamic features for a Japanese phrase “chi- isanaunagi”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.2 Relation between probability density function and generated param- eter for a Japanese phrase “unagi” (top: static, middle: delta, bot- tom: delta-delta). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3 Generated spectra for a sentence fragment “kiNzokuhiroo.” . . . . . . 37 4.4 Spectra obtained from 1-mixture HMMs and 8-mixture HMMs. . . . 37 4.5 The result of the pair comparison test. . . . . . . . . . . . . . . . . . 38 5.1 Calculation of dynamic features for F0. . . . . . . . . . . . . . . . . . 40 5.2 Feature vector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.3 structure of HMM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 vii
  9. 9. 5.4 Decision trees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455.5 Examples of decision trees. . . . . . . . . . . . . . . . . . . . . . . . . 466.1 The block diagram of the HMM-based TTS. . . . . . . . . . . . . . . 506.2 Generated spectra for a phrase “heikiNbairitsu” (top: natural spec- tra, bottom: generated spectra). . . . . . . . . . . . . . . . . . . . . . 526.3 Generated F0 pattern for a sentence “heikiNbairitsuwo sageta keisekiga aru” (top: natural F0 pattern, bottom: generated F0 pattern). . . . . 536.4 Generated spectra for a phrase “heikiNbairitsu.” . . . . . . . . . . . . 536.5 Generated F0 pattern for a sentence “heikiNbairitsuwo sageta keisekiga aru.” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546.6 Effect of dynamic feature. . . . . . . . . . . . . . . . . . . . . . . . . 556.7 Effect of difference of initial model. . . . . . . . . . . . . . . . . . . . 566.8 Generated spectra for an utterance “/t-o-k-a-i-d-e-w-a/” with differ- ent speaking rates (top : ρ = −0.1, middle : ρ = 0, bottom : ρ = 0.1). 577.1 Traditional excitation model. . . . . . . . . . . . . . . . . . . . . . . . 597.2 Multi-band mixing model. . . . . . . . . . . . . . . . . . . . . . . . . 607.3 Mixed excitation model. . . . . . . . . . . . . . . . . . . . . . . . . . 617.4 Structure of a feature vector modeled by HMM. . . . . . . . . . . . . 627.5 Structure of a HMM. . . . . . . . . . . . . . . . . . . . . . . . . . . . 637.6 Effect of postfiltering(dots: before postfiltering, solid: after postfilter- ing β = 0.5). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657.7 Example of generated excitation for phrase “sukoshizutsu.” (top: tra- ditional excitation bottom: mixed excitation) . . . . . . . . . . . . . 667.8 Comparison of traditional and mixed excitation models. . . . . . . . . 677.9 Effect of mixed excitation and postfiltering. . . . . . . . . . . . . . . 688.1 Block diagram of speech synthesis system with speaker interpolaiton. 708.2 A space of speaker individuality modeled by HMMs. . . . . . . . . . . 72 viii
  10. 10. 8.3 Comparison between method (a), (b) and (c) with regard to interpo- lation between two multi-dimensional Gaussian distributions. . . . . . 748.4 Comparison between method (a), (b) and (c) with regard to inter- polation between two Gaussian distributions p1 and p2 with inter- polation ratios A: (a1 , a2 ) = (1, 0), B: (a1 , a2 ) = (0.75, 0.25), C: (a1 , a2 ) = (0.5, 0.5), D: (a1 , a2 ) = (0.25, 0.75), E: (a1 , a2 ) = (0, 1). . . 758.5 Generated spectra of the sentence “/n-i-m-o-ts-u/”. . . . . . . . . . . 778.6 Subjective distance between samples. . . . . . . . . . . . . . . . . . . 78A.1 Examples of decision trees for mel-cepstrum. . . . . . . . . . . . . . . 93A.2 Examples of decision trees for F0. . . . . . . . . . . . . . . . . . . . . 94A.3 Examples of decision trees for bandpass voicing strength. . . . . . . . 95A.4 Examples of decision trees for Fourier magnitude. . . . . . . . . . . . 96A.5 Examples of decision trees for duration. . . . . . . . . . . . . . . . . . 97 ix
  11. 11. x
  12. 12. AbstractA text-to-speech(TTS) system is one of the human-machine interfaces using speech.In recent years, TTS system is developed as an output device of human-machineinterfaces, and it is used in many application such as a car navigation system, in-formation retrieval over the telephone, voice mail, a speech-to-speech translationsystem and so on. However, although most text-to-speech systems still cannot syn-thesize speech with various voice characteristics such as speaker individualities andemotions. To obtain various voice characteristics in text-to-speech systems based onthe selection and concatenation of acoustical units, a large amount of speech data isnecessary. However, it is difficult to collect, segment, and store it. From these pointsof view, in order to construct a speech synthesis system which can generate variousvoice characteristics, an HMM-based text-to-speech system has been proposed. Thisdissertation presents the construction of the HMM-based text-to-speech system, inwhich spectrum, fundamental frequency and duration are modeled simultaneouslyin a unified framework of HMM.In the system, mainly three techniques are used; (1) a mel-cepstral analysis/synthesistechnique, (2) speech parameter modeling using HMM and (3) a speech parametergeneration algorithm from HMM. Since the system uses above three techniques,the system has several capabilities. First, since the TTS system uses the speechparameter generation algorithm, the generated spectral and pitch paramters fromthe trained HMMs can be similar to those of real speech. Second, by transformingHMM parameters appropriately, voice characteristics of synthetic speech can bechanged since the system generates speech from the HMMs. Third, this systemis trainable. In this thesis, first, the above three techniques are presented, andsimultaneous modeling of phonetic and prosodic parameters in a framework of HMMis proposed.Next, to improve of the quality of synthesized speech, the mixed excitation model ofthe speech coder MELP and postfilter are incorporated into the system. Experimen-tal results show that the mixed excitation model and postfilter significantly improvethe quality of synthesized speech.Finally, for the purpose of synthesizing speech with various voice characteristics 1
  13. 13. such as speaker individualities and emotions, the TTS system based on speakerinterpolation is presented. 2
  14. 14. Abstract in Japanese (TTS) HMMHMM (1) (2)HMM (3)HMM 3 3 HMM HMM HMM (1),(2),(3) HMM MELP 3
  15. 15. Chapter 1Introduction1.1 Text-to-speech(TTS) systemSpeech is the natural form of human communication, and it can enhance human-machine communication. A text-to-speech(TTS) system is one of the human-machineinterface using speech. In recent years, TTS system is developed an output deviceof human-machine interface, and it is used in many application such as a car naviga-tion system, information retrieval over the telephone, voice mail, a speech-to-speechtranslation system and so on. The goal of TTS system is to synthesize speech withnatural human voice characteristics and, furthermore, with various speaker individ-ualities and emotions (e.g., anger, sadness, joy).The increasing availability of large speech databases makes it possible to constructTTS systems, which are referred to as data-driven or corpus-based approach, byapplying statistical learning algorithms. These systems, which can be automaticallytrained, can generate natural and high quality synthetic speech and can reproducevoice characteristics of the original speaker.For constructing such a system, the use of hidden Markov models (HMMs) has arisenlargely. HMMs have successfully been applied to modeling the sequence of speechspectra in speech recognition systems, and the performance of HMM-based speechrecognition systems have been improved by techniques which utilize the flexibil-ity of HMMs: context-dependent modeling, dynamic feature parameters, mixturesof Gaussian densities, tying mechanism, speaker and environment adaptation tech-niques. HMM-based approaches to speech synthesis can be categorized as follows: 1. Transcription and segmentation of speech database [1]. 2. Construction of inventory of speech segments [2]–[5]. 4
  16. 16. 3. Run-time selection of multiple instances of speech segments [4], [6]. 4. Speech synthesis from HMMs themselves [7]–[10].In approaches 1–3, by using a waveform concatenation algorithm, e.g., PSOLA al-gorithm, a high quality synthetic speech could be produced. However, to obtainvarious voice characteristics, large amounts of speech data are necessary, and it isdifficult to collect, segment, and store the speech data. On the other hand, in ap-proach 4, voice characteristics of synthetic speech can be changed by transformingHMM parameters appropriately. From this point of view, parameter generationalgorithms [11], [12] for HMM-based speech synthesis have been proposed, and aspeech synthesis system [9], [10] has been constructed using these algorithms. Actu-ally, it has been shown that voice characteristics of synthetic speech can be changedby applying a speaker adaptation technique [13], [14] or a speaker interpolationtechnique [15]. The main feature of the system is the use of dynamic feature: byinclusion of dynamic coefficients in the feature vector, the dynamic coefficients ofthe speech parameter sequence generated in synthesis are constrained to be realistic,as defined by the parameters of the HMMs.1.2 Proposition of this thesisThe proposed TTS system is shown in Fig. 1.1. This figure shows the training andsynthesis parts of the HMM-based TTS system. In the training phase, first, spectralparameters (e.g., cepstral coefficients) and excitation parameters (e.g., fundamen-tal frequency) are extracted from speech database. The extracted parameters aremodeled by context-dependent HMMs. In the systhesis phase, a context-dependentlabel sequence is obtained from a input text by text analysis. A sentence HMM isconstructed by concatenating context dependent HMMs according to the context-dependent label sequence. By using the parameter generation algorithm, spectraland excitation parameters are generated from the sentence HMM. Finaly, by usinga synthesis filter, speech is synthesized from the generated spectral and excitationparameters.In this thesis, it is roughly assumed that spectral and excitation parameters includephonetic and prosodic information, respectively. If these phonetic and prosodic pa-rameters are modeled in a unified framework of HMMs, it is possible to apply speakeradaptation / interpolation techniques to phonetic and prosodic information, simul-taneously, and synthesize speech with various voice characteristics such as speakerindividualities and emotions. From the point of view, the phonetic and prosodicparameter modeling technique and the voice conversion technique are proposed inthis thesis. 5
  17. 17. Speech signal SPEECH DATABASE Excitation Spectral parameter parameter extraction extraction Excitation parameter Spectral parameter Training of HMM Training part Label Synthesis part Context dependent TEXT HMMs Text analysis Label Parameter generation from HMM Excitation parameter Spectral parameter Excitation Synthesis SYNTHESIZED generation filter SPEECH Figure 1.1: The scheme of the HMM-based TTS system.The remainder of this thesis is organized as follows:In Chapters 2–4, the following fundamental techniques of the HMM-based speechsynthesis are described: • Mel-cepstral analysis and synthesis technique (Chapter 2) • Speech parameter modeling based on HMM (Chapter 3) • Speech parameter generation from HMM (Chapter 4)Chapter 5 presents construction of the proposed HMM-based TTS system in whichspectrum, fundamental frequency (F0), duration are modeled by HMM simultane-ously, and Chapter 6 discribes how to synthesize speech. In Chapter 7, mixedexcitation model is incorporated into the proposed TTS system in order to improve 6
  18. 18. quality of synthesized speech. Chapter 8 presents how to synthesize speech withvarious voice characteristics, appling speaker interpolation technique to HMM-basedTTS. Finally Chapter 9 draws overall conclusions and describes possible futureworks.1.3 Original contributionsThis thesis describes new approaches to synthesize speech with natural human voicecharacteristics and with various voice characteristics such as speaker individualityand emotion. The major original contributions are as follows: • Speech parameter generation using multi-mixture HMM. • Duration modeling for the HMM-based TTS system. • Simultaneous modeling of spectrum, F0 and duration. • Training of context dependent HMM using MDL principle. • F0 parameter generation using dynamic features. • Automatic training of the HMM-based TTS system. • Improvement of the quality of the synthesized speech by incorporating the mixed excitation model and postfilter into the HMM-based TTS system. • Voice conversion using a speaker interpolation technique. 7
  19. 19. Chapter 2Mel-cepstral Analysis andSynthesis TechniqueThe proposed TTS system is based on source-filter model. In order to constructthe system, first, it is necessary to extract feature parameters, which describe thevocal tract, from speech database for training. For all the work in this thesis, themel-cepstral analysis [16] is used for spectral estimation. This chapter describesthe mel-cepstral analysis, how feature parameters, i.e., mel-cepstral coefficients, areextracted speech signals and how speech is synthesized from the mel-cepstral coef-ficients.2.1 Source-filter modelTo treat a speech waveform mathematically, source-filter model is generally usedto represent sampled speech signals, as shown in 2.1. The transfer function H(z)models the structure of vocal tract. The excitation source is chosen by a switchwhich controls the voiced/unvoiced character of the speech. The excitation signalis modeled as either a periodic pulse train for voiced speech, or a random noisesequence for unvoiced speech. To produce speech signal x(n), the parameters of themodel must change with time. The excitation signal e(n) is filtered by a time-varyinglinear system H(z) to generate speech signals x(n).The speech x(n) can be computed from the excitation e(n) and the impulse responseh(n) of the vocal tract using the convolution sum expression x(n) = h(n) ∗ e(n) (2.1)where the symbol ∗ stands for discrete convolution. The details of digital signalprocessing and speech processing are given in Ref. [17] 8
  20. 20. Fundamental frequency Vocal Tract Parameter Periodic pulse train Synthesized Synthesis Filter e(n) speech White noise h(n) x(n) = h(n) ∗ e(n) Figure 2.1: Source-filter model.2.2 Mel-cepstral analysisIn mel-cepstral analysis[16], the model spectrum H(ejw ) is represented by the M -thorder mel-cepstral coefficients c(m) as follows: ˜ M H(z) = exp c(m)˜−m , ˜ z (2.2) m=0where −1 z −1 − α z = ˜ , |a| < 1. (2.3) 1 − αz −1The phase characteristic of the all-pass transfer function z−1 = e−j ω is given by ˜ ˜ (1 − α2 ) sin ω ω = tan−1 ˜ . (2.4) (1 + α2 ) cos ω − 2αFor example for a sampling frequency of 16kHz, ω is a good approximation to the ˜mel scale based on subjective pitch evaluations when α = 0.42(Tab. 2.1).To obtain an unbiased estimate, we use the following criterion [18] and minimize itwith respect to c(m)M ˜ m=0 1 π E= exp R(ω) − R(ω) − 1dω, (2.5) 2π −πwhere 2 R(ω) = log IN (ω) − log H(ejω ) , (2.6)and IN (ω) is the modified periodogram of a weakly stationary process x(n) with atime window of length N . To take the gain factor K outside from H(z), we rewriteEq.(2.2) as: M H(z) = exp b(m)Φm (z) = K · D(z), (2.7) m=0 9
  21. 21. Table 2.1: Examples of α for approximating auditory frequency scales. Sampling frequency Mel scale Bark scale 8kHz 0.31 0.42 10kHz 0.35 0.47 12kHz 0.37 0.50 16kHz 0.42 0.55where K = exp b(0), (2.8) M D(z) = exp b(m)Φm (z), (2.9) m=0and c(m) m=M b(m) = (2.10) c(m) − αb(m + 1) 0 ≤ m < M    1 m=0 2 −1 Φm (z) = (1 − α )z (2.11)   z −(m−1) m ≥ 1 ˜ 1 − αz −1Since H(z) is a minimum phase system, we can show that the minimization of Ewith respect to c(m)M is equivalent to that of ˜ m=0 1 π IN (ω) ε= dω, (2.12) 2π −π |D(ejω )|with respect to b = [b(1), b(2), · · · , b(M )]T . (2.13) ∂EThe gain factor K that minimizes E is obtained by setting ∂K = 0: √ K= εmin (2.14)where εmin is the minimized value of ε.2.3 Synthesis filterThe synthesis filter D(z) of Eq.(2.9) is not a rational function and, therefore, it can-not be realized directly. Howerver, using Mel Log Spectrum Approximation filter 10
  22. 22. Table 2.2: Coefficients of R4 (ω). l A4,l 1 4.999273 × 10−1 2 1.067005 × 10−1 3 1.170221 × 10−2 4 5.656279 × 10−4(MLSA filter) [19], the synthesis filter D(z) can be approximated with sufficient ac-curacy and becomes minimum phase IIR system. The complex exponential functionexp ω is qpproximated by a rational function exp ω RL (F (z)) L 1+ AL,l ω l l=1 = L . (2.15) 1+ AL,l (−ω)l l=1Thus D(z) is approximated as follows: RL (F (z)) exp(F (z)) = D(z) (2.16)where F (z) is defined by M F (z) = b(m)Φm (z). (2.17) m=1The filter structure of F (z) is shown in Fig. 2.2(a). Figure 2.2(b) shows the blockdiagram of the MLSA filter RL (F (z)) for the case of L = 4.When we use the coefficients A4,l show in Tab. 2.2, R4 (F (z)) is stable and becomesa minimum phase system under the condition |F (ejω )| ≤ 6.2. (2.18)Further more, we can show that the approximation error | log D(ejω )−log R4 (F (ejω ))|does not exceed 0.24dB[20] under the condition |F (ejω )| ≤ 4.5. (2.19)When F (z) is expressed as F (z) = F1 (z) + F2 (z) (2.20) 11
  23. 23. the exponential transfer function is approximated in a cascade form D(z) = exp F (z) = exp F1 (z) · exp F2 (z) RL (F1 (z)) · RL (F2 (z)) (2.21)as shown in Fig. 2.2(c). If max |F1 (ejω )|, max |F2 (ejω )| < max |F (ejω )|, (2.22) ω ω ωit is expected that RL (F1 (ejω )) · RL (F2 (ejω )) approximates D(ejω ) more accuratelythan RL (F (ejω )).In the following experiments, we let F1 (z) = b(1)Φ1 (z) (2.23) M F2 (z) = b(m)Φm (z). (2.24) m=2Since we empirically found that max |F1 (ejω )|, max |F2 (ejω )| < 4.5 (2.25) ω ωfor speech sounds, RL (F1 (z))·RL(F2 (z)) approximates the exponential transfer func-tion D(z) with sufficient accuracy and becomes a stable system. 12
  24. 24. α α α 1 − α2Input Z −1 Z −1 Z −1 Z −1 b(1) b(2) b(3) Output (a) Basic filter F (z) OutputInput F (z) F (z) F (z) F (z) A4,1 A4,2 A4,3 A4,4 (b) RL (F (z)) D(z), L=4 Input output R4 (F1 (z)) R4 (F2 (z)) (c) Two stage cascade structure Figure 2.2: Implementation of Synthesis filter D(z). 13
  25. 25. Chapter 3Speech Parameter ModelingBased on HMMThe most successful and widely used acoustic models in recent years have beenHidden Markov Models (HMMs). Practically all major speech recognition systemsare generally implemented using HMM. This chapter describes how to model spectraland excitation parameters in a framework of HMM.3.1 Spectral parameter modeling3.1.1 Continuous density HMMIn this thesis, a continuous density HMM is used for the vocal tract modeling in thesame way as speech recognition systems. The continuous density Markov model is afinite state machine which makes one state transition at each time unit (i.e, frame).First, a decision is made to which state to succeed (including the state itself). Thenan output vector is generated according to the probability density function (pdf) forthe current state. An HMM is a doubly stochastic random process, modeling statetransition probabilities between states and output probabilities at each state.One way of interpreting HMMs is to view each state as a model of a segment ofspeech. Figure 3.1 shows an example of representation of a speech utterance using aN -state left-to-right HMM where each state is modeled by a multi-mixture Gaussianmodel. Assume that this utterance(typically parameterized by speech analysis asthe D-dimensional observation vector ot ) is divided into N segments di which arerepresented by the states Si . The transition probability aij defines the probabilityof moving from state i to state j and satisfies aii + aij = 1. Then, each state can be 14
  26. 26. a11 a22 a33 a44 a55 a12 a23 a34 a45 S1 S2 S3 S4 S5 HMM b1 (ot ) b2 (ot ) b3 (ot ) b4 (ot ) b5 (ot )Frequency (kHz) 0 2 4 Spectrum 6 8 d1 d2 d3 d4 d5 Figure 3.1: Representation of a speech utterance using a five-state HMM.modeled by a M -mixtures Gaussian density function: M bj (ot ) = cjk N (ot , µjk , Σjk ) k=1 M 1 1 = cjk D exp − (ot − µjk )T Σ−1 (ot − µjk ) , 1 jk (3.1) k=1 (2π) |Σjk | 2 22where cjk , µjk and Σjk are the mixture coefficient, D-dimensional mean vector andD × D covariance matrix(full covariance matrix) for the k-th mixture componentin the j-th state, respectively. This covariance can be restricted to the diagonalelements (diagonal covariance matrix) when the elements of the feature vector areassumed to be independent. |Σjk | is the determinant of Σjk , and Σ−1 is the inverse jkof Σjk . The mixture gains cjk satisfy the stochastic constraint M cjk = 1, 1≤j≤N (3.2) k=1 cjk ≥ 0, 1 ≤ j ≤ N, 1 ≤ k ≤ M (3.3) 15
  27. 27. so that the pdf is properly normalized, i.e., ∞ bj (o)do, 1≤j≤N (3.4) −∞Since the pdf of Eq. (3.1) can be used to approximate, arbitrarily closely, any finite,continuous density function, it can be applied to a wide range of problems and iswidely used for acoustic modeling.For convenience, to indicate the complete parameter set of the model, we use thecompact notation λ = (A, B, π), (3.5)where A = {aij }, B = {bj (o)} and π = {πi }. πi is the initial state distribution ofstate i, and it have the property 0, i = 1 π= (3.6) 1, i = 1in the left-to-right model.3.1.2 Probability calculationFor calculation of P (O|λ), which is the probability of the observation sequence O =(o1 , o2 , · · · , oT ) given the model λ, forward-backward algorithm is generally used.If we calculate P (O|λ) directly without this algorithm, it requires on the order of2T N 2 calculation. On the other hand, calculation using forward-backward algorithmrequires on the order of N 2 T calculations, and it is computationally feasible. In thefollowing part, forward-backward algorithm is described.The forward algorithmConsider the forward variable αt (i) defined as αt (i) = P (o1 , o2 , · · · , ot , qt = i|λ) (3.7)that is, the probability of the partial observation sequence from 1 to t and state iat time t, given the model λ. We can solve for αt (i) inductively, as follows: 1. Initialization α1 (i) = πi bi (o1 ), 1 ≤ ileqN. (3.8) 16
  28. 28. 2. Induction N 1≤t≤T −1 αt+1 (j) = αt (i)aij bj (ot+1 ), (3.9) i=1 1≤j≤N 3. Termination N P (O|λ) = αT (i). (3.10) i=1The backward algorithmIn the same way as forward algorithm, consider the backward variable βt (i) definedas βt (i) = P (ot + 1, ot + 2, · · · , oT |qt = i, λ) (3.11)that is, the probability of the partial observation sequence from t to T , given statei at time t and the model λ. We can solve for βt (i) inductively, as follows: 1. Initialization βT (i) = 1, 1 ≤ i ≤ N. (3.12) 2. Induction N t = T − 1, T − 2, · · · , 1 βt (i) = aij bj (ot+1 )βt+1 (j), (3.13) i=1 1≤i≤N 3. Termination N P (O|λ) = β1 (i). (3.14) i=1The forward-backward probability calculation is based on the trellis structure shownin Fig. 3.2. In this figure, the x-axis and y-axis represent observation sequence andstates of Markov model, respectively. On the trellis, all the possible state sequencewill remerge into these N nodes no matter how long the observation sequence. Inthe case of the forward algorithm, at times t = 1, we need to calculate values ofα1 (i), 1 ≤ i ≤ N . At times t = 2, 3, · · · , T , we need only calculate values of αt (j),1 ≤ j ≤ N , where each calculation involves only the N previous values of αt−1 (i)because each of the N grid points can be reached from only the N grid points atthe previous time slot. As the result, the forward-backward algrorithm can reduceorder of probability calculation. 17
  29. 29. N Si 3 State 2 1 1 2 3 4 T Observation sequence otFigure 3.2: Implementation of the computation using forward-backward algorithmin terms of a trellis of observation t and state i.3.1.3 Parameter estimation of continuous density HMMIt is difficult to determine a method to adjust the model parameters (A, B, π) tosatisfy a certain optimization criterion. There is no known way to analytical solve forthe model parameter set that maximizes the probability of the observation sequencein a closed form. We can, however, choose λ = (A, B, π) such that its likelihood,P (O|λ), is locally maximized using an iterative procedure such as the Baum-Welchmethod(also known as the EM(expectation-maximization method))[21], [22].To describe the procedure for reestimation of HMM parameters, first, the probabilityof being in state i at time t, and state j at time t + 1, given the model and theobservation sequence, is defined as follows: ξt (i, j) = P (qt = i, qt+1 = j|O, λ), (3.15)From the definitions of the forward and backward variables, ξt (i, j) is writen in theform P (qt = i, qt+1 = j|O, λ) ξt (i, j) = P (O|λ) αt (i)aij bj (ot+1 βt+1 (j)) = N N (3.16) αt (i)aij bj (ot+1 βt+1 (j)) i=1 j=1 18
  30. 30. Using ξt (i, j), the probability of being in state i at time t, given the entire observationand the model, is represented by N γt (i) = ξt (i, j). (3.17) j=1Q-functionThe reestimation formulas can be derived directly by maximizing Baum’s auxiliaryfunciton Q(λ , λ) = P (O, q|λ ) log P (O, q|λ) (3.18) qover λ. Because Q(λ , λ) ≥ Q(λ , λ) ⇒ P (O, q|λ ) ≥ P (O, q|λ) (3.19)We can maximize the function Q(λ , λ) over λ to improve λ in the sense of increasingthe likelihood P (O, q|λ).Maximization of Q-functionFor given observation sequence O and model λ , we derive parameters of λ whichmaximize Q(λ , λ). P (O, q|λ) can be written as T P (O, q|λ) = πq0 aqt−1 qt bqt (ot ) (3.20) t=1 T T log P (O, q|λ) = log πq0 + log aqt−1 qt + log bqt (ot ) (3.21) t=1 t=1Hence Q-function (3.18) can be written as Q(λ , λ) = Qπ (λ , pi) T + Qai (λ , ai ) t=1 T + Qbi (λ , bi ) t=1 N = P (O, q0 = i|λ ) log πi i=1 N T + P (O, qt−1 = i, qt = j|λ ) log aij j=1 t=1 T + P (O, qt = i|λ ) log bi (ot ) (3.22) t=1 19
  31. 31. where π = [π1 , π2 , · · · , πN ] (3.23) ai = [ai1 , ai2 , · · · , aiN ] , (3.24)and bi is the parameter vector that defines bi (·). The parameter set λ which maxi-mizes (3.22), subject to the stochastic constraints N πj = 1, (3.25) j=1 N aij = 1, (3.26) j=1 M cjk = 1, (3.27) k=1 ∞ bj (o)do = 1, (3.28) −∞can be derived as α0 (i)β0 (i) πi = N = γ0 (i) (3.29) αT (j) j=1 T T αt−1 (i)aij bj (ot )βt (j) ξt−1 (i, j) t=1 t=1 aij = T = T (3.30) αt−1 (i)βt−1 (i) γt−1 (i) t=1 t=1The reestimation formulas for the coefficients of the mixture density, i.e, cjk , µjkand Σjk are of the form T γt (j, k) t=1 cij = T M (3.31) γt (j, k) t=1 k=1 T γt (j, k) · ot t=1 µjk = T (3.32) γt (j, k) t=1 T γt (j, k) · (ot − µjk )(ot − µjk ) t=1 Σjk = T (3.33) γt (j, k) t=1 20
  32. 32. voiced region (continuous value) 200Frequency (Hz) 150 100 50 0 1 2 3 4 5 time (s) unvoiced region (discrete symbol) Figure 3.3: Example of F0 pattern. (3.34)where γt (j, k) is the probability of being in state j at time t with the kth mixturecomponent accounting for ot , i.e.,        α (j)β (j)   c N (o , µ , Σ )   t t   jk t jk jk  γt (j, k) =  N  M . (3.35)       αt (j)βt (j) N (ot , µjk , Σjk ) j=1 m=13.2 F0 parameter modelingThe F0 pattern is composed of continuous values in the “voiced” region and adiscrete symbol in the “unvoiced” region (Fig.3.3). Therefore, it is difficult to applythe discrete or continuous HMMs to F0 pattern modeling. Several methods havebeen investigated [23] for handling the unvoiced region: (i) replacing each “unvoiced”symbol by a random vector generated from a probability density function (pdf) witha large variance and then modeling the random vectors explicitly in the continuousHMMs [24], (ii) modeling the “unvoiced” symbols explicitly in the continuous HMMsby replacing “unvoiced” symbol by 0 and adding an extra pdf for the “unvoiced”symbol to each mixture, (iii) assuming that F0 values is always exist but they cannotobserved in the unvoiced region and applying the EM algorithm [25]. In this section,A kind of HMM for F0 pattern modeling, in which the state output probabilitiesare defined by multi-space probability distributions (MSDs), is described. 21
  33. 33. 3.2.1 Multi-Space Probability DistributionWe consider a sample space Ω shown in Fig. 3.4, which consists of G spaces: G Ω= Ωg (3.36) g=1where Ωg is an ng -dimensional real space Rng , and specified by space index g. Eachspace Ωg has its probability wg , i.e., P (Ωg ) = wg , where G wg = 1. If ng > 0, each g=1space has a probability density function Ng (x), x ∈ Rng , where Rng Ng (x)dx = 1.We assume that Ωg contains only one sample point if ng = 0. Accordingly, lettingP (E) be the probability distribution, we have G G P (Ω) = P (Ωg ) = wg Ng (x)dx = 1. (3.37) g=1 g=1 RngIt is noted that, although Ng (x) does not exist for ng = 0 since Ωg contains onlyone sample point, for simplicity of notation, we define as Ng (x) ≡ 1 for ng = 0.Each event E, which will be considered in this thesis, is represented by a randomvariable o which consists of a continuous random variable x ∈ Rn and a set of spaceindices X, that is, o = (x, X) (3.38)where all spaces specified by X are n-dimensional. The observation probability ofo is defined by b(o) = wg Ng (V (o)) (3.39) g∈S(o)where V (o) = x, S(o) = X. (3.40)Some examples of observations are shown in Fig. 3.4. An observation o1 consists ofthree-dimensional vector x1 ∈ R3 and a set of space indices X1 = {1, 2, G}. Thusthe random variable x is drawn from one of three spaces Ω1 , Ω2 , ΩG ∈ R3 , and itsprobability density function is given by w1 N1 (x) + w2 N2 (x) + wG NG (x).The probability distribution defined in the above, which will be referred to as multi-space probability distribution (MSD) in this thesis, is the same as the discrete distri-bution and the continuous distribution when ng ≡ 0 and ng ≡ m > 0, respectively.Further, if S(o) ≡ {1, 2, . . . , G}, the continuous distribution is represented by aG-mixture probability density function. Thus multi-space probability distributionis more general than either discrete or continuous distributions. 22
  34. 34. Sample Space Ω Space Index Space pdf of x Observation Ω1 = R3 1 w1 o1 = ( x1, {1, 2, G}) N1 ( x ) x1 ∈ R3 Ω 2 = R3 2 w2 N2 ( x ) o2 = ( x2 , {1, G}) x2 ∈ R3 Ω 3 = R5 3 w3 N3 ( x ) o3 = ( x3 , {3}) x3 ∈ R 5 Ω G = R3 wG G NG ( x ) Figure 3.4: Multi-space probability distribution and observations.3.2.2 Multi-space distribution HMMThe output probability in each state of MSD-HMM is given by the multi-spaceprobability distribution defined in the previous section. An N -state MSD-HMM λis specified by initial state probability distribution π = {πj }N , the state transition j=1probability distribution A = {aij }Nj=1 , and state output probability distribution i,B = {bi (·)}N , where i=1 bi (o) = wig Nig (V (o)), i = 1, 2, . . . , N. (3.41) g∈S(o)As shown in Fig. 3.5, each state i has G probability density functions Ni1 (·), Ni2 (·),. . ., NiG (·), and their weights wi1 , wi2 , . . ., wiG . 23
  35. 35. 1 2 3 w11 w21 w31 Ω1 = R n1 N11 ( x ) N21 ( x ) N31 ( x ) w12 w22 w32 Ω 2 = Rn2 N12 ( x ) N22 ( x ) N32 ( x ) w1G w2 G w3G Ω G = RnG N1G ( x ) N2 G ( x ) N3 G ( x ) Figure 3.5: An HMM based on multi-space probability distribution.Observation probability of O = {o1 , o2 , . . . , oT } is written as T P (O|λ) = aqt−1 qt bqt (ot ) all q t=1 T = aqt−1 qt wqt lt Nqt lt (V (ot )) (3.42) all q,l t=1where q = {q1 , q2 , . . . , qT } is a possible state sequence, l = {l1 , l2 , . . . , lT } ∈ {S(o1 )×S(o2 )×. . .×S(oT )} is a sequence of space indices which is possible for the observationsequence O, and aq0 j denotes πj .The forward and backward variables: αt (i) = P (o1 , o2 , . . . , ot , qt = i|λ) (3.43) βt (i) = P (ot+1 , ot+2 , . . . , oT |qt = i, λ) (3.44)can be calculated with the forward-backward inductive procedure in a manner simi-lar to the conventional HMMs. According to the definitions, (3.42) can be calculatedas N N P (O|λ) = αT (i) = β1 (i). (3.45) i=1 i=1 24
  36. 36. The forward and backward variables are also used for calculating the reestimationformulas derived in the the next section3.2.3 Reestimation algorithm for MSD-HMM trainingFor a given observation sequence O and a particular choice of MSD-HMM, the ob-jective in maximum likelihood estimation is to maximize the observation likelihoodP (O|λ) given by (3.42), over all parameters in λ. In a manner similar to [21], [22], wederive reestimation formulas for the maximum likelihood estimation of MSD-HMM.Q-functionAn auxiliary function Q(λ , λ) of current parameters λ and new parameter λ isdefined as follows: Q(λ , λ) = P (O, q, l|λ ) log P (O, q, l|λ) (3.46) all q,lIn the following, we assume Nig (·) to be the Gaussian density with mean vector µigand covariance matrix Σig .Theorem 1 Q(λ , λ) ≥ Q(λ , λ ) → P (O, λ) ≥ P (O, λ )Theorem 2 If, for each space Ωg , there are among V (o1 ), V (o2 ), . . ., V (oT ), ng +1observations g ∈ S(ot ), any ng of which are linearly independent, Q(λ , λ) has aunique global maximum as a function of λ, and this maximum is the one and onlycritical point.Theorem 3 A parameter set λ is a critical point of the likelihood P (O|λ) if andonly if it is a critical point of the Q-function.We define the parameter reestimates to be those which maximize Q(λ , λ) as afunction of λ, λ being the latest estimates. Because of the above theorems, thesequence of resetimates obtained in this way produce a monotonic increase in thelikelihood unless λ is a critical point of the likelihood. 25
  37. 37. Maximization of Q-functionFor given observation sequence O and model λ , we derive parameters of λ whichmaximize Q(λ , λ). From (3.42), log P (O, q, l|λ) can be written as log P (O, q, l|λ) T = log aqt−1 qt + log wqt lt + log Nqt lt (V (ot )) . (3.47) t=1Hence Q-function (3.46) can be written as N Q(λ , λ) = P (O, q1 = i|λ ) log πi i=1 N T −1 + P (O, qt = i, qt+1 = j|λ ) log aij i,j=1 t=1 N G + P (O, qt = i, lt = g|λ ) log wig i=1 g=1 t∈T (O,g) N G + P (O, qt = i, lt = g|λ ) log Nig (V (ot )) i=1 g=1 t∈T (O,g) (3.48)where T (O, g) = {t | g ∈ S(ot )}. (3.49)The parameter set λ = (π, A, B) which maximizes (3.48), subject to the stochasticconstraints N πi = 1, N aij = 1 and G wg = 1, can be derived as i=1 j=1 g=1 πi = γ1 (i, g) (3.50) g∈S(o1 ) T −1 ξt (i, j) t=1 aij = T −1 (3.51) γt (i, g) t=1 g∈S(ot ) γt (i, g) t∈T (O,g) wig = G (3.52) γt (i, h) h=1 t∈T (O,h) γt (i, g) V (ot ) t∈T (O,g) µig = , ng > 0 (3.53) γt (i, g) t∈T (O,g) 26
  38. 38. γt (i, g)(V (ot ) − µig )(V (ot ) − µig )T t∈T (O,g) Σig = , γt (i, g) t∈T (O,g) ng > 0 (3.54)where γt (i, h) and ξt (i, j) can be calculated by using the forward variable αt (i) andbackward variable βt (i) as follows: γt (i, h) = P (qt = i, lt = h|O, λ) αt (i)βt (i) wih Nih (V (ot )) = N · (3.55) wig Nig (V (ot )) αt (j)βt (j) g∈S(ot ) j=1 ξt (i, j) = P (qt = i, qt+1 = j|O, λ) αt (i)aij bj (ot+1 )βt+1 (j) = N N (3.56) αt (h)ahk bk (ot+1 )βt+1 (k) h=1 k=1From the condition mentioned in Theorem 2, it can be shown that each Σig ispositive definite.3.2.4 Application to F0 pattern modelingThe MSD-HMM includes the discrete HMM and the continuous mixture HMMas special cases since the multi-space probability distribution includes the discretedistribution and the continuous distribution. If ng ≡ 0, the MSD-HMM is the sameas the discrete HMM. In the case where S(ot ) specifies one space, i.e., |S(ot )| ≡ 1,the MSD-HMM is exactly the same as the conventional discrete HMM. If |S(ot )| ≥ 1,the MSD-HMM is the same as the discrete HMM based on the multi-labeling VQ[26]. If ng ≡ m > 0 and S(o) ≡ {1, 2, . . . , G}, the MSD-HMM is the same asthe continuous G-mixture HMM. These can also be confirmed by the fact that ifng ≡ 0 and |S(ot )| ≡ 1, the reestimation formulas (3.50)-(3.52) are the same as thosefor discrete HMM of codebook size G, and if ng ≡ m and S(ot ) ≡ {1, 2, . . . , G},the reestimation formulas (3.50)-(3.54) are the same as those for continuous HMMwith m-dimensional G-mixture densities. Further, the MSD-HMM can model thesequence of observation vectors with variable dimension including zero-dimensionalobservations, i.e., discrete symbols.While the observation of F0 has a continuous value in the voiced region, there existno value for the unvoiced region. We can model this kind of observation sequenceassuming that the observed F0 value occurs from one-dimensional spaces and the 27
  39. 39. “unvoiced” symbol occurs from the zero-dimensional space defined in Section 3.2.1,that is, by setting ng = 1 (g = 1, 2, . . . , G − 1), nG = 0 and {1, 2, . . . , G − 1}, (voiced) S(ot ) = , (3.57) {G}, (unvoiced)the MSD-HMM can cope with F0 patterns including the unvoiced region withoutheuristic assumption. In this case, the observed F0 value is assumed to be drawnfrom a continuous (G − 1)-mixture probability density function. 28
  40. 40. Chapter 4Speech parameter generation fromHMMThe performance of speech recognition based on HMMs has been improved by incor-porating the dynamic features of speech. Thus we surmise that, if there is a methodfor speech parameter generation from HMMs which include the dynamic features, itwill be useful for speech synthesis by rule. This chapter derives a speech parametergeneration algorithm from HMMs which include the dynamic features.4.1 Speech parameter generation based on maxi- mum likelihood criterionFor a given continuous mixture HMM λ, we derive an algorithm for determiningspeech parameter vector sequence O = o1 , o2 , . . . , oT (4.1)in such a way that P (O|λ) = P (O, Q|λ) (4.2) all Qis maximized with respect to O, where Q = {(q1 , i1 ), (q2 , i2 ), . . . , (qT , iT )} (4.3)is the state and mixture sequence, i.e., (q, i) indicates the i-th mixture of stateq. We assume that the speech parameter vector ot consists of the static featurevector ct = [ct (1), ct (2), . . . , ct (M )] (e.g., cepstral coefficients) and dynamic feature 29
  41. 41. vectors ∆ct , ∆2 ct (e.g., delta and delta-delta cepstral coefficients, respectively), thatis, ot = [ct , ∆ct , ∆2 ct ] , where the dynamic feature vectors are calculated by (1) L+ ∆ct = w (1) (τ )ct+τ (4.4) (1) τ =−L− (2) L+ ∆2 ct = w (2) (τ )ct+τ . (4.5) (2) τ =−L−We have derived algorithms [11], [12] for solving the following problems:Case 1. For given λ and Q, maximize P (O|Q, λ) with respect to O under the conditions (4.4), (4.5).Case 2. For a given λ, maximize P (O, Q|λ) with respect to Q and O under the conditions (4.4), (4.5).In this section, we will review the above algorithms and derive an algorithm for theproblem:Case 3. For a given λ, maximize P (O|λ) with respect to O under the conditions (4.4), (4.5).4.1.1 Case 1: Maximizing P (O|Q, λ) with respect to OFirst, consider maximizing P (O|Q, λ) with respect to O for a fixed state and mixturesequence Q. The logarithm of P (O|Q, λ) can be written as 1 log P (O|Q, λ) = − O U −1 O + O U −1 M + K (4.6) 2where U −1 = diag U −1 1 , U −1 2 , ..., U −1,iT q1 ,i q2 ,i qT (4.7) M = µq1 ,i1 , µq2 ,i2 , ..., µqT ,iT (4.8)µqt , it and U qt , it are the 3M × 1 mean vector and the 3M × 3M covariance ma-trix, respectively, associated with it -th mixture of state qt , and the constant K isindependent of O. 30
  42. 42. It is obvious that P (O|Q, λ) is maximized when O = M without the conditions(4.4), (4.5), that is, the speech parameter vector sequence becomes a sequence ofthe mean vectors. Conditions (4.4), (4.5) can be arranged in a matrix form: O = WC (4.9)where C = [c1 , c2 , . . . , cT ] (4.10) W = [w 1 , w2 , . . . , wT ] (4.11) (0) (1) (2) wt = wt , wt , wt (4.12) (n) (n) wt = [ 0M ×M , . . . , 0M ×M , w (n) (−L− )I M ×M , 1st (n) (t−L− )-th (n) . . . , w(n) (0)I M ×M , . . . , w(n) (L+ )I M ×M , t-th (n) (t+L+ )-th 0M ×M , . . . , 0M ×M ] , n = 0, 1, 2 (4.13) T -th (0) (0)L− = L+ = 0, and w (0) (0) = 1. Under the condition (4.9), maximizing P (O|Q, λ)with respect to O is equivalent to that with respect to C. By setting ∂ log P (W C|Q, λ) = 0, (4.14) ∂Cwe obtain a set of equations W U −1 W C = W U −1 M . (4.15)For direct solution of (4.15), we need O(T 3M 3 ) operations1 because W U −1 W is aT M × T M matrix. By utilizing the special structure of W U −1 W , (4.15) can besolved by the Cholesky decomposition or the QR decomposition with O(T M3 L2 )operations2 , where L = maxn∈{1,2},s∈{−,+} L(n) . Equation (4.15) can also be solved sby an algorithm derived in [11], [12], which can operate in a time-recursive manner[28].4.1.2 Case 2: Maximizing P (O, Q|λ) with respect to O and QThis problem can be solved by evaluating max P (O, Q|λ) = max P (O|Q, λ)P (Q|λ)for all Q. However, it is impractical because there are too many combinations of 1 When U q,i is diagonal, it is reduced to O(T 3 M ) since each of the M -dimensions can becalculated independently. 2 (1) (1) When U q,i is diagonal, it is reduced to O(T M L 2 ). Furthermore, when L− = −1, L+ = 0,and w(2) (i) ≡ 0, it is reduced to O(T M ) as described in [27]. 31
  43. 43. Q. We have developed a fast algorithm for searching for the optimal or sub-optimalstate sequence keeping C optimal in the sense that P (O|Q, λ) is maximized withrespect to C [11], [12].To control temporal structure of speech parameter sequence appropriately, HMMsshould incorporate state duration densities. The probability P (O, Q|λ) can be writ-ten as P (O, Q|λ) =P (O, i|q, λ)P (q|λ), where q = {q1 , q2 , . . . , qT }, i = {i1 , i2 ,. . . , iT }, and the state duration probability P (q|λ) is given by N log P (q|λ) = log pqn (dqn ) (4.16) n=1where the total number of states which have been visited during T frames is N , andpqn (dqn ) is the probability of dqn consecutive observations in state qn . If we determinethe state sequence q only by P (q|λ) independently of O, maximizing P (O, Q|λ) =P (O, i|q, λ)P (q|λ) with respect to O and Q is equivalent to maximizing P (O, i|q, λ)with respect to O and i. Furthermore, if we assume that state output probabilitiesare single-Gaussian, i is unique. Therefore, the solution is obtained by solving (4.15)in the same way as the Case 1.4.1.3 Case 3: Maximizing P (O|λ) with respect to OWe derive an algorithm based on an EM algorithm, which find a critical point ofthe likelihood function P (O|λ). An auxiliary function of current parameter vectorsequence O and new parameter vector sequence O is defined by Q(O, O ) = P (O, Q|λ) log P (O , Q|λ). (4.17) all QIt can be shown that by substituting O which maximizes Q(O, O ) for O, thelikelihood increases unless O is a critical point of the likelihood. Equation (4.17)can be written as 1 Q(O, O ) = P (O|λ) − O U −1 O + O U −1 M + K (4.18) 2where U −1 = diag U −1 , U −1 , . . . , U −1 1 2 T (4.19) U −1 = t γt (q, i)U −1 q,i (4.20) q,i U −1 M = U −1 µ1 , U −1 µ2 , . . . , U −1 µT 1 2 T (4.21) U −1 µt = t γt (q, i) U −1 µq,i q,i (4.22) q,i 32
  44. 44. and the constant K is independent of O . The occupancy probability γt (q, i) defined by γt (q, i) = P (qt = (q, i)|O, λ) (4.23) can be calculated with the forward-backward inductive procedure. Under the con- dition O = W C , C which maximizes Q(O, O ) is given by the following set of equations: W U −1 W C = W U −1 M . (4.24) The above set of equations has the same form as (4.15). Accordingly, it can be solved by the algorithm for solving (4.15). The whole procedure is summarized as follows:Step 0. Choose an initial parameter vector sequence C.Step 1. Calculate γt (q, i) with the forward-backward algorithm.Step 2. Calculate U −1 and U −1 M by (4.19)–(4.22), and solve (4.24).Step 3. Set C = C . If a certain convergence condition is satisfied, stop; otherwise, goto Step 1. From the same reason as Case 2, HMMs should incorporate state duration densi- ties. If we determine the state sequence q only by P (q|λ) independently of O in a manner similar to the previous section, only the mixture sequence i is assumed to be unobservable3 . Further, we can also assume that Q is unobservable but phoneme or syllable durations are given. 4.2 Example A simple experiment of speech synthesis was carried out using the parameter gener- ation algorithm. We used phonetically balanced 450 sentences from ATR Japanese speech database for training. The type of HMM used was a continuous Gaussian model. The diagonal covariances were used. All models were 5-state left-to-right models with no skips. The heuristic duration densities were calculated after the training. Feature vector consists of 25 mel-cepstral coefficients including the zeroth coefficient, their delta and delta-delta coefficients. Mel-cepstral coefficients were ob- tained by the mel-cepstral analysis. The signal was windowed by a 25ms Black man window with a 5ms shift. 3 For this problem, an algorithm based on a direct search has also been proposed in [29]. 33
  45. 45. Frequency (kHz) 0 2 4 6 8 (a) without dynamic features Frequency (kHz) 0 2 4 6 8 (b) with delta parameters Frequency (kHz) 0 2 4 6 8 (c) with delta and delta-delta parametersFigure 4.1: Spectra generated with dynamic features for a Japanese phrase “chi-isanaunagi”.4.2.1 Effect of dynamic featureFirst, we observed the parameter generation in the case 1, in which parametersequence O maximizes P (O|Q, λ). State sequence Q was estimated from the re-sult of Veterbi alignment of natural speech. Fig. 4.1 shows the spectra calculatedfrom the mel-cepstral coefficients generated by the HMM, which is composed byconcatenation of phoneme models. Without the dynamic features, the parame-ter sequence which maximizes P (O|Q, λ) becomes a sequence of the mean vectors(Fig. 4.1(a)). On the other hand, Fig. 4.1(b) and Fig. 4.1(c) show that appropriateparameter sequences are generated by using the static and dynamic feature. Look-ing at Fig. 4.1(b) and Fig. 4.1(c) closely, we can see that incorporation of delta-deltaparameter improves smoothness of generated speech spectra.Fig 4.2 shows probability density functions and generated parameters for the zero-thmel-cepstral coefficients. The x-axis represents frame number. A gray box and itsmiddle line represent standard deviation and mean of probability density function,respectively, and a curved line is the generated zero-th mel-cepstral coefficients.From this figure, it can be observed that parameters are generated taking accountof constraints of their probability density function and dynamic features. 34
  46. 46. 7.2 c(0) 4.9 0.38 ∆c(0) -0.31∆2 c(0) 0.070 -0.068 frame standard deviation generated parameter meanFigure 4.2: Relation between probability density function and generated parameterfor a Japanese phrase “unagi” (top: static, middle: delta, bottom: delta-delta). 35
  47. 47. 4.2.2 Parameter generation using multi-mixuture HMMIn the algorithm of the case 3, we assumed that the state sequence (state and mixturesequence for the multi-mixture case) or a part of the state sequence is unobservable(i.e., hidden or latent). As a result, the algorithm iterates the forward-backwardalgorithm and the parameter generation algorithm for the case where state sequenceis given. Experimental results show that by using the algorithm, we can reproduceclear formant structure from multi-mixture HMMs as compared with that producedfrom single-mixture HMMs.It has found that a few iterations are sufficient for convergence of the proposedalgorithm. Fig. 4.3 shows generated spectra for a Japanese sentence fragment “kiN-zokuhiroo” taken from a sentence which is not included in the training data. Fig. 4.4compares two spectra obtained from single-mixure HMMs and 8-mixture HMMs, re-spectively, for the same temporal position of the sentence fragment. It is seen fromFig. 4.3 and Fig. 4.4 that with increasing mixtures, the formant structure of thegenerated spectra get clearer.We evaluated two synthesized speech obtained from single-mixure HMMs and 8-mixture HMMs by listening test, where fundamental frequency and duration is ob-tained from natural speech. Figure 4.5 shows the result of pair comparison test.From the listening test, it has been observed that the quality of the synthetic speechis considerably improved by increasing mixtures.When we use single-mixture HMMs, the formant structure of spectrum correspond-ing to each mean vector µq,i might be vague since µq,i is the average of differentspeech spectra. One can increase the number of decision tree leaf clusters. However,it might result in perceivable discontinuities in synthetic speech since overly largetree will be overspecialized to training data and generalized poorly. We expect thatthe proposed algorithm can avoid this situation in a simple manner. 36
  48. 48. 0 2(kHz) 4 6 8 (a) 1-mixture 0 2(kHz) 4 6 8 (b) 2-mixture 0 2(kHz) 4 6 8 (c) 4-mixture 0 2(kHz) 4 6 8 (d) 8-mixture Figure 4.3: Generated spectra for a sentence fragment “kiNzokuhiroo.” 100 Log magnitude (dB) mix1 80 mix8 60 40 20 0 2 4 6 8 Frequency (kHz) Figure 4.4: Spectra obtained from 1-mixture HMMs and 8-mixture HMMs. 37
  49. 49. 1-mixture 40.88-mixtures 59.2 0 20 40 60 80 100 Preference score (%) Figure 4.5: The result of the pair comparison test. 38
  50. 50. Chapter 5Construction of HMM-basedText-to-Speech SystemPhonetic parameter and prosodic parameter are modeled simultaneously in a unifiedframework of HMM. In the proposed system, mel-cepstrum, fundamental frequency(F0) and state duration are modeled by continuous density HMMs, multi-spaceprobability distribution HMMs and multi-dimensional Gaussian distributions, re-spectively. The distributions for spectrum, F0, and the state duration are clusteredindependently by using a decision-tree based context clustering technique. Thischapter describes feature vector modeled by HMM, structure of HMM and how totrain context-dependent HMM.5.1 Calculation of dynamic featureIn this thesis, mel-cepstral coefficient is used as spectral parameter. Mel-cepstralcoefficient vectors c are obtained from speech database using a mel-cepstral analysistechnique [16]. Their dynamic feature ∆c and ∆2 c are calculated as follows: 1 1 ∆ct = − ct−1 + ct+1 , (5.1) 2 2 2 1 1 1 ∆ ct = ct−1 − ct + ct+1 . (5.2) 4 2 4In the same way, dynamic features for F0 are calculated by 1 1 δpt = − pt−1 + pt+1 , (5.3) 2 2 1 1 1 δ 2 pt = pt−1 − pt + pt+1 . (5.4) 4 2 4 39
  51. 51. represents a continous value ( voiced ). represents a discrete symbol ( unvoiced ). Unvoiced region Voiced region Unvoiced region Delta-delta δ 2 pt Delta δpt Static pt−1 pt pt+1 1 t T Frame number Figure 5.1: Calculation of dynamic features for F0.where, in unvoiced region, pt , δpt and δ 2 pt are defined as a discrete symbol. Whendynamic features at the boundary between voiced and unvoiced can not be calcu-lated, they are defined as a discrete symbol. For example, if dynamic features are pcalculated by Eq.(5.3)(5.4), δpt and δt at the boundary between voiced and unvoicedas shown Fig. 5.1 become discrete symbol.5.2 Spectrum and F0 modelingIn the chapter 3, it is described that sequence of mel-cepstral coefficient vector andF0 pattern are modeled by a continuous density HMM and multi-space probabilitydistribution HMM, respectively.We construct spectrum and F0 models by using embedded training because theembedded training does not need label boundaries when appropriate initial modelsare available. However, if spectrum models and F0 models are embedded-trainedseparately, speech segmentations may be discrepant between them.To avoid this problem, context dependent HMMs are trained with feature vectorwhich consists of spectrum, F0 and their dynamic features (Fig. 5.2). As a result,HMM has four streams as shown in Fig. 5.3. 40
  52. 52. c Stream 1 ∆c Continuous probability distribution(Spectrum part) 2 ∆ c p Multi−space probability distribution Stream 2 δp Multi−space probability distribution (F0 part) 2 δ p Multi−space probability distribution Figure 5.2: Feature vector. Mel-cepstrum N-dimensional Gaussian distribution Voiced Unvoiced 1-dimensional Gaussian Voiced distribution F0 Unvoiced 0-dimensional Gaussian Voiced distribution Unvoiced Figure 5.3: structure of HMM. 41
  53. 53. 5.3 Duration modeling5.3.1 OverviewThere have been proposed techniques for training HMMs and their state durationdensities simultaneously (e.g., [30]). However, these techniques require a large stor-age and computational load. In this thesis, state duration densities are estimatedby using state occupancy probabilities which are obtained in the last iteration ofembedded re-estimation [31].In the HMM-based speech synthesis system described above, state duration densitieswere modeled by single Gaussian distributions estimated from histograms of statedurations which were obtained by the Viterbi segmentation of training data. In thisprocedure, however, it is impossible to obtain variances of distributions for phonemeswhich appear only once in the training data.In this thesis, to overcome this problem, Gaussian distributions of state durationsare calculated on the trellis(Section 3.1.2) which is made in the embedded trainingstage. State durations of each phoneme HMM are regarded as a multi-dimensionalobservation, and the set of state durations of each phoneme HMM is modeled bya multi-dimensional Gaussian distribution. Dimension of state duration densities isequal to number of state of HMMs, and nth dimension of state duration densitiesis corresponding to nth state of HMMs1 . Since state durations are modeled bycontinuous distributions, our approach has the following advantages: • The speaking rate of synthetic speech can be varied easily. • There is no need for label boundaries when appropriate initial models are available since the state duration densities are estimated in the embedded training stage of phoneme HMMs.In the following sections, we describe training and clustering of state duration mod-els, and determination of state duration in the synthesis part.5.3.2 Training of state duration modelsThere have been proposed techniques for training HMMs and their state durationdensities simultaneously, however, these techniques is inefficient because it requireshuge storage and computational load. From this point of view, we adopt anothertechnique for training state duration models. 1 We assume the left-to-right model with no skip. 42
  54. 54. State duration densities are estimated on the trellis which is obtained in the em-bedded training stage. The mean ξ(i) and the variance σ2 (i) of duration density ofstate i are determined by T T χt0 ,t1 (i)(t1 − t0 + 1) t0 =1 t1 =t0 ξ(i) = T T , (5.5) χt0 ,t1 (i) t0 =1 t1 =t0 T T χt0 ,t1 (i)(t1 − t0 + 1)2 t0 =1 t1 =t0 σ 2 (i) = T T − ξ 2 (i), (5.6) χt0 ,t1 (i) t0 =1 t1 =t0respectively, where χt0 ,t1 (i) is the probability of occupying state i from time t0 to t1and can be written as t1 χt0 ,t1 (i) = (1 − γt0 −1 (i)) · γt (i) · (1 − γt1 +1 (i)), (5.7) t=t0where γt (i) is the occupation probability of state i at time t, and we define γ−1 (i) =γT +1(i) = 0.5.4 Context dependent model5.4.1 Contextual factorsThere are many contextual factors (e.g., phone identity factors, stress-related factors,locational factors) that affect spectrum, F0 and duration. In this thesis, followingcontextual factors are taken into account: • mora2 count of sentence • position of breath group in sentence • mora count of {preceding, current, succeeding} breath group • position of current accentual phrase in current breath group • mora count and accent type of {preceding, current, succeeding} accentual phrase 2 A mora is a syllable-sized unit in Japanese. 43

×