The document discusses unit selection text-to-speech synthesis in MaryTTS. It provides an overview of unit selection TTS and the MaryTTS implementation. The MaryTTS implementation is modularized with unit selection packages divided into groups like base classes, data, selection/cost functions, and weighting functions. It describes the unit selection process in MaryTTS step-by-step, including loading a voice's properties, selecting units with a Viterbi algorithm, and concatenating units. Finally, it asks how the system could be improved, such as restructuring the codebase.

1. Unit Selection Synthesis in
MaryTTS

2. Overview
● Unit Selection Synthesis
● MaryTTS implementation/structure
● How can we improve it?

3. TTS – Unit Selection (U.S.)
Input text NLP tasks
Unit
Selection
Speech
Database
Speech

4. TTS – Unit Selection (U.S.)
● Input text is processed and split into tokens.
● The best possible candidates for each token
are then selected from the speech database.
● The desired target utterance is then created by
determining the best chain of these candidate
units and concatenating them together →
Viterbi algorithm(using join and target costs).

5. MaryTTS implementation
● MaryTTS is a “modularised” system
● U.S. packages are currently embedded within
the marytts-runtime module.

6. MaryTTS implementation
● These packages are divided into the following groups:
- Base classes (marytts.unitselection)
- Data (marytts.unitselection.data)
- Selection & Cost Functions (marytts.unitselection.select)
- Viterbi (marytts.unitselection.select.viterbi)
- Weighting Functions
(marytts.unitselection.weightingfunctions)
- Target Features (marytts.features)
- Voice Properties (marytts.server)

7. MaryTTS Implementation
UML Schema of Unit Selection Packages in MaryTTS - woah!
***Only modelling the Unit selection part...concatenation excluded

8. MaryTTS U.S. step-by-step:
● A request is made to the mary server to output
audio from some input using a unit selection
voice.
● The Synthesis module is called to process the
input data → calls voice.synthesize()
● In this case, the unit selection voice (extension
of voice class) calls the unit selection
synthesizer.

9. MaryTTS U.S. step-by-step:
UnitSelectionSynthesizer.synthesize(tokens, UnitSelVoice):
→ Processes the tokens into audio by calling on the voice's
database, unit selector and concatenator.
– UnitSelectionVoice loads these objects by reading in
properties from the voice's .config file using MaryProperties
– The UnitDatabase class contains the target and join cost
functions, as well as a way to access the speech database
to retrieve target candidates etc.
– Unit selector selects the units
– Unit Concatenator concatenates these units into a single
audio stream

10. Unit Selector
● Contains a reference to the voice's database
● .selectUnits(tokens, voice):
- tokens converted to targets
- target feature vectors computed for each
target
- viterbi algorithm applied to find best path

11. How can we improve the system?
● Restructuring of codebase?