A local virtual signer project, LINNE, is proposed several years ago. However, to process a huge amount of sound-bank data is big problem. Here we make use of the python tool lib., PyMIR and SciKit-Learn, to help us extract the necessary information that needed for a song synthesizer, ex. UTAU.

1. Project Linne 徵音梅林
–– Virtual Singer Sound-bankVirtual Singer Sound-bank
Processing with PythonProcessing with Python
Yuan CHAO ( 趙元 )
PyCon TW
2016/06/03-05
ㄓ ˇ

2. Who am I ？
Yuan CHAO (John)
YChao
...

3. A researcher
working on HEP
using OSS...

4. No physics today
Let's talk about
Virtual Singer (VS)...

5. Familiar with TTS?
(Text-to-Speech synth.)
Siri, Ok Google

6. Virtual Singer –
Note-to-Vocal synth.

7. YAMAHA Vocaloid™
初音未來

8. Sound-bank for VS?

9. Sound bank
From commercial company
or volunteers
Editor
Notes and Lyrics
Vocal Synth.
Synthesized
song
Block diagram for
a VS system
Songs

10. 飴屋 P - UTAU の基本的アルゴリズムと開発経緯
http://udn.utau-synth.com/documents/kouen/20120325/
Step 1: cut the recored sounds through into sound
elements (phonons)

11. 飴屋 P - UTAU の基本的アルゴリズムと開発経緯
http://udn.utau-synth.com/documents/kouen/20120325/
Step 2: connect the elements following the lyrics

12. 飴屋 P - UTAU の基本的アルゴリズムと開発経緯
http://udn.utau-synth.com/documents/kouen/20120325/
Step 3: Adjust the pitches and lengths of the lyrics

13. Sound-bank Parameters

14. Consonants / Vowel
子音 / 母音
聲母 / 韻母
(fixed / variable length)

15. Start
Beat matching
position
Beginning of Vowel
Consonants
End
Next sound
Sound Parameters

16. Sound bank for Japanese

17. The 50 Sounds
五十音

18. Japanese
50 sounds x 2
(voiced/unvoiced)
( 清濁 )

19. Japanese
Fifty sounds x 2
(voiced/unvoiced) +
Half-voiced, palatalized...
拗音、半濁音及其他輔音

20. Japanese
Fifty sounds x 2 +
others
~150 basic sounds

21. Japanese
with connected vowels
~150 sounds x 6

22. Japanese
With connected vowels
~150 sounds x 6
Total ~ 1000 sounds

23. Japanese
With connected vowels
~150 sounds x 6
Total ~ 1000 sounds
(~10 samples/hr. for well trained people)

24. Chinese

25. Chinese
If counted with bopomofo
ㄅㄆㄇㄈ

26. Chinese
21 consonants, 16 vowels
聲母 21 個、韻母 16 個

27. Chinese
All possible sound
combinations
~ 450

28. Chinese
with connected vowels
~450 sounds x 9
一ㄨㄩㄚㄛㄜㄝㄦ n ( ㄣㄥㄢㄤ )
( ㄞㄟㄠㄡ )

29. Chinese
With connected vowels
~450 sounds x 9
~4000 sounds

30. https://github.com/benlau/linne-analyzer
http://www.gnu.org/software/octave/
https://github.com/jsawruk/pymir
Analysis framework by
Ben Lau
PyMIR lib
.wav I/O
Feature extraction
GNU Octave
visualization

31. Simple Analyzer
http://guhy.csie.ntust.edu.tw/pap/07_TWN_Mandarin_SingingVoice_Synthesis_BasedOn_ExpressionParameter_Analyzing.pdf
過零率
Zero-cross
rate
頻譜
變異數
Spectrum
variance

32. Simple Analyzer
http://guhy.csie.ntust.edu.tw/pap/07_TWN_Mandarin_SingingVoice_Synthesis_BasedOn_ExpressionParameter_Analyzing.pdf
過零率
Zero-cross
rate
頻譜
變異數
Spectrum
variance

33. Threshold method doesn't
give good results

34. Try in frequency domain

35. Fourier Transformation
https://en.wikipedia.org/wiki/Fourier_transform

36. https://en.wikipedia.org/wiki/Vowel
Spectrum Patterns

37. Time domain vs. Frequency domain

38. Thousands of samples
to be processed...

39. Try with ML tools –
SciKit Learn

40. Sampling with
sliding window

41. Using SVM to determine
the vowel positions
http://www.cmlab.csie.ntu.edu.tw/~cyy/learning/tutorials/SVM3.pdf
https://en.wikipedia.org/wiki/Support_vector_machine
http://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html

42. Using SVM to set vowels
あいうえおん

43. Using SVM to set vowels
ㄚㄛㄜㄝ一ㄨㄩㄦ
ㄢㄣㄤㄥ (n)

44. Supervised learning -
Training sample?
https://github.com/yuanchao/linne-analyzer/blob/vowel_det/src/linne/analyzer/cmd/linne-train2.py

45. Take part of the data as
the training sample –
Data-driven Analysis
https://github.com/yuanchao/linne-analyzer/blob/vowel_det/src/linne/analyzer/cmd/linne-spect2.py

46. N
U
E
O
I
A
N
U
E
O
I
A

47. A-A-I-A-U
N
U
E
O
I
A
https://github.com/yuanchao/linne-analyzer/blob/vowel_det/src/linne/analyzer/cmd/linne-test2.py
Detecting connected vowels
fen-fia-fou-a-fe
ㄈㄣ - ㄈ一ㄚ - ㄈㄡ - ㄚ - ㄈㄜ
N
ㄩ
ㄨ
一
ㄝ
ㄜ
ㄛ
ㄚ
Still some room for
improvements

48. Fork Me on GitHub!
https://github.com/yuanchao/linne-analyzer/tree/vowel_det/src/linne/analyzer/cmd

49. ㄓ ˇ ㄧㄣㄇㄟ ˊ ㄌㄧㄣ ˊ
徵音梅林開發計畫
https://github.com/ProjectMeilin

50. ち おんメイリン
徵音梅林開發計畫
Free and open VS platform
痴音

51. ㄓ ˇ
徵音梅林開發計畫
Software: Paul Liu, MGDesigner,
Ben Lau, Atsushieno, Yuan Chao

52. ㄓ ˇ
徵音梅林開發計畫
Vocal: 羅竺 License: CC-BY
https://www.youtube.com/watch?v=OZNrVq50wEY

53. Welcome

54. 示範曲播放
Live DEMO!!!
https://soundcloud.com/ychao/umbrella-linne2
http://www.nicovideo.jp/watch/sm26831479
https://soundcloud.com/ychao/utau-celluloid-linne-zh

55. 以上
謝謝

56. Remerci de
Votre
Attention