Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Intro for project Meilin and linne platform

462 views

Published on

open source virtual singer(or 'v'ocaloid)

Published in: Technology
  • Be the first to comment

Intro for project Meilin and linne platform

  1. 1. Virtual singer 徴音梅林 & LINNE CC-BY-NC
  2. 2. Slide author ● 張正一 (Chou Shouichi)/ MGdesigner ● Paul Liu and me organize 徴音梅林開発委員會 ● Wikimedia.tw: member of board of directors (and direct tech development ) ● A programmer ● A musician (Jazz ukulele, DTM) ● Shoichi.chou@gmail.com
  3. 3. Everyone knows her
  4. 4. Powered by Yamaha Vocaloid2 engine
  5. 5. So Why a FOSS 'v'ocaloid?
  6. 6. If you buy an instrument You can play any song, Do anything.
  7. 7. play
  8. 8. By teeth
  9. 9. Break !
  10. 10. burn
  11. 11. In any Vocaloid product EULA You didn't get whole rights ● no “anti-society” (反社会) works (so,What works are “anti-society”?) ● Trademarks protection (images, keywords) (ex: 'Vocaloid' ,' 初音ミク ',' 初音ミク ''s image)
  12. 12. No using Miku images=not popular musicians are controlled No freedom
  13. 13. Be ruled ● Using a Gibson guitar,you are its master. ● Using Vocaloid products, You are their slave.
  14. 14. INDIE DIE
  15. 15. UTAU ?
  16. 16. A free vocaloid-like ● DIY a 『 vocaloid 』 ● Programs: editor(frontend)+resampler+wavtool ● data: vocal DB - oto.ini + wav samples ● Vocal DB is an open spec ,many people DIY
  17. 17. vocaloid programs working flow 1.Editor: compose the melody(many notes) 2.Resampler: modulate a sample to Specified pitch,or other parameters (velocity...). 3.Wavtool: combine these modulated wavs Finally,we get a song vocal wav file,and mix into a song
  18. 18. but ● Charge no fee,not freedom ● Default resampler work badly ● DB bad international support (S-JIS) ● Oto.ini no implementing ini comments “;“ ● UTAU always auto sort oto.ini (hard collaboration) ● Hard UI control ● Not open source ● Its development is very private
  19. 19. And you know ... Yamaha owns many super patents
  20. 20. A nice “free vocaloid“ Is Very difficult
  21. 21. During 2011-2012 One day, Paul Liu talked to me ● New Algorithm, 'World' better than Vocaloid2 ● Author: 日本山梨大學 Doctor 森勢将雅 ● Patent free ● EFB-GW(Synthesizer) for UTAU ● Open source(old version GPL,newer is BSD) ● https://github.com/mmorise/World ● During Dec,Dr. 森勢 'll do another great upgrade
  22. 22. How good is World algorithm? ● very awesome 'autotune' (original official test is a realtime Karaoke autotune for 音痴 s. ) ● Modulate a sample to any pitch without distortion (Keep F0 well) (Vocaloid2 can't ,so Miku need 3 different range versions of each sample) ● Very fast ,no need to pre-preapre frequency tables (Just do it real time) ● If X86, Even works good on older machines(maybe on ARM)
  23. 23. Ok Let's do it!
  24. 24. Finally we made her...
  25. 25. Listen... Hear “MAMA” cover
  26. 26. 徵音梅林 「徵音」 :Ancient Chinese,Japanese Pentatonic scale note. (Do Re Mi Sol La) 宮(きゅう)、商(しょう)、 角(かく)、 徴( Also means We 'recruit' a voice actor(and also a Jazz singer) from Internet 「梅林」: Merlin(super wizard) 林: Linux
  27. 27. http://projectmeilin.github.io/
  28. 28. Project Meilin Features ● CC-BY ● Utau compatible ● Professional recording(in studio) ● Src:24bits 48000hz wavs ● VCV 連続音 、 VC 単独音 (V - Vowel c - Consonant) ● Recorded: Japanese,Mandarin(Taiwan style)
  29. 29. How good? A test ● Commercial Miku VS. open content Meilin ● V2 Miku each sample recorded high,middle,low versions VS. Meilin each sample just record 1 version.
  30. 30. Listen to the comparing video… ( song: 歌い手様総合テスト , Start from 0:44) Especially check super low pitch and super high pitch if is distorted (失真) ?
  31. 31. fact Miku DB: ● 1gb+ ● Only Japanese Meilin DB: ● 627mb ● Japanese+Mardarin ● Mardarlin DB is 3 倍 of JP DB
  32. 32. thank to Dr. 森勢 Without his effort and kindness, a good FOSS virtual singer is imposible
  33. 33. 2 more Special features
  34. 34. 1: 14 Special effects Defined in oto.ini ● 3 breath : br1,br2,br3 ( ex:Miku only have these breath. ) ● Spanish 'R' rolling: trill ● Cough: cough ● Cry,dry tears:drytears ● Blownose: blownose ● Sucking: suck ● sigh( 嘆 ):sgn1,sgn2,sgn3,sgn4 ● Whistle :whsl ● clean throat: clnt
  35. 35. 2: 日本方言 possible ● EX: 円唇母音'う' in 関西弁 (video) ● in Mandarin ,there is the same 'u' ● Just borrow what we recorded. ● also can borrow other Mandarin samples for synthesizing 方言 or some foreign languages. (ex: 1 or 2 foreign lyrics in a Japanese song)
  36. 36. 'v'ocaloid also can do speech synthesis Better than traditional speech synthesis ● Accent(= pitch,velocity,rhythm,speed) controllable ● Could do many emotion(melody lines) : cry,angry... ● TTS,story telling,emotional ' うかがか' possible ● Some tests which I have done by Miku: 1,2,3,4 based on my scale algorithm. 'Auto render' possible,but…. ● If use Vocaloid to do this,you need to beg YAMAHA for opening API. But our software stack are open source. She could do more than singing.
  37. 37. How made?
  38. 38. Recorded in a pro studio
  39. 39. Thanks to sponsor 阿怪 (Aguai),my master (A famous POP song producer in TW.)
  40. 40. About the vocal ● Her name is 羅竺 (Lo Chu). ● We choose her voice from 20 girls from on internet. ● She is a singer in a JAZZ / anime cover song band. ● Also vocal acting trained. ● Japanese accent not bad. ● Japanese friend ATsushi 發音指導
  41. 41. But very hard work Japanese recording need 3~4 hours. But Intact Madarin(possibility on math ,then minus repeated samples by Phonology) Madarin recording needs days.
  42. 42. The final day
  43. 43. LINNE platform ● We defined the FOSS 'v'ocaloid stack ● Of course opensource ● Compatible with Utau DB (but UTF-8) ● resampler+wavtool+editor(interface)+DB -making tools ● May include 'hardware'
  44. 44. Hardware Ex: Doll robot
  45. 45. Our Oto.ini DB spec ● You can use ';' for comments ● Editors programs shouldn't resort the file ● UTF-8 ● IPA based (International Phonetic Alphabet) ● By IPA,Different languages could use common pronunciation samples (no more re-recording again, keep the DB size smaller, more storage efficiency )
  46. 46. Engine (now is xvsqExec ,may need to be changed) Jcadencii Linne-editor (in dev) (song editor,front end) Wavtool-pl (GPL wavtool) tn_fnds_yc (gpl) (resampler,EFB-GW variant ) World lib Other programs in the future ex: linne-TTS
  47. 47. The chart may need evolution.
  48. 48. Problem now: the editor(frontend) ● Cadencii is written by .net with binding too many Windows native calls ● Jcadencii is very slow (Cadencii java port) ● Upstream dev stopped. We also give it up. ● Another open Utau frontend: http://fluidvocalsynth.weebly.com/ (also .Net)
  49. 49. ●linne-editor(frontend) ● https://github.com/marty1885/linne-editor ● In very earily development
  50. 50. fact We don't have enough manpower about interface coding… ● When normal users edit, still need wine+Utau ● Similar to early Linux dev in Minix >_<
  51. 51. Wish more people join Make LINNE More complete!
  52. 52. 徴音梅林開発委員会 ● Open source community ● OSS programmers,musicians,a physicist,Phonologists,artists... ● Members are international(TW,JP)
  53. 53. welcome ● Official Site ● Github: https://github.com/ProjectMeilin/ ● Slack (tech talk): https://meilin.slack.com/ (email me for invitation : shoichi.chou@gmail.com ) ● FB fan page ● FB group (more about DB making and musician) ● Youtube channel
  54. 54. Q&A Listen another songs? :P Hackasong.JP Umbrella カバー END

×