Successfully reported this slideshow.
Your SlideShare is downloading. ×

Making machines that make music

Upcoming SlideShare
Carnatic Music Synthesis
Carnatic Music Synthesis
Loading in …3

Check these out next

1 of 87 Ad

More Related Content

Recently uploaded (20)


Making machines that make music

  1. 1. Making Machines that Make Music Srihari Sriraman nilenso
  2. 2. should we listen to some now?
  3. 3. Why I do this I Sing, I do computers Bleeding edge research Dreamy ambitions Interesting By-products
  4. 4. What this talk is about Melody Modelling Synthesis Generation Melody Modelling SynthesisGeneration
  5. 5. Melody ˈmɛlədi noun a sequence of single notes that is musically satisfying; a tune.
  6. 6. Carnatic Music
  7. 7. Carnatic Music Kalyani, Extempore MS Gopalakrishnan, Violin
  8. 8. Khamas, Thillana Abhishek Raghuram
  9. 9. Carnatic Music South Indian classical music Ragas, Gamakams Vocal Tradition Rich in compositions Extempore / Manodharma
  10. 10. Tanpura Veena Mridangam
  11. 11. The foundations of musical abstractions in Carnatic music
  12. 12. Shruthi Tonic note Choice of artist Swarams are relative to this
  13. 13. Laya Rhythm concepts Similar to time signatures A rather mature system
  14. 14. Sa Ri Ga Ma Pa Da SaNi Swarams
  15. 15. Sa Ri Ga Ma Pa Da SaNi S R G M P D SN R1 R2 R3 G1 G2 G3 M1 M2 D1 D2 D3 N1 N2 N3 Notation Pronunciation Variations
  16. 16. Swarams The 12 semitones Elements of a raga Simples are sung Prescriptive notation
  17. 17. Rāgā Kaapi, Extempore TM Krishna
  18. 18. Raga Has a name Rule to ascend Rule to descend Not necessarily symmetric Not necessarily linear Grouped into families
  19. 19. Raga Has a name Rule to ascend Rule to descend Not necessarily symmetric Not necessarily linear Grouped into families
  20. 20. Demo of the fundamental abstractions.
  21. 21. Tools & Libraries
  22. 22. Fuzzy search For indic languages Needs to be fast Primary stitching mechanism Helps with multi-source data
  23. 23. A quick recap Play the scale of a raga Fuzzy find a raga Play a phrase Play a phrase in the context of a raga Play some prescriptive notation
  24. 24. But.. ..that doesn’t sound like Carnatic music, does it?
  25. 25. Synthesis
  26. 26. Enter Melographs Me Machine
  27. 27. another phrase Me Machine
  28. 28. Prescriptive vs Descriptive
  29. 29. Gamakams
  30. 30. Sphuritam Orikai Jaaru Kampitam Sphuritam Nokku Ravai Kandippu Ullasitam Etra-jaru Iraka-jaru Odukkal Orikai Vali Kampitam
  31. 31. Gamakams in SSP
  32. 32. Gamakams in SSP
  33. 33. Gaayaka | S, N D | N S R G | ((P S,,)) , ((S , S>>> S)) -((D. S. D)) ((S , S>> S))- S R ((G<< G , ,)) Subramanian, 2009 Database of phrases Automatic Gamakam feature – guided
  34. 34. Modelling Gamakams
  35. 35. Me Machine Back to this…
  36. 36. PASR Srikumar 2013 Pitch, Attack, Sustain, Release Vector specifies the PASR vars for each prescriptive note
  37. 37. Me Machine Rendering PASR…
  38. 38. Rendering PASR…
  39. 39. Generation
  40. 40. Random | Within a raga
  41. 41. Random | Within a raga
  42. 42. Get data
  43. 43. Kosha An Open Carnatic Music Database
  44. 44. Study data
  45. 45. Melographs
  46. 46. Melographs kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram
  47. 47. Pitch Histograms
  48. 48. Pitch Histograms kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg
  49. 49. Pitch Histograms Kalyani - Vocal Kalyani - Violin Mohana - MandolinMohana - Vocal Revati - Vocal Revati - Instrumental
  50. 50. Extract Music Information
  51. 51. Midi Histogram Normalised Midi Histogram
  52. 52. Tonic note identification Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012) A knowledge based signal processing approach to tonic identification in indian classical music.
 Bellur, A., and H. A. Murthy (2013) Automatic tonic identification in classical music using melodic characteristics and tuning of the drone. Srihari, S. (2016) * Pick the most frequent note, it mostly just works. * not really, no
  53. 53. Tonic note identification
  54. 54. Tonic note identification
  55. 55. Swaram Histogram Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S
  56. 56. Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S Revati S, R1, M1, P, N2, S. S., N2, P, M1, R1, S Mohana S, R2, G3, P, D2, S. S., D2, P, G3, R2, S
  57. 57. Generation with weighted probabilities
  58. 58. In comparison with random Random Single Swaram Weighted
  59. 59. Melody insights #1 Tonic note is prominent Sa, and Pa have higher and sharper peaks Other note peaks are blunt Probabilities of all swarams in a raga are not the same Probabilities across octaves are not the same
  60. 60. Two swaram probabilities
  61. 61. Two swaram probabilities Prominence of adjacency Encoded rules of Arohanam and Avarohanam
  62. 62. Two swaram probabilities
  63. 63. Single swaram vs Two swarams Two Swaram Weighted Single Swaram Weighted
  64. 64. Melody insights #2 Swarams close to each other are more melodious The rules of Arohanam, Avarohanam are encoded We begin to see gamakams Sometimes, the in-between is worse than either extreme
  65. 65. Three swaram probabilities and more A simple markov chain
  66. 66. First Order Matrix Markov Chains in Music Second Order Matrix
  67. 67. Markov Chains in Music
  68. 68. Melody insights #3 Generic markov chains don’t really work LSTMs also don’t work, probably
  69. 69. By-products
  70. 70. Automatic Transcription
  71. 71. Automatic Transcription (:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1 :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 : ..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1 :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3 :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n 3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1 :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g 1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3 :..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)
  72. 72. Raga Identification
  73. 73. Goodness of fit test
  74. 74. :base mohanam-base :samples mohanam-files (12.39 3.84 11.14 6.46 9.88 7.02 9.41 12.61 13.22 1.58) :base mohanam-base :samples kalyani-files (10.95 28.66 25.61 15.26 27.32 21.53 16.42 18.58 24.80 23.80) :base mohanam-base :samples revati-files (46.56 57.19 65.69 55.21 38.61 78.10 56.27 42.99 70.92 58.39)
  75. 75. Raga Identification Revati sample vs Revati base Mohana sample vs Revati base
  76. 76. What next Model insights as melodic abstractions Use synthesis models with generative music Experiment with Rhythm Synthesise Human Voice Deep learning (Recurrent variational auto encoders)
  77. 77. Is this music though?
  78. 78. Behag Dasarapada Abhishek Raghuram
  79. 79. Making Machines that Make Music Srihari Sriraman nilenso