Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EEGsynth: a tool for brain-controlled sound synthesis and performance

33 views

Published on

Presentation of the EEGsynth for IRCAM, Paris, with a focus on commonality between sound synthesis and EEG analysis

Published in: Technology
  • Be the first to comment

  • Be the first to like this

EEGsynth: a tool for brain-controlled sound synthesis and performance

  1. 1. EEGsynth: a tool for brain-controlled sound synthesis and performance Stephen Whitmarsh Laboratoire de Neurosciences Cognitives Département d'Études Cognitives École Normale Supérieure 12-12-2017 IRCAM www.stephenwhitmarsh.com www.oneplusoneisthree.org www.eegsynth.org
  2. 2. Decomposition / Analysis
  3. 3. Decomposition / Analysis Composition / Synthesis
  4. 4. Decomposition / Analysis Composition / Synthesis
  5. 5. Motor cortex Somatosensory cortex Visual cortex
  6. 6. pre-synaptic potential ~ 2 ms
  7. 7. pre-synaptic potential post-synaptic potential ~ 2 ms ~ 10 ms
  8. 8. x10.000
  9. 9. A-Ref A Ref
  10. 10. A-Ref A Ref Time (ms) microvolts
  11. 11. A-Ref A Ref A
  12. 12. B-Ref Ref A B
  13. 13. C-Ref Ref A B C
  14. 14. Raw EEG A-Ref A Ref
  15. 15. Raw EEG Frequency Decomposition A-Ref A Ref
  16. 16. Raw EEG Frequency Decomposition A-Ref A Ref Amplitude envelopes (ADSR)
  17. 17. Raw EEG Frequency Decomposition Phase-power modulation (AM) Amplitude envelopes (ADSR) A-Ref A Ref
  18. 18. Raw EEG Phase-coupling (sync) B A Ref
  19. 19. Raw EEG Phase-power modulation (AM) A Ref B Phase-coupling (sync)
  20. 20. Raw EEG Power-Power correlation B A Ref Phase-power modulation (AM) Phase-coupling (sync)
  21. 21. Spike rate modulation (FM) Phase-coupling (sync)
  22. 22. pre-synaptic potential post-synaptic potential Spike rate modulation (FM)
  23. 23. pre-synaptic potential post-synaptic potential Spike rate modulation (FM) Population encoding (stochastics)
  24. 24. Modularity in synthesizers • Creativity in patching • Multiple uses of same module • Compatibility: CV/gate • Distributed development: Eurorack
  25. 25. Modularity in Mind (Fodor, 1983) • Domain-Specific • Informational Encapsulation • Obligatory firing • Fast speed, Innate Specification • Shallow outputs (Simple) • Characteristic Development • Characteristic Breakdown • Fixed Neural Architecture Modularity in synthesizers • Creativity in patching • Multiple uses of same module • Compatibility: CV/gate • Distributed development: Eurorack
  26. 26. Van Essen & Gallant (1994) Neuron
  27. 27. Modularity in Mind (Fodor, 1983) • Domain-Specific • Informational Encapsulation • Obligatory firing • Fast speed, Innate Specification • Shallow outputs (Simple) • Characteristic Development • Characteristic Breakdown • Fixed Neural Architecture Modularity in synthesizers • Creativity in patching • Multiple uses of same module • Compatibility: CV/gate • Distributed development: Eurorack Modularity in the EEGsynth • All of the above! • Compatibility: Redis • Distributed development: Python & Github
  28. 28. Public domain Electrophysiology Analogue synthesis EEGsynth hardware prototype
  29. 29. CV/gate MIDI OSCArtNet MIDI keyboard [Volca Beats] Pulsegenerator [Volca Bass] [Volca Keys] Playback Ctrl Sequencer Audio [Avmixer] [OpenBCI] FieldTrip Buffer Rec. & Play Muscle Brain Heartrate Accelerometer Eyeblink DMX Postprocessing Preprocessing Quantizer[Launchcontrol] Redis Recording Analysis Live Control Transformation Output to Hardware & Software EEGsynth
  30. 30. EEGsynth philosophy • Inspired by analogues between EEG analysis and sound synthesis • Principles of time-series analysis vs. synthesis provide common ‘language’ • Common modularity of brain, mind, and analogue synthesizers • Circular causality and emergent properties • Open-source development • Teaching • Center for Interdisciplinary Research (CRI) • Workshop IRCAM, Music Hacking, November 8-10 • Performance • 1+1=3
  31. 31. oneplusoneisthree.org
  32. 32. α brain oscillations ~ 1 second ~ 10Hz
  33. 33. Adrian & Matthews (1934) The Berger rhythm: potential changes from the occipital lobes in man. Brain Hans Berger (1873 – 1941)
  34. 34. Adrian & Matthews (1934) The Berger rhythm: potential changes from the occipital lobes in man. Brain
  35. 35. Adrian & Matthews (1934) The Berger rhythm: potential changes from the occipital lobes in man. Brain
  36. 36. Alvin Lucier, Music for Solo Performer (orig. 1964)
  37. 37. David Rosenboom preparing: Ecology of the Skin Automation House, NYC, 1970
  38. 38. Adapted from Jensen & Mazaheri (2010) Frontiers Neurosci. a b c
  39. 39. a b Adapted from Jensen & Mazaheri (2010) Frontiers Neurosci. c
  40. 40. Adapted from Jensen & Mazaheri (2010) Frontiers Neurosci. a b c
  41. 41. Samaha, Sprague & Postle (2016) Journal of Cognitive Neuroscience
  42. 42. Neuro-feedback
  43. 43. Neuro-feedback
  44. 44. The Science of Consciousness conference, Tucson, USA, April 2016 Selen Atasoy
  45. 45. The Science of Consciousness conference, Tucson, USA, April 2016
  46. 46. X H audienceaudience speaker speaker Performer Hypnotist Musician Scientist
  47. 47. The Science of Consciousness conference, April 2015 5 Car Garage, LA, CA, April 2016
  48. 48. Reference
  49. 49. Huhta home studio, December 2015
  50. 50. The Ship In The Woods, Escondido, CA, USA – April 20
  51. 51. www.stephenwhitmarsh.com  publications
  52. 52. proto-notes proto-durations www.stephenwhitmarsh.com  publications
  53. 53. www.stephenwhitmarsh.com  publications original rearranged
  54. 54. www.danieladepaulis.com www.eegsynth.org
  55. 55. Sound wave (44.1kHz) Interstellar Radio transmission (GHz) gTec EEG headset + Oculus Rift
  56. 56. ASTRON Dwingelo Radio Telescope Radio transmission simulation Operator room
  57. 57. Look into the future • Just getting started: developed in our free time, out of personal interest • Three commissioned performances this summer/fall • Open to a growing community of brain-music hacking (Nov. 8-10 @ IRCAM) • Personally: very excited about synthesis & composition! ArtNeuroscience Knowledge Creativity Performance Community
  58. 58. Special thanks to: Core team Robert Oostenveld Per Huttner Jean-Louis Huhta Hypnotist LA Marco Luytens Brain Control Club CRI Hernan Anllo Ignacio Rebollo Liburn Jupolli COGITO Daniela de Paulis Guillaume Dumas Robert Oostenveld ASTRON Dancer / Choreographer Carima Neusser The Ship in the Woods RJ Brooks Will Reid Lianne Thompson Mueller Dan Fauchier Lidia Rossner Science of Consciousness Conference Abi Behar Montefiore Institutions Vision Forum (SWE) Ecole Normale Superieure (FR) Donders Center for Cognitive Neuroimaging (NL) Karolinska Institutet (SWE) Past sponsors Swedish Arts Grant Council Innovativ Kultur Kulturbryggan Stockholm County Council

×