A wearable MIDI interface using a wireless sensor network


Published on

The idea behind this project is to use a wearable WSN to produce music. The user wears a certain number of sensors, sampling accelerometric data: these data are mapped into MIDI messages, that are routed toward a musical software. The WSN is seen as a standard input MIDI peripheral.
The platform allows new expressive forms based on gestuality: using more sensors placed on the body (with an appropriate mapping), it could be possible to link different forms of expressivity, such dance and music.

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A wearable MIDI interface using a wireless sensor network

  1. 1. A WEARABLE MUSICALINTERFACE USING A WIRELESSSENSOR NETWORKFrancesco Corucci – – 2012MSc in Computer Engineering – University of Pisa
  2. 2. Introduction Goal: implement a platform that allows the use of awireless sensor network as a wearable interfacethat can be used for producing music Requirements: Modular Flexible MIDI-compliant2
  3. 3. What is MIDI MIDI (Musical Instrument Digital Interface) is astandard technology for the interaction betweenmusical devices It standardizes: A communication protocol An hardware interface Versatile technology, widely used since the 80’s and stillreliable after more than 30 years3
  4. 4. Shimmer sensor nodes For setting up the WSN we choose Shimmer sensornodes : Suitable for wearable applications Integrate a 3-axis accelerometer Optional gyroscope shield(not used in this project) Bluetooth or 802.15.4communication Support TinyOS4
  5. 5. Shimmer sensor nodes5
  6. 6. Overview of the systemUSB802.15.4• Two or more wireless sensors on the body• The sensors sample accelerometric dataassociated with movements• Data are then collected from the sink andforwarded to the computerSink6
  7. 7. Overview of the system (cont’d)Oscilloscope application(extended with MIDI map module)Virtual MIDIdeviceMusical softwareData from sink• Filtering• Displaying• Mapping in MIDI commands7
  8. 8. Detailed architectureSerial forwarderComputerUSBTCP/IPOscilloscopeRadioMIDI mapMIDI forwarderVirtual MIDI portTCP/IPMusical sw(VST-host)VSTPluginVSTPluginMIDIMIDI8…
  9. 9. MIDI controllerThis way the WSN become a standard MIDI controller,that can interoperate with every musical software9
  10. 10. How to choose the mapping The way to map sensed data into MIDI messages isabsolutely arbitrary, and can lead to a multitude ofexpressive results Every data channel can be mapped in a different instrument effect note/chord etc… …and this can be done in a variety of ways!10
  11. 11. Demonstration time(http://youtu.be/jIrE5j2Mdtw)11
  12. 12. Features Modular, extensible: Is it possible to use an arbitrary number of sensor nodes, andchoose which on-board sensors must be used as data sources Flexible: Many degrees of freedom in the mapping Many expressive possibilities Sound generation is left to specialized and mature softwares Standard: Since the WSN becomes a MIDI controller, it is able tointeroperate with a moltitude of software platforms12
  13. 13. Possible applicationsThe platform can be used for several purposes: New expressive forms based on gestuality In combination with traditional instruments(e.g. influence of on-stage movements on played music) Any application that asks for sonification of human movements: Sport training Rehabilitation …13
  14. 14. References Shimmer sensor nodes http://www.shimmer-research.com/ General information about MIDI http://en.wikipedia.org/wiki/MIDI VST http://en.wikipedia.org/wiki/Virtual_Studio_Technology rtMidi, a cross-platform C++ library for MIDI http://www.music.mcgill.ca/~gary/rtmidi/ Sonification http://en.wikipedia.org/wiki/Sonification14
  15. 15. Thank you for your attention :-)15
  16. 16. Implementation details16
  17. 17. OscilloscopeAppC - overview OscilloscopeAppC is the TinyOS application runningon sensor nodes Receives commands from the sink:start/stop/change_freq Samples the on-board tri-axis accelerometer usingthree ADC channels in DMA mode When a block of samples has been collected, thesensor node sends it to the sink via radio17
  18. 18. OscilloscopeAppC - architecture18
  19. 19. OscilloscopeAppC - interactions19…
  20. 20. BasestationC - overview BasestationC is the TinyOS application running onthe sink node Acts as a bridge between the serial and the radiolink Implements queues in both direction to handletraffic spikes20
  21. 21. BasestationC - architecture21
  22. 22. MIDI map module Offers a simple GUI: the user can choose how to mapthe active data channels into MIDI messages from apredefined set of mappings22 The module then catchesincoming data, maps it intoMIDI command accordingto user-definedpreferences, and forwardsthem via TCP/IP sockettoward the MIDI forwarder
  23. 23. MIDI forwarder Implements a virtualMIDI device: the WSN isseen as an input MIDIperipheral plugged intothe computer It then acts as a bridge: Waits for incoming MIDImessages from a TCP/IPsocket Forwards MIDI messagesto the virtual MIDI port23
  24. 24. Screenshots: Oscilloscope + MidiMap24
  25. 25. Screenshots25