1.
LinnStrument:
the ultimate open-source hacker instrument
Geert Bevin

2.
Who am I?
• Geert Bevin, Twitter @gbevin
• XRebel Product Manager at ZeroTurnaround
• Java Champion, Musician, Composer, Arranger, Producer, Singer,
Guitarist, Gamer, Kung-Fu
• Many open-source projects including Gentoo Linux, OpenLaszlo,
RIFE, Juce, …

3.
Musical instrument hacker

4.
Software for Eigenharp

5.
GECO for Leap Motion

6.
LinnStrument firmware and tools

7.
What is the LinnStrument?

8.
Revolutionary Music Performance
Controller with 3D Note Expression

9.
DEMO

10.
What’s inside LinnStrument?

11.
Front-panel
Chassis + circuit boards
Translucent silicone sheet
Final prototype before production - actual units are slightly different

12.
Metal chassis with wooden sides
Sensor board
Final prototype before production - actual units are slightly different

13.
LEDs board
Final prototype before production - actual units are slightly different

14.
Connection between circuit boards
Final prototype before production - actual units are slightly different

15.
Arduino Due’s
ARM chip Serial <-> USB MIDI Shield
Final prototype before production - actual units are slightly different

16.
What you need to know about
the Arduino Due

17.
ARM Cortex-M3
32-bit 84MHz
CPU
SPI signals
shared by
LED/Sensor
Digital 33-37
Footpedals
DIN <-> USB
LED control
MIDI <-> Serial

18.
Arduino Due and LinnStrument
• 32-bit core, 4 bytes wide operations within single CPU clock
• CPU Clock at 84Mhz
• 96 KBytes of SRAM
• 512 KBytes of Flash memory for code
• Digital I/O pins
• Serial Peripheral Interface (SPI) pins with Slave Select

19.
Very simple Arduino program
// the setup function runs once when you press reset or power the board
void setup() {
// initialize digital pin 13 as an output.
pinMode(13, OUTPUT);
}
// the loop function runs over and over again forever
void loop() {
digitalWrite(13, HIGH); // turn the LED on (HIGH is the voltage level)
delay(1000); // wait for a second
digitalWrite(13, LOW); // turn the LED off by making the voltage LOW
delay(1000); // wait for a second
}

20.
Arduino code
• language based on C/C++
• concise reference with all language structures, values and functions
• Arduino IDE to get started
• the ‘setup’ function runs once, when the board starts up
• the ‘loop’ function runs at 100% CPU, over and over again

21.
Only one execution thread
• what happens in the ‘loop’ function is all that’s happening
• if something takes too long, something else isn’t happening
• guerrilla coding tactics: powerful, short strikes and get out of there
• design your algorithms to be sub-dividable
• avoid dynamic memory allocation, it’s very slow

22.
How to use the Arduino IDE

23.
Source code in editor

24.
Select target board

25.
Select serial port

26.
Compile and Upload

27.
I don’t like the Arduino IDE
• non-HDPI fonts
• extremely basic editor
• bad OS integration
• I use Sublime Text 3 with the Stino plugin

28.
How the LinnStrument hardware
is accessed through software

29.
Easy LED functions
• Change the color and brightness of a single LED
void setLed(byte col, // Column of LED to be changed
byte row, // Row of LED to be changed
byte color, // Color of LED to be changed
byte brightness) // Brightness of LED (0, 1, 2 or 3)
• Light up a single LED with the default color and brightness
void lightLed(byte col, // Column of LED to be changed
byte row ) // Row of LED to be changed
• Clear a single LED
void clearLed(byte col, // Column of LED to be changed
byte row ) // Row of LED to be changed

30.
Details of LED control
• LED control is done through SPI using pin 10, with mode 0, running at 21MHz
SPI.begin(10);
SPI.setDataMode(10, SPI_MODE0);
SPI.setClockDivider(10, 4); // 84MHz divided by 4
• Digital pin 37 is an output and connected to the LED driver chips
pinMode(37, OUTPUT);
• Write 32-bit data structure to SPI to control the LEDs, refreshed every 100μs
digitalWrite(37, HIGH); // enable the outputs of the LED driver chips
SPI.transfer(10, column, SPI_CONTINUE); // send column (left-shifted 2 bits + special bit 7)
SPI.transfer(10, blue, SPI_CONTINUE); // send blue byte
SPI.transfer(10, green, SPI_CONTINUE); // send green byte
SPI.transfer(10, red); // send red byte
digitalWrite(37, LOW); // disable the outputs of the LED driver chips

31.
Easy sensor functions
• Send 16-bit word over SPI to touch sensor to set the analog switches
void selectSensorCell(byte col, // Column used by analog switches
byte row, // Row used
byte switch) // Switch to read X (0), Y (1) or Z (2)
• Read stable raw X value at the current col and row (returns 0-4095)
int readX()
• Read stable raw Y value at the current col and row (returns 0-127)
int readY()
• Read stable raw Z value at the current col and row (returns 0-127, 255 feather)
byte readZ()

32.
Details of touch sensor control
• Touch sensor control is done through SPI using pin 4,
with mode 0, running at 21MHz
SPI.begin(4);
SPI.setDataMode(4, SPI_MODE0);
SPI.setClockDivider(4, 4); // 84MHz divided by 4
• Write 16-bit data to SPI to set analog switches (see ls_sensor.ino)
SPI.transfer(4, lsb, SPI_CONTINUE); // first byte of data structure
SPI.transfer(4, msb); // second byte of data structure

33.
Read touch sensor data
• Touch sensor A/D input is using SPI through pin 52,
with mode 0, running at 21MHz
SPI.begin(52);
SPI.setDataMode(52, SPI_MODE0);
SPI.setClockDivider(52, 4); // 84MHz divided by 4
• Read sensor data
delayUsec(7); // wait for stable current after sensor
// control changes the analog switches
// delay different for each analog switch
byte msb = SPI.transfer(4, 0, SPI_CONTINUE); // first byte of sensor data
byte lsb = SPI.transfer(4, 0); // second byte of sensor data
int raw = (int(msb) << 8 | lsb) >> 2; // pack into int, shift from 16 to 14 bit

34.
Reading the foot pedals
• Done in ls_switches.ino, modify this method to add custom behavior
void handleFootSwitchState(byte whichSwitch, boolean state)
• Digital pin 33 and 34, respectively for left and right foot switches, configured as pull-up
inputs (inverted inputs: high is off, low is on)
pinMode(33, INPUT_PULLUP);
pinMode(34, INPUT_PULLUP);
• Read the digital value of foot pedal states (typically every 20ms)
boolean leftPedalState = digitalRead(33);
boolean rightPedalState = digitalRead(34);

35.
Details of MIDI / Serial - USB / DIN
• Setting digital switches changes the communication methods from the LinnStrument
to the outside world
• Digital pin 35 switches between Serial and MIDI
pinMode(35, OUTPUT);
digitalWrite(35, HIGH); // high switches to Serial input/output
digitalWrite(35, LOW); // low switches to MIDI input/output
• Digital pin 36 switches between USB and DIN connectors
pinMode(36, OUTPUT);
digitalWrite(36, HIGH); // high switches to USB input/output
digitalWrite(36, LOW); // low switches to DIN input/output

36.
That’s all the hardware stuff!

37.
Serial Monitor for debugging

38.
Uncomment line in ls_debug.h
Recompile and upload

39.
Enable Update OS mode
Which essentially turns on Serial instead of MIDI

40.
Click on the Looking Glass icon
To enable Serial Monitor and reset LinnStrument

41.
Column 17 selects the debug level
By default it’s -1, meaning that no debug info is printed

42.
Row 2 selects debug level 0

43.
Row 3 selects debug level 1

44.
Row 4 selects debug level 2

45.
This is printed in the Serial Console
Along with the actual debug messages of corresponding levels

46.
The relevant debug code
if (sensorCol == 17 && sensorRow < 4) {
debugLevel = sensorRow - 1;
DEBUGPRINT((-1,"debugLevel = "));
DEBUGPRINT((-1,debugLevel));
DEBUGPRINT((-1,"n"));
}
void handleNewTouch(byte z) {
DEBUGPRINT((1,"handleNewTouch"));
DEBUGPRINT((1," col="));DEBUGPRINT((1,(int)sensorCol));
DEBUGPRINT((1," row="));DEBUGPRINT((1,(int)sensorRow));
DEBUGPRINT((1," z="));DEBUGPRINT((1,(int)z));
DEBUGPRINT((1,"n"));
void handleTouchRelease() {
DEBUGPRINT((1,"handleTouchRelease"));
DEBUGPRINT((1," col="));DEBUGPRINT((1,(int)sensorCol));
DEBUGPRINT((1," row="));DEBUGPRINT((1,(int)sensorRow));
DEBUGPRINT((1,"n"));

47.
Overview of firmware files

48.
Core Files
• linnstrument-firmware.ino: global data structures, setup and main loop
• ls_displayModes.ino: illuminate LEDs for the different display modes
• ls_handleTouches.ino: driven by main loop, handles touch tracking
• ls_rtos.ino: primitive scheduler calling continuous tasks during delay
• ls_settings.ino: switch behavior from UI, store and recall settings
• ls_touchInfo.ino: encapsulate sensor data into touched cells

49.
Low-level Files
• ls_calibration.ino: calibration procedure and data conversion
• ls_leds.ino: low-level communication with the LEDs
• ls_midi.ino: MIDI input, NRPN control, clock, output and queue
• ls_sensor.ino: low-level touch sensing with bias and curve
• ls_test.ino: debug functions and low-level reports

50.
Auxilliary Features Files
• ls_arpeggiator.ino: arpeggiator logic, tied to internal MIDI clock
• ls_faders.ino: MIDI CC faders touch handling and data sending
• ls_font.ino: tiny, small, and big font display, including scrolling
• ls_lowRow.ino: continuous cell evaluation for low-row features, driven by
the main touch tracking
• ls_noteTouchMapping.ino: track MIDI notes to touched cells, mainly used
by arpeggiator
• ls_switches.ino: handles control switches and foot pedals

51.
Support Header Files
• ls_bytebuffer.h: circular byte buffer with independent push and pop
locations, used by ls_midi.ino output queue
• ls_channelbucket.h: hands out MIDI channels from a bucket of
allowed channel numbers.
• ls_debug.h: debug macros and defines
• ls_midi.h: standard MIDI status codes, used by ls_midi.ino

52.
Useful global functionalities

53.
Global variables
• byte sensorCol: the column number of the current sensor cell
• byte sensorRow: the row number of the current sensor cell
• byte sensorSplit: the split of the current sensor cell (0: left, 1: right)
• DisplayMode displayMode: the active display mode
(see DisplayMode enum in linnstrument-firmware.ino for the values)

54.
Global functions
• TouchInfo &cell(): the last touch data for the current cell
• TouchInfo &cell(byte col, byte row): the last touch data for a cell
• FocusCell &focus(byte split, byte channel): the current cell that has
the focus for a particular split and channel (FocusCell is a structure
that just contains col and row)

55.
Some specific firmware
examples

56.
Per-finger touch-tracking
• With a general purpose CPU, you’d model this as touch ‘sessions’
that are dynamically created and have a life of their own
• Too much memory churn for Arduino, too much book-keeping also
• Instead, have a touch state for each cell
• Transfer data between cells since we don’t support two fingers
touching the same cell

57.
Inside ls_handleTouches.ino
void handleNewTouch(byte z) {
// ... snip ...
// check if the new touch could be an ongoing slide to the right
if (potentialSlideTransferCandidate(sensorCol-1)) {
// if the pressure gets higher than adjacent cell,
// the slide is transitioning over
if (isReadyForSlideTransfer(sensorCol-1)) {
transferFromSameRowCell(sensorCol-1);
handleXYZupdate(z);
}
// otherwise act as if this new touch never happened
else {
cellTouched(transferCell);
}
}
// similar for slide to the left

58.
Check potential slide transfer
boolean potentialSlideTransferCandidate(int col) {
if (col < 1) return false;
if (sensorSplit != getSplitOf(col)) return false;
if (!isLowRow() && (!Split[sensorSplit].sendX ||
!isFocusedCell(col, sensorRow))) return false;
if (isLowRow() && !lowRowRequiresSlideTracking()) return false;
if (isStrummingSplit(sensorSplit)) return false;
// the sibling cell has an active touch
return cell(col, sensorRow).touched != untouchedCell &&
// either a release is pending to be performed, or
(cell(col, sensorRow).pendingReleaseCount ||
// both cells are touched simultaneously on the edges
abs(cell().calibratedX() - cell(col, sensorRow).calibratedX())
< TRANSFER_SLIDE_PROXIMITY);
}

59.
Is the sibling cell ready for the transfer?
boolean isReadyForSlideTransfer(int col) {
// there's a pending release waiting
return cell(col, sensorRow).pendingReleaseCount ||
// the cell pressure is higher
cell().rawZ > cell(col, sensorRow).rawZ;
}

60.
Perform the data transfer 1/3
void transferFromSameRowCell(byte col) {
cell().initialX = cell(col, sensorRow).initialX;
cell().initialReferenceX = cell(col, sensorRow).initialReferenceX;
cell().lastMovedX = cell(col, sensorRow).lastMovedX;
cell().fxdRateX = cell(col, sensorRow).fxdRateX;
cell().rateCountX = cell(col, sensorRow).rateCountX;
cell().initialY = cell(col, sensorRow).initialY;
cell().note = cell(col, sensorRow).note;
cell().channel = cell(col, sensorRow).channel;
cell().fxdPrevPressure = cell(col, sensorRow).fxdPrevPressure;
cell().fxdPrevTimbre = cell(col, sensorRow).fxdPrevTimbre;
cell().velocity = cell(col, sensorRow).velocity;
cell().vcount = cell(col, sensorRow).vcount;
noteTouchMapping[sensorSplit]
.changeCell(cell().note, cell().channel, sensorCol, sensorRow);

61.
Perform the data transfer 2/3
if (cell(col, sensorRow).touched != untouchedCell) {
cell(col, sensorRow).touched = transferCell;
}
cell(col, sensorRow).initialX = -1;
cell(col, sensorRow).initialReferenceX = 0;
cell(col, sensorRow).lastMovedX = 0;
cell(col, sensorRow).fxdRateX = 0;
cell(col, sensorRow).rateCountX = 0;
cell(col, sensorRow).initialY = -1;
cell(col, sensorRow).note = -1;
cell(col, sensorRow).channel = -1;
cell(col, sensorRow).fxdPrevPressure = 0;
cell(col, sensorRow).fxdPrevTimbre = 0;
cell(col, sensorRow).velocity = 0;
cell(col, sensorRow).pendingReleaseCount = 0;
// do not reset vcount!

62.
Perform the data transfer 3/3
// transfer the focus if this was the focused cell
byte channel = cell().channel;
if (channel != -1 && col == focus(sensorSplit, channel).col &&
sensorRow == focus(sensorSplit, channel).row) {
focus(sensorSplit, channel).col = sensorCol;
focus(sensorSplit, channel).row = sensorRow;
}
}

63.
Finding the right MIDI channels
• Note-per-channel distributes up to 16 MIDI channels across notes
• Notes should reuse same channel as late as possible (release trails)
• Intuitively you’d scan all the active notes and determine which channel is
available for a new note, which is again too much overhead
• We use a bucket of available channels, channels bubble up or sink down
• A new note merely has to take the next channel from the top
• Fully encapsulated inside ls_channelbucket.h

64.
Low-row functionalities
• Intuitively you’d detect a touch on low-row cells when it’s active
• Then evaluate state of every other cell and trigger behavior
• This is again too much overhead
• Instead keep track of low-row start/stop in a state machine
• Piggy-back when processing each cell in the main loop to evaluate
appropriate low-row behavior

65.
Inside ls_handleTouches.ino
void handleXYZupdate(byte z) {
// ... snip ...
handleLowRowState(z);
if (isLowRow()) {
if (newVelocity) {
lowRowStart();
}
return;
}
// ... snip ...
}
void handleTouchRelease() {
// ... snip ...
if (isLowRow()) {
lowRowStop();
}
// ... snip ...
}

66.
Inside ls_lowRow.ino 1/2
void lowRowStart() {
switch (Split[sensorSplit].lowRowMode) {
case lowRowStrum:
lowRowState[sensorCol] = pressed;
break;
// ... snip, different for each low-row mode
}
}
void lowRowStop() {
switch (Split[sensorSplit].lowRowMode) {
case lowRowStrum:
lowRowState[sensorCol] = inactive;
break;
// ... snip, different for each low-row mode
}
}

67.
Inside ls_lowRow.ino 2/2
void handleLowRowState(byte z) {
// this is a low-row cell
if (isLowRow()) {
// send out the continuous data for low-row cells
if (cell().velocity) {
// ... snip, different for each low-row mode
}
}
// this is a non low-row cell
else {
switch (Split[sensorSplit].lowRowMode) {
case lowRowStrum:
// uses lowRowState to correlate with column
handleLowRowStrum();
break;
// ... snip, other cases
}
}
}

68.
Sending MIDI bytes
• MIDI was causing the LEDs to flicker
• Too much time was spent at once (need more guerrilla!)
• Created a MIDI queue to continuously send byte-by-byte from our
RTOS
• Arduino UART classes still caused problems: synchronous wait for
readiness when sending

69.
Patched UARTClass
--- UARTClass.cpp 2014-11-10 14:55:10.000000000 +0100
+++ UARTClass.cpp 2014-10-10 19:39:43.000000000 +0200
@@ -109,9 +109,15 @@
size_t UARTClass::write( const uint8_t uc_data )
{
+ return write(uc_data, true);
+}
+
+size_t UARTClass::write( const uint8_t uc_data, const bool wait )
+{
// Check if the transmitter is ready
- while ((_pUart->UART_SR & UART_SR_TXRDY) != UART_SR_TXRDY)
- ;
+ while ((_pUart->UART_SR & UART_SR_TXRDY) != UART_SR_TXRDY) {
+ if ( !wait ) return 0;
+ }
// Send character
_pUart->UART_THR = uc_data;

70.
Queuing of messages
ByteBuffer<4096> midiOutQueue;
// called for each MIDI message that is sent
void queueMidiMessage(MIDIStatus type, byte param1, byte param2, byte channel) {
param1 &= 0x7F; param2 &= 0x7F;
midiOutQueue.push((byte)type | (channel & 0x0F));
midiOutQueue.push(param1);
if (type != MIDIProgramChange && type != MIDIChannelPressure) {
midiOutQueue.push(param2);
}
}
// continuously called by our RTOS
void handlePendingMidi() {
if (!midiOutQueue.empty()) {
if (Serial.write(midiOutQueue.peek(), false) > 0) { // patched UART method
midiOutQueue.pop();
}
}
}

71.
More information at
http://www.rogerlinndesign.com
Questions?
@gbevin