- The document discusses designing a sensor-based experiment using the Brownie framework. It focuses on integrating heart rate biofeedback from sensors into the experiment. - It provides steps for creating sensor configuration and recorder classes to initialize and record data from a Bioplux sensor, and code examples for configuring sampling rates and storing data files. - The document aims to teach experimenters how to design experiments that measure and provide real-time biofeedback of physiological signals like heart rate to support research.

1. KIT – Universität des Landes Baden-Württemberg und
nationales Forschungszentrum in der Helmholtz-Gemeinschaft
INSTITUTE OF INFORMATION SYSTEMS AND MARKETING (IISM)
www.kit.edu
Know your Sensors
Designing your physiological experiment
Dr. Anuja Hariharan
/

2. Anuja Hariharan – Untersuchungsdesign und Implementierung 2Institute of Information Systems
and Marketing (IISM)
Learning objectives
Create an experiment with Sensor data
Content:
Design and Implement a Sensor-Based Lab Experiment with the Java-Framework BROWNIE.
Integrate the element of heart-rate based Biofeedback in your Experiment UI.
Literatur:
Jung D, Adam M, Dorner V, Hariharan A (2017): A Practical Guide for Human Lab Experiments in Information
Systems Research: A Tutorial with Brownie. Journal of Systems and Information Technology
(JSIT), Vol. 19 Issue: 3/4, pp.228-256.
Hariharan, A., Adam, M. T. P., Lux, E., Pfeiffer, J., Dorner, V., Müller, M. B., & Weinhardt, C. (2017). Brownie:
A platform for conducting NeuroIS experiments. Journal of the Association for Information Systems, 18(4),
264.
Material:
Downlod main repository from Brownie‘s bitbucket page
https://bitbucket.org/kit-iism/experimenttool/src
Java 8 JDK and Eclipse Neon
Exkurs

3. Anuja Hariharan – Untersuchungsdesign und Implementierung 3Institute of Information Systems
and Marketing (IISM)
Quick survey
Experience
With behavioral experiments…?
With physiological experiments…?
With Brownie…?
Other platforms / programming background…?
What is your intended purpose with Brownie..?
Brownie is just a tool. The research question is the key..
Interest in hands-on part / background?

4. Anuja Hariharan – Untersuchungsdesign und Implementierung 5Institute of Information Systems
and Marketing (IISM)
CASE:
IMPLEMENT THE WEB
SHOPPING EXPERIENCE
Add sensor data measurement to your experiment

5. Anuja Hariharan – Untersuchungsdesign und Implementierung 6Institute of Information Systems
and Marketing (IISM)
The case of the compulsive Buyer
In this world of product choices, market designs encourage
emotional buying.
However, this does not always favor the consumer since it
encourages impulsive buying
Can we make the consumer aware of his emotions and
support wiser purchase choices?

6. Anuja Hariharan – Untersuchungsdesign und Implementierung 7Institute of Information Systems
and Marketing (IISM)
Enter – the era of Biofeedback
A personal „drive-safe“ aide

7. Anuja Hariharan – Untersuchungsdesign und Implementierung 8Institute of Information Systems
and Marketing (IISM)
What building blocks do we need for
such an experiment?
Hardware to measure sensors ….
Bioplux in this case
Can also be camera, audio, mouse pressure, etc.
Hardware configuration for the sensor…..
How to record the data…
Store the sensor data (if required)….
Process sensor data and Biofeedback display….

8. Anuja Hariharan – Untersuchungsdesign und Implementierung 9Institute of Information Systems
and Marketing (IISM)
Implementation Steps on Brownie
Preparing your experiment for Biofeedback
How to create a SensorConfiguration file
How to create a SensorRecorder file
Specifying Sensor Data storage
Testing start/stop recording on clients
Coding and Configuring Biofeedback
Preparing for trouble: hardware
connectivity and data quality

9. Anuja Hariharan – Untersuchungsdesign und Implementierung 10Institute of Information Systems
and Marketing (IISM)
Use the ISensorRecorderConfiguration
to specify configuration variables
public class BiopluxRecorderConfiguration
extends ISensorRecorderConfiguration {
…….
}
1 We begin by creating a class in the sensor package extending
ISensorRecorderConfiguration
!

10. Anuja Hariharan – Untersuchungsdesign und Implementierung 11Institute of Information Systems
and Marketing (IISM)
2 Specify variables in the class, which correspond to sensor configuration
parameters, as required by the sensor recording API
Use the ISensorRecorderConfiguration
to specify configuration variables
public class BiopluxRecorderConfiguration extends
ISensorRecorderConfiguration {
public int samplingRate;
public int frameSize = 1000;
public int resolution;
public void setDefaultValues() {
samplingRate = 1000;
resolution = 12;
}
}
Set default values, if
any.

11. Anuja Hariharan – Untersuchungsdesign und Implementierung 12Institute of Information Systems
and Marketing (IISM)
Use the ISensorRecorderConfiguration
to specify configuration variables
public class BiopluxRecorderConfiguration extends
ISensorRecorderConfiguration {
@SensorConfigurationElement(name = "Sampling Rate",
description = "Value in Hz. Max value is 1000.")
public int samplingRate;
@SensorConfigurationElement(name = "Sampling Frame Size",
description = "frame buffer size that is transfered from the
device to the pc per request (in bits). Max value is 1000.")
public int frameSize = 1000;
@SensorConfigurationElement(name = "Measurement Resulution",
description = "Value rage of measurements in bits. E.g. 12 =
12 bit = measurement data range of 4096. Max value is 12.")
public int resolution;
public void setDefaultValues() {
samplingRate = 1000;
resolution = 12;
}
}
3 State the name and description to be displayed for the
experimenter, using @SensorConfigurationElement

12. Anuja Hariharan – Untersuchungsdesign und Implementierung 13Institute of Information Systems
and Marketing (IISM)
View on Brownie UI

13. Anuja Hariharan – Untersuchungsdesign und Implementierung 14Institute of Information Systems
and Marketing (IISM)
The case of Bioplux
How to create a SensorConfiguration file
How to create a SensorRecorder file
Specifying Sensor Data storage
Testing start/stop recording on clients
Coding and Configuring Biofeedback
Preparing for trouble: hardware
connectivity and data quality

14. Anuja Hariharan – Untersuchungsdesign und Implementierung 15Institute of Information Systems
and Marketing (IISM)
Creating your own Sensor Recorder
public class BiopluxRecorder extends
ISensorRecorder<BiopluxRecorderConfiguration> {
…..
}
1 • Specify behavior for standard sensor functions
(start recording, stop recording, specifying sensor configuration, etc. )
• These standard functions are defined in the interface ISensorRecorder
• Create a class which extends this interface, and link up your Sensor
Configuration that you just created

15. Anuja Hariharan – Untersuchungsdesign und Implementierung 16Institute of Information Systems
and Marketing (IISM)
2 Ádd implementations for the abstract methods
public class BiopluxRecorder extends
ISensorRecorder<BiopluxRecorderConfiguration> {
@Override
public String getMenuText() {
return „Bioplux";
}
}
Creating your own Sensor Recorder
getMenuText():
Provides the menu text for this
sensor, shown in the server GUI

16. Anuja Hariharan – Untersuchungsdesign und Implementierung 17Institute of Information Systems
and Marketing (IISM)
3 Ádd implementations for the abstract methods (configureRecorder)
public class BiopluxRecorder extends
ISensorRecorder<BiopluxRecorderConfiguration> {
@Override
public boolean
configureRecorder(BiopluxRecorderConfiguration
configuration) {
this.macAddress =
extractMadAdressFromProperties
(Constants.getComputername());
dev = new Device(macAddress);
fileWrite =
FileManager.getInstance().getFileWriter
(getMenuText());
fileWrite.write("ReceiveTime");
…….
fileWrite.write("n");
return true;
}
}
Creating your own Sensor Recorder
configureRecorder (…):
Configures the sensor recorder with
all parameters required to initiate the
hardware, all related parameters to
hardware, any initialization tests, and
data storage specifications (like file
format derivation, file header).
This example:
• Specifies macaddress for Bluetooth
connection.
• Initializes a Bioplux Device object
(from the Bioplux API)
• Creates file object and file header
for local sensor data storage.
• Return true if all operations are
successfully completed / initialized.

17. Anuja Hariharan – Untersuchungsdesign und Implementierung 18Institute of Information Systems
and Marketing (IISM)
3 Compare with equivalent code for AudioRecorder
Creating your own Sensor Recorder
public class AudioRecorder extends
ISensorRecorder<AudioRecorderConfiguration> {
@Override
public boolean
configureRecorder(AudioRecorderConfiguration
configuration) {
audioFormat = new AudioFormat
(configuration.sampleRate,
configuration.sampleSize,
configuration.channels, configuration.signed,
configuration.bigEndian);
info = new DataLine.Info(TargetDataLine.class,
audioFormat);
// checks if system supports the audio data line
if (!AudioSystem.isLineSupported(info)) {
(Log..and throw exception..)
return false;
}
return true;
}
}
configureRecorder (…):
This example:
• Creates an audioFormat object
from Sensor configuration
information
• Obtains Dataline for Audio
recording .
• Check is line is supported
• Returns true if all operations
successfully completed
!

18. Anuja Hariharan – Untersuchungsdesign und Implementierung 19Institute of Information Systems
and Marketing (IISM)
3 Some tips to know which initializations / configurations are needed:
Creating your own Sensor Recorder
• Find out if your purchased Sensor is accompanied with a documented API with
initialization requirements
• Find out if there is an open source API available for standard/in-built sensors – such as
Audio/Webcam
• Catch and log all exceptions that might occur, to help troubleshooting
• Log all necessary information to aid Troubleshooting, especially when running in the
lab.
• Test each step carefully before proceeding.
• Think of different hardware scenarios in the lab
• Issues due to 64-bit or 32-bit libraries
• Issues due to Java version requirements
• Differences in OS (Developing in mac, deploying on Windows, etc.)

19. Anuja Hariharan – Untersuchungsdesign und Implementierung 20Institute of Information Systems
and Marketing (IISM)
public class BiopluxRecorderConfiguration extends ISensorRecorderConfiguration {
public void Recording() throws RecordingException {
FileManager.getInstance().updateStartTime(fileWrite);
Device.Frame[] frames = new Device.Frame[config.frameSize];
for (int i = 0; i < config.frameSize; i++) {
frames[i] = new Device.Frame();
}
samplingTime = new Date();
dev.BeginAcq(config.samplingRate, config.channels, config.resolution);
while (this.isCapturingActive()) {
dev.GetFrames(config.frameSize, frames);
receiveTime = new Date();
//ShowConsoleOutput if needed
// convert acquired data in frames [i] into a desired
// string format, and write to File
for (int i = 0; i < config.frameSize; i++) {
samplingTime.setTime(samplingTime.getTime() + 1);
fileWrite.write(getStringFromFrame(frames[i]));
fileWrite.write("n");
}
fileWrite.flush();
}// end of while loop
} //end of Recording()
} // end of Class
4 Create the logic for starting the recording
Creating your own Sensor Recorder
Recording (…):
This example:
• More initializations
• Saves the start time into a variable
• Begins the acquisition on the initialized
Bioplux Device object (call to the Bioplux
API)
• Runs a loop till the capturing is active
• Fetches each frame of the specified
frameSize and stores it in the “frames”
array.
• Converts each frame to a desired string
format and stores a frame into the created
file.
• Flushes (i.e. frees up) the file write after
each write call

20. Anuja Hariharan – Untersuchungsdesign und Implementierung 21Institute of Information Systems
and Marketing (IISM)
4 Compare with equivalent code for AudioRecorder
Creating your own Sensor Recorder
public class AudioRecorder extends
ISensorRecorder<AudioRecorderConfiguration> {
@Override
public void Recording() throws RecordingException {
targetDataLine = (TargetDataLine)
AudioSystem.getLine(info);
targetDataLine.open(audioFormat);
// start capturing
targetDataLine.start();
AudioInputStream audioInputStream = new
AudioInputStream(targetDataLine);
// start recording
FileOutputStream fileOutputStream =
FileManager.getInstance().getFileOutputStream
(getMenuText());
AudioSystem.write(audioInputStream,
audioFileFormatType, fileOutputStream);
} // end of Recording
} // end of class
Recorder (…):
This example:
• Creates and opens an audio
data line in the specified
audio format.
• Starts the acquisition using
the “start” method in the
target dataline.
• Since the start method is a
thread call by itself, no check
is needed in a while loop.
• Writes Audio stream output
to a file.
• Also a suitable method to add data quality checks, and store different information in
the file according to the estimated quality of the data!

21. Anuja Hariharan – Untersuchungsdesign und Implementierung 22Institute of Information Systems
and Marketing (IISM)
5 Ádd implementations for the abstract methods (cleanupRecorder)
public class BiopluxRecorder extends
ISensorRecorder<BiopluxRecorderConfiguration> {
@Override
public void cleanupRecorder() {
dev.Close();
fileWrite.close();
}
}
Creating your own Sensor Recorder
cleanupRecorder():
Closes the Sensor acquisition
objects
Bioplux example:
- Closes the Device Object
Audio Recorder example:
- Stops and Closes the audio
data line
public class AudioRecorder extends
ISensorRecorder<AudioRecorderConfiguration> {
@Override
public void cleanupRecorder() {
targetDataLine.stop();
targetDataLine.close();
}
}

22. Anuja Hariharan – Untersuchungsdesign und Implementierung 23Institute of Information Systems
and Marketing (IISM)
The case of Bioplux
How to create a SensorConfiguration file
How to create a SensorRecorder file
Specifying Sensor Data storage
Testing start/stop recording on clients
Coding and Configuring Biofeedback
Preparing for trouble: hardware
connectivity and data quality

23. Anuja Hariharan – Untersuchungsdesign und Implementierung 24Institute of Information Systems
and Marketing (IISM)
Setup mcaddresses.properties and run
server
• Browse to the following location and edit the file
ExpSensorssrcedukitexpsensorbiopluxmcaddresses.properties
• Add entry for each hostname in your experiment, and the mcaddress it
should connect to. (<Hostname> = <Macaddress of Bioplux>

24. Anuja Hariharan – Untersuchungsdesign und Implementierung 25Institute of Information Systems
and Marketing (IISM)
Configure sensor for entire experiment
• Run Brownie in Server mode
• Select the experiment, a list of available sensors appear
• Double click „Bioplux“ to add it to the experiment

25. Anuja Hariharan – Untersuchungsdesign und Implementierung 26Institute of Information Systems
and Marketing (IISM)
Configure Sensor and apply changes
• Click „Configure selected sensor“ to edit the configuration
• Modify any parameters, save, and apply changes to the experiment

26. Anuja Hariharan – Untersuchungsdesign und Implementierung 27Institute of Information Systems
and Marketing (IISM)
Run session & connect clients

27. Anuja Hariharan – Untersuchungsdesign und Implementierung 28Institute of Information Systems
and Marketing (IISM)
View Bioplux connectivity on server side

28. Anuja Hariharan – Untersuchungsdesign und Implementierung 29Institute of Information Systems
and Marketing (IISM)
CASE:
ADD BIOFEEDBACK ELEMENT
TO YOUR EXPERIMENT

29. Anuja Hariharan – Untersuchungsdesign und Implementierung 30Institute of Information Systems
and Marketing (IISM)
The case of Bioplux
Steps
How to create a SensorConfiguration file
How to create a SensorRecorder file
Specifying Sensor Data storage
Testing start/stop recording on clients
Coding and Configuring Biofeedback
Preparing for trouble: hardware
connectivity and data quality

30. Anuja Hariharan – Untersuchungsdesign und Implementierung 31Institute of Information Systems
and Marketing (IISM)
1 Add a UI element to display the Live Biofeedback element on your experiment
screen
public class GaugeMeterScreen extends Screen {
JPanel panel = new JPanel();
panel.setBackground(Color.GRAY);
panel.setBounds((int)bounds.getWidth() - 250, 0,
250, bounds.height);
panel.setBounds(bounds);
add(panel);
panel.setLayout(null);
final RadialBargraph gauge = new RadialBargraph();
System.out.println((int)bounds.getWidth() + ";" +
bounds.getHeight());
gauge.setBounds((int)bounds.getWidth() - 200, 300,
180, 200);
gauge.setTitle("Heart Beats");
gauge.setLcdVisible(true);
gauge.setLedVisible(true);
panel.add(gauge);
}
In the Screen class
Position the UI
element
appropriately

31. Anuja Hariharan – Untersuchungsdesign und Implementierung 32Institute of Information Systems
and Marketing (IISM)
Configure BiopluxRecorder to send LBF
update events
public class BiopluxRecorder {
….
Recording() {
……
// make a call to your biosignal processor passing in the data frames
processor(config.frameSize, frames);
double processedValue = 0.0;
//( Result from Heart rate processing module)
// Add Value to LBFManager. Use name of this class as value-key.
LBFManager.getInstance().updateLBFValue(BiopluxRecorder.class.getName(),
processedValue );
……
}
….
}
2

32. Anuja Hariharan – Untersuchungsdesign und Implementierung 33Institute of Information Systems
and Marketing (IISM)
3
Receive updates from LBFManager via the BiopluxRecorder class
Write logic to update the UI element based on received values
public class GaugeMeterScreen extends Screen {
LBFManager.getInstance().addLBFUpdateEvent(BiopluxRecorder.class.getName(),
new LBFUpdateEvent() {
@Override
public void LBFValueUpdate(Object value) {
double gaugeValue = ((double) value) * 100;
gauge.setValue(gaugeValue);
if (gaugeValue < 50) {
gauge.setBarGraphColor(ColorDef.GREEN);
} else if (gaugeValue < 80) {
gauge.setBarGraphColor(ColorDef.YELLOW);
} else {
gauge.setBarGraphColor(ColorDef.RED);
}
}
});
}
In the Screen class, handle received
events

33. Anuja Hariharan – Untersuchungsdesign und Implementierung 34Institute of Information Systems
and Marketing (IISM)
Complete the code to integrate gauge in
Browser
public class GaugeMeterScreen extends Screen {
WebBrowser browser = new WebBrowser(true);
Rectangle bounds = MainFrame.getCurrentScreen().getBounds();
browser.setBounds(new Rectangle((int) bounds.getX(), (int) bounds.getY(), (int)
bounds.getWidth() - 250,
(int) bounds.getHeight()));
/* Set screen layout to null => full screen web browser */
setLayout(null);
add(browser);
/* Load initial web page */
browser.loadURL("https://www.amazon.de/");
/* Send client respone to end the experiment after a certain amount of time */
new java.util.Timer().schedule(new java.util.TimerTask() {
@Override
public void run() {
ClientGuiController.getInstance().sendClientResponse(parameter,
gameId, screenId);
}
}, DURATION_OF_SCREEN);
……
}

34. Anuja Hariharan – Untersuchungsdesign und Implementierung 35Institute of Information Systems
and Marketing (IISM)
Run the configured Bioplux experiment
again

35. Anuja Hariharan – Untersuchungsdesign und Implementierung 36Institute of Information Systems
and Marketing (IISM)
Basic concepts about Biofeedback
• LBFManager – A singleton class (i.e., only one instance exists across the
application) that contains a list of LBFValues and a map of LBF Events
• LBFUpdateEvent – An abstract Event listener that must be handled in each screen to
specify how to handle incoming values
RecorderClass (method Recording())
• Processes incoming values and
raises an event using the
udpateLBFValue call
LBFManager.getInstance().updateLBFVa
lue
(BiopluxSensorRecorder.class.getName(
), dummyValue);
Screen
• Registers the update event, along with
the corresponding RecorderClass
• Listener logic to specify how to react to
events
LBFManager.getInstance().addLBFUpdateE
vent
(BiopluxSensorRecorder.class.getName(),
new LBFUpdateEvent())

36. Anuja Hariharan – Untersuchungsdesign und Implementierung 37Institute of Information Systems
and Marketing (IISM)
Data processing considerations
- A Baseline recording phase is necessary, with an initial cool down period of
atleast 5 minutes to calibrate the values of each subject with his/her own
baseline
- Regular baselines (such as at the end of each round) can be thought of as
well
- Use of standardized packages (such as xAffect) to process the data
- More flexibility in using own algorithms for real-time processing
(if this is part of the research focus)
- Specify a time window where LBF value error is tolerable (such as 5
seconds), yet the update frequency is not too high on the screen.

37. Anuja Hariharan – Untersuchungsdesign und Implementierung 38Institute of Information Systems
and Marketing (IISM)
How to create a SensorConfiguration file
How to create a SensorRecorder file
Specifying Sensor Data storage
Testing start/stop recording on clients
Coding and Configuring Biofeedback
Preparing for trouble: hardware
connectivity and data quality

38. Anuja Hariharan – Untersuchungsdesign und Implementierung 39Institute of Information Systems
and Marketing (IISM)
Hands-on component

39. Anuja Hariharan – Untersuchungsdesign und Implementierung 40Institute of Information Systems
and Marketing (IISM)
Outlook
Fun future ideas with Brownie
• Camera based
• QR Code integration – (Zxing java library)
• Image recognition – (TensorFlow Library)
• EEG based
• Brainathon- Brain wave controlled 2 player game
• Signal processing
• JMathStudio – useful collection of signal processing functions