1. Improving Decision-
making via Wearable
Biosensors
Xiaoying Pu1 Elliot Radsliff1
Advisor: Evan M. Peck, Ph.D.1
1Department of Computer Science
Bucknell University
Susquehanna Valley Undergraduate Research Symposium
August 4, 2015
2. Decision-making
under Stress
• Chocolate cake v.s. fruit salad
[Shiv and Fedorikhin, 1999]
Shiv, Baba, and Alexander Fedorikhin. "Heart and mind in conflict: The interplay of affect and cognition in consumer decision making." Journal of consumer Research 26.3 (1999): 278-292.
3. Decision-making
under Stress with
serious consequences
• Implicit bias
• Racial biases in opioid prescription more
likely to manifest under high levels of
cognitive load [Staats et. al., 2015].
Staats, C., Capatosto, K., Wright, R. A., & Contractor, D. (2015). STATE OF THE SCIENCE: IMPLICIT BIAS REVIEW 2015.
4. Good News
Physiological signals reflect people’s
cognitive workload Low workloadHigh workload
Normalized heart
rate
Solovey, E. T., Zec, M., Garcia Perez, E. A., Reimer, B., & Mehler, B. (2014). Classifying driver workload using physiological and driving performance data. Proceedings of
the 32nd Annual ACM Conference on Human Factors in Computing Systems - CHI ’14, 4057–4066. doi:10.1145/2556288.2557068
6. Project Overview
Improving Decision-making via Wearable
Biosensors
N-BACK
‘CALIBRATOR’
FLYLOOP
DECISION-
MAKING
How can we get physiological
data in everyday contexts?
How can we extract meaning out
of raw physiological data in real
time?
How can we use data to
improve decision-making?
7. How can we get
physiological data
from everyday living?
• Portable
• E-4 wristband
• Smart phone application
• Lightweight messaging (mqtt)
• Wealth of information
• Temperature
• Heart rate
• Electrodermal activity (EDA)
• Acceleration
mqtt.bucknell.edu
Empatica E-4
21. High workload?
Low workload?
How can we teach
computer the meaning
of raw physiological data
in real time?
FILTER
DATASOURCE
OUTPUT
FLYLOOP
CALIBRATOR
LEARNER
22. FlyLoop learns from raw physiological data
Heart
Rate
CSV
SLOPE
AVG
Peck, Evan M., et al. "FlyLoop: a micro framework for rapid development of physiological computing systems." Proceedings of the 7th ACM SIGCHI Symposium on Engineering
Interactive Computing Systems. ACM, 2015.
Extracting important features,
aggregated over every trial from n-back
k1, x̄1
23. FlyLoop learns from raw physiological data
Heart
Rate
TEMP
EDA CSV
SLOPE
AVG
One example
k1, x̄1
k2, x̄2
k3, x̄3
24. FlyLoop learns from raw physiological data
One example
k1, x̄1
k2, x̄2
k3, x̄3
One example
k1, x̄1
k2, x̄2
k3, x̄3
One example
k1, x̄1
k2, x̄2
k3, x̄3…
Training set
N-BACK
25. FlyLoop learns from raw physiological data
High workload
k1, x̄1
k2, x̄2
k3, x̄3
Low workload
k1, x̄1
k2, x̄2
k3, x̄3
High workload
k1, x̄1
k2, x̄2
k3, x̄3…
N-BACK
Training set
26. FlyLoop learns from raw physiological data
CALIBRATOR
LEARNER
N-BACK
“States”
High workload
k1, x̄1
k2, x̄2
k3, x̄3
Low workload
k1, x̄1
k2, x̄2
k3, x̄3
High workload
k1, x̄1
k2, x̄2
k3, x̄3…
Support Vector Machine
(SVM) as the “Learner”
Training set
27. FlyLoop learns from raw physiological data
Heart
Rate
TEMP
EDA CSV
SLOPE
AVG CALIBRATOR
LEARNER Classification
N-BACK
Data
“States”
After training…
28. How can we use data
to improve decision-
making?
Hypothesis:
Quality of decision-making
can be improved by an
adaptive decision delivery
system that understands
users’ physiological states.
29. Take away message
Improving Decision-making via Wearable
Biosensors
N-BACK
‘CALIBRATOR’
FLYLOOP
DECISION-
MAKING
Physiological signals Machine learning for
cognitive states
Decision-making,
mediated by computers
30. Thank you
Questions?
Xiaoying Pu xp002@bucknell.edu
Elliot Radsliff efr004@bucknell.edu
Evan Peck emp017@bucknell.edu
Low workload,
here’s a decision…
Editor's Notes
Open the conversation with decisions
We are gonna start with an aspect of decision-making. It’s widely recognized that people are bad decision-makers under stress.
Here is a famous experiment that a Stanford professor did with several dozen college students.
"One group was given a two-digit number to remember,
while the second group was given a seven-digit number.
Then they were told to walk down the hall, where they were presented with two different snack options:
a slice of chocolate cake or a bowl of fruit salad."
The only thing that's different is that...
The students with seven digits to remember were nearly twice as likely to choose the cake as students given two digits. The reason, according to Professor Shiv, is that those extra numbers took up making it that much harder to resist a decadent dessert. In other words, willpower is so weak, and the prefrontal cortex is so overtaxed, that
valuable space in the brain — they were a "cognitive load" —
Chocolate cake seems to be harmless,
When else is this happening?
Explanation? Cognitive load, limited resources, -> chose the intuitively more appealing option over the long-term benefits of having fruit salad?
College student’s final week
Trauma surgeons, police officers
More susceptible to implicit bias
If only we could detect workload in real time!
MIT researchers….
Conditions: participants were to memorize numbers while driving.
Heart rate alone …. (about 80% classfication accuracy)
However, bulky -> improvement?
Apple watch/ Fitbit -> more to it
signals coming out, can we combine those two ideas: stress detection and decision-making
Can we use the apple watch to make better decisions?
Here’s where our study comes in..
How can we interpret the raw physiological data in real time?
Recognize the momoents
Green box: A task through which we could see the difference in decision making under diff. conditions
(Show off the E-4)
Device and framework that we use
How the parts are communicating (mqtt.bucknell.edu), publish-subscribe model
We built the capability to communicate wirelessly. phone ~ app ~= server
The characteristics of Flyloop:
Micro framework written in Java
Modular, simple, flexible
teach the computer: learn first then make classification.
So how does it learn / label data with the states
We built this
Raw data shows difference in high and low workload
The characteristics of Flyloop:
Micro framework written in Java
Modular, simple, flexible
We BUILT
teach the computer: learn first then make classification.
So how does it learn / label data with the states
Teaching the computer to learn
Dataflow
Real time classification
Show the computer examples (teaching)
Filter: extracting what we think are important features of the signal
By the end of the day, we can have state classification in real time
Dataflow
Real time classification
Filter: extracting what we think are important features of the signal
By the end of the day, we can have state classification in real time
Dataflow
Real time classification
Filter: extracting what we think are important features of the signal
Calibrate / build model, then classify user data in real time
By the end of the day, we can have state classification in real time
Measuring tool to see the difference, or improvement in decision-making
Built the ability to interface with it
We are in the process of organizing the experiment
Remember back in the story, researchers had people do tasks while taking a decision,
Now we want to detect and avoid the high workload time and …
H1: Decision-making quality is different under H/L workload (by looking at the trajectories)
(* Main ) H2: By doing something (delaying?), quality of decisions can be improved.
MouseTracker
“Rich cognitive output”
Tool:
MouseTracker
H1: Decision-making quality is different under high / low workload
(observable with MouseTracker)
Big idea: use new information to make better decisions
First step in a really big problem..
Proposed a possibility to use portable sensors like to acquire new information , teach computers learn, and … gain insights into our cognitive states to mediate decision-making
not only the health app
How to introduce machine learning
Wait, why did we choose those parameters in the first place?
The importance of aggregating data
Previous research to support that it works
![Decision-making
under Stress
• Chocolate cake v.s. fruit salad
[Shiv and Fedorikhin, 1999]
Shiv, Baba, and Alexander Fedorikhin. "Heart and mind in conflict: The interplay of affect and cognition in consumer decision making." Journal of consumer Research 26.3 (1999): 278-292.](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)