Thesis Project undertaken at QUT 2016. Final report from the project.

1. Wearable Technology and Stress Detection for students
with Autism
Pre-Empting Breakdowns and Enabling Self-Corrective
Behaviour
Torren Lamont
Supervisor
Dr Bernd Ploderer
1

2. 1 Executive Summary
The aim of this project is to research, design and develop a device to be used for stress
detection for students with autism in the classroom. This project is centred around early
detection of stress before it has manifested in behaviour and likely before the stressed person
is aware that they are becoming stressed, this project aims to pre-empt and counteract the
negative impacts that stress has for students with autism in the classroom by communicating
the stress-level of the student in real time with both the student and teacher, as well as any
other relevant parties. Since this project centres on stress detection, a significant component
will be researching and understanding the bio-signals associated with stress in autism and
how they can be detected. This project will center on using an individualised approach to
detect stress in a specific student and then use this detected state in a system approach to
communicate this information to relevant parties. In order to acheive this dynamic behaviour
will be implemented into the solution such that the system can change in real time based on
user feedback so that a desired level of individualisation is realised.
2 Introduction
Students with ASD are often more susceptible to becoming stressed in the classroom, for
students with ASD many things that are part of day to day life act is significant stressors
and can lead to regular meltdowns, these meltdowns inhibit the learning experience for
the student who has the meltdown as well as other students in the class. In the paper
I will discuss research, development and implementation of a stress detection application
using the Microsoft Band 2. This cross platform application will enable the student to
self regulate and the teacher to become aware of the students mental state and potentially
to pre-empt classroom meltdowns. This project consists of two main sections. First, the
research component in which literature has been assessed, other persons with experience
relevant to this project have and will be met with and their understanding and opinions
discussed. Existing technology has also been assessed and the Micosoft Band 2 has been
chosen because of it’s array of sensors, ease of development and low cost. Second, the device
and designed multi-user interface system will be omplemented in a real world setting and
testing completed. Prior to testing in Humpybong State School, the system will first be tested
in a study conducted with QUT students. As this paper will demonstrate their lacks a body
of work around the bio-signals that precede a breakdown for persons with ASD. Much work
exists on the detection of stress in persons with ASD. The development of a more effective
algorithm for detecting and pre-empting breakdowns is unlikely to be achievable given the
scope of this project, however the distinction between pre-meltdown states and stressed states
is worth bearing in mind while assessing literature. Their exists another gap in the literature
and existing device implementations, this gap is seen by the lack of individualised approach
in stress detection projects as well as the lack of device implementation in a complete systems
approach with a goal to assisting teachers and students in understanding and pre-empting
autistic meltdowns so that students with autism are provided with the feedback and support
2

3. needed for an effective classroom education.
3 Literature Review
Stress
What is it that we mean when we talk about stress? Stress as it is understood colloquially,
generally refers to the negative emotional state (called stress response) caused by some sort of
external stimulus (called a stressor). However stress does not necessarily produce a negative
emotional state nor is it associated with a negative stimulus. As outlined by Hyland, M., &
Richardson, S. (2014) stress occurs in many activities seen as pleasant that people choose to
do, such as weddings and marathon running.
Stressors are external stimuli, either seen as positive or negative, that create an alerting
response. Whether the stressor is seen as positive or negative does not alter the physiological
response associated with stress. Thus stress detection will not necessarily imply that a
negative emotional state is being reached, but rather that an alerting response is present in
the subject.
For persons on the ASD spectrum, stress presents a more frequent and difficult problem, with
stressors specific to persons on the ASD spectrum being things which are often experienced
in day to day life by all people. As well as presenting a more severe and frequent problem
for persons with ASD, stress also appears differently in persons with ASD. A Study from
(Kushki et al., 2013) shows a difference in both physiological response to stress and baseline
readings in persons with ASD when compared to persons without. Its important to note
that this higher baseline might mean that a person with ASD would not be able to tell that
he/she was becoming stressed, given then higher baseline.
Stress Response exists in two main forms over time. Acute Stress Response to short term
stressors and long term stress response to chronic stressors. When someone experiences
an Acute Stress Response their perceptual awareness increases while their cognitive ability
decreases (Hyland, M., & Richardson, S. 2014). This has to do with the relationship between
arousal and cognitive performance which is represented by an inverted U curve, meaning some
level of arousal is required for cognitive tasks, but the arousal associated with short term
stress response is too great and begins to appear cognitive functioning.
3

4. Acute Stress response is characterised by a short term change in some bodily function which
reverts back to normal once the Acute Stressor is removed (Hyland, M., & Richardson,
S. 2014, page 62). This type of stress does not present long term problems and is even
subjectively desirable in many cases.
Long term stress response is characterised by a change in some bodily functions over time
creating a new baseline. This stress response is caused by the build up over time of stress.
This form of stress response is always undesirable and leads to physical and mental health
problems over time.
How do we measure Stress?
Detection of stress centers around the use of sensors which measure certain physiological
markers associated with the experience of stress. This data is then processes with an algorith
to ascertain the extent (or lack thereof) to which this person is experiencing stress.
For persons with ASD it has been shown that the biosignals associated with stress can be at
levels well above baseline before the person is showing any behavioural manifestation of this
stress. In their paper (Goodwin et al., 2006) showed a very high heart rate of 120bpm in
persons with ASD who appeared completely calm, this level is well above the usual baseline
of 70bpm. As well as heart rate, EDA has been shown to be well above the usual baseline in
persons with ASD who appear calm and without any behavioural or physical manifestation
(Hirstein, Iversen, & Ramachandran, 2001).
The autonomic nervous system is responsible for regulating a person’s physiological state.
The parasympathetic nervous system is the component of the autonomic nervous system
which regulates the physiological markers associated with increased arousal of which stress
is a possible state. It is worth noting that stress is not the only possible reason for increased
parasympathetic activation, this system is also activated during cognitive tasks as well as
many other emotional states both positive and negative. One physiological marker which
may be indicative of stress is Galvanic Skin Response insofar as an appropriate baseline is
known and set for each individual (Villarejo, Zapirain, & Zorrilla, 2012) as a raw value of
GSR is not necessarily indicative of stress, rather this value must be weighted against an
individual baseline.
Another possible indicator of stress is Heart Rate Variablility(HRV). HRV is a meausre of
the variation in time between heart beats, it differs from heart rate which is a measure of
4

5. the averaged number of beats per minute. HRV variability is measured using Electrocardio-
gram(ECG) sensors which measure the electrical signal of the heart. In their paper MacLean
et al. (2013) used a combination of ECG and GSR sensors to detect stress in the devices
wearer.
As well as GSR and HRV, skin temperature is also indicative of parasympathetic activation
and thus can be used to detect stress. In their study (Yamakoshi et al., 2008) reported a
significant drop in skin temperature when a person experienced a drop in parasympatheic
activation. Their paper investigated drivers whos initial stressed state indicated by a higher
than baseline skin temperature which began to drop during a monotonous task.
Autism
Autism Spectrum Disorder(ASD) is typically defined as a biologically based condition which
is defined by abnormalities in social interaction and social communication along with the
repetitive behaviours. ASD, however has not and still is not easy to define clearly.
Within the spectrum of ASD exists Aspergers. Aspergers shares similarities with Autism.
It is understood as a higher functioning end of the ASD spectrum, individuals with Asperg-
ers share the abnormalities in social interaction and communication; however they do not
share the language or cognitive development difficulties that persons further along the ASD
spectrum exhibit.
Within the scientific research surrounding Autism there is a divide between different inter-
pretations of the physiological data associated with Autism, some believing that the physi-
ological markers are the cause of autism and others believing the physiological markers just
indicate the presence of autism.
Stress poses a more serious problem for person with ASD, specifically long term stress re-
sponse. This long term stress response plays a major role in the childs ability function and
also comes with negative effects on health (Corbett et al. 2009).
Many things in day to day life which are not generally significant stressors for persons without
ASD can act is as serious stressors for persons with ASD. The classroom, by it’s very nature
is a stressful environment, things in that classroom environment which are generally easily
dealt with such as the sound of pen being tapped on a desk or a humming noise in the distance
5

6. can act as major stressors which can build up over time and cause a meltdown. There are
almost endless things like the aforementioned noises which can cause meltdowns, the key
similarity to take note of is best described by the concept of Novel Situations as outlined
in (Lipsky, 2011, p. 161). A novel situation is any event which is off plan or unexpected.
These occur frequently in the classroom, often without the teacher even being aware, and
can cause serious stress response in persons with ASD and often Lead to meltdowns.
An important distinction needs to be made between Autistic Meltdowns and Autistic Tantrums.
An Autistic Meltdown is caused by an accumulation of stressors causing a stress response
in the individual. They are not conscious decisions nor do they have an intended response
(Lipsky, 2011, p. 135). Autistic Meltdowns do not
Autistics Tantrums differ greatly to Autistic Meltdowns. Autistic Tantrums are not caused
by external stressors but instead are an attempt at manipulating a person or situation so
that the person throwing the Tantrum can get what he/she wants (Lipsky, 2011, p. 136).
Tantrums are also not bound to persons with ASD, they occur in persons without ASD as
well.
Autistic Meltdowns do not occur uniformly within the Spectrum of persons with ASD. Some
appear to internalise the negative effects of accumulating stressors and shut down as opposed
to those who externalise this in the form of a meltdown while some appear to not be plagued
at all by meltdowns (Lipsky, 2011, p. 107). Even though a reaction of an individual with
ASD may take the form of a shutdown rather than a meltdown, and this would not cause a
disruption in the classroom. Both of these manifestations of a stressed state are undesirable
in a learning environment and thus a device capable of communicating in real time with a
teacher so that pre-emptive action can be taken could prove highly beneficial in a classroom
environment.
Existing Technology
Some similar and relevant devices exist already. These include the Empatica Wristband and
Empatica Watch which implement sensors for GSR, skin temperature and Photoplethysmog-
raphy Sensors from which HRV can be calculated. The Empatica e4 wristband represents
the most ideal system for application in this project. It uses all necessary sensors for the
scope of this project and offers it all housed in an aesthetic package.
As well as the Empatica devices, Microsoft also produces a device that is capable of measuring
some relevant Bio-Signals including GSR and Skin Temperature. It also includes a heart rate
6

7. monitor, and rr interval from which Heart Rate Variability can be calculated. . The Microsoft
Band 2 also includes a large 320x128 pixel which can be used for a means of communicating
with the user. This will be acheived by changing the screen colour based on stress level to
communicate with both teacher and student.
As well as these aforementioned devices that are directly relevant and have been utilised for
this project, there also exist peripherally relevant devices and papers, one of these is the
EEG headset as researched and outlined in(Garzotto et al., 2016). This paper outlines the
work done by Garzotto(2016) which centres on the development of an individualised school
room, which responds to a childs level of arousal as given by the EEG Headset. Though this
paper is not directly related to stress detection it is directly related to the classroom and to
children with special needs.
The device chosen is the Microsoft Band 2 because of it’s relative low cost and large array of
sensors. Other advantages include the Visual Studio Universal Application software which
is used for developing the Microsoft Band 2, this includes a wide ranging SDK and Universal
application development meaning that applications developed for the Microsoft Band 2 can
run on Microsoft PC’s, Tablets and Phones
Literature Gap
Much work exists on the detection of stress using real time analysis of Bio-Signals including
some of which centres specifically on stress detection for persons with ASD (Picard, 2009).
The existing body of work centres on stress detection for real time updates on stress levels.
However as can be seen in the work by Goodwin et al. (2006) and Hirstein et al. (2001)
extremely high readings of some Bio-Signals associated with stress can be seen in individuals
with ASD who appear calm. These high readings are indicative of stress but not necessarily
indicative of a pre-meltdown state. By its very nature a classroom is a stressful environment,
this is not undesirable as stress is not necessarily a negative state. This is a key concept to
account for when designing the individualised baseline and thresholds for zones of stress.
Bio-Signal processing, by its very nature is complex, and is made more complex when one
attempts to use a one size fits all approach to inferring mental states. Within the existing
literature there are some key useful areas that have not been addressed. First is the dis-
tinction which ought to be made between detecting and inferring stress and detecting the
physiological state that precedes an Autistic Meltdown so that pre-emptive action can be
taken. Research needs to be conducted on this distinction so that a wearable device that
warns the teacher of an impending meltdown can do just that, and not merely warn the
7

8. teacher every time a student is becoming stressed. It is possible that the complex and indi-
vidual nature of the Bio-Signals associated with a pre-meltdown state are too individual for
a generic approach, and thus there is a need for individualisation of the detection algorithm
in order for affective pre-emption of Autistic Meltdowns in the classroom.
For the scope of this project, however, addressing the aforementioned lack of clear detection
of pre-meltdown states in persons with ASD is not likely to yield better results than what has
been achieved by others, however it is worth noting this gap when assessing literature and
interpreting results. There is another gap within the existing literature and technology which
we will aim to address throughout this project. That is the development of an individualised
algorithm and complete system capable of detecting stressed and pre-meltdowns states in
a student with ASD and once detected, informing the student by way of haptic feedback
and a changing colour which will enable the student to develop self-corrective behaviour.
The teacher will also have this information communicated to them by simply glancing at
the students wrist band. The special response unit(SEP) will also be notified through the
development of a tablet/computer user interface which updates in real time. This will be
discussed in further detail in the following section.
4 Research Aim
This project centres on the early detection of stress in school students with ASD who may
not be aware they are becoming stressed and are not displaying any visual signs of a potential
for a breakdown. This information will then be communicated with the classroom teacher
so that she can take pre-emptive action before a breakdown or disruption occurs in the
class room. In undertaking this project it is necessary to understand some key relevant
areas. These are as follows; stress, autism, stress related bio-signals and there detection,
stress for persons with autism and human computer interaction. Learning about the current
information in each of these areas will enable an effective and useful crossplatform system
to be devloped using the Microsoft Band 2 and other necessary Windows devices. Once
developed the system will be implemented first with QUT students. This will serve two key
purposes, first to assess the effectiveness of the system and get feedback on the system design,
how the user interacts with the system, whether this interaction is positive or negative for
the user and hopefully some constructive feedback to allow the design to be changed or
refined for better user experiece.
8

9. 5 Project Plan
The Aim of this project is comprised of three specific sections. First, to assess the body of
relevant literature surrounding Autistic Meltdowns, stress and the methods by which it can
be detected using Bio-Signals. Second, to implement this understanding in development of
a crossplatform solution using the microsft band 2 with a dynamic baseline setup such that
the system updates and changes baselines and thresholds in real time based on user input.
Third, to change the colour of the Band itself as well as a desktop application run in the
SEP room based on the detected emotional state to allow the student to identify his or her
current state and also to make the teacher aware of the students emotinal state. There is
also a fourth component to this project which may or may not come to fruition depending on
the deemed scope of this unit. That is to develop a computer or tablet based game/activity
designed to calm the stressed student down and prevent a meltdown from occuring.
In order to meet the first task, relevant literature has been assessed so as to understand
the nature of the problem as deeply as possibly. As well as this persons with first hand
experience in relevant areas such as teachers who work with students with ASD have been
interviewed to build on the understanding elicited from the relevant literature, this included
a trip to Humpybong State School in which the project team members met with teachers
and had a meeting to build understanding of stress, students with autism and the classroom
setting.
Semester 1 consisted of the research component of the project as well as initial implemen-
tation of the Microsoft Band including preliminary development of the software for this
project. During this semester the conceptualisation of the project changed repeatedly as
new relevant information was discovered. The literature review has been completed. During
this first semester, interviews were also conducted with the school teacher with whom we
will be working to implement the device in the classroom.
Semester 2 consisted of the design and implementation component of the project. From
the understanding gained during the research undetaken in semester one a system has been
devloped which utilises the mircosoft band 2 for reading user bio-signals and interacting with
both the teacher and student. The application is run on a native Windows device and is
implemented in Visual Stusio using a Universal Application.
The system architecture that has been developed is as follows. The band changes colour
based on the users perceived state, based on HR and/or GSR. If the band is in the Green
9

10. State the interface is not continuously updated to preserve battery life, when the band enters
or remains in either the amber or red state the device is periodically updated so that the
device screen remains on and not locked. When the device enters the amber or red state the
user is notified and he or she is prompted to go to the ’tile’ application run on the Microsoft
Band 2(Show in the figure above). This application consists of three buttons(one green,
one amber and one red) the user then clicks the button that he or she identifies as being
there current stress level. If this user input contradicts the bands current colour then the
thresholds are changed accordingly. For example, if the device changes to amber because the
users HR is greater than ’HRThreshold1’, the band then prompts the user to confirm their
state in the tile app, if the user clicks the green button then the value of ’HRThreshold1’ is
changed to equal the current users HR and naturally the device changes back to green. This
process of allowing the user to confirm or disconfirm the current colour displayed gives the
system dynamic capabilities which should improve in accuracy and effectiveness over time,
this will be explained further in the following section. Along with this, a prototype for the
application for the SEP room has also been developed.
The application which runs on the host computer will double as the display as seen in the
SEP room. This consists of a single button to connect the Microsoft Band 2, a real time
heart rate reading, real time GSR value and a large colour block which changes to green,
amber or red synchronously with the Microsoft Band display. The following 2 figures show
the application prior to connection, and after connection.
10

11. 11

12. The key component in this project is developing an algorith which is able to infer states of
stress or pre-meltdown reasonably effectively. This is not an area which we will be able to
perfect, therefore we used an Engineering approach to develop a preliminary solution which
can be developed further with more research.
Below is a conceptual sketch of the system setup for our project. The system has been
designed to be used with only one student. However, if more students were to be handles
an interface such as the one shown below could be used by replacing the N/A’s that appear
in the 3 other segments of the screen. The system can be revised so additional students can
easily be added to the display in the SEP room.
12

13. 13

14. The device has also been programmed to provide haptic feedback to the student at either
red or orange and red zones. This can be changed based on a speicific student.
The system has been designed to also consist of a laptop/tablet which will display the
previously shown application interface with the Green, Amber, Red zones, this device will
sit in the special response unit(SEP) so that they too are aware of the state of the student
and can act accordingly. The user-interface could be changed for the SEP unit such that
multiple students scale values and colour zones can be displayed.
14

15. 6 Research Approach
6.1 Initial Design
The first iteration of the system utilized the Microsoft Band 2 which relied upon a windows
laptop running a Visual Studio Application. The system had thresholds which were set
based upon wearing the band ourselves and seeing which numbers made sense for a generic
threshold. Four threshold values were used, these are UserGSR1, UserGSR2, UserHR1 and
UserHR2.
Each of these were set based on the readings taken from team members wearing the device
while sitting in a weekly meeting. We found that for each team member values taken at dur-
ing the meeting ranged from 50-70bpm for Heart Rate and 10,000 to 60,000 KOhms for GSR.
Before continuing its worth noting that GSR is inversely proportional to parasympathetic
nervous system activation and therefore inversely proportional to stress.
As can be seen, the variation in hear rate is low whereas the variation in GSR is much larger.
This showed the absolute need to develop the system in a way such that the constants
UserGSR1, UserGSR2, UserHR1 and UserHR2 are all individualised based on the user.
The first prototype did not have these individualised but instead had them hard coded,
userGSR1 being 8000 KOhms, UserGSR2 being 6000KOhms, UserHR1 being 85bpm and
UserHR2 being 105bpm.
This system was tested with the three project colleagues as an initial barometer as well as
a means to get some initial readings from each team member as they underwent a specific
task designed to increase cognitive load which would therefore, lead to a decrease in GSR
(Shi, Ruiz, Taib, Choi, & Chen, 2007). The below figure shows the GSR data for the third
team member is they underwent a cognitive task. The box bound by green is the data from
the meeting without and specific task being undertaken, at 15:17 a series of Brain Training
games from brainhq.com that are well known to be very cognitively stimulating and stressful.
Bound by the red box is the data which was taken during these activities.
15

16. 16

17. From these graphs, a few clear points of understanding were realized. GSR corresponds very
accurately to cognitive load and is significantly different in magnitude from a relaxed state
that it can be used to detect cognitive load. This is both good and bad, such significant
changes are seen in GSR for a participant undergoing cognitive load would result in the
system changing states too often if the thresholds are not set with sufficient spectrum to
encapsulate these as Green State readings.
From the HR data, no significant change is seen, more research is needed to see if a deeply
stressful situation would cause significant enough changes in HR for it to be used alongside
GSR. If this is the case HR could act as a grounding factor that would prevent GSR from
causing the system to change states too often when a user is merely studying or concentrating
on a difficult task.
6.2 Second Iteration of Design
The following changes were made to the system before testing it with University Students.
The four thresholds (UserGSR1, UserGSR2, UserHR1 and UserHR2) were made to be dy-
namic and individualised. This was achieved using a one minute initial baseline reading
period. The participant was to put on the device and upon starting the application several
prompts would appear to initialise the system. After this was done GSR and HR readings
were taken over the course of one minute and these values were averaged. These initial in-
dividual readings were taken in a relaxed state and were translated into the four thresholds
by four coefficients which could easily be changed later.
For example, if a wearers average values were as follows; GSR = 10000 KOHms, HR = 70
bpm. These would be multiplied by four coefficients (GSRT1 = 0.85, GSRT2 = 0.7, HRT1
= 1.15, HRT2 = 1.4) that would result in a UserGSR1 of 8500 KOhms, UserGSR2 of 7000
KOhms, a UserHR1 of 81 bpm and a UserHR2 of 98 bpm. In this way, the systems thresholds
are dynamically set based on the users own baseline readings.
Further changes were implemented before testing the system with university students. These
included the implementation of the user interaction tile to allow the user to change their
threshold values in real time. This is best explained within the context of the aforementioned
example. Just considering GSR for example, the system works as follows. If the user GSR
reading drops to 8000 KOhms then the system will change from the green to amber state as
it has passed UserGSR1. If the user then disconfirms this as accurate by clicking the green
tile on the band application user interface, then UserGSR1 is changed so as to encapsulate
the current reading in the green zone(ie UserGSR1 = 7000*0.95 = 7600). This new value is
17

18. set at 5% below the current user reading so as to encapsulate this value in the green zone.
The same is true for HR, the key difference being that since HR is directly proportional to
stress, whereas GSR is inversely proportional to stress, the new threshold value will be set
for HR by a 5% increase rather than decrease.
This system behaviour was implemented for all cases of a user disconfirming the current
system state, for example the user clicking the amber tile when their reading is red, the user
clicking the amber tile when the system is currently green, the user clicking the green tile
when the system is red etc. will all lead to the system changing the users four threshold
values to produce the revised understanding of the users current readings.
The testing for this phase was completed with GSR being the sole factor responsible for
changes is system state. This was done because, if HR was included it may have caused the
system to change states very rarely or never rendering the user interaction component of the
testing obsolete. The results for this testing are outlined in the following section.
6.3 Second Iteration Results
6.3.1 User Readings
The following data plot was taken from the first University student who took part in the
testing. The data has three key sections that have been highlighted. The green box covers
data 9:55am to 10:00am. In this time the participant was being driven to the UQ bridge.
The amber box covers 10:00am through to 10:15am during which time the participant was
walking across the bridge and through the university to campus to the laboratory. The red
box is from the remainder of the time spent using the device in which the participant was
reading scientific papers and studying. The first block of data which has no bounding box
corresponds to the participant getting ready for university and eating breakfast.
This data demonstrates the extent to which temperature effects GSR values. An initial drop
as seen almost immediately upon leaving the cool house and going outside to get into a hot
car. An upwards trend is then seen after the participants leaves the car and begins to walk.
However, this upwards trend is short in duration and the participant’s GSR begins to fall
again as they walk in the hot sun. From this data there is a very clear conclusions. Firstly,
temperature change and/or physical exertion effects GSR readings to such a degree that the
system cannot be said to remain accurate between the different environments that a user
would experience in day to day use, nor can it be said to be accurate for different levels
physical exertion.
18

19. 19

20. 6.3.2 User Experience and Feedback
The first prototype has been tested with a research participant. The system has been tested
overall with a university student (in this case a PhD). The Microsoft Band 2, accompanied
by a windows phone on which the MS band 2 runs has been given out for field testing in
everyday life for a student.
After spending a part day wearing and using the device, the student was interviewed, specif-
ically with regards to how they interact with the technology and whether they find it useful.
From this first test, there were some key areas for improvement that were highlighted by the
feedback given.
Firstly, the algorithm itself is not accurate and does not serve as useful for the student,
though the system is dynamic and can be updated by the user, the continual process of user
input was seen as overly complicated and time consuming. The haptic feedback to the user
was over used and served more as a source of frustration than as a useful form of feedback.
Secondly, though more easily fixed, the band itself was too large in size and as a result was
not comfortable to use, this of course is easily remedied as the Microsoft Band 2 comes in
three sized.
In order to improve upon these issues, the algorithm was edited to take into account both
GSR and HR. GSR alone is far too variable based on too many uncontrollable factors such as
temperature change between rooms, physical movement such as walking etc. Incorporating
HR will not allow the system to change colour unless significant increases in HR are seen.
This of course, is still subject to change based on physical exertion but should not change
when merely walking or moving into a different environment or room. The number of
conditions that haptic feedback is sent with will also be reduced.
These changes were made to hopefully see an improvement in the user interaction with the
technology and will allow the system to be useful for students in further research studies.
20

21. 6.4 Third Iteration Design
The algorithm was altered so as to use HR as well as GSR after the feedback given by the
previous participant. In theory this should have allowed the system to be less invasive and
irritating to the user as it would be less prone to fluctuations in GSR that are unrelated to
stress causing the system to change states (colour).
6.5 Third Iteration Results
6.5.1 User Readings
The following data plot is from another participant who wore the device throughout his day
at the university. There are three notable sections from this days data. The participant is a
PhD student who was preparring for and giving a talk at the weekly meeting for his group.
The meeting commenced at 3:00pm and his talk commenced at 3:30pm. His talk went from
3:30pm to 4:00pm after which time he sat back down for the remainder of the meeting.
within this data there is no significant changes in GSR or HR that could be detected to
indicate the participants stress level. The participant later reported that he felt stressed for
the majority of the day and especially so during his talk.
Unfortunately, the device did not take and GSR readings for the period 1:20pm through to
the beginning of the meeting. If this data was present it would have provided some context
for the data from his talk which may have allowed for better understanding and analysis
of the participants data. The participant did change working locations throughout the day
including for the meeting, this could have altered the users GSR readings and may not have
allowed for meaningful analysis of the data in a way that pertains to stress.
21

22. 6.5.2 User Feedback
The participant for this design iteration testing noted many of the same things that were
noted by the participant from the previous section. The device was found to be inaccurate
in its assessment of the wearers stress level. This inaccuracy caused the system to be more of
a hindrance rather than a help to the participant as they felt they were spending too much
time clicking the green tile to take the system out of the amber and red states.
Though HR was used alongside GSR in this section to mitigate the effect that fluctuations in
GSR have on the system state, this did not prove to have the desired effect. The participant
reported that when he got up, walked around or underwent any change of environment or
physical exertion that the system would change states. Furthermore he reported that the
system appeared to change states sometimes when he felt nothing had changed at all in
terms of his environment of physical exertion.
22

23. These findings could be due to changes in cognitive load causing changes in GSR, or because
of outlier values. In order to mitigate this system behaviour a mean operation could be
applied to blocks of GSR data to account for outliers and fluctuations.
7 Conclusion
Throughout the research, design and implementation phases of this project a lot has been
learned. The way in which stress it related to measurable biosignals is well understood,
though not in a way such as to distinguish it from other types of stimulation that cause
activation of the para-sympathetic nervous system. Furthermore, a very clear distinction
needs to be made between the biosignal readings associated with stress and the biosignal
readings associated with a student with autism prior to a meltdown. This distinction is not
made in the existing literature and is necessary to explore if a fully functional system is to
be realised.
From the testing that was conducted, it is clear that changes in temperature and physica
exertion experienced by the wearer when the band is worn throughout the day causes such
a significant change in GSR to occur that it causes the system to detect stress. This is
due to these factors causing similar changes in GSR to the changes that would be seen
when a person is becoming stressed. This finding still would mean that the system could be
implemented in a classroom scenario but would not be suitable for an entire days use.
8 Future Work
There are several key areas for future work with this system. First and most simple would
be to use the system as it currently exists in a classroom setting with a student who has
asd. Collecting data for this time and then interviewing the student afterwards would yield
insights into the changes in GSR and HR that one would expect to see for a student with
autism in a classroom setting. Alternatively the user interface and haptic feedback could be
disabled for this initial testing to prevent any potential frustration caused by an inaccurate
system.
The key area that underpins this entire project is understanding the relationship that Gal-
vanic Skin Response and Heart Rate have to pre-meltdown states in students with ASD.
Without this are being clearly understood the algorithm will remain unrefined and the sys-
tem will not be sufficiently functional. Because of this, testing or data collection and analysis
from many students who have ASD is the most important next step in the system develop-
ment.
23

24. Once some preliminary findings have been made the system can be implemented in a class-
room situation to test its efficacy in enabling students to self identify with their emotional
state and in allowing teachers an insight into their students emotional state. Done correctly
this would allow for a more functional and enjoyable learning environment for all involved.
24

25. References
Cheol-Hong, M., Youngchun, K., Tewfik, A., & Kelly, A. (2009). Detection of self-stimulatory
behaviors of children with autism using wearable and environmental sensors. Journal of Med-
ical Devices, 3(2), 027506 (027501 pp.). doi: 10.1115/1.3134931
Garzotto, F., et al. (2016). Using Brain Signals in Adaptive Smart Spaces for Disabled
Children. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors
in Computing Systems. Santa Clara, California, USA, ACM: 1684-1690.
Goodwin, M. S., Groden, J., Velicer, W. F., Lipsitt, L. P., Baron, M. G., Hofmann, S.
G., & Groden, G. (2006). Cardiovascular Arousal in Individuals With Autism. Focus on
Autism and Other Developmental Disabilities, 21(2), 100-123.
Hamlin, T., & Ratey, J. (2015). Autism and the Stress Effect : A 4-step lifestyle approach to
transform your childs health, happiness and vitality Retrieved from http://QUT.eblib.com.au/patron/Fu
Hernandez, J., McDuff, D. J., & Picard, R. W. (2015, 9-12 June 2015). BioInsights: Extract-
ing personal data from &x201C;Still&x201D; wearable motion sensors. Paper presented at
the Wearable and Implantable Body Sensor Networks (BSN), 2015 IEEE 12th International
Conference on.
Hirstein, W., Iversen, P., & Ramachandran, V. S. (2001). Autonomic responses of autis-
tic children to people and objects. Proceedings of the Royal Society of London B, 268,
1883-1888. Hyland, M., & Richardson, S. (2014). Stress : All That Matters Retrieved from
http://QUT.eblib.com.au/patron/FullRecord.aspx?p=1897161
Kientz, J. A., Goodwin, M. S., Hayes, G. R., & Abowd, G. D. (2013). Interactive Tech-
nologies for Autism. Synthesis Lectures on Assistive, Rehabilitative, and Health-Preserving
Technologies, 2(2), 1-177. doi: 10.2200/S00533ED1V01Y201309ARH004
Kirsch, D. L. (2014). Stress in Health and Disease, An Issue of Psychiatric Clinics of North
America Retrieved from http://QUT.eblib.com.au/patron/FullRecord.aspx?p=1911857
Kushki, A., Drumm, E., Pla Mobarak, M., Tanel, N., Dupuis, A., Chau, T., & Anag-
nostou, E. (2013). Investigating the Autonomic Nervous System Response to Anxiety in
Children with Autism Spectrum Disorders. PLoS ONE, 8(4), e59730. doi: 10.1371/jour-
nal.pone.0059730
Lipsky, D. (2011). From anxiety to meltdown: How individuals on the autism spectrum
deal with anxiety, experience meltdowns, manifest tantrums, and how you can intervene
effectively: Jessica Kingsley Publishers.
25

26. MacLean, D., Roseway, A., & Czerwinski, M. (2013). MoodWings: a wearable biofeed-
back device for real-time stress intervention. Paper presented at the Proceedings of the
6th International Conference on PErvasive Technologies Related to Assistive Environments,
Rhodes, Greece.
Picard, R. W. (2009). Future affective technology for autism and emotion communication.
Philosophical Transactions of the Royal Society of London B: Biological Sciences, 364(1535),
3575-3584.
Tang, T. B., Yeo, L. W., & Lau, D. J. H. (2014). Activity awareness can improve continuous
stress detection in galvanic skin response. Paper presented at the 13th IEEE SENSORS
Conference, SENSORS 2014, November 2, 2014 - November 5, 2014, Valencia, Spain.
Tiinanen, S., A, M., x00E, tt, x00E, Silfverhuth, M., . . . nen. (2011, Aug. 30 2011-
Sept. 3 2011). HRV and EEG based indicators of stress in children with asperger syndrome
in audio-visual stimulus test. Paper presented at the Engineering in Medicine and Biology
Society, EMBC, 2011 Annual International Conference of the IEEE.
Villarejo, M. V., Zapirain, B. G., & Zorrilla, A. M. (2012). A Stress Sensor Based on
Galvanic Skin Response (GSR) Controlled by ZigBee. Sensors (Basel, Switzerland), 12(5),
6075-6101. doi: 10.3390/s120506075
Vittorias, J., Petrantonakis, P., Bolis, D., Tsiligkyri, A., Kosmidou, V., & Hadjileontiadis,
L. J. (2008, 1-5 July 2008). NOESIS: An Enhanced Educational Environment for Kids with
Autism Spectrum Disorders. Paper presented at the Advanced Learning Technologies, 2008.
ICALT ’08. Eighth IEEE International Conference on.
Welch, K. C. (2012). Physiological signals of autistic children can be useful. IEEE In-
strumentation & Measurement Magazine, 15(1), 28-32. doi: 10.1109/MIM.2012.6145259
Westeyn, T., Vadas, K., Bian, X., Starner, T., & Abowd, G. D. (2005, 18-21 Oct. 2005).
Recognizing mimicked autistic self-stimulatory behaviors using HMMs. Paper presented at
the Wearable Computers, 2005. Proceedings. Ninth IEEE International Symposium on.
Yamakoshi, T., Yamakoshi, K., Tanaka, S., Nogawa, M., Park, S. B., Shibata, M., . . .
Hirose, Y. (2008, 20-25 Aug. 2008). Feasibility study on driver’s stress detection from
differential skin temperature measurement. Paper presented at the 2008 30th Annual Inter-
national Conference of the IEEE Engineering in Medicine and Biology Society.
Zhai, J., & Barreto, A. (2006, Aug. 30 2006-Sept. 3 2006). Stress Detection in Computer
Users Based on Digital Signal Processing of Noninvasive Physiological Variables. Paper pre-
26

27. sented at the Engineering in Medicine and Biology Society, 2006. EMBS ’06. 28th Annual
International Conference of the IEEE.
Shi, Y., Ruiz, N., Taib, R., Choi, E., & Chen, F. (2007). Galvanic skin response (GSR)
as an index of cognitive load. Paper presented at the CHI ’07 Extended Abstracts on Hu-
man Factors in Computing Systems, San Jose, CA, USA.
27

28. 9 Time Plan
The final timeplan for this project is shown below.
28

29. 29