Frustration is an emotional reaction to the thwarting of the completion of a task or the achievement of a goal, the intensity of which depends on goal commitment and self-efficacy. The experience of frustration is correlated with undesirable behavioral consequences; in particular, frustration has been shown to impede progress in learning difficult subject matter. The emotional state of frustration reflects heightened arousal of the autonomic nervous system, which can be detected by a variety of physiological tests. Affective computing is a new subdiscipline of human-computer interaction studies that strives to train computer algorithms to automatically detect frustration, to initiate early countermeasures and cognitive, affective, and behavioral interventions. The goal is to make learning more pleasurable and less cumbersome, and ultimately improve the learning outcome. Presently, the accuracy of the affective assessments made by trained and tested artificially intelligent systems, based on fused sensor data, are comparable to the quality of judgments made by professionals specializing in human behavior. Affective computing offers the possibility of employing non-invasive technologies, such as web cams, to monitor the affective states of computer users and to initiate emotional and behavioral interventions when a counterproductive intensity of frustration is detected in a user.

1. Templin 1
Thomas Templin
Professor Sheila Sandapen
ENGL 102-196
20 March 2017
Automated Detection of Frustration
Frustration is an emotional response to an obstacle or barrier to achieving a desired goal. If a person fails
in reaching a goal, she experiences frustration because hopes have been thwarted and satisfaction has not
been achieved. Human actions are explicitly or implicitly goal-oriented, and any interruption in the
completion of a task can cause frustration. External impediments to goal attainment include the physical
environment, social, cultural, or legal barriers, and other people’s behavior. Deficiencies specific to the
individual, such as a lack of knowledge, skill, physical ability, or resources, constitute internal obstacles
(Lazar, Jones, and Shneiderman 240).
The severity of the emotional reaction and the motivation to overcome the obstruction is
dependent on goal commitment. Individuals are typically highly committed to a goal when the goal is
important to them and they believe that they can achieve it. When encountering an obstacle, a high degree
of personal identification with a goal can lead to both a high level of frustration and great efforts on the
part of the individual to surmount the impediment. In this context, the concept of self-efficacy denotes the
belief in one's personal abilities to overcome an obstacle to achieving a goal. Generally, the level of
frustration experienced depends on the interplay between the situation or environment and the
psychological make-up of the person, including her coping skills, creativity, and flexibility in behavioral
response (Locke 119, Latham and Locke 218–219).
Different affective states can have different effects on learning success or failure. While the
experience of flow, where the learner is totally absorbed in the learning process, or confusion, in which
the experience of cognitive disequilibrium forces the learner to think harder, has been found to be

2. Templin 2
positively correlated with learning, other states such as frustration, boredom, anxiety, and despair can
impede learning. Students who exhibit low self-efficacy and doubt their ability to succeed may avoid
learning or give up on the goal of mastering a field of study. The student becomes demotivated and
disengages from the material being studied (Lehman, Matthews, D’Mello, and Person 51).
Ekman and Friesen distinguish six basic emotions: fear, anger, happiness, sadness, disgust, and
surprise. However, studies on learning have shown that boredom, confusion, delight, flow, and frustration
are more prevalent in learning than the six basic emotions (D'Mello, Picard, and Graesser 54). In fact, a
study by Grafsgaard et al. found that frustration was the only feature included in the experiment that had a
significant effect on learning outcome; the correlation between degree of frustration and learning progress
was negative (162).
Some authors describe “breakdown” scenarios, due to seemingly conflicting information, where
confusion appears in a learner. Initially, this type of cognitive dissonance can have positive effects and
stimulate the student to work harder. However, if the misconception persists for a long time, remains
unresolved, and leads to frustration and/or boredom, it can “backfire” and make the student avoid or give
up on the task. This line or argumentation has been used to advocate against pure discovery-based
learning, where students “figure it out” by themselves with only minimal guidance and instruction from a
teacher. Such a learning philosophy might be too onerous on the student, lead to confusion and
disappointment, and ultimately frustration and boredom, alongside task avoidance (Kennedy and Lodge
319).
Frustration, as well as restrictive intervention styles employed by parents or teachers, may result
in effort avoidance. Generally speaking, the more pressure the student experiences, the more she resists
engaging with the learning material. Symptoms of defensive effort avoidance include statements made by
the student that she is not smart enough to understand the material, slow work, fast, but sloppy, work,
terminating work when praised, the expression of resignation, and various excuses for not doing the work
(Snow, Corno, and Jackson 275).

3. Templin 3
In the authors’ words, “Effort avoidance can be distinguished from low need for achievement
characterized by laziness or high fear of failure characterized by striving to achieve; a person motivated
by effort avoidance shows active mental or physical escape, that is, mindful avoidance, with no intention
to succeed. Effort avoidance tendencies seem to derive from frustrating early experiences in a task
domain, so the construct is usually domain-specific. But experiencing frustration in many school activities
presumably leads to generalized effort avoidance” (275).
Intelligent effort avoidance, on the other hand, denotes the budgeting of effort to reach a goal
with the minimum expenditure of resources. This type of effort avoidance, which manifests itself by
giving up on some tasks or setting lower standards, can be an adaptive and healthy response to exhausting
or extremely difficult tasks (Snow, Corno, and Jackson 275).
When a student has problems with understanding difficult technical material, the arousal of her
autonomic nervous system increases, and the student experiences unpleasant emotions, which may
demotivate or discourage the student or make her give up on pursuing the task she has been involved in.
Besides frustration, difficulties related to learning may also cause irritation, anger, rage, or despair,
uncomfortable emotions that may also interfere with task completion (D'Mello, Picard, and Graesser 53).
In order to inspire learning and prevent students from giving up on mastering challenging subject matter,
it would be important to prevent emotions inimical to learning, such as frustration, from developing in a
student or to detect them early, so that countermeasures can be taken to shift the emotional experience
into a more positive direction.
For this reason, attempts have been made, since the beginning of this century, to automatically
detect stress responses, in particular, frustration. Automated detection means that the assessment of the
emotional state of a person is made by some type of machine (broadly defined), as opposed to the person
herself or a human observer. The assessment is typically made using algorithms from artificial
intelligence, machine learning, or pattern recognition (e.g., Naïve Bayes, Decision Tree, or Support
Vector Machine) (Barreto, Zhai, and Adjouadi 29). In one study, measurements were baseline-adjusted

4. Templin 4
and had to be above an empirically determined detection threshold to be included in the analysis. A
statistical prediction model was built based on the findings (Grafsgaard et al. 161–162).
Studies have focused on the emotional states experienced by computer users. For this reason, the
topic has mostly been treated as a subdiscipline of human-computer interaction studies. In particular, the
emergent field of affective computing strives to provide computers with the capability to identify and
produce adaptive responses to the affective and cognitive states of their users, in order to provide a more
comfortable and effective user experience (Barreto, Zhai, and Adjouadi 29). In the case of detecting
stress, the computer and software would request feedback from the user concerning the cause of the
frustration and/or offer assistance or guidance (Puri et al. 1725). Researchers involved in affective
computing are typically from the fields of computer science, technology education, or psychology.
A frequently used tool in the automated detection of frustration and affective computing is an
(artificially) intelligent tutoring system, called AutoTutor, which has the ability to adapt its teaching
strategy to a learner’s progress in mastering the topic of instruction. Affective computing attempts to go a
step further and take the emotional component of the learning experience into account as well. The
objective of this augmentation is to recognize and respond to students' emotional states, in order to
achieve superior learning outcomes (D'Mello, Picard, and Graesser 53–60).
The physiological stress levels of users are measured using tests such as blood volume pulse,
galvanic skin response, skin temperature, electromyelography, and pupil diameter, to record learners'
physiological states that might be related to learning success or failure (Barreto, Zhai, and Adjouadi 29;
Puri et al. 1725). The physiological reaction reflects the “fight or flight” response of the autonomic
nervous system, which is not fully accessible to conscious mental control. Experimental measurement
devices are multiple-modality sensors that include instruments attached to the hand, cameras, an eye
tracking system, a body-posture measurement system, and the typed dialog with the computer (Barreto,
Zhai, and Adjouadi 32; Grafsgaard et al. 161). Posture sensors were best suited to detect low-arousal
states (boredom, flow), while facial-feature sensors were better at detecting states that are accompanied

5. Templin 5
by significant arousal (confusion, delight). Frustration was best detected by examining the textual dialog
between learner and tutor (D'Mello, Picard, and Graesser 57–59).
D'Mello, Picard, and Graesser state that “[t]he next step in our research will be to combine the
information from the different sensor channels into one emotion classifier for use in AutoTutor. We
envision two ways to achieve this goal. The first is to acknowledge that each sensor is best at classifying a
particular set of emotions. If so, the posture sensor would detect boredom and flow, facial-feature tracking
would detect confusion and delight, and we could classify frustration on the basis of dialogue features.
However, this approach leaves no room for improvement because we would be committed to the
maximum accuracy values affiliated with each sensor. Perhaps a more attractive alternative is to combine
features from different sensors to determine whether sensor fusion increases classification accuracy. An
initial step at sensor fusion is to develop and test four additional classification models
• dialogue + posture,
• dialogue + face,
• posture + face, and
• dialogue + posture + face” (59).
Initially, emotional states are evaluated by the users themselves, other (lay)people, and/or trained
judges. The human assessments (the “gold standard”) are correlated with the sensor measurements, in
order to train the artificially intelligent machines to accurately identify users’ affective states. The results
from multiple sensors are fused in order to improve classification accuracy (D'Mello, Picard, and
Graesser 53–60). It is challenging to detect frustration using this approach. Users tended to report
“frustration at a higher rate when they verbalized their own emotions […] than when trained judges
determined their emotions […]. This could be due to social pressures that cause people to disguise
negative emotions, such as frustration, making these emotions difficult for judges to detect. In contrast,
when participants freely reflect and report their affect, as in […] emote-aloud stud[ies], such barriers

6. Templin 6
drop” (D'Mello, Picard, and Graesser 55). Upon completion of training and testing, the evaluations of
emotional states made by computational systems rival the judgments made by humans (59).
A technique to elicit frustration and stress in computer users, frequently used in research, is the
Stroop Color Word Conflict Test. In this scheme, the name of a color is displayed in a discordant
coloration (e.g., “blue” in orange) on a computer screen. The users are asked to record the color of the
word with less and less time available to complete the task (Barreto, Zhai, and Adjouadi 29–38). In one
study, both baseline and elevated Stroop stress levels were determined by measuring oxygen consumption
with a gas mask. The idea is that increased stress leads to increased energy metabolism. Then, the authors
used an infrared, thermal-imaging camera to determine the thermal signature of the user's face and the
amount of blood flow to different areas of the face. The results show that increased user frustration was
correlated with increased blood flow to the frontal vessels of the forehead and an increased forehead
temperature. The correlation with the oxygen-consumption measurements was high. The authors propose
constant thermal monitoring of computer users in order to improve the users' experience (Puri et al. 1726–
1728).
Investigators conducting other studies identified facial indicators of frustration, including the
facial expressions outer brow raise, brow lowering, and mouth dimpling, as well as pupil diameter as
predictor of frustration. The performance of the detection algorithms substantially deteriorated when the
pupil-diameter variable was excluded. Consequently, the authors advocate the use of web cams to monitor
and react to the affective/cognitive states of computer users. Web cams and thermal-imaging cameras are
non-invasive ways of measuring affective states and thus are presumably more acceptable and tolerable to
people (Grafsgaard et al. 159–165).
In conclusion, frustration is an emotional reaction to the thwarting of the completion of a task or
the achievement of a goal, the intensity of which depends on goal commitment and self-efficacy. The
experience of frustration is correlated with undesirable behavioral consequences; in particular, frustration
has been shown to impede progress in learning difficult subject matter. The emotional state of frustration
reflects heightened arousal of the autonomic nervous system, which can be detected by a variety of

7. Templin 7
physiological tests. Affective computing is a new subdiscipline of human-computer interaction studies
that strives to train computer algorithms to automatically detect frustration, to initiate early
countermeasures and cognitive, affective, and behavioral interventions. The goal is to make learning more
pleasurable and less cumbersome, and ultimately improve the learning outcome. Presently, the accuracy
of the affective assessments made by trained and tested artificially intelligent systems, based on fused
sensor data, are comparable to the quality of judgments made by professionals specializing in human
behavior. Affective computing offers the possibility of employing non-invasive technologies, such as web
cams, to monitor the affective states of computer users and to initiate emotional and behavioral
interventions when a counterproductive intensity of frustration is detected in a user.

8. Templin 8
Works Cited
Barreto, Armando, Jing Zhai, and Malek Adjouadi. “Non-intrusive Physiological Monitoring for
Automated Stress Detection in Human-Computer Interaction.” Human-Computer Interaction,
edited by Michael Lew, Nicu Sebe, Thomas S. Huang, and Erwin M. Bakker, Springer, 2007, 29–
38.
D'Mello, Sidney, Rosalind Picard, and Arthur Graesser. “Toward an Affect-Sensitive AutoTutor.” IEEE
Intelligent Systems, vol. 22, no. 4, 2007, 53–61.
Grafsgaard, Joseph F., Joseph B. Wiggins, Kristy Elizabeth Boyer, Eric N. Wiebe, and James C. Lester.
“Automatically Recognizing Facial Indicators of Frustration: A Learning-Centric Analysis.”
Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII),
2013, 159–165.
Kennedy, Gregor and Jason M. Lodge. “All Roads Lead to Rome: Tracking Students’ Affect as They
Overcome Misconceptions.” Show Me the Learning, edited by S. Barker, S. Dawson, A. Pardo,
and C. Colvin, Proceedings ASCILITE ’16, 2016, 318–328.
Lazar, Jonathan, Adam Jones, and Ben Shneiderman. “Workplace User Frustration with Computers: An
Exploratory Investigation of the Causes and Severity.” Behaviour & Information Technology,
vol. 25, no. 3, 2006, 239–251.
Locke, Edwin A., 1996, “Motivation through Conscious Goal Setting.” Applied Preventative Psychology,
vol. 5, no. 2, 117–124.
Latham, Gary P. and Edwin A. Locke, “Self-Regulation through Goal Setting.” Organizational Behavior
and Human Decision Processes, vol. 50, no. 2, 1991, 212–247.

9. Templin 9
Lehman, Blair, Melanie Matthews, Sidney D’Mello, and Natalie Person. “What Are You Feeling?
Investigating Student Affective States During Expert Human Tutoring Sessions.” Intelligent
Tutoring Systems, edited by Beverley P. Woolf, Esma Aïmeur, Roger Nkambou, and Susanne
Lajoie, ITS ’08, Springer, 2008, 50–59.
Puri, Colin, Leslie Olson, Ionannis Pavlidis, James Levine, and Justin Starrn. “StressCam: Non-contact
Measurement of Users' Emotional States through Thermal Imaging.” CHI '05 Extended Abstracts
on Human Factors in Computing Systems, 2005, 1725–1728.
Snow, Richard E., Lyn Corno, and Douglas Jackson III. “Individual Differences in Affective and
Conative Functions.” Handbook of Educational Psychology, edited by David C. Berliner and
Robert C. Calfee, Routledge, 1996, 243–310.