1. Emotion Recognition: Benefits and Challenges
Sujith Kumar Anand
Affective Computing and
Human Robot Interaction
University College London
sujith.anand.14@ucl.ac.uk
ABSTRACT
The project aims to capture the body postures with sensing
devices and video recorder for automatic recognition of
affective states and motivate them; through feedback and
personal exercise plan.
INTRODUCTION
An injury caused in the act of participating in a sport is a
traumatic experience for the person who has dedicated their
time and energy into their fitness and achievements[1]. The
most common type of knee injury among sport
professionals are ACL, or anterior cruciate ligament[2]. A
targeted set of strength and training fitness activities can be
found useful for the personal who have ACL injuries[3].
Researchers have tried to generalize the emotional aspect of
a sport person injury[4]. But Crossman and Smith[5][6],
thinks, the post-injury responses of an ACL injured person
will be different and more complex.
Some of the commonly shown emotions to injury are
disengagement, frustration, depression, anger, tension,
disbelief, fear, rage, depression, and fatigue [6]. Johnston
and Carroll[7] studied the dissimilarities between uninjured
and injured sports athletes and pointed out the injured
athlete showed greater affect of negativism, inferior self-
esteem and very great amount of disengagement, depression
and anxiety.
ACL participants require regular fitness activities during
their rehab period to overcome their short or long term
outcomes w.r.t to their pain. However, emotions such as
fear from the pain and/or re-injury, counteracts many
individuals suffering from ACL, to hold on to an exercise
regime[4][8].
Being an athlete requires commitment, determination, and,
most importantly, a passion. During these periods, athlete
may question themselves about their identity of being an
athlete they were if they are unable to practice and
compete[6].
Squat is one of the exercises used by the physiotherapists
for ACL athletes during their rehab period. This exercise
puts every muscle below the waist into work when
performed correctly[9].
The use of technology in the field of physical activity in
increased over time but their uses in chronic pain people
performing physical activity is still in its infancy[10].
Luckily, the last research performed by [11][12][13]
showed use of pedometers to automatically track exercise
an motivate a person to perform physical activity. However,
pedometers are used in strength training exercise are not
comparable fitness activities such as running[14].
Large companies (Nike, FitBit, Microsoft) have come
forward with their own devices to support fitness activities
with the help of wearable fitness sensors such as the
FuelBand, Flex and so on. These could be using while
performing an exercise to track their calorie outflow and
overall activity information. But they fail to information
such as (reps, sets, time taken)[14]
The use of camera along with sensor technology doesn’t
robustly handle different body motions and postures while
performing strength-training exercises such as squats.
In this work, the goal is to understand the advantages and
disadvantages of using sensing device i.e., Microsoft Kinect
sensor and external devices such as empatica and video
camera to build a design system in identifying the affective
states i.e., engagement and disengagement (in our case)
with the context of fitness.
BACKGROUND & RELATED WORK
What is ACL?
The full form of ACL is Anterior Cruciate Ligament. This
injury occurs in the knee and this is the most complex and
largest joint in the human body. The knee depends on
ligaments, muscles, tendons and secondary ligaments to
function consistently [15].
How is the ACL Injured?
The most common way for an ACL person to be injured is
through a direct contact to the knee, which usually occurs in
football. This case occurs when one or more ligaments are

2. torn between the knee and another object from an abnormal
position. There are chances where ACL injury can occur
without the external contact of another object such as a
running athlete changing direction or hyperextends their
knee while landing from a jump [15].
What does ACL injury recovery entail?
ACL reconstruction is conducted through rehabilitation,
which requires time and hard work. A minimum of 6 weeks
to 6 months are required varying on the individual activity
and severity level. Many research studies have proved 90
percent of the ACL patients are able to normal sporting
duty without the symptoms of knee instability [15].
Body Expression
de Gelder[16] notes, 95% of the studies conducted in the
field of emotion in humans are suing facial expression
whereas the information from sounds such as voice, music
and environment, makes up the remaining 5% on whole-
body expressions.
Kleinsmith et al[17] cite body expressions are identified as
more important for non-verbal communication. The change
in body posture tends to change in the person’s affective
state[18][19]. Additionally, some affective state expressions
can be very well communicated via body than face[20].
Research studies have expressed people tend to control
facial expressions more than body expressions[21] and
similarly people trust body expressions more when
compared with facial expressions[22]. Therefore, we focus
extensively on body expressions to identify the participant’s
engagement in performing a squat.
Studies in the field of physiology are using body expression
to evaluate the features of the attributed body to recognize
the affective states that need to be identified. Aronoff et al.
[23] use acted movements with angular and diagonal
postures to signify threatening behavior. James[24]
expresses the importance of body openness, leaning
direction and head position for distinguishing several
affective states. Similarly, Wallbott[19] uses the arm,
shoulder, and head positions to distinguish between 14
emotions and Coulson[25] used computer-generated avatars
to examine postural features for assigning affect to body
posture. Similarly, in our work we used body expression to
identify affective states
Recognition of Body Expressions
Acted sets
Most of the automatic recognition systems to identify
affective states in the current era are based on the Laban
movement analysis [26]. Camurri et al. [27] examined body
expressions to identify emotional expressions in dance. The
results for automatic recognition of four emotions were
ranging between 31-46%, whereas 56% was recognized for
observers rating [28].
Berthouze et al. [29] use acted postures, which are labeled
by observers to recognize emotions based on low-level
features from the body joints. The recognition model
yielded 90% accuracy.
Similarly, Berhardt and Robinso [30], Pollick et al, [31]
build recognition model using acted sets. In our work, we
use acted sets as automatic label without the observers to
identify affective states.
Unacted or non-acted sets
In Picard's studies [32] [33], non-acted sets to build
multimodal recognition model from body expressions to
identify the various levels of interest [32] and self-narrated
frustration [33]. They obtained 55.1% recognition accuracy
for facial expressions, body postures, and game-state
expressions.
Similarly, Kleinsmith et al, [17] proposed a computational
model to identify affective states from low-level non-acted
body postures. In this study, external observers are used for
extensive analysis agreement to identify base rate
In our work, we use non-acted body postures to recognize
engagement and disengagement affective states. This
labeling of affective states for non-acted is performed by
observers.
Technology support for physical activity in ACL
Self-supervision and physical motivation activity are
increasingly in terms technology support[10]. However,
they fall short for addressing chronic pain such as fear of
movement. Existing technology are not detailed in terms of
to help chronic pain users with detailed valued help, which
is valued by users[34]. Smartphones apps are better to acute
pain rather chronic pain.
WebMD PainCouch
(www.webmd.com/webmdpaincoach.app) was developed
along with the healthcare professional to enable users to
monitor their pain and set and track activity goals,
generating related messages. Similarly, Google PACO, a
self-monitoring app allows users to create personalized
monitoring for specific exercises and identifying relevant
factors. These apps are such supportive to monitoring
physical and psychological states but fail in support and
engaging behavior present in chronic pain people i.e.,
guarding. People with chronic pain suggest friendly
experiences help with recovering from pain[10].

3. This suggests a need for supportive experience in pain
management with better understanding of engagement.
Other technologies for physical activity
Consoles such as Nintendo Wii are used in the areas of
rehabilitation such as stroke therapy. The Microsoft Kinect
has used for helping physical activity in older adults by
prototype games[10]. Use of sensing devices to provide
information of the correctness of the movement and to
increase the fitness level are increasing such as riablo
(www.corehab.com). Riablo uses accelerometer data used
by patients and remotely send the information to clinicians
to monitor. However, a technology that adapts along with
people emotions is still immature with no use in the
physical context of rehabilitation[35].
In our study, Microsoft Kinect sensor is used to record the
physical activity i.e., squat to create the recognition model
along with two other devices (explained in below section).
Full squat vs. Half squat
Drinkwater et al, [36] explains fitness industry have poorly
understood squat and, in particular, squat depth. There is
still confusion in understanding what a full squat implies to
parallel squat or half squat for some people. Here are few
definitions[37]:
Half or Parallel squat: This is where a person performing a
squat will use his hip joint going down as far as being
parallel to their knees. This formation leads to the formation
of 90% of knee flexion.
Deep or Full squat: This is where the hip joint goes down
are well below the knee joint.
Partial Squat: This is where the hip joint goes down till they
are just above the knees.
The squat can be performed using weights just above the
shoulder either front or back, which is ideal for the gym but
ideal for rehabilitation setup.
In our study, understanding the context of the ACL athletes.
The full, half and partial squats are considered and used for
labeled of affective states (explained in below section).
Physiotherapy exercises
Athletes having ACL go through a rehab process by
exercising and keeping them healthy. Physiotherapy is the
process, they go through to recover and manage the pain to
improve their strength and flexibility. With physiotherapy,
they try to understand the problem and advise and help the
athletes to perform a variety of simple and exercises. Some
of the exercises are used by physiotherapist are Leg stretch,
Leg cross, leg raises, sit/stands, step ups, knee squats or half
squat and so on[38] [39].
Ideally, ACL person will be performing squat only in their
advanced treatment due to its complexness and considering
the person capability to withstand the pain[39].
DATA COLLECTION
Use of sensors
Which ones?
Two different sensing devices were used for collecting
body postures i.e., Microsoft Kinect Xbox One and
Empatica. Alongside, these two devices an external video
recorder from the apple laptop was also used.
Why were selected?
The sensor from the Kinect Xbox was used to collect the
physiological data or the various body joint positions while
the user in action i.e., different body postures. Empatica
was considered to see the possibility identifying
information accelerometer and galvanic sensor. These two
sensing devices were considered to collect the numerical
data to identify the affective states of the users. However,
after initial assessment empatica data was disregarded due
to the complexity in identifying squat numerical data.
However, numerical data from Kinect provided clear
differentiation between various squats performed by the
users. A video recorder was also used as an external device
to record users performing squats. Observers were provided
with non-acted squat sets for labeling the affective state of
the users while performing the squat i.e., engaged or
disengaged.
Setup
The data was set up in a closed university room, similar to a
laboratory setup. The room was spacious and comfortable
to accommodate the involved participants. The participants
were allowed to wear clothes comfortable to them. A
Kinect sensor was placed above the video camera and in
front of the users at certain position to identify all the body
joint positions. The main idea was to capture the whole
body of the user performing squat in front of the device.
Empatica device was asked to wear either on their left or
right hand depending on their choice. The majority of the
times users wore the device on their right wrist. During the
feature labeling and extraction, it was understood a fixed
position to keep both feet’s were needed. Since all the
participants had different foot positions while performing
each squats section. Due to this, normalization was very
difficult to be created and was agreed to be neglected.
Additionally, their body measurements were also different
from each other making it difficult to identify the ground
truth.

4. Participants
Due to the ethical limitations ACL users were not involved
in this study whereas students who are performing this
study were only considered as participants i.e., 3 people i.e.,
P1, P2, P3. The participant’s age group range between the
years of 25 to 28 with previous knowledge of performing a
squat. The students involved were healthy while performing
the squat activity.
Process
The participants were asked to perform two sets i.e., acted
and non-acted. Four affective states were considered
initially, i.e., engaged, disengaged, energetic and tired.
During the acted sets, the participants were advised to
perform 5 squats of acted affective state squats. A clear
verbal instruction was exchanged between the participants
on how to perform four affective state squats. After
completion of acted sets, the participants were asked to
perform 40 squats of non-acted squats with an interval of 2
minutes after the 20th squat. The whole process is repeated
twice with the three participants. There was a total of 120
acted and 240 non-acted sets at the end of the process. This
data Kinect and video recorder were individually
segregated to ease the further process.
FEATURE LABELLING
The participants labeled the acted sets automatically. Since
participants were explained and agreed together about the
instruction of performing squat in their respective affective
state. However, this might not be an idealistic approach
since there are high chances where the participants have
failed to portray the actual affective state correctly and
might have inter performed the affective states.
The non-acted sets were 240 in total after the process. Each
participant has performed 40 sets. Due to the time involved
in labeling the non-acted sets by external observers. The
non-acted sets were further divided into segments. 12
segments were created with each segment containing 5 non-
acted squats (i.e., a total of 60 non-acted squats) from each
participant. Also, it was made user the each segment would
contain at least one non-acted squat from each participant.
Twelve observers were recruited from the Ifor Evans Hall,
Camden road, London (a UCL University Residence). The
observers were consented to participate in this labeling. The
whole process was explaining and pointing out the research
would yield not benefits. The age group of the observers
was between 21 to 28 years. These 12 observers were asked
to label 4 affective states from 2 segments of 10 non-acted
squats i.e., 5 squats as engaged or disengaged and the rest
of the 5 squats with energetic and tired. The non-acted
squats recorded in the video recorder were considered for
observer labeling. The squats were cropped individually
with a time ranging from 30-50 seconds. The whole process
of observer labeling lasted between 2 to 4 minutes. The
observers were screened and selected to participate if they
had a previous knowledge of performing squats. There
faces in the video recording of users performing squat were
not covered, so there are chances of facial expression
biasing and performing labeling without their full interest
and could have used sensor blob video. One important
comment from the observer was why didn’t we perform
squat sideways to the Kinect and video recorder instead of
facing forward to the setup.
The four affective states i.e., engaged, disengaged,
energetic and tired. Since from the literature it showed
many ACL athletes are engaged, disengaged, energetic and
tired, affective states were disregarded in the furthered
process since the there two states were not ideal or best
suited with the ACL chronic pain users.
FEATURE EXTRACTION
We used skeleton data to capture captured skeleton data
from Kinect sensor, which is comprised of three-
dimensional coordinates (see Figure 2) for the markers or
attributes depicted in Figure 1.
The Kinect Xbox sensor has identified 75 markers or
attributes from each participant (see table 1). Furthermore,
the attributes and be classified into 5 segments i.e., cervical
and thoracic spine, left arm, right arm, left leg and right leg.
Figure 1. Skeleton data

5. Figure 2. Skeleton data (3D)
Segments Attributes
Cervical and thoracic spine Head, neck, spine, spine
base, spine shoulder
Left arm Left shoulder, left elbow,
left hand, left wrist, left
fingertip, left thumb
Left leg Right shoulder, Right
elbow, Right hand, Right
wrist, Right fingertip, Right
thumb
Right arm Right shoulder, Right
elbow, Right wrist, Right
hand, Right fingertip, Right
thumb
Right leg Left shoulder, Left elbow,
Left wrist, Left hand, Left
fingertip, Left thumb
Table 1. Markers identified from Kinect Xbox sensor
As explained in literature, half squat is performed only in
vertical motion only in the direction of y-axis (See Figure
2) i.e., where a person performing a squat will use his hip
joint going down as far as being parallel to their knees. This
formation leads to formation of 90% of knee flexion (See
Figure 3).
Figure 3. Half Squat
So wr.t to the above reason we disregarded x and z-axis
attributes and thus 25 attributes remains.
Furthermore, we disregard 4 attributes from y-axis i.e., right
ankle, left ankle, right foot, left foot. Since they have very
minimal movement in y-axis more or less they remain
motionless compared to other attributes that have very high
variation (See Figure 4), thus 25 attributes reduces to 21
attributes.
The actual sensor data records 30 frames per second. This
actually leads to unwanted frames since there were
situations where the sensor was not stopped when
participant was taking rest or pre and post squat phase,
where the participant will be motionless. So we later we
refined each attribute into 3 attributes according to each
squat i.e., upper 1, lower and upper 2.
This position is considered when a person is starting to
perform the squat i.e., upper 1. Later, when the participant’s
hip joint going down as far as being parallel to their knees
or to the lowest point of the hip from the standing position
before he pulls back himself, this position in the attribute is
considered as lower and finally when the participant comes
back to the initial position, this is called as upper 2. This
refinement is performed for each squat i.e., for both acted
(63 squats) and non-acted sets (61 squats). Finally, a total of
63 attributes are created after the consideration of 3 points
for each squat, i.e., each squat will be having attributes for
e.g., spine_base Y_Upper_1, spine_base Y_Lower and
spine_base Y_Upper_2
Figure 4. A chart of 25 attributes over y-axis, where four
attributes are highlighted in red color to point out the
inconsistency from the remaining attributes.
FEATURE MODELLING
We tested our model i.e., 63 attributes with seven
classification algorithms, ZeroR, OneR, NaiveBayes, SMO,
J48 and Random Forest to identify the most discriminative
attributes identifying the affective states. To facilitate the
implementation, Weka version 3.6.12[40], The acted and
non-acted data sets were evaluated with all the algorithms
mentioned above with 10 fold cross-validation method
except ZeroR as training set, which was used to identify the
baseline accuracy.

6. Method Accuracy
ZeroR 51%
OneR 57%
Naïve Bayes 68%
Nearest neighbor (IBK) 75%
Random Forest 73%
J48 81%
SMO 78%
Table 2. Various algorithms used to acted sets to identify the
algorithm with best accuracy.
a b <-- classified as
27 4 a = Yes
8 24 b = No
Table 3. Confusion matrix for J48 algorithm with 10-cross
validation training.
In acted sets, The performance accuracy from J48 was
highest i.e., 81%; compared to other algorithm used. Also
all the algorithm’s accuracy was higher than the baseline
accuracy (J48).
Method Accuracy
ZeroR 65%
OneR 47%
Naïve Bayes 43%
Nearest neighbor (IBK) 52%
Random Forest 55%
J48 60%
SMO 60%
Table 4. Various algorithms used on non-acted sets to identify
the algorithm with best accuracy.
In this case of non-acted sets, its clearly identifiable the
accuracy of OneR, NaiveBayes, SMO, J48 and Random
Forest are below the accuracy (65%) of the ZeroR.
According to Ian[41], if the baseline accuracy i.e., the
accuracy of ZeroR more than the algorithms used then the
used dataset is not good. External observers who might
have resulted to this situation performed the labeling of the
un-acted data sets.
Method Accuracy
Zero R 50%
OneR 38%
Naïve Bayes 43%
Nearest neighbor (IBK) 52%
Random Forest 52%
J48 52%
SMO 67%
Table 5. Leave one person out, where participant P3 was used
as test set to validate the training set of P1+P2.
Method Accuracy
Zero R 50%
OneR 65%
Naïve Bayes 45%
Nearest neighbor (IBK) 55%
Random Forest 50%
J48 45%
SMO 55%
Table 6. Leave one person out, where participant P1 was used
as test set to validate the training set of P2+P3.

7. Method Accuracy
Zero R 50%
OneR 55%
Naïve Bayes 55%
Nearest neighbour (IBK) 50%
Random Forest 73%
J48 82%
SMO 59%
Table 7. Leave one person out, where participant P2 was used
as test set to validate the training set of P1+P3.
Cases Accuracy
1 67%
2 65%
3 82%
Table 8. Leave one person out, maximum accuracies identified
from the above three possibilities..
To conclude, we performed leave on person out wherein the
acted sets from two participants P1and P2 were considered
to test with the P3. This was repeated with all the
possibilities (See Figure 5,6,7) and finally the highest
accuracy was considered from the three possibilities (See
Figure8) to identify the mean performance of the model i.e.,
71%.
The leave one person out was considered for non-acted
because of the baseline accuracy problem explained above.
However, during the acted set validation, J48 algorithm
provides 81% accuracy with 5 leaves and 9 trees and
discriminative attributes identified are
right_fingertip_lower (Parent node),
right_wrist_lower (child node),
right_elbow_lower (grandchild node).
Few more iterations were performed by remove dominant
attributes but the accuracy dropped drastically. However,
the dominant attributes were either from left or right finger,
hands, shoulders i.e., left or right hands
So we agreed this presented model is better than the rest
and our model requires hands to automatically predict the
affective states i.e., engaged and disengaged.
PERSONALIZED EXERCISE PLAN
As explained, by Johnston and Carroll[7] injured sports
athletes showed greater amount of disengagement,
depression and anxiety. As a result, engaged and
disengaged were considered, leaving out energetic and
tired. Also, asking ACL patients to be energetic in the
initial phase of their treatment is impossible.
As pointed before ACL can be cured and treated with
rehabilitation process. For ACL patients, strengthening of
the muscles in the close range of knee, particularly
hamstrings are required.
As explained in the lecture 5 of the class, persuasion can be
considered in an attempt to an attempt to change attitudes or
behaviors for creating a personalized exercise plan for an
ACL patient e.g., motivating patient to perform a perfect
half squat.
It is also important to perform the squat in correct
positioning. So ACL patients can be provided with a
consent sheet informing, how to perform a squat or through
assistance. Here are some of the positions to remember
while performing a squat [42].
Neck: It is important to keep the neck in neutral position.
The ideal way is to look forward and stick to it until the
squat or a set is finished.
Lower Back: Similar to neck, keeping lower back in neutral
position is important. However, to achieve this position,
chest can be squeezed up as much as possible to have a
neutral lower back.
Back at the hips first: The effective way to perform the
squat is by sticking the butt back instead of pushing knees
forward. This provides better stability to hamstrings and
glutes by putting less stress on knee.
Knee: The knee must be kept inline with toes.
Stay on the heels: If “back at the hip first” can’t be followed
then this can be followed i.e., to stay on heal while
performing the squat.
After carful considerations, as majority of the ACL patients
visit physiotherapist or knee doctors for checkups only once
a week and learn home exercise program to be performed at
their home environment[39].

8. As explained in sportshealth.com, ACL patients will be able
to perform half, partial squat and wall squat during their 2 -
4th
weeks of their rehab sessions.
Keeping this in mind, for the week 2, asking the patients to
perform half squat is too pushy. So as explained in the
lecture 5, its better to use macrosuasion for patients to
perform at their own pace and ability. Also, it’s
recommended, to perform each squat for minimum of 10
seconds with a repetition of 10 times.
During the week two, patients can be asked to perform wall
squats or partial squat once every day for three sets or more
with the support of wall. With wall squats, user stand on
their back against a wall, keeping feet at shoulder width
apart with 40 centimeters away from the wall. Then
patients are allowed to slide down until they start to sense
the pain in their knee and hold for 5 seconds to see
improvement (See Figure 5).
Figure 5. An ACL patient performing wall squat during
the week two of their exercise plan.
In week 3, they can perform partial squat without the
support of wall. In this squat, they follow the same
procedure of performing 10 times of 3 sets while holding
for 5 seconds when starting to feel pain in their knee while
pushing their hip downwards[43] (See Figure 6).
Figure 6. An ACL patient performing partial squat
during week three of their exercise plan without support
of wall.
Finally, in week 4 they are asked to perform partial squat
with the same procedure of performing 10 times of 3 sets
with holding time of 10 seconds.
Figure 7. An ACL patient performing half squat during
week four of their exercise plan.
FEEDBACK
However, setting the exercise plan for ACL patients in
devices ambitious since they are already in a pain to walk
or not at all in a possible to move legs with or without
support of hands, so handheld devices to display an exercise
plan is difficult. We could use the functional triad model or
persuasive tools to identify the best-suited way of display
personalized display panel. However, a mounted display or
anthropomorphic interfaces can be used to provide
feedback.
The current work on chronic pain[44] and use of
technology for chronic pain[45] shows sound feedback
eases self-analysis and anxiety reduction. The body
function increases to teach mindfulness skills using aural
feedback[45]. Sound feedback, have an advantage over
visual display due to its flexibility of providing feedback
with movement and doesn’t require fixation on a
display[10]. Sound feedback do have a positive effect over
motor rehabilitation e.g. smartphone devices [46] and
sonification [47] such as introducing movement,
facilitating coordination and performance improvement.
So the sound feedback suits best for our study where people
are with chronic pain in their knees and they introduce
movements to perform activities. Aural feedback can be
used similarly to the work. Similarly, [10][48] have used
aural feedback in their work where they provide feedback
to patients when the chronic patients bend forward and
backward accordingly.
As pointed, the affective states such as engaged and
disengaged can be considered to identify the person. The
ACL patients perform engaged and disengaged squats in the
vertical position. So an aural feedback can be provided

9. based on the patient's movement up and down (See Figure
7).
However, a problem arises during wall squat. This is the
phase where ACL patients are in their starting phase of
recovery and use the wall to perform squats, but patients
use hands supports themselves by resting on the wall to
perform squat. But, in our feature modeling left or right
hand had the dominant impact. However, after second
iteration spine was the dominant attribute. So this actually
helps us to put the sensor or the feedback devices on the
central hip position (See Figure8) where it could
automatically identify affective states from squat.
Figure 8. A look-a-alike wearing sensing device over hip.
Figure 9. Sound feedback and exercise poses.
As shown in the above Figure 8 and 9[48], the feedback
device can be placed behind or front of the hip. The patient
could set the minimum and maximum posture performing
up and down while performing during the weeks 2 to 4 via
external button. The pitch rises sound while the patients
reaching down and the reduces while coming back to
normal position. If the patients perform the squat faster,
then a faster sound can be generated to inform the patient to
slow down. This way patient’s self-analysis will increase
effectively and reduces over-active movements.
DISCUSSION
Identifying the ground truth with our affective states was
really difficult as pointed in the lecture too[49]. User of
external observer could have been handled better way. With
the current observer labeling the data was not good to
automatic recognition of affective states. Similarly, we
could have faced opposite to Kinect while performing squat
rather then facing forward. Also, we could have prevented
biased observer labeling by providing blob videos from
Kinect rather than video recording.
CONCLUSION
[1] “A Pain in the Brain: The Psychology of Sport and
Exercise Injury,” www.ideafit.com. [Online].
Available: http://www.ideafit.com/fitness-library/a-
pain-in-the-brain-the-psychology-ofsport-and-
exercise-injury. [Accessed: 28-May-2015].
[2] “Statistics on ACL Injuries in Athletes,”
LIVESTRONG.COM. [Online]. Available:
http://www.livestrong.com/article/548782-statistics-
on-acl-injuries-in-athletes/. [Accessed: 28-May-
2015].
[3] A. W. Kiefer, A. M. Kushner, J. Groene, C.
Williams, M. A. Riley, and G. D. Myer, “A
Commentary on Real-Time Biofeedback to Augment
Neuromuscular Training for ACL Injury Prevention
in Adolescent Athletes,” J. Sports Sci. Med., vol. 14,
no. 1, pp. 1–8, Jan. 2015.
[4] C. Klenk, “Psychological response to injury,
recovery, and social support: A Survey of athletes at
an NCAA Division I University.,” 2006.
[5] J. Crossman, “Psychological rehabilitation from
sports injuries,” Sports Med. Auckl. NZ, vol. 23, no.
5, pp. 333–339, May 1997.
[6] A. M. Smith, S. G. Scott, and D. M. Wiese, “The
psychological effects of sports injuries. Coping,”
Sports Med. Auckl. NZ, vol. 9, no. 6, pp. 352–369,
Jun. 1990.
[7] L. Johnston and D. Carroll, “The psychological
impact of injury: effects of prior sport and exercise
involvement,” Br. J. Sports Med., vol. 34, no. 6, pp.
436–439, Dec. 2000.
[8] M. L. Shuer and M. S. Dietrich, “Psychological
effects of chronic injury in elite athletes.,” West. J.
Med., vol. 166, no. 2, pp. 104–109, Feb. 1997.
[9] “Research Review: Front or back squats,” Precision
Nutrition. .
[10] A. Singh, A. Klapper, J. Jia, A. Fidalgo, A. Tajadura-
Jiménez, N. Kanakam, N. Bianchi-Berthouze, and A.
Williams, “Motivating People with Chronic Pain to
Do Physical Activity: Opportunities for Technology
Design,” in Proceedings of the 32Nd Annual ACM
Conference on Human Factors in Computing
Systems, New York, NY, USA, 2014, pp. 2803–2812.
[11] D. M. Bravata, C. Smith-Spangler, V. Sundaram, A.
L. Gienger, N. Lin, R. Lewis, C. D. Stave, I. Olkin,
and J. R. Sirard, “Using pedometers to increase
physical activity and improve health: a systematic
review,” JAMA, vol. 298, no. 19, pp. 2296–2304,
Nov. 2007.
[12] C. B. Chan, D. A. J. Ryan, and C. Tudor-Locke,
“Health benefits of a pedometer-based physical

10. activity intervention in sedentary workers,” Prev.
Med., vol. 39, no. 6, pp. 1215–1222, Dec. 2004.
[13] D. Merom, C. Rissel, P. Phongsavan, B. J. Smith, C.
Van Kemenade, W. J. Brown, and A. E. Bauman,
“Promoting walking with pedometers in the
community: the step-by-step trial,” Am. J. Prev.
Med., vol. 32, no. 4, pp. 290–297, Apr. 2007.
[14] D. Morris, T. S. Saponas, A. Guillory, and I. Kelner,
“RecoFit: Using a Wearable Sensor to Find,
Recognize, and Count Repetitive Exercises,” in
Proceedings of the 32Nd Annual ACM Conference on
Human Factors in Computing Systems, New York,
NY, USA, 2014, pp. 3225–3234.
[15] P. on Wed and O. 27, “The Anterior Cruciate
Ligament (ACL).” [Online]. Available:
http://www.foundrysportsmedicine.com/our-
blog/bid/47663/The-Anterior-Cruciate-Ligament-
ACL. [Accessed: 30-May-2015].
[16] B. de Gelder, “Why bodies? Twelve reasons for
including bodily expressions in affective
neuroscience,” Philos. Trans. R. Soc. Lond. B. Biol.
Sci., vol. 364, no. 1535, pp. 3475–3484, Dec. 2009.
[17] A. Kleinsmith, N. Bianchi-Berthouze, and A. Steed,
“Automatic Recognition of Non-Acted Affective
Postures,” Trans Sys Man Cyber Part B, vol. 41, no.
4, pp. 1027–1038, Aug. 2011.
[18] A. Mehrabian and J. T. Friar, “Encoding of attitude
by a seated communicator via posture and position
cues,” J. Consult. Clin. Psychol., vol. 33, no. 3, pp.
330–336, 1969.
[19] H. G. Wallbott and K. R. Scherer, “Cues and
channels in emotion recognition,” J. Pers. Soc.
Psychol., vol. 51, no. 4, pp. 690–699, 1986.
[20] P. Ekman and W. V. Friesen, “Nonverbal leakage and
clues to deception,” Psychiatry, vol. 32, no. 1, pp.
88–106, Feb. 1969.
[21] P. Ekman and W. V. Friesen, “Detecting deception
from the body or face,” J. Pers. Soc. Psychol., vol.
29, no. 3, pp. 288–298, 1974.
[22] C. C. R. J. van Heijnsbergen, H. K. M. Meeren, J.
Grèzes, and B. de Gelder, “Rapid detection of fear in
body expressions, an ERP study,” Brain Res., vol.
1186, pp. 233–241, Dec. 2007.
[23] J. Aronoff, B. A. Woike, and L. M. Hyman, “Which
are the stimuli in facial displays of anger and
happiness? Configurational bases of emotion
recognition,” J. Pers. Soc. Psychol., vol. 62, no. 6,
pp. 1050–1066, 1992.
[24] W. T, “A study of the expression of bodily posture,”
J. Gen. Psychol., vol. 7, pp. 405–437, 1932.
[25] M. Coulson, “Attributing Emotion to Static Body
Postures: Recognition Accuracy, Confusions, and
Viewpoint Dependence,” J. Nonverbal Behav., vol.
28, no. 2, pp. 117–139, Jun. 2004.
[26] R. von Laban, The mastery of movement. Macdonald
and Evans, 1980.
[27] A. Camurri, B. Mazzarino, M. Ricchetti, R. Timmers,
and G. Volpe, “Multimodal Analysis of Expressive
Gesture in Music and Dance Performances,” in
Gesture-Based Communication in Human-Computer
Interaction, A. Camurri and G. Volpe, Eds. Springer
Berlin Heidelberg, 2004, pp. 20–39.
[28] A. Camurri, I. Lagerlöf, and G. Volpe, “Recognizing
emotion from dance movement: comparison of
spectator recognition and automated techniques,” Int.
J. Hum.-Comput. Stud., vol. 59, no. 1–2, pp. 213–
225, Jul. 2003.
[29] P. R. De Silva and N. Bianchi-Berthouze, “Modeling
Human Affective Postures: An Information Theoretic
Characterization of Posture Features: Research
Articles,” Comput Animat Virtual Worlds, vol. 15,
no. 3–4, pp. 269–276, Jul. 2004.
[30] D. Bernhardt and P. Robinson, “Detecting Affect
from Non-stylised Body Motions,” in Affective
Computing and Intelligent Interaction, A. C. R.
Paiva, R. Prada, and R. W. Picard, Eds. Springer
Berlin Heidelberg, 2007, pp. 59–70.
[31] Y. Ma, H. M. Paterson, and F. E. Pollick, “A motion
capture library for the study of identity, gender, and
emotion perception from biological motion,” Behav.
Res. Methods, vol. 38, no. 1, pp. 134–141, Feb. 2006.
[32] A. Kapoor, R. W. Picard, and Y. Ivanov,
“Probabilistic combination of multiple modalities to
detect interest,” in Proceedings of the 17th
International Conference on Pattern Recognition,
2004. ICPR 2004, 2004, vol. 3, pp. 969–972 Vol.3.
[33] A. Kapoor, W. Burleson, and R. W. Picard,
“Automatic Prediction of Frustration,” Int J Hum-
Comput Stud, vol. 65, no. 8, pp. 724–736, Aug. 2007.
[34] B. A. Rosser, P. McCullagh, R. Davies, G. A.
Mountain, L. McCracken, and C. Eccleston,
“Technology-mediated therapy for chronic pain
management: the challenges of adapting behavior
change interventions for delivery with pervasive
communication technology,” Telemed. J. E-Health
Off. J. Am. Telemed. Assoc., vol. 17, no. 3, pp. 211–
216, Apr. 2011.
[35] M. S. H. Aung, B. Romera-Paredes, A. Singh, S.
Lim, N. Kanakam, A. C. de C Williams, and N.
Bianchi-Berthouze, “Getting RID of pain-related
behaviour to improve social and self perception: A
technology-based perspective,” in 2013 14th
International Workshop on Image Analysis for
Multimedia Interactive Services (WIAMIS), 2013, pp.
1–4.
[36] E. J. Drinkwater, N. R. Moore, and S. P. Bird,
“Effects of changing from full range of motion to
partial range of motion on squat kinetics,” J. Strength
Cond. Res. Natl. Strength Cond. Assoc., vol. 26, no.
4, pp. 890–896, Apr. 2012.
[37] “How are partial squats and full squats different?,”
Strength & Conditioning Research. .

11. [38] G. St Thomas, “Physiotherapy following anterior
cruciate ligament (ACL) reconstruction.” © 2013
Guy’s and St Thomas’ NHS Foundation Trust, Feb-
2013.
[39] “Your Quick Guide to ACL Rehab Exercises,”
About.com Health, 15-Dec-2014. [Online].
Available:
http://sportsmedicine.about.com/od/surgeryrehab/a/A
CLSurgeryRehab.htm. [Accessed: 30-May-2015].
[40] “Weka 3 - Data Mining with Open Source Machine
Learning Software in Java.” [Online]. Available:
http://www.cs.waikato.ac.nz/ml/weka/. [Accessed:
31-May-2015].
[41] “Data Mining with Weka (2.4: Baseline accuracy) -
YouTube.” [Online]. Available:
https://www.youtube.com/watch?v=MrQhW4FyNW
4. [Accessed: 31-May-2015].
[42] “How To Do Squats Properly - The Ultimate Guide.”
[Online]. Available: http://johnalvino.com/how-to-
do-squats/. [Accessed: 31-May-2015].
[43] “Iowa Governor’s Council on Physical Fitness &
Nutrition Exercise #6 HALF SQUAT.” .
[44] M. S. Cepeda, D. B. Carr, J. Lau, and H. Alvarez,
“Music for pain relief,” in Cochrane Database of
Systematic Reviews, John Wiley & Sons, Ltd, 1996.
[45] J. Vidyarthi, B. E. Riecke, and D. Gromala, “Sonic
Cradle: Designing for an Immersive Experience of
Meditation by Connecting Respiration to Music,” in
Proceedings of the Designing Interactive Systems
Conference, New York, NY, USA, 2012, pp. 408–
417.
[46] B. A. Rosser and C. Eccleston, “Smartphone
applications for pain management,” J. Telemed.
Telecare, vol. 17, no. 6, pp. 308–312, Sep. 2011.
[47] K. Vogt, D. Pirrò, I. Kobenz, R. Höldrich, and G.
Eckel, “PhysioSonic - Evaluated Movement
Sonification as Auditory Feedback in
Physiotherapy,” in Auditory Display, S. Ystad, M.
Aramaki, R. Kronland-Martinet, and K. Jensen, Eds.
Springer Berlin Heidelberg, 2010, pp. 103–120.
[48] “Chronic Pain Rehabilitation Assistant | Ye’s
Stories.” [Online]. Available:
http://yelin.in/works/chronic.html. [Accessed: 31-
May-2015].
[49] R. W. Picard, E. Vyzas, and J. Healey, “Toward
machine emotional intelligence: analysis of affective
physiological state,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 23, no. 10, pp. 1175–1191, Oct.
2001.
The important factor for an ACL athlete is to be motivated.
They expressed lack of motivation in performing strength-
training exercises. This paper also points out the lack of
technology support towards chronic pain patients especially
towards ACL athletes. A need to build a dedicated system
for ACL patients can be seen expressed in this paper
highlight the various problem we faced though sensing
devices and video recorder to automatic recognition of
affective states. However, the acted sets provided a
accuracy of 81% which is really good. The discriminative
features identified are arms in particular hands. Finally a
personalized exercise and feedback plan was proposed to
increase the motivation of the ACL patients.
REFERENCES
[1] “A Pain in the Brain: The Psychology of Sport and
Exercise Injury,” www.ideafit.com. [Online].
Available: http://www.ideafit.com/fitness-library/a-
pain-in-the-brain-the-psychology-ofsport-and-
exercise-injury. [Accessed: 28-May-2015].
[2] “Statistics on ACL Injuries in Athletes,”
LIVESTRONG.COM. [Online]. Available:
http://www.livestrong.com/article/548782-statistics-
on-acl-injuries-in-athletes/. [Accessed: 28-May-
2015].
[3] A. W. Kiefer, A. M. Kushner, J. Groene, C.
Williams, M. A. Riley, and G. D. Myer, “A
Commentary on Real-Time Biofeedback to Augment
Neuromuscular Training for ACL Injury Prevention
in Adolescent Athletes,” J. Sports Sci. Med., vol. 14,
no. 1, pp. 1–8, Jan. 2015.
[4] C. Klenk, “Psychological response to injury,
recovery, and social support: A Survey of athletes at
an NCAA Division I University.,” 2006.
[5] J. Crossman, “Psychological rehabilitation from
sports injuries,” Sports Med. Auckl. NZ, vol. 23, no.
5, pp. 333–339, May 1997.
[6] A. M. Smith, S. G. Scott, and D. M. Wiese, “The
psychological effects of sports injuries. Coping,”
Sports Med. Auckl. NZ, vol. 9, no. 6, pp. 352–369,
Jun. 1990.
[7] L. Johnston and D. Carroll, “The psychological
impact of injury: effects of prior sport and exercise
involvement,” Br. J. Sports Med., vol. 34, no. 6, pp.
436–439, Dec. 2000.
[8] M. L. Shuer and M. S. Dietrich, “Psychological
effects of chronic injury in elite athletes.,” West. J.
Med., vol. 166, no. 2, pp. 104–109, Feb. 1997.
[9] “Research Review: Front or back squats,” Precision
Nutrition. .
[10] A. Singh, A. Klapper, J. Jia, A. Fidalgo, A. Tajadura-
Jiménez, N. Kanakam, N. Bianchi-Berthouze, and A.
Williams, “Motivating People with Chronic Pain to
Do Physical Activity: Opportunities for Technology
Design,” in Proceedings of the 32Nd Annual ACM
Conference on Human Factors in Computing
Systems, New York, NY, USA, 2014, pp. 2803–2812.
[11] D. M. Bravata, C. Smith-Spangler, V. Sundaram, A.
L. Gienger, N. Lin, R. Lewis, C. D. Stave, I. Olkin,
and J. R. Sirard, “Using pedometers to increase
physical activity and improve health: a systematic

12. review,” JAMA, vol. 298, no. 19, pp. 2296–2304,
Nov. 2007.
[12] C. B. Chan, D. A. J. Ryan, and C. Tudor-Locke,
“Health benefits of a pedometer-based physical
activity intervention in sedentary workers,” Prev.
Med., vol. 39, no. 6, pp. 1215–1222, Dec. 2004.
[13] D. Merom, C. Rissel, P. Phongsavan, B. J. Smith, C.
Van Kemenade, W. J. Brown, and A. E. Bauman,
“Promoting walking with pedometers in the
community: the step-by-step trial,” Am. J. Prev.
Med., vol. 32, no. 4, pp. 290–297, Apr. 2007.
[14] D. Morris, T. S. Saponas, A. Guillory, and I. Kelner,
“RecoFit: Using a Wearable Sensor to Find,
Recognize, and Count Repetitive Exercises,” in
Proceedings of the 32Nd Annual ACM Conference on
Human Factors in Computing Systems, New York,
NY, USA, 2014, pp. 3225–3234.
[15] P. on Wed and O. 27, “The Anterior Cruciate
Ligament (ACL).” [Online]. Available:
http://www.foundrysportsmedicine.com/our-
blog/bid/47663/The-Anterior-Cruciate-Ligament-
ACL. [Accessed: 30-May-2015].
[16] B. de Gelder, “Why bodies? Twelve reasons for
including bodily expressions in affective
neuroscience,” Philos. Trans. R. Soc. Lond. B. Biol.
Sci., vol. 364, no. 1535, pp. 3475–3484, Dec. 2009.
[17] A. Kleinsmith, N. Bianchi-Berthouze, and A. Steed,
“Automatic Recognition of Non-Acted Affective
Postures,” Trans Sys Man Cyber Part B, vol. 41, no.
4, pp. 1027–1038, Aug. 2011.
[18] A. Mehrabian and J. T. Friar, “Encoding of attitude
by a seated communicator via posture and position
cues,” J. Consult. Clin. Psychol., vol. 33, no. 3, pp.
330–336, 1969.
[19] H. G. Wallbott and K. R. Scherer, “Cues and
channels in emotion recognition,” J. Pers. Soc.
Psychol., vol. 51, no. 4, pp. 690–699, 1986.
[20] P. Ekman and W. V. Friesen, “Nonverbal leakage and
clues to deception,” Psychiatry, vol. 32, no. 1, pp.
88–106, Feb. 1969.
[21] P. Ekman and W. V. Friesen, “Detecting deception
from the body or face,” J. Pers. Soc. Psychol., vol.
29, no. 3, pp. 288–298, 1974.
[22] C. C. R. J. van Heijnsbergen, H. K. M. Meeren, J.
Grèzes, and B. de Gelder, “Rapid detection of fear in
body expressions, an ERP study,” Brain Res., vol.
1186, pp. 233–241, Dec. 2007.
[23] J. Aronoff, B. A. Woike, and L. M. Hyman, “Which
are the stimuli in facial displays of anger and
happiness? Configurational bases of emotion
recognition,” J. Pers. Soc. Psychol., vol. 62, no. 6,
pp. 1050–1066, 1992.
[24] W. T, “A study of the expression of bodily posture,”
J. Gen. Psychol., vol. 7, pp. 405–437, 1932.
[25] M. Coulson, “Attributing Emotion to Static Body
Postures: Recognition Accuracy, Confusions, and
Viewpoint Dependence,” J. Nonverbal Behav., vol.
28, no. 2, pp. 117–139, Jun. 2004.
[26] R. von Laban, The mastery of movement. Macdonald
and Evans, 1980.
[27] A. Camurri, B. Mazzarino, M. Ricchetti, R. Timmers,
and G. Volpe, “Multimodal Analysis of Expressive
Gesture in Music and Dance Performances,” in
Gesture-Based Communication in Human-Computer
Interaction, A. Camurri and G. Volpe, Eds. Springer
Berlin Heidelberg, 2004, pp. 20–39.
[28] A. Camurri, I. Lagerlöf, and G. Volpe, “Recognizing
emotion from dance movement: comparison of
spectator recognition and automated techniques,” Int.
J. Hum.-Comput. Stud., vol. 59, no. 1–2, pp. 213–
225, Jul. 2003.
[29] P. R. De Silva and N. Bianchi-Berthouze, “Modeling
Human Affective Postures: An Information Theoretic
Characterization of Posture Features: Research
Articles,” Comput Animat Virtual Worlds, vol. 15,
no. 3–4, pp. 269–276, Jul. 2004.
[30] D. Bernhardt and P. Robinson, “Detecting Affect
from Non-stylised Body Motions,” in Affective
Computing and Intelligent Interaction, A. C. R.
Paiva, R. Prada, and R. W. Picard, Eds. Springer
Berlin Heidelberg, 2007, pp. 59–70.
[31] Y. Ma, H. M. Paterson, and F. E. Pollick, “A motion
capture library for the study of identity, gender, and
emotion perception from biological motion,” Behav.
Res. Methods, vol. 38, no. 1, pp. 134–141, Feb. 2006.
[32] A. Kapoor, R. W. Picard, and Y. Ivanov,
“Probabilistic combination of multiple modalities to
detect interest,” in Proceedings of the 17th
International Conference on Pattern Recognition,
2004. ICPR 2004, 2004, vol. 3, pp. 969–972 Vol.3.
[33] A. Kapoor, W. Burleson, and R. W. Picard,
“Automatic Prediction of Frustration,” Int J Hum-
Comput Stud, vol. 65, no. 8, pp. 724–736, Aug. 2007.
[34] B. A. Rosser, P. McCullagh, R. Davies, G. A.
Mountain, L. McCracken, and C. Eccleston,
“Technology-mediated therapy for chronic pain
management: the challenges of adapting behavior
change interventions for delivery with pervasive
communication technology,” Telemed. J. E-Health
Off. J. Am. Telemed. Assoc., vol. 17, no. 3, pp. 211–
216, Apr. 2011.
[35] M. S. H. Aung, B. Romera-Paredes, A. Singh, S.
Lim, N. Kanakam, A. C. de C Williams, and N.
Bianchi-Berthouze, “Getting RID of pain-related
behaviour to improve social and self perception: A
technology-based perspective,” in 2013 14th
International Workshop on Image Analysis for
Multimedia Interactive Services (WIAMIS), 2013, pp.
1–4.
[36] E. J. Drinkwater, N. R. Moore, and S. P. Bird,
“Effects of changing from full range of motion to
partial range of motion on squat kinetics,” J. Strength

13. Cond. Res. Natl. Strength Cond. Assoc., vol. 26, no.
4, pp. 890–896, Apr. 2012.
[37] “How are partial squats and full squats different?,”
Strength & Conditioning Research. .
[38] G. St Thomas, “Physiotherapy following anterior
cruciate ligament (ACL) reconstruction.” © 2013
Guy’s and St Thomas’ NHS Foundation Trust, Feb-
2013.
[39] “Your Quick Guide to ACL Rehab Exercises,”
About.com Health, 15-Dec-2014. [Online].
Available:
http://sportsmedicine.about.com/od/surgeryrehab/a/A
CLSurgeryRehab.htm. [Accessed: 30-May-2015].
[40] “Weka 3 - Data Mining with Open Source Machine
Learning Software in Java.” [Online]. Available:
http://www.cs.waikato.ac.nz/ml/weka/. [Accessed:
31-May-2015].
[41] “Data Mining with Weka (2.4: Baseline accuracy) -
YouTube.” [Online]. Available:
https://www.youtube.com/watch?v=MrQhW4FyNW
4. [Accessed: 31-May-2015].
[42] “How To Do Squats Properly - The Ultimate Guide.”
[Online]. Available: http://johnalvino.com/how-to-
do-squats/. [Accessed: 31-May-2015].
[43] “Iowa Governor’s Council on Physical Fitness &
Nutrition Exercise #6 HALF SQUAT.” .
[44] M. S. Cepeda, D. B. Carr, J. Lau, and H. Alvarez,
“Music for pain relief,” in Cochrane Database of
Systematic Reviews, John Wiley & Sons, Ltd, 1996.
[45] J. Vidyarthi, B. E. Riecke, and D. Gromala, “Sonic
Cradle: Designing for an Immersive Experience of
Meditation by Connecting Respiration to Music,” in
Proceedings of the Designing Interactive Systems
Conference, New York, NY, USA, 2012, pp. 408–
417.
[46] B. A. Rosser and C. Eccleston, “Smartphone
applications for pain management,” J. Telemed.
Telecare, vol. 17, no. 6, pp. 308–312, Sep. 2011.
[47] K. Vogt, D. Pirrò, I. Kobenz, R. Höldrich, and G.
Eckel, “PhysioSonic - Evaluated Movement
Sonification as Auditory Feedback in
Physiotherapy,” in Auditory Display, S. Ystad, M.
Aramaki, R. Kronland-Martinet, and K. Jensen, Eds.
Springer Berlin Heidelberg, 2010, pp. 103–120.
[48] “Chronic Pain Rehabilitation Assistant | Ye’s
Stories.” [Online]. Available:
http://yelin.in/works/chronic.html. [Accessed: 31-
May-2015].
[49] R. W. Picard, E. Vyzas, and J. Healey, “Toward
machine emotional intelligence: analysis of affective
physiological state,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 23, no. 10, pp. 1175–1191, Oct.
2001.